In CUDA extension, input tensor is same to normal C++ array and can use cublas directly?

Ziyu_Huang: <p>As mentioned in the title~<br>
Thank you!!!</p>

ptrblck: <p>You would need to pass device pointers to the cublas operation as described in e.g. <a href="https://docs.nvidia.com/cuda/cublas/index.html#cublas-GemmEx"><code>gemmEx</code></a>.</p>

Ziyu_Huang: <p>Thank you very much!!!</p>

Ziyu_Huang: <aside class="onebox allowlistedgeneric" data-onebox-src="https://github.com/pytorch/extension-cpp">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="32" height="32">

      <a href="https://github.com/pytorch/extension-cpp" target="_blank" rel="noopener nofollow ugc">GitHub</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/345;"><img src="https://opengraph.githubassets.com/55097934f32dd055b0ed45dea13e3a829d89a92396d4fd5fe1ee879e6e575062/pytorch/extension-cpp" class="thumbnail" width="690" height="345"></div>

<h3><a href="https://github.com/pytorch/extension-cpp" target="_blank" rel="noopener nofollow ugc">GitHub - pytorch/extension-cpp: C++ extensions in PyTorch</a></h3>

  <p>C++ extensions in PyTorch. Contribute to pytorch/extension-cpp development by creating an account on GitHub.</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>Above is the official tutorial of pytorch for combining the CUDA and Pytorch…Seems the torch library can not be used in GPU, right? Like at:: torch:: aten::, they can only be used in cpp file?</p>
<p>So on GPU, the only library we can use is… provided by NVIDIA company like cublas, cusparse?</p>
<p>Thank you!!!</p>

ptrblck: <aside class="quote no-group" data-username="Ziyu_Huang" data-post="4" data-topic="146183">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ziyu_huang/48/14158_2.png" class="avatar"> Ziyu_Huang:</div>
<blockquote>
<p>Seems the torch library can not be used in GPU, right?</p>
</blockquote>
</aside>
<p>No, you can use <code>libtorch</code> with a GPU. Take a look at e.g. <a href="https://github.com/pytorch/extension-cpp/blob/1031028f3b048fdea84372f3b81498db53d64d98/cuda/lltm_cuda_kernel.cu"><code>lltm_cuda_kernel.cu</code></a> in the linked tutorial.</p>
<aside class="quote no-group" data-username="Ziyu_Huang" data-post="4" data-topic="146183">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ziyu_huang/48/14158_2.png" class="avatar"> Ziyu_Huang:</div>
<blockquote>
<p>Like at:: torch:: aten::, they can only be used in cpp file?</p>
</blockquote>
</aside>
<p><code>Aten</code> methods work on different devices (similar to how the Python API can use tensors on the CPU and GPU).</p>
<aside class="quote no-group" data-username="Ziyu_Huang" data-post="4" data-topic="146183">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ziyu_huang/48/14158_2.png" class="avatar"> Ziyu_Huang:</div>
<blockquote>
<p>So on GPU, the only library we can use is… provided by NVIDIA company like cublas, cusparse?</p>
</blockquote>
</aside>
<p>You are of course able to call into cublas etc. manually or you could use the <code>ATen</code> method (e.g. <code>at::matmul</code>) which will call into <code>cublas</code> internally. Other methods would call into other libs or a native PyTorch CUDA kernel.</p>

Ziyu_Huang: <p>Thank you very much!!!</p>

Ziyu_Huang: <p><a>Uploading: image.png…</a><br>
Well, I think torch can not be used in GPU, I mean, kernel function with <strong>global</strong>, like here, bugs show:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/2/b24266a5b613eebf604415f18fc61901334040da.png" data-download-href="https://discuss.pytorch.org/uploads/default/b24266a5b613eebf604415f18fc61901334040da" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/2/b24266a5b613eebf604415f18fc61901334040da_2_690x442.png" alt="image" data-base62-sha1="pqXfH7AtoOZGmtHvkOHyvcw9Vx0" width="690" height="442" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/2/b24266a5b613eebf604415f18fc61901334040da_2_690x442.png, https://discuss.pytorch.org/uploads/default/optimized/3X/b/2/b24266a5b613eebf604415f18fc61901334040da_2_1035x663.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/b/2/b24266a5b613eebf604415f18fc61901334040da.png 2x" data-dominant-color="333232"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1249×801 104 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/f/9f26716f89aee9788cddd9b34b7438882b4a79ea.png" data-download-href="https://discuss.pytorch.org/uploads/default/9f26716f89aee9788cddd9b34b7438882b4a79ea" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/9/f/9f26716f89aee9788cddd9b34b7438882b4a79ea.png" alt="image" data-base62-sha1="mHUhtag5BaNuLPDiBbkvLdmEnwe" width="596" height="500" data-dominant-color="272626"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1239×1038 112 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
error: no instance of constructor “at::Tensor::Tensor” matches the argument list<br>
argument types are: (int, int, int)</p>

Ziyu_Huang: <p>Well, I see lots of torch function in the .cu file (in official tutorial), but they are in the <strong>host</strong> function, which is on CPU… And no torch function on <strong>global</strong> or <strong>device</strong> function~</p>

Ziyu_Huang: <p>Well, I noticed even cublas is running in <strong>host</strong> function, I guess, even in <strong>host</strong> function does not necessarily mean this is running on CPU, just like if we use torch::mm in <strong>host</strong> function, if the data are on GPU, the calculation will be done on GPU? (Guarantee? should we obviously set calculation on GPU?)</p>
<p>Thank you!!!</p>

ptrblck: <p>Take a look at e.g. the <a href="https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#kernels">CUDA programming model - Kernels</a>, which explains the <code>__global__</code> declaration specifier and explains how code is executed on the GPU.</p>
<p>In summary: PyTorch can use the GPU through native kernels, extensions, as well as CUDA libraries.</p>