Questions on GPU/ CPU tensor transfer

username: Yeb02
cooked: <p>Using libtorch gpu and CUDA, windows 10.</p>
<p>I have several questions on this topic. I feel like answers should be in the doc but I could not find them, both in libtorch and pytorch, if I missed them please point me towards the right place.</p>
<p>1- How can I create a Module on GPU directly ? I know that you can transfer it after creation on CPU, but I would like to avoid this overhead. <a href="https://pytorch.org/cppdocs/api/classtorch_1_1nn_1_1_module.html#exhale-class-classtorch-1-1nn-1-1-module" rel="noopener nofollow ugc">doc</a></p>
<p>2- Does <code>tensor A = B.clone()</code> create <code>A</code> on GPU if <code>B</code> was on GPU ?</p>
<p>3- How do I cast a c10::device into a CUDA device ?</p>
<p>4- How do I copy the content of tensor A into the storage pointed to by tensor B (created with “from_blob”, or “normally”, with eye for instance)? A and B have the same sizes. Is std::copy the best way ?</p>

username: J_Johnson
cooked: <p>Will answer some of your questions.</p>
<ol>
<li>Every module allows you to pass a <code>device</code> argument. For example:</li>
</ol>
<pre><code class="lang-auto">model = nn.Linear(10, 5, device = 'cuda:0')
</code></pre>
<ol start="2">
<li>
<p>Yes, the clone will happen on whichever device the current tensor is located.</p>
</li>
<li>
<p>No idea.</p>
</li>
<li>
<p>Not exactly clear on what you mean here. “storage pointed to by tensor B”. You can obtain the device of any tensor via <code>tensor.device</code>. For example:</p>
</li>
</ol>
<pre><code class="lang-auto">import torch

A = torch.rand((2,2), device = 'cuda:0')
B = torch.tensor(A.clone().detach(), device = A.device)
print(A)
print(B)
</code></pre>
<p>Pytorch also has some useful functions when you wish to make a tensor of the same size as A all in ones or zeros with the same dtype and device.</p>
<pre><code class="lang-auto">A = torch.rand((2,2), device = 'cuda:0')
B = torch.zeros_like(A)
C = torch.ones_like(A)
D = torch.rand_like(A)
</code></pre>

username: Yeb02
cooked: <p>Thank you for your reply.</p>
<p>I am using libtorch, not pytorch. So c++, not python. Even though they share the same backend (and documentation ?!).</p>
<p>1- There is the relevant documentation for the c++ equivalent of nn.linear : <a href="https://pytorch.org/cppdocs/api/classtorch_1_1nn_1_1_linear_impl.html" rel="noopener nofollow ugc">Class LinearImpl — PyTorch main documentation</a> There is the doc of the construction parameters struct : <a href="https://pytorch.org/cppdocs/api/structtorch_1_1nn_1_1_linear_options.html#structtorch_1_1nn_1_1_linear_options" rel="noopener nofollow ugc">Struct LinearOptions — PyTorch main documentation</a> Unless I miss the obvious, there is no way to do it as python does, i.e. the <code>device = 'cuda:0'</code> argument you suggested.</p>
<p>2- Thank you, I expect it to work the same for libtorch then. If someone could confirm …</p>
<p>3- Cant find anything on that. =(</p>
<p>4-</p>
<blockquote>
<ol start="2">
<li>Not exactly clear on what you mean here. “storage pointed to by tensor B”.</li>
</ol>
</blockquote>
<p>I dont think one can access raw storage in python without external modules (?). In libtorch (C++), the underlying storage of a tensor is accessed with something like <code>.data_ptr&lt;float&gt;()</code>. I want to make sure I am never reallocating a when copying b, using <code>tensor a = b.clone()</code>, given a and b have the same sizes. And especially when a and b are on GPU.</p>