Moving tensor to cuda

username: Intel_Novel
cooked: <p>Hi, this works, <code>a = torch.LongTensor(1).random_(0, 10).to("cuda")</code>. but this won’t work:</p>
<pre><code class="lang-auto">a = torch.LongTensor(1).random_(0, 10)
a.to(device="cuda")
</code></pre>
<p>Is this per design, maybe I am simple missing something to convert tensor from CPU to CUDA?</p>

username: ptrblck
cooked: <p>If you are pushing tensors to a device or host, you have to reassign them:</p>
<pre><code class="lang-python">a = a.to(device='cuda')
</code></pre>
<p><code>nn.Module</code>s push all parameters, buffers and submodules recursively and don’t need the assignment.</p>

username: Swarnadeep_Bhar
cooked: <p>Hi,<br>
I try running this simple block of code:</p>
<p>list_torch = [1,2,3,4]<br>
tenso = torch.tensor(list_torch).view(-1,1)<br>
tenso = tenso.to(device=‘cuda’)</p>
<p>I get this error:<br>
RuntimeError                              Traceback (most recent call last)<br>
 in ()<br>
----&gt; 1 tenso = tenso.to(device=‘cuda’)</p>
<p>RuntimeError: CUDA error: device-side assert triggered<br>
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.<br>
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.</p>
<p>The code works fine in CPU though, it throws error in COLAB as well as in my local machine</p>

username: ptrblck
cooked: <p>CUDA operations are executed asynchronously, so you are most likely to run into a previous error.<br>
Did you rerun the code with <code>CUDA_LAUNCH_BLOCKING=1</code> as suggested in the error message?</p>

username: Swarnadeep_Bhar
cooked: <p>This code was a part of a larger block, I created a Encoder module with a input  layer of size n where as my vocab was n+2, fixing this solved the error.<br>
I was confused as the IDE kept pointing at the cuda conversion lines as the bugs, I adapted the following:</p>
<ol>
<li>Shifted the model to cpu</li>
<li>used try except and printed the exception string</li>
</ol>
<p>Although, I could not understand how the layer mismatch gave rise to this error</p>

username: ptrblck
cooked: <p>The CPU can run ahead, since CUDA operations are executed asynchronously in the background.<br>
Unless you are blocking the code via <code>CUDA_LAUNCH_BLOCKING=1</code>, the stack trace will point to the current line of code executed on the host, which is often wrong.<br>
In any case, good to hear you’ve narrowed it down.</p>

username: kuraga
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> , but what if I don’t touch <code>CUDA_LAUNCH_BLOCKING</code> but set <code>non_blocking</code> argument to <code>False</code>?</p>

username: ptrblck
cooked: <p>In this case compute kernels will still be executed asynchronously and you should not blindly trust the stacktrace.</p>

username: kuraga
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> , thanks for reply! I just see (remember I don’t touch <code>CUDA_LAUNCH_BLOCKING</code> and <code>non_blocking=False</code>) that <code>.to(device='cuda')</code> changes the <code>device</code> value to <code>cuda:0</code> but would increase the GPU’s memory consumption if and only if <code>copy=True</code>. So, is moving to GPU lazy in the <code>copy=False</code> case?</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="kuraga" data-post="9" data-topic="39318">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/kuraga/48/65217_2.png" class="avatar"> kuraga:</div>
<blockquote>
<p>but would increase the GPU’s memory consumption if and only if <code>copy=True</code></p>
</blockquote>
</aside>
<p>That’s generally not true and only the case when the tensor is already on the desired device. From the docs:</p>
<blockquote>
<p>When copy is set, a new Tensor is created even when the Tensor already matches the desired conversion.</p>
</blockquote>

username: kuraga
cooked: <p>Ok, seems like the Google Colab GPU RAM measuring is… strange.</p>