Why are newly spawned tensors inside a GPU-loaded model not located on the GPU?

javadr: <p>I am encountering an issue with my PyTorch model that has been moved to the GPU using <code>to('cuda:0')</code>. When the model creates a new tensor inside its <code>forward</code> method, the tensor does not get placed on the GPU and instead remains on the CPU.</p>
<p>To better illustrate the problem, I have provided an MWE  below:</p>
<pre><code class="lang-python">import torch

class Model(torch.nn.Module):
    def __init__(self):
        super().__init__()

    def forward(self, x):
        y = torch.Tensor(x.shape)
        y[::2] = x[::2]
        y[1::2] = x[1::2]
        return x, y

model = Model().to('cuda:0')
z = torch.rand(4).to('cuda:0')
x, y = model(z)
print(x.device, y.device)
</code></pre>
<p>The output of this code indicates that <code>x</code> is located on the GPU (<code>cuda:0</code>), but <code>y</code> remains on the CPU (<code>cpu</code>).</p>
<p>I was expecting both <code>x</code> and <code>y</code> to be located on the GPU (<code>cuda:0</code>). Can someone please help me understand why the newly spawned tensor <code>y</code> does not get placed on the GPU even though the model is on the GPU?<br>
I’ve also tried <code>y = torch.nn.Parameter(torch.Tensor(x.shape), requires_grad=False)</code>, but it didn’t work too!</p>
<p>I appreciate any insights or suggestions to resolve this issue. Thank you for your assistance!</p>

johannes-lee: <p>Tensors are placed on CPU by default. The forward function does not change the default.</p>
<p>You can try <code>y = torch.empty_like(x)</code> or <code>y = torch.empty(x.shape, device=x.device)</code> (or use the zeros instead of empty version if desired).</p>

javadr: <p>Thanks, I thought the model parameters would be saved on the same device with it, I mean in case of <code>torch.nn.Parameters</code>!</p>

ptrblck: <aside class="quote no-group" data-username="javadr" data-post="1" data-topic="185135">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/javadr/48/37793_2.png" class="avatar"> javadr:</div>
<blockquote>
<p>even though the model is on the GPU?</p>
</blockquote>
</aside>
<p>That’s not necessarily true. While <code>.to()</code> will be applied recursively to all registered submodules, parameters, and buffers, you are still free to move different parameters to different devices, e.g. to implement model sharding. The model thus does not have a <code>.device</code> attribute, but every parameter and buffer etc.</p>

javadr: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/4/a4a805089bd7fedab77507bcab96e16c464c6185.png" data-download-href="https://discuss.pytorch.org/uploads/default/a4a805089bd7fedab77507bcab96e16c464c6185" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/4/a4a805089bd7fedab77507bcab96e16c464c6185_2_690x416.png" alt="image" data-base62-sha1="nuChHVnysGSUaePCqlgons1831H" width="690" height="416" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/4/a4a805089bd7fedab77507bcab96e16c464c6185_2_690x416.png, https://discuss.pytorch.org/uploads/default/original/3X/a/4/a4a805089bd7fedab77507bcab96e16c464c6185.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/a/4/a4a805089bd7fedab77507bcab96e16c464c6185.png 2x" data-dominant-color="F1F1F2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">979×591 49.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>I’m unsure what you want to tell with your image since this behavior is expected as already mentioned. You are also neither using the <code>_like</code> methods nor pass the <code>device</code> attribute to the tensor creation but instead are using the same failing code from before.</p>

javadr: <p>Thank you for your consideration. I have sent the picture for this phrase: <code>That’s not necessarily true.</code> Perhaps there was a misunderstanding on my part.</p>
<p>By the way, as I mentioned before, I expected to get the tensor on the GPU when the model was on the GPU, but I was mistaken. I have resolved my issue with <code>device</code>.</p>