Tensor view not consistent between GPU and CPU

Julian_Buchel: <p>I created a minimal working example of the bug/issue that I have that can be run easily.</p>
<pre><code class="lang-auto">import torch
device = "cuda:0"

class Test(torch.nn.Module):
    def __init__(self):
        super(Test, self).__init__()
        w = torch.randn(20,20)
        self.x = torch.nn.Parameter(w)
        self.y = torch.nn.Parameter(w.view(-1))

test = Test().to(device)
assert torch.allclose(test.x.flatten(),test.y), "This works"
test.x.data *= 2.
# assert torch.allclose(test.x.flatten(),test.y), "This does not"

test2 = Test()
assert torch.allclose(test2.x.flatten(),test2.y), "This works"
test2.x.data *= 2.
test2.to(device)
assert torch.allclose(test2.x.flatten(),test2.y), "This also works"
</code></pre>
<p>Why is that the case? I feel like when the network is on cpu, the memory is indeed shared. But when the data is on the GPU, updating the base weight does not update the view of it. Why is that? And how can I solve it?</p>

InnovArul: <p>When the data movement happens from the source device to the target device (from CPU to GPU, or vice versa), for different parameters, a different memory location is allocated in the target device, even if they share the same memory in the source device.</p>
<p>for example,</p>
<pre><code class="lang-auto">test = Test()  # create in CPU
print(test.x.data_ptr(), test.y.data_ptr()) # prints 1763172664192 1763172664192

test.cuda() # move to GPU
print(test.x.data_ptr(), test.y.data_ptr()) # prints 30150754304 30150756352
</code></pre>
<p>I am not sure if it’s intended behavior or a bug.<br>
<a class="mention" href="/u/ptrblck">@ptrblck</a> do you know if it’s the intended behavior?</p>

ptrblck: <p>The usage of the <code>.data</code> attribute is deprecated and should not be used.<br>
Based on your code you are trying to manipulate a leaf variable inplace, which isn’t supported, so could you explain your use case a bit more?</p>

Julian_Buchel: <p>My use-case is the following: I re-implemented the <code>Conv2d</code> layer so that it also has a <code>Linear</code> layer. The linear layer basically implements the same <code>conv</code> operation, just using an <code>MVM</code>. Why I do that is not important for this, but in the end they always share the same parameters.<br>
The code that I posted here is just to reproduce this quickly. In practice, this happens when I load a <code>state_dict</code>. The <code>state_dict</code> does not contain that linear layer (I want to hide that from the user) since the information is already in the conv weights.<br>
So, when I initialize the model on a GPU, the parameters do not share the same core data anymore. As a result, when I load the state dict, only the conv weights and not the linear ones get updated correctly. This is not a problem when the model is on the CPU.</p>

ptrblck: <p>Thanks for the description. Based on the use case, I think the cleanest way to implement it would be to either register the parameter explicitly and use the functional API for the <code>F.conv2d</code> and <code>F.linear</code> operations or to initialize the <code>nn.Conv2d</code> module and use its <code>weight</code> in <code>F.linear</code>.</p>

Julian_Buchel: <p>Unfortunately not an option since the <code>Linear</code> is not always equivalent to a <code>Linear</code> module. It is a custom class that inherits from another class, which in turn inherits from <code>torch.nn.Module</code>. In <code>train</code> and <code>eval</code> mode, your solution would work fine, but I have another mode where this does not work anymore.</p>

Julian_Buchel: <p>Is there something like an “update hook”? I.e. everytime the data tensor of the conv parameter gets updated to some value, I also update the data of the linear parameter.<br>
But this created too much overhead during training, since I only need the linear layer to be correct during inference mode.</p>