What just happened?

Hannibal046: <p>Hi,</p>
<p>Can anyone explain it to me please?<br>
thx!</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/9/4978a4de9f05f78716c5f3d89e22229d90bd9953.png" data-download-href="https://discuss.pytorch.org/uploads/default/4978a4de9f05f78716c5f3d89e22229d90bd9953" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/9/4978a4de9f05f78716c5f3d89e22229d90bd9953_2_690x173.png" alt="image" data-base62-sha1="atXkHLQzxBwa9ZwQHqsSjhwCPBx" width="690" height="173" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/9/4978a4de9f05f78716c5f3d89e22229d90bd9953_2_690x173.png, https://discuss.pytorch.org/uploads/default/optimized/3X/4/9/4978a4de9f05f78716c5f3d89e22229d90bd9953_2_1035x259.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/4/9/4978a4de9f05f78716c5f3d89e22229d90bd9953_2_1380x346.png 2x" data-dominant-color="F4F5F5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1948×490 37.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>But it turns out that in another situation, this tensor transfer between different gpu is doable:</p>
<p>In the following code, I put the encoder and decoder of a auto encoder in different gpu ,and in forward function I manually change the data device:</p>
<pre data-code-wrap="python"><code class="lang-python">f = True
class LitAutoEncoder(nn.Module):

    def __init__(self):
        super().__init__()
        self.encoder = nn.Sequential(
            nn.Linear(28*28, 64),
            nn.ReLU(),
            nn.Linear(64, 3)
        )
        self.decoder = nn.Sequential(
            nn.Linear(3, 64),
            nn.ReLU(),
            nn.Linear(64, 28*28)
        )

        self.encoder = self.encoder.cuda(0)
        self.decoder = self.decoder.cuda(1)

    def forward(self, batch, batch_idx=None):
        global f
        x, y = batch
        x = x.view(x.size(0), -1)
        x = x.cuda(0)

        z = self.encoder(x)
        if f:
            print(z.cuda(1))    
            f = False
        x_hat = self.decoder(z.cuda(1))
        loss = F.mse_loss(x_hat.cuda(0), x)
        return loss

dataset = MNIST(os.getcwd(), download=True, transform=transforms.ToTensor())
train_loader = DataLoader(dataset,num_workers=4,batch_size=12)

autoencoder = LitAutoEncoder()
optimizer = torch.optim.Adam(autoencoder.parameters(), lr=1e-3)
losses = []
print_cnt = 0
for batch in train_loader:
    optimizer.zero_grad()
    loss = autoencoder(batch)
    loss.backward()
    losses.append(loss.item())
    optimizer.step()
</code></pre>
<p>and here is the output:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/8/480bff4d4d972eb1f433b2eb9ee81bcb30113d2a.png" data-download-href="https://discuss.pytorch.org/uploads/default/480bff4d4d972eb1f433b2eb9ee81bcb30113d2a" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/4/8/480bff4d4d972eb1f433b2eb9ee81bcb30113d2a.png" alt="image" data-base62-sha1="ahm54RSMXfxBajPPbf0a2yk4bnc" width="690" height="245" data-dominant-color="E6E6E6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1056×376 25.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I seemed to find some clue, but I am not so sure.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/9/a9b1e5a51147957d6c88f6d37894727ff1748b8b.png" data-download-href="https://discuss.pytorch.org/uploads/default/a9b1e5a51147957d6c88f6d37894727ff1748b8b" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/9/a9b1e5a51147957d6c88f6d37894727ff1748b8b_2_690x288.png" alt="image" data-base62-sha1="odbPMqdK1o0wVYE8aBcBTK2lb1F" width="690" height="288" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/9/a9b1e5a51147957d6c88f6d37894727ff1748b8b_2_690x288.png, https://discuss.pytorch.org/uploads/default/optimized/3X/a/9/a9b1e5a51147957d6c88f6d37894727ff1748b8b_2_1035x432.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/a/9/a9b1e5a51147957d6c88f6d37894727ff1748b8b_2_1380x576.png 2x" data-dominant-color="F4F5F5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1632×682 59.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/5/653ca37096004ade9b6cb069075dc3918bdbff67.png" data-download-href="https://discuss.pytorch.org/uploads/default/653ca37096004ade9b6cb069075dc3918bdbff67" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/5/653ca37096004ade9b6cb069075dc3918bdbff67_2_690x225.png" alt="image" data-base62-sha1="erA8MWZgj4YYG7VZosgHQoKI0mj" width="690" height="225" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/5/653ca37096004ade9b6cb069075dc3918bdbff67_2_690x225.png, https://discuss.pytorch.org/uploads/default/optimized/3X/6/5/653ca37096004ade9b6cb069075dc3918bdbff67_2_1035x337.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/6/5/653ca37096004ade9b6cb069075dc3918bdbff67_2_1380x450.png 2x" data-dominant-color="F3F4F4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1678×548 60.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
The result is different in different version</p>
<p>emm…<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/2/225bbc6154f8abc8a6f4627c7786a7cee2de79c8.png" data-download-href="https://discuss.pytorch.org/uploads/default/225bbc6154f8abc8a6f4627c7786a7cee2de79c8" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/2/225bbc6154f8abc8a6f4627c7786a7cee2de79c8_2_690x252.png" alt="image" data-base62-sha1="4TWMbrbEqjACL7FlPPn7jHLHBrO" width="690" height="252" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/2/225bbc6154f8abc8a6f4627c7786a7cee2de79c8_2_690x252.png, https://discuss.pytorch.org/uploads/default/optimized/3X/2/2/225bbc6154f8abc8a6f4627c7786a7cee2de79c8_2_1035x378.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/2/2/225bbc6154f8abc8a6f4627c7786a7cee2de79c8_2_1380x504.png 2x" data-dominant-color="F4F4F4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1674×612 58 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>If I remember it correctly, we’ve seed a similar issue in this forum which pointed to a hardware defect. Which devices are you using and could you run some sanity checks on them?<br>
Also, are you only seeing this issue in GPU1 and if so, could you swap both devices and check the behavior in again?</p>

Hannibal046: <p>Oh, thanks for your reply. It seems like a hardware defect.</p>
<p>When I changed to another machine, everything works fine.</p>
<p>The previous GPU I used is 1080Ti. And it still return 0 in GPU1. I did some searching in forum for the issue you mentioned, but I just couldn’t find it.</p>
<p>This is a really hard debugging process. Again, thanks very much!</p>