Issue with gather from multiple devices in nn.DataParallel

username: basingse
cooked: <p>Hello,</p>
<p>I have used <code>nn.DataPrallel</code> to use 2 gpus while training.</p>
<p>This is how my model code looks (roughly)</p>
<pre data-code-wrap="python"><code class="lang-python">class Model(nn.Module):
    def forward(self, inputs):
        logger.info(f"shape of unique values in inputs = {torch.unique(inputs).shape}: {inputs.device}")
        logger.info(f"mean of inputs = {torch.mean(inputs)}: {inputs.device}")
        # some calculation using conv and linear layers
        # after this I get two variables loss and per
        # both are size of 1. loss is output of loss function (mean squared error)
        logger.info(f"per: {per}: {per.device}")
        logger.info(f"loss: {loss}: {loss.device}")

        return loss, per
</code></pre>
<p>My training loop looks like this</p>
<pre data-code-wrap="python"><code class="lang-python">model = Model()

model = nn.DataParallel(model, device_ids=[0, 1])
model.to("cuda:0")

optimizer = Adam(model.parameters())

for data in dataloader:
    data = data.to("cuda:0")
    loss, per = model(data)
    logger.info(f"per tensor: {per}")
    logger.info(f"loss tensor: {loss}")
    # compute the average loss
    x = loss.mean()
    # do backward
    x.backward()

    optimizer.step()
</code></pre>
<p>So I ran this code in 2 different servers (both have same configuration and versions. The details are given below).</p>
<p>The logs for 1st iteration in 1st and 2nd server is given below (# are some of my comments)</p>
<pre data-code-wrap="bash"><code class="lang-bash">Server 1 LOG
shape of unique values in inputs = torch.size([366633886]): cuda:0
mean of inputs = 0.00390: cuda:0
per: 1.0: cuda:0
loss: 0.18566: cuda:0
shape of unique values in inputs = torch.size([1]): cuda:1
mean of inputs = 0: cuda:1   # observe that for cuda:1 num(unique) = 1 and mean = 0
per: 1.0: cuda:1
loss: 0: cuda:1
per tensor: tensor([1., 0.], device='cuda:0')  # the output has to be [1.0, 1.0] I guess
loss tensor: tensor([0.1857, 0.0000], device='cuda:0')
</code></pre>
<pre data-code-wrap="bash"><code class="lang-bash">Server 2 LOG
shape of unique values in inputs = torch.size([35358244]): cuda:0
mean of inputs = 0.0002569: cuda:0
per: 1.0: cuda:0
loss: 0.1797265: cuda:0
shape of unique values in inputs = torch.size([35318931]): cuda:1
mean of inputs = 0.0002706056: cuda:1
per: 1.0: cuda:1
loss: 0.179730564: cuda:1
per tensor: tensor([1., 1.], device='cuda:0')
loss tensor: tensor([0.1797, 0.1797], device='cuda:0')
</code></pre>
<p>As you can see in <strong>server 1</strong>, I am receving inputs as all zero in <code>device = cuda:1</code> (because number of unique value is 1 and mean is 0) but everything seems working fine in <strong>server 2</strong></p>
<h2><a name="setup-1" class="anchor" href="#setup-1"></a>SETUP</h2>
<p>Both servers (operation system is Windows Server 16) uses the same data and same code. The version of torch and torchvision for both servers is (both were installed by downloading the wheel file and then using <code>pip</code>)</p>
<pre data-code-wrap="bash"><code class="lang-bash">torch : 1.8.1+cu111-cp36-cp36m-win_amd64.whl
torchvision: 0.9.1+cu111-cp36-cp36m-win_amd64
</code></pre>
<p>The GPU details are given below (both are same for servers)<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/5/85f6c53a5279b04e1adf81d16b86f0f9999957a1.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/85f6c53a5279b04e1adf81d16b86f0f9999957a1" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/8/5/85f6c53a5279b04e1adf81d16b86f0f9999957a1.jpeg" alt="image" data-base62-sha1="j76cOMoC07716213auaP7jPoI2B" width="690" height="314" data-dominant-color="4E3B69"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1061×484 55.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>The only <strong>difference</strong> between two servers is that is in server 1, three versions  of CUDA (<code>10.0</code>, <code>10.1</code> and <code>11.2</code>) are installed but in server 2 only <code>10.2</code> CUDA version is installed.</p>
<p>But according to these posts (<a href="https://discuss.pytorch.org/t/please-help-me-understand-installation-for-cuda-on-linux/14217/4">1</a> and <a href="https://stackoverflow.com/a/61034368/9812353" rel="noopener nofollow ugc">2</a>) I think it doesn’t matter what versions of CUDA are installed in the system as we are using wheel file to install <code>torch</code> which will come with pre-build CUDA and CUDNN.</p>
<p>Can I know why in <strong>server 1</strong> the output is <strong>wrong</strong> ?</p>
<p>Note : Apology for a very long post as I did work on this sometime and wanted to share my thoughts as well. Also I understand the forward computation and training loop is minimal as I dont know if I can share the organization code here. If you need any more details about the server configurations, environment variables in system I can provide them as well</p>
<p>Thanks !!</p>

username: pritamdamania87
cooked: <p>Although the data is the same on both servers, I’m wondering if there is non-determinism in the dataloader where the input is shuffled causing the order in which the inputs are loaded to be different. To narrow down the issue, can you take a fixed tensor on both servers and run it as input through both servers?</p>

username: basingse
cooked: <p>Hello,</p>
<p>Its right that the data loader in both servers load different batch ( due to random seeds ) but what my point was that how come in server 1 and <code>cuda:1</code>, I am receiving all zeros as inputs ( as all my input data contains images ) and also how come <code>per</code> tensor is <code>[1, 0]</code> in server 1 (which means there is some issue with gather from 2 gpus)</p>