ParameterList assigned to 1 GPU only (?)

ortho-stice: <p>Hey folks,</p>
<p>I am new to pytorch and I am trying to parallelize my network. Using <code>nn.DataParallel</code> seems to work as expected for the <code>nn.modules</code> living inside my class, however, it looks like the <code>nn.ParameterLists</code> that I’m defining as class members are listed as sitting in <code>(GPU 0)</code> only, when I print out the module’s parameters:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/a/4ab6504d922e1f0c172c7a21cf004209d1263d8a.png" data-download-href="https://discuss.pytorch.org/uploads/default/4ab6504d922e1f0c172c7a21cf004209d1263d8a" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/a/4ab6504d922e1f0c172c7a21cf004209d1263d8a_2_688x500.png" alt="image" data-base62-sha1="aEVW9Yu8TNMqkNc49YzJ6Uvup0C" width="688" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/a/4ab6504d922e1f0c172c7a21cf004209d1263d8a_2_688x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/4/a/4ab6504d922e1f0c172c7a21cf004209d1263d8a_2_1032x750.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/4/a/4ab6504d922e1f0c172c7a21cf004209d1263d8a.png 2x" data-dominant-color="100E0E"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1182×859 127 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Is this expected behaviour and why are they not listed on both of the GPUs I’m using? Could somebody please explain what is going on here?</p>
<hr>
<ul>
<li><code>torch.cuda.device_count</code> returns <code>2</code> as expected.</li>
</ul>
<p>My code looks something like the following:</p>
<pre><code class="lang-auto">class Network(nn.Module):
    def __init__(self):
        ...
        self.templates = nn.ModuleList([nn.ParameterList([nn.Parameter(template_init, requires_grad=True) for i in range(n)]) for n in self.num_t])

...

self.Network = nn.DataParallel(self.Network)
self.Network.to(self.device)
</code></pre>

mrshenli: <p>Hi <a class="mention" href="/u/ortho-stice">@ortho-stice</a></p>
<p>This is expected behavior. Here is the source code of DataParallel: <a href="https://github.com/pytorch/pytorch/blob/46539eee0363e25ce5eb408c85cefd808cd6f878/torch/nn/parallel/data_parallel.py#L148-L153" rel="nofollow noopener">https://github.com/pytorch/pytorch/blob/46539eee0363e25ce5eb408c85cefd808cd6f878/torch/nn/parallel/data_parallel.py#L148-L153</a></p>
<p>What happens is that, in every forward pass, DataParallel will</p>
<ol>
<li>scatters the input to all GPUs</li>
<li>replicate the model to all GPUs</li>
<li>launch parallel_apply so that every GPU will run its own forward pass using its input data split in parallel.</li>
<li>gather all outputs to the output device</li>
</ol>
<p>So the model replication only occurs in the forward pass, and hence you won’t see those model replicas outside the forward function.</p>
<p>BTW, we do recommend using DistributedDataParallel which only replicates the model once in constructor instead of in every forward invocation.</p>