Discrepancy between manual parameter registration vs using nn.ModuleList when parallelizing

username: timgianitsos
cooked: <p>In an initializer to an <code>nn.Module</code>, I usually get the same behavior depending on whether I append the submodules to an <code>nn.ModuleList</code>, or whether I append them to a regular python list while registering manually with <code>self.add_module</code>.</p>
<p>However, I notice a discrepancy when I try to run experiments on multiple GPUs. The following code results in <code>RuntimeError: Expected all tensors to be on the same device</code>.</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn

class MyNetwork(nn.Module):
    def __init__(self):
        super().__init__()
        self.layers = []

        for i in range(3):
            layer = nn.Linear(10, 10, bias=False)
            self.add_module(f'layer_{i}', layer)
            self.layers.append(layer)

    def forward(self, x):
        for layer in self.layers:
            x = layer(x)
        return x

model = MyNetwork()
model = model.to(torch.device('cuda'))

p_model = nn.DataParallel(model, device_ids=['cuda:0', 'cuda:1'])
p_model(torch.zeros((5, 10)))
</code></pre>
<p>When I change <code>self.layers = []</code> to <code>self.layers = nn.ModuleList()</code> and delete <code>self.add_module(f'layer_{i}', layer)</code>, the code works fine.</p>
<p>Why does <code>DataParallel</code> recognize the submodules differently in these two scenarios?</p>

username: ptrblck
cooked: <p>I don’t know what <code>DataParallel</code> is using internally, but note that it’s deprecated and replaced by <code>DistributedDataParallel</code> for performance reasons and an even memory usage.</p>

username: timgianitsos
cooked: <p>I understand <code>DistributedDataParallel</code> is better than <code>DataParallel</code>, but I’m asking more about the implicit contract - automatic parameter registration (i.e. from <code>self.__setattr__</code>) <em>should</em> always be the same as manual parameter registration (i.e. from <code>self.add_module</code>).</p>
<p>That is, how is it possible for <em>any</em> wrapper to know the difference between manual parameter registration and automatic parameter registration? I feel like some implicit contract is broken to observe them behaving differently.</p>
<p>The lesson I just learned as a user is that automatic registration is not the same as manual registration, and I have evidence to always prefer the former. Since I have an instance of the latter failing, I’ll probably never use manual registration again, and I am confused why it is not deprecated. Therefore, to state my question differently: “Am I drawing the wrong lesson here?”</p>
<p>I just found this comment which argues this is “expected” behavior, but a contract breach should not be expected behavior. If I understand this comment correctly, this seems to be happening because <code>DataParallel</code> uses the module’s attributes from its <code>__dict__</code> instead of referring to the module’s registered parameters directly. Perhaps I should I file this as a Github issue.</p><aside class="onebox githubissue" data-onebox-src="https://github.com/pytorch/pytorch/issues/3174#issuecomment-337856808">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/issues/3174#issuecomment-337856808" target="_blank" rel="noopener nofollow ugc">github.com/pytorch/pytorch</a>
  </header>

  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue">
	  <svg width="60" height="60" class="github-icon" viewbox="0 0 14 16" aria-hidden="true"><path fill-rule="evenodd" d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/pytorch/pytorch/issues/3174#issuecomment-337856808" target="_blank" rel="noopener nofollow ugc">Data parallel failure when using add_module</a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2017-10-19" data-time="08:57:26" data-timezone="UTC">08:57AM - 19 Oct 17 UTC</span>
      </div>

        <div class="date">
          closed <span class="discourse-local-date" data-format="ll" data-date="2017-10-19" data-time="09:45:39" data-timezone="UTC">09:45AM - 19 Oct 17 UTC</span>
        </div>

      <div class="user">
        <a href="https://github.com/zxytim" target="_blank" rel="noopener nofollow ugc">
          <img alt="zxytim" src="https://avatars.githubusercontent.com/u/888911?v=4" class="onebox-avatar-inline" width="20" height="20">
          zxytim
        </a>
      </div>
    </div>

    <div class="labels">
    </div>
  </div>
</div>

  <div class="github-row">
    <p class="github-body-container">code:
```python
#!/usr/bin/env python3

import torch
from torch import nn
<span class="show-more-container"><a href="" rel="noopener" class="show-more">…</a></span><span class="excerpt hidden">from torch.autograd import Variable



class Model(nn.Module):
    def __init__(self):
        super().__init__()
        self.subs = [
            nn.Sequential(
                nn.Linear(2, 3), nn.ReLU(),
                nn.Linear(3, 1), nn.ReLU(),
            )
        ]

        for i, s in enumerate(self.subs):
            self.add_module('sub_{}'.format(i), s)

    def forward(self, x):
        return [s(x) for s in self.subs]


m = Model()
m = nn.DataParallel(m, device_ids=[0, 1])
m.cuda()

inpvar = Variable(torch.zeros(8, 2)).cuda()
out = m(inpvar)
```

output:
```shell
$ ./test_data_parallel.py                                                                                                                                                  1
Traceback (most recent call last):
  File "./test_data_parallel.py", line 31, in &lt;module&gt;
    out = m(inpvar)
  File "/home/zxy/.local/lib/python3.5/site-packages/torch/nn/modules/module.py", line 224, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/zxy/.local/lib/python3.5/site-packages/torch/nn/parallel/data_parallel.py", line 60, in forward
    outputs = self.parallel_apply(replicas, inputs, kwargs)
  File "/home/zxy/.local/lib/python3.5/site-packages/torch/nn/parallel/data_parallel.py", line 70, in parallel_apply
    return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)])
  File "/home/zxy/.local/lib/python3.5/site-packages/torch/nn/parallel/parallel_apply.py", line 75, in parallel_apply
    raise output
  File "/home/zxy/.local/lib/python3.5/site-packages/torch/nn/parallel/parallel_apply.py", line 50, in _worker
    output = module(*input, **kwargs)
  File "/home/zxy/.local/lib/python3.5/site-packages/torch/nn/modules/module.py", line 224, in __call__
    result = self.forward(*input, **kwargs)
  File "./test_data_parallel.py", line 23, in forward
    return [s(x) for s in self.subs]
  File "./test_data_parallel.py", line 23, in &lt;listcomp&gt;
    return [s(x) for s in self.subs]
  File "/home/zxy/.local/lib/python3.5/site-packages/torch/nn/modules/module.py", line 224, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/zxy/.local/lib/python3.5/site-packages/torch/nn/modules/container.py", line 67, in forward
    input = module(input)
  File "/home/zxy/.local/lib/python3.5/site-packages/torch/nn/modules/module.py", line 224, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/zxy/.local/lib/python3.5/site-packages/torch/nn/modules/linear.py", line 53, in forward
    return F.linear(input, self.weight, self.bias)
  File "/home/zxy/.local/lib/python3.5/site-packages/torch/nn/functional.py", line 554, in linear
    return torch.addmm(bias, input, weight.t())
  File "/home/zxy/.local/lib/python3.5/site-packages/torch/autograd/variable.py", line 924, in addmm
    return cls._blas(Addmm, args, False)
  File "/home/zxy/.local/lib/python3.5/site-packages/torch/autograd/variable.py", line 920, in _blas
    return cls.apply(*(tensors + (alpha, beta, inplace)))
  File "/home/zxy/.local/lib/python3.5/site-packages/torch/autograd/_functions/blas.py", line 26, in forward
    matrix1, matrix2, out=output)
RuntimeError: arguments are located on different GPUs at /pytorch/torch/lib/THC/generic/THCTensorMathBlas.cu:232

```</span></p>
  </div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


username: ptrblck
cooked: <p>The issue I’m seeing is that <code>DataParallel</code> is deprecated. So even if you claim a contract is broken, I doubt <code>DataParallel</code> will receive any fixes.</p>

username: timgianitsos
cooked: <p>Is it correct to say that <code>DataParallel</code> is deprecated? I interpreted the docs to just be giving a recommendation to use <code>DistributedDataParallel</code>. Also, that recommendation has existed in the docs for at least 3 years so I would think it’d be removed by now if it were deprecated.</p>
<p>I know why this recommendation exists. <code>DataParallel</code> uses multithreading which means it gets a bit stymied from Python’s Global Interpreter Lock whereas <code>DistributedDataParallel</code> uses multi-process to get around this. But I never notice much of a performance hit.</p>
<p>Are there any docs somewhere that I am missing that say it is on its way out?</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="timgianitsos" data-post="5" data-topic="181055">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/t/cab0a1/48.png" class="avatar"> timgianitsos:</div>
<blockquote>
<p><code>DataParallel</code> uses multithreading which means it gets a bit stymied from Python’s Global Interpreter Lock whereas <code>DistributedDataParallel</code> uses multi-process to get around this. But I never notice much of a performance hit.</p>
</blockquote>
</aside>
<p>Additionally to the GIL issues, your workload would also suffer from an imbalanced GPU memory usage (the default device would have a higher usage) and the additional communication overhead as the model is copied to all devices in each forward pass which is not the case for <code>DDP</code>.</p>
<aside class="quote no-group" data-username="timgianitsos" data-post="5" data-topic="181055">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/t/cab0a1/48.png" class="avatar"> timgianitsos:</div>
<blockquote>
<p>Is it correct to say that <code>DataParallel</code> is deprecated?</p>
</blockquote>
</aside>
<p>Yes, based on e.g. <a href="https://github.com/pytorch/pytorch/issues/94636#issuecomment-1426545929">this comment</a>:</p>
<blockquote>
<p>No new features will be added to DataParallel, it’s in maintenance mode <a href="https://github.com/pytorch/pytorch/issues/65936">#65936</a>. Wrappers make launching distributed jobs really easy, give them a try.</p>
</blockquote>

username: timgianitsos
cooked: <p>Thanks! Appreciate the additional detail about the communication overhead.</p>
<p>On the broader point about the contract, should automatic parameter registration always behave the same as manual parameter registration? (i.e. anytime they do not is because of a bug)</p>

username: ptrblck
cooked: <p>Yes, my expectation would be that both APIs should yield the same results and if a supported module depends on one particular way of registering a parameter I would see it as a bug.</p>