Issues on using nn.DataParallel with Python 3.10 and PyTorch 1.11

nullgeppetto: <p>I modify <code>nn.DataParallel</code> so as it handles a modified network (a GenForce GAN generator in my case), as follows:</p>
<pre><code class="lang-auto">class DataParallelPassthrough(nn.DataParallel):
    def __getattr__(self, name):
        try:
            return super(DataParallelPassthrough, self).__getattr__(name)
        except AttributeError:
            return getattr(self.module, name)
</code></pre>
<p>If I use standard <code>nn.DataParallel</code>, I get the following errors:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  [...]
  File "/home/***/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1185, in __getattr__
    raise AttributeError("'{}' object has no attribute '{}'".format(
AttributeError: 'DataParallel' object has no attribute 'dim_z'
</code></pre>
<p>that is, errors about attributes that I have added to my network but standard <code>nn.DataParallel</code> isn’t “aware of”.</p>
<p><code>DataParallelPassthrough</code> used to work just fine, but after upgrading to Python 3.10 (‘Python 3.10.2’) and PyTorch 1.11 (‘1.11.0+cu102’), I get the following error when I actually use data parallelism:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  [...]
  File "/home/***/lib/python3.10/site-packages/torch/cuda/nccl.py", line 51, in _check_sequence_type
    if not isinstance(inputs, collections.Container) or isinstance(inputs, torch.Tensor):
AttributeError: module 'collections' has no attribute 'Container'
</code></pre>
<p>If I don’t call <code>DataParallelPassthrough</code> (i.e., if the batch size allows using a single GPU), everything is fine.</p>
<p>Any ideas on how I could fix this? Any insight on a better way to parallelize my (modified) network?</p>
<p>Thank you!</p>

ptrblck: <p>Based on <a href="https://stackoverflow.com/a/70631361">this comment</a> it seems to be related to Python 3.10.<br>
Could you downgrade your Python version and check if 3.9 would work (in a new virtual environment)?</p>

nullgeppetto: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Thanks for your comment, I was aware of it being Python3.10-related but I thought I should ask here in case there are any insights on how to solve this, or even whether there’s a “better” way to parallelize my model.</p>
<p>Indeed, with python 3.9 I had no problems (not tested with python 3.9 AND PyTorch 1.11 though).</p>

ptrblck: <p>I think the nightly binary could work with Python 3.10 as <a href="https://github.com/pytorch/pytorch/pull/72239">this PR</a> seems to have fixed the issue.</p>

nullgeppetto: <p>Just to confirm, I just tried to install the nightly binary as:</p>
<pre><code class="lang-auto">pip3 install --pre torch torchvision torchaudio -f https://download.pytorch.org/whl/nightly/cu113/torch_nightly.html
</code></pre>
<p>and checked again, but it has not been fixed.</p>

ptrblck: <p>That’s strange, since the failing line of code in the current <a href="https://github.com/carmocca/pytorch/blob/f176dc9515726c9c75f14bd295fd0e7ef8c0182a/torch/cuda/nccl.py#L51">master</a> is using:</p>
<pre><code class="lang-python">def _check_sequence_type(inputs: Union[torch.Tensor, Sequence[torch.Tensor]]) -&gt; None:
    if not isinstance(inputs, collections.abc.Container) or isinstance(inputs, torch.Tensor):
        raise TypeError("Inputs should be a collection of tensors")
</code></pre>
<p>now and doesn’t fit your error message anymore:</p>
<pre><code class="lang-python">if not isinstance(inputs, collections.Container) or isinstance(inputs, torch.Tensor):
</code></pre>

nullgeppetto: <p>Indeed I get the same error</p>
<pre><code class="lang-auto">  File "/home/***/lib/python3.10/site-packages/torch/cuda/nccl.py", line 51, in _check_sequence_type
    if not isinstance(inputs, collections.Container) or isinstance(inputs, torch.Tensor):
AttributeError: module 'collections' has no attribute 'Container'
</code></pre>
<p>Apparently I have a different pytorch version before. I think you linked a forked (by carmocca) version before, but this seems to be the case in the <a href="https://github.com/pytorch/pytorch/blob/master/torch/cuda/nccl.py#L51" rel="noopener nofollow ugc">master</a> of the original repo as well.</p>
<p>Apologies if this is a stupid question, but shouldn’t I get this version (i.e., the master branch) by installing pytorch simply using an entry <code>torch</code> in the requirements.txt file in a venv? I think I’ve missed something trivial here…</p>

ptrblck: <aside class="quote no-group" data-username="nullgeppetto" data-post="7" data-topic="146745">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/nullgeppetto/48/40331_2.png" class="avatar"> nullgeppetto:</div>
<blockquote>
<p>I think you linked a forked (by carmocca) version before, but this seems to be the case in the <a href="https://github.com/pytorch/pytorch/blob/master/torch/cuda/nccl.py#L51">master</a> of the original repo as well.</p>
</blockquote>
</aside>
<p>Yes, you are right and I just followed the PR file change. The <code>pytorch/master</code> branch however also has this change as you’ve noticed.</p>
<aside class="quote no-group" data-username="nullgeppetto" data-post="7" data-topic="146745">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/nullgeppetto/48/40331_2.png" class="avatar"> nullgeppetto:</div>
<blockquote>
<p>by installing pytorch simply using an entry <code>torch</code> in the requirements.txt file in a venv?</p>
</blockquote>
</aside>
<p>No, I don’t think setting <code>torch</code> as a required package would install the nightly binary.</p>

nullgeppetto: <p>Sorry, my bad, but shouldn’t the nightly binary be installed as</p>
<pre><code class="lang-auto">pip3 install --pre torch torchvision torchaudio -f https://download.pytorch.org/whl/nightly/cu113/torch_nightly.html
</code></pre>
<p>I just checked again (after uninstalling torch and installing again using the above line) and the condition if still as <code>isinstance(inputs, collections.Container)</code>.</p>

ptrblck: <p>I see the expected change in the current nightly, so I guess you are still looking into a wrong installation:</p>
<pre><code class="lang-python">root@4031504dc1c7:/workspace# pip install --pre torch -f https://download.pytorch.org/whl/nightly/cu113/torch_nightly.html
Looking in links: https://download.pytorch.org/whl/nightly/cu113/torch_nightly.html
Collecting torch
  Downloading https://download.pytorch.org/whl/nightly/cu113/torch-1.12.0.dev20220317%2Bcu113-cp38-cp38-linux_x86_64.whl (1623.1 MB)
     |████████████████████████████████| 1623.1 MB 1.8 kB/s 
Requirement already satisfied: typing-extensions in /opt/conda/lib/python3.8/site-packages (from torch) (3.10.0.0)
Installing collected packages: torch
Successfully installed torch-1.12.0.dev20220317+cu113
</code></pre>
<pre><code class="lang-python">root@4031504dc1c7:/workspace# python -c "import torch; print(torch.__path__)"
['/opt/conda/lib/python3.8/site-packages/torch']
root@4031504dc1c7:/workspace# sed -n 51p /opt/conda/lib/python3.8/site-packages/torch/cuda/nccl.py 
    if not isinstance(inputs, collections.abc.Container) or isinstance(inputs, torch.Tensor):
</code></pre>

nullgeppetto: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> even though I was using the same nightly binary, for some strange reason it refused to update.</p>
<p>Eventually, I uninstalled all torch* from the venv, reinstalled and now I’m seeing the correct line.</p>
<p>Thanks for your help and patience!</p>

mingun0112: <p><a class="mention" href="/u/nullgeppetto">@nullgeppetto</a> I think today many people having this problem because python 3.10 and torch 1.11 are lately update version!<br>
Maybe lately version of torch/cuda with nccl doesn’t support (collections.Container) so we should use (collections.abc.Container) instead of  (collections.Container)<br>
There is no need to change the version of Pytorch!<br>
I’m glad it worked out ^^</p>

nullgeppetto: <p>Hi <a class="mention" href="/u/mingun0112">@mingun0112</a>, indeed I observed the same thing today – changed to <code>collections.abc.Container</code> as a dirty but quick workaround.</p>

jure: <p>Thanks for providing a solution <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"> But could you please also tell how and where it has to be changed?</p>

nullgeppetto: <p>Hi <a class="mention" href="/u/jure">@jure</a>, apologies for the late reply.</p>
<p>So, you need to change <code>lib/python3.10/site-packages/torch/cuda/nccl.py</code>, at line 51:</p>
<pre><code class="lang-auto">def _check_sequence_type(inputs: Union[torch.Tensor, Sequence[torch.Tensor]]) -&gt; None:
    if not isinstance(inputs, collections.abc.Container) or isinstance(inputs, torch.Tensor):
        raise TypeError("Inputs should be a collection of tensors")
</code></pre>

jure: <p>Thank you so much <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>