Access att. of model wrapped within torch.nn.DataParallel: maximum recursion depth exceeded

username: sbelharbi
cooked: <p>Hello,<br>
I am trying to access to a model’s attributes that has been wrapped by <code>torch.nn.DataParallel</code>.<br>
Following the <a href="https://pytorch.org/tutorials/beginner/former_torchies/parallelism_tutorial.html" rel="nofollow noopener">tutorial</a>, overriding <code>__getattr__()</code> is enough. The simple following example shows a recursive issue, unless I am missing something:</p>
<pre><code class="lang-python">import torch
import torch.nn as nn


class Model(nn.Module):

    def __init__(self):
        super(Model, self).__init__()
        self.block1 = nn.Linear(10, 20)
        self.block2 = nn.Linear(20, 20)
        self.block3 = nn.Linear(20, 20)

    def forward(self, x):
        x = self.block1(x)
        x = self.block2(x)
        x = self.block3(x)
        return x


class MyDataParallel(torch.nn.DataParallel):
    """
    Allow nn.DataParallel to call model's attributes. (supposedly)
    https://pytorch.org/tutorials/beginner/former_torchies/parallelism_tutorial.html
    """
    def __getattr__(self, name):
        print("Name att.: {}".format(name))
        return getattr(self.module, name)


device = torch.device("cpu")

model = Model()
model = MyDataParallel(model)
model.to(device)

x = torch.rand(20, 10)
x = x.to(device)
model(x)
</code></pre>
<p>Output:</p>
<pre><code class="lang-bash">Name: module
Name: module
Name: module
Name: module
Name: module
Name: module
Name: module
Name: module
.... (more. Deleted for clarity)
Traceback (most recent call last):
  File "test-data-parallel.py", line 37, in &lt;module&gt;
    model(x)
  File "/home/brian/Venvs/pytorch.1.0.1/lib/python3.7/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/brian/Venvs/pytorch.1.0.1/lib/python3.7/site-packages/torch/nn/parallel/data_parallel.py", line 138, in forward
    return self.module(*inputs, **kwargs)
  File "test-data-parallel.py", line 26, in __getattr__
    return getattr(self.module, name)
  File "test-data-parallel.py", line 26, in __getattr__
    return getattr(self.module, name)
  File "test-data-parallel.py", line 26, in __getattr__
    return getattr(self.module, name)
  [Previous line repeated 491 more times]
  File "test-data-parallel.py", line 25, in __getattr__
    print("Name: {}".format(name))
RecursionError: maximum recursion depth exceeded while calling a Python object
</code></pre>
<p>The line <code>getattr(self.module, name)</code> calls recursively <code>MyDataParallel.__getattr__()</code>, since it calls <code>self.module</code>.</p>
<p>Any idea how to allow accessing to the attributes of a model that was wrapped within <code>torch.nn.DataParallel</code>?</p>
<p>The trivial solution of changing, within the entire code, the call <code>model.attribute</code> to <code>model.module.attribute</code> is not considered.</p>
<p>Thank you!</p>

username: sbelharbi
cooked: <p><a href="https://github.com/pytorch/pytorch/issues/16885#issuecomment-494165559" rel="nofollow noopener">This</a> workaround breaks down the recursion:</p>
<pre><code class="lang-python">class MyDataParallel(torch.nn.DataParallel):
    """
    Allow nn.DataParallel to call model's attributes.
    """
    def __getattr__(self, name):
        try:
            return super().__getattr__(name)
        except AttributeError:
            return getattr(self.module, name)
</code></pre>