nn.DataParallel replicating torch.Tensor instead nn.Parameters for network weight and bias

username: dahyun
cooked: <p>Hi,<br>
I’m using nn.Dataparallel and suffering for network weight replicating issue.<br>
Pytorch version is 1.3.1.</p>
<p><code>Linear_fw</code> is a child class of <code>nn.Linear</code>, which becomes a part of the network architecture.<br>
The key feature of my interest is to register a member named <code>fast</code> for <code>weight</code> and <code>bias</code> in <code>nn.Linear</code>.<br>
i.e.</p>
<pre><code class="lang-auto">class Linear_fw(nn.Linear): # child class of nn.Linear.
    def __init__(self, in_features, out_features):
        super(Linear_fw, self).__init__(in_features, out_features)
        self.weight.fast = None # &lt;-- register a member, supposed to be assigned an nn.Parameter outside.
        self.bias.fast = None #&lt;-- register a member, supposed to be assigned an nn.Parameter outside.
        self.fast_flag = True
</code></pre>
<p>The error occurs during forwarding with nn.DataParallel</p>
<pre><code class="lang-auto">class Linear_fw(nn.Linear): # child class of nn.Linear.
    def __init__(self, in_features, out_features):
        super(Linear_fw, self).__init__(in_features, out_features)
        self.weight.fast = None # &lt;-- register a member, supposed to be assigned an nn.Parameter outside.
        self.bias.fast = None #&lt;-- register a member, supposed to be assigned an nn.Parameter outside.
        self.fast_flag = True

    def forward(self, x):
        if self.fast_flag:
            out = F.linear(x, self.weight.fast, self.bias.fast) # &lt;-- Attribute error occurs
        else:
            out = super(Linear_fw, self).forward(x)
        return out
</code></pre>
<p>An attribute error raises saying that:</p>
<pre><code class="lang-auto">    out = F.linear(x, self.weight.fast, self.bias.fast)
        AttributeError: 'Tensor' object has no attribute 'fast'
</code></pre>
<p>Without using nn.DataParallel, <code>self.weight</code> is <code>nn.Parameters</code> and the code runs. But with using nn.DataParallel, <code>self.weight</code> is <code>torch.Tensor</code>, thus, accessing to its member <code>self.weight.fast</code> raise the error. This error raise only with nn.DataParallel.</p>
<p>Here, the easiest way is to code (somehow magically) to replicate the network weight and bias of <code>nn.Parameters</code> instead <code>torch.Tensor</code> during <code>nn.DataParallel.forward</code>.<br>
Is it possible somehow? I sincerely appreciate your advices.</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="dahyun" data-post="1" data-topic="74578">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/dahyun/48/22090_2.png" class="avatar"> dahyun:</div>
<blockquote>
<p>Here, the easiest way is to code (somehow magically) to replicate the network weight and bias of <code>nn.Parameters</code> instead <code>torch.Tensor</code> during <code>nn.DataParallel.forward</code> .<br>
Is it possible somehow? I sincerely appreciate your advices.</p>
</blockquote>
</aside>
<p>Could you explain a bit, what you mean by this approach?</p>
<p>How are you setting the parameters to the initially empty attributes?<br>
Note that you cannot set these parameters in the <code>forward</code> method using <code>nn.DataParallel</code>, as these parameters will only be registered in the replica models, not the root model.</p>

username: dahyun
cooked: <p>Hi!<br>
What I wondering is: is it possible to make <code>self.weight</code> and <code>self.bias</code> to be passed as the type of <code>nn.Parameter</code> in multi-GPU setting?<br>
At the beginning of <code>forward</code>, <code>self.weight.__class__</code> and <code>self.bias.__class__</code> are <code>torch.Tensor</code> type at the multi-GPU setting, which looses access their member variable <code>"fast"</code> <img src="https://discuss.pytorch.org/images/emoji/apple/cry.png?v=9" title=":cry:" class="emoji" alt=":cry:">. In the single-GPU settings, <code>self.weight.__class__</code> and <code>self.bias.__class__</code> keep their type <code>nn.Parameter</code> at <code>forward</code> as they were made, thus accessing <code>fast</code> is not a problem. In other words, setting those to <code>nn.Parameter</code> inside of <code>forward</code> is not desired in this case.<br>
Thank you.</p>