nn.Module classes instantiation and weights sharing

username: 0x97
cooked: <p>I am dealing with two modules, both of them contain different sub-modules and one of these must be the same, as follows:</p>
<pre><code class="lang-auto">class Module1(nn.Module):
    def __init__(self, net1, net_shared):
        super(Module1,self).__init__()
        self.net1 = net1
        self.net_shared = net_shared
    def forward(self, x):
        ...
        return y

class Module2(nn.Module):
    def __init__(self, net2, net_shared):
        super(Module2,self).__init__()
        self.net2 = net2
        self.net_shared = net_shared
    def forward(self, x):
        ...
        return y

net1 = nn.Sequential()
net2 = nn.Sequential()
net_shared = nn.Sequential()

module1 = Module1(net1, net_shared)
module2 = Module2(net2, net_shared)

</code></pre>
<p>net1, net2, net_shared are preinstantiated nn.Sequential modules that will be passed to the constructor of Module1 and Module2. So, does net_shared will be treated as a single net? So that backpropagating through Module2, net_shared weights in Module1 will be updated, or by instantiating two different classes it will be treated as two deep copies instead of a shallow one?</p>

username: KFrank
cooked: <p>Hi 151!</p>
<aside class="quote no-group" data-username="0x97" data-post="1" data-topic="167548" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/0/ecc23a/48.png" class="avatar"> 0x97:</div>
<blockquote>
<p>So, does net_shared will be treated as a single net? So that backpropagating through Module2, net_shared weights in Module1 will be updated,</p>
</blockquote>
</aside>
<p>Yes and yes.</p>
<aside class="quote no-group">
<blockquote>
<p>or by instantiating two different classes it will be treated as two deep copies instead of a shallow one?</p>
</blockquote>
</aside>
<p>No, nothing is being copied (other than the python <em>references</em> themselves).<br>
The line of code, <code>net_shared = nn.Sequential()</code>, instantiates a single<br>
instance of the class <code>Sequential</code> and sets the python reference <code>net_shared</code><br>
to refer to it.  When, instantiating instances of <code>Module1</code> and <code>Module2</code>, you<br>
pass in <code>net_shared</code>, <code>module1.net_shared</code> and <code>module2.net_shared</code> get<br>
created as python references that refer to the same object-instance as the<br>
<code>net_shared</code> reference that is passed in.</p>
<p>There is only one (not counting <code>net1</code> and <code>net2</code>) instance of <code>Sequential</code><br>
and it is indeed shared between <code>module1</code> and <code>module2</code> – no copies,<br>
neither deep nor shallow are made.</p>
<p>You can probe this by looking at the object  <code>id()</code>'s of your <code>net_shared</code>s and<br>
you will see that they are the same:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'1.13.0'
&gt;&gt;&gt; net_shared = torch.nn.Sequential()
&gt;&gt;&gt; id (net_shared)
2253492592912
&gt;&gt;&gt; module1 = torch.nn.Sequential (net_shared)
&gt;&gt;&gt; module2 = torch.nn.ModuleList ([net_shared])
&gt;&gt;&gt; id (module1[0])
2253492592912
&gt;&gt;&gt; id (module2[0])
2253492592912
</code></pre>
<p>Best.</p>
<p>K. Frank</p>

username: 0x97
cooked: <p>Thanks KFrank for the very clear explaination!</p>