Multiple model instances with common weight updates

username: utsavdutta98
cooked: <p>I have a model called <code>CombinedModel()</code> which itself consists of 3 smaller blocks (named submodel1, submodel2, submodel3).</p>
<p>I created multiple instances of this <code>CombinedModel()</code> as so:</p>
<pre><code class="lang-auto">model1 = CombinedModel()
model2 = CombinedModel()
model3 = CombinedModel()

</code></pre>
<ol>
<li>
<p>When checking the layer weights individually by calling them as <code>model1.submodel1.layer[0].weight</code> I observed that the weights in all three model instances are the same. Is this purely due to having a similar random initialization with the same random seed?</p>
</li>
<li>
<p>I proceeded to train <strong>only</strong> the first model <code>model1</code>:</p>
</li>
</ol>
<pre><code class="lang-auto">def train_model(model,optimizer):

## Loop over epochs
    model.train()
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

# Call instances of model and optimizer
model1 = CombinedModel()
optimizer = torch.optim.Adam(model1.parameters())

# Run function
train_model(model1,optimizer)

</code></pre>
<p>After doing so I checked the layer weights of all 3 models and found that the weights were again the same. How does training model1 independently affect the other weights as well?</p>
<pre><code class="lang-auto">model1.submodule1.layer[0].weight
model2.submodule1.layer[0].weight
model3.submodule1.layer[0].weight
</code></pre>
<p>The above three are all the same, despite model2 and model3 not being trained.</p>
<p>Even when I create a new model instance (<code>model4 = CombinedModel()</code>) <strong>after</strong> finishing <code>model1</code> training, model4 also has the same weights as the other 3 models.</p>
<p>Why is it that each instance has the same weights? And how can I avoid this weight dependency among instances?</p>

username: utsavdutta98
cooked: <p>Not sure, but the problem might be arising in how the <code>CombinedModel()</code> is created.</p>
<pre><code class="lang-auto">class CombinedModel(nn.Module):

	def __init__(self,mod1 = submodule1(), mod2 = submodule2(), mod3 = submodule3()):
		super().__init__()

		self.mod1 = mod1
		self.mod2 = mod2
		self.mod3 = mod3

	def forward(self,x):
		
		x = self.mod3(self.mod2(self.mod1(x)))

		return x
</code></pre>
<p>Is the fact that I am calling the submodule instances inside the creation of the combined module class making it point to the same weight tensors?</p>

username: utsavdutta98
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> any chance you could have a look when you get the time?</p>

username: ariG23498
cooked: <p>Hey <a class="mention" href="/u/utsavdutta98">@utsavdutta98</a><br>
Could you share a colab notebook of sorts where the whole snippet would be easily accessible?</p>
<p>I would really want to look at the <code>submodule</code> class.</p>

username: utsavdutta98
cooked: <p><a class="mention" href="/u/arig23498">@ariG23498</a>  Yeah sure!<br>
<a href="https://colab.research.google.com/drive/1BMLnhSCQoAOuw6jHQ_iLcL63sM6NvKTE?usp=sharing" rel="noopener nofollow ugc">Colab Notebook</a></p>
<p>The first three cells contain the modules, and the final one combines them.</p>
<p>I have printed the weight tensors of each afterwards too.</p>
<p>Thanks for your help <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: ariG23498
cooked: <p>Hey <a class="mention" href="/u/utsavdutta98">@utsavdutta98</a><br>
I can confirm that the way the CustomModel is created makes all the difference.</p>
<p>I made this <a href="https://gist.github.com/ariG23498/7f0a1025d795234fa209e337688dc073" rel="noopener nofollow ugc">GIST</a> to help you find the difference.</p>
<p>It is a wise choice to not create an object through functional arguments.</p>
<p>I did a quick python snippet to investigate the issue:</p>
<pre><code class="lang-python">class A:
    def __init__(self):
        pass

class B:
    def __init__(self, a=A()):
        self.a = a

###############################
obj1 = B()
obj1.a

&lt;__main__.A at 0x7f315079e910&gt;


obj2 = B()
obj2.a
&lt;__main__.A at 0x7f315079e910&gt;
</code></pre>
<p>They are the same object. Hope this helps! <img src="https://discuss.pytorch.org/images/emoji/apple/grinning_face_with_smiling_eyes.png?v=9" title=":grinning_face_with_smiling_eyes:" class="emoji" alt=":grinning_face_with_smiling_eyes:"></p>

username: utsavdutta98
cooked: <p><a class="mention" href="/u/arig23498">@ariG23498</a> ah that clears it up!</p>
<p>I’m not too strong in OOP theory, is there any underlying reason why assigning the variable inside the functional argument makes all instances point to the same object?</p>
<p>Either way, I will definitely keep this in mind from a practical standpoint.<br>
Thanks a lot <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>