Multiple instances of same model!?

akskuchi: <p>Hello,</p>
<p>I have a situation to work with multiple instances of the same model, like this:</p>
<pre><code class="lang-auto">class Decoder(nn.Module):
    pass

decoders = []
for _ in range(some_number):
    decoders.append(Decoder())
</code></pre>
<p>Some weird doubts:</p>
<ol>
<li>All the decoder instances are independent of each other as long as they don’t share a tensor. True/False?</li>
<li>To save them to disk and reload them I’ll have to loop through and do something like this:</li>
</ol>
<pre><code class="lang-auto">'decoder0': decoders[0].state_dict(),
.
.
</code></pre>
<p>is there a better way?</p>
<p>Primarily, I am trying to understand the independence amongst the decoders themselves, and the conditional dependence that comes in, when there is a shared model (like an encoder).</p>
<p>Cheers.</p>

ptrblck: <ol>
<li>Yes, all <code>Decoder</code> instances are independent, since you’ve initialized each one separately.</li>
<li>In your current approach, yes. However, you could also use an <code>nn.ModuleList</code>, which will return all internal submodule states using its <code>state_dict()</code> method:</li>
</ol>
<pre><code class="lang-python">modules = nn.ModuleList()
for _ in range(10):
    modules.append(nn.Linear(1, 1))

modules.state_dict()
&gt; OrderedDict([('0.weight', tensor([[-0.0277]])),
             ('0.bias', tensor([-0.3542])),
             ('1.weight', tensor([[0.2417]])),
             ('1.bias', tensor([0.2794])),
             ('2.weight', tensor([[0.6173]])),
             ('2.bias', tensor([0.7524])),
             ('3.weight', tensor([[-0.9020]])),
             ('3.bias', tensor([0.7507])),
             ('4.weight', tensor([[-0.2359]])),
             ('4.bias', tensor([0.6560])),
             ('5.weight', tensor([[-0.8661]])),
             ('5.bias', tensor([-0.9012])),
             ('6.weight', tensor([[0.7482]])),
             ('6.bias', tensor([0.6804])),
             ('7.weight', tensor([[0.7841]])),
             ('7.bias', tensor([-0.6375])),
             ('8.weight', tensor([[0.6187]])),
             ('8.bias', tensor([-0.3414])),
             ('9.weight', tensor([[0.2675]])),
             ('9.bias', tensor([0.0969]))])
</code></pre>

akskuchi: <p>Great! Thanks for that clarification.</p>
<p>And for the case when there is an encoder intrusion?</p>
<pre><code class="lang-auto">class Encoder(nn.Module):
    pass

class Decoder(nn.Module):
    pass

encoder = Encoder()

decoders = []
for _ in range(some_number):
    decoders.append(Decoder())

intermediate = encoder(input)

output0 = decoders[0](intermediate)
output1 = decoders[1](intermediate)
.
.
</code></pre>
<p>In this scenario, the decoders are still independent of each other, but the encoder is dependent on all the decoders. Correct?</p>

ptrblck: <p>Yes, each output tensor will be attached to a computation graph, which involves the same <code>encoder</code> instance. If you calculate some losses based on these outputs and call <code>backward</code> on them, the gradients will be accumulated in the parameters of <code>encoder</code>:</p>
<pre><code class="lang-python">encoder = nn.Linear(1, 1)

decoders = nn.ModuleList()
for _ in range(3):
    decoders.append(nn.Linear(1, 1))

x = torch.randn(1, 1)
intermediate = encoder(x)

output0 = decoders[0](intermediate)
output1 = decoders[1](intermediate)
output2 = decoders[2](intermediate)

output0.backward(retain_graph=True)
print(encoder.weight.grad)
output1.backward(retain_graph=True)
print(encoder.weight.grad)
output2.backward()
print(encoder.weight.grad)
</code></pre>

akskuchi: <p>Makes sense. Thank you : )</p>