Create new Model from some of layers of already Pre-trained model

anuj_singh1: <p>class Autoencoder(nn.Module):<br>
def <strong>init</strong>(self):<br>
super().<strong>init</strong>()<br>
<span class="hashtag">#self</span>.linear = nn.Linear(input_size, output_size)<br>
self.layer1 = nn.Linear(input_size, 10)<br>
self.layer2 = nn.Linear(10,6)<br>
self.layer3 = nn.Linear(6, output_size)<br>
self.act = nn.ReLU()</p>
<pre><code>def forward(self, xb):
    #out = self.linear(xb)
    xb = F.relu(self.layer1(xb))
    xb = F.relu(self.layer2(xb))
    out = F.sigmoid(self.layer3(xb))
    #out = self.layer3(xb)
    return out
</code></pre>
<p>Now can I create new torch model with learned weights of only 2 layers of above model:<br>
self.layer1 = nn.Linear(input_size, 10)<br>
self.layer2 = nn.Linear(10,6)</p>

Andrei_Cristea: <p>Hello Anuj, check <a href="https://pytorch.org/tutorials/beginner/saving_loading_models.html#warmstarting-model-using-parameters-from-a-different-model" rel="noopener nofollow ugc">this link</a> out.</p>
<p>As an illustration, suppose this is the other model you want to create (it has its own <code>layer1</code> and <code>layer2</code>).</p>
<pre><code class="lang-auto">class OtherModel(nn.Module):
    def __init__(self, input_size, output_size):
        super(OtherModel, self).__init__()
        self.layer_new_0 = nn.Linear(input_size, input_size)
        self.layer1 = nn.Linear(input_size, 10)
        self.layer2 = nn.Linear(10,6)
        self.layer_new_1 = nn.Linear(6, output_size)
        self.act = nn.ReLU()

    def forward(self, xb):
        xb = F.relu(self.layer_new_0(xb))
        xb = F.relu(self.layer1(xb))
        xb = F.relu(self.layer2(xb))
        out = F.sigmoid(F.relu(self.layer_new_1(xb)))
        return out
</code></pre>
<p>Initiate both an old model and a new one.</p>
<pre><code class="lang-auto">model = Autoencoder(10, 3)
model_new = OtherModel(10, 10)
</code></pre>
<p>Let’s have a look at the old model’s state (parameters) by using the <code>state_dict</code> method.</p>
<pre><code class="lang-auto">print(model.state_dict().keys())
odict_keys(['layer1.weight', 'layer1.bias', 'layer2.weight', 'layer2.bias', 'layer3.weight', 'layer3.bias'])
</code></pre>
<p>The parameters we care about are the ones whose names start with <code>layer1. and layer2.</code></p>
<p>We confirm that the new model’s layer1.weight parameter doesn’t have the same values as the old model layer1.weight parameter yet.</p>
<pre><code class="lang-auto">print(torch.allclose(model.layer1.weight, model_new.layer1.weight))
False
</code></pre>
<p>Let’s use the <code>load_state_dict</code> method to set the new model’s relevant parameters equal to the old model’s. Here I’m using a dictionary comprehension but you can do it many other ways.</p>
<pre><code class="lang-auto">state_dict = {k: model.state_dict()[k] for k in filter(lambda x: x.startswith("layer1.") or x.startswith("layer2."), model.state_dict())}
model_new.load_state_dict(state_dict, strict=False)
</code></pre>
<p>Now you can confirm that the layer1.weight parameter has indeed been copied over.</p>
<pre><code class="lang-auto">print(torch.allclose(model.layer1.weight, model_new.layer1.weight))
True
</code></pre>

anuj_singh1: <p>Hi <a href="/u/Andrei_Cristea">Andrei_Cristea</a> thank you very much for your time and effort to answer my question.<br>
This solution seems very good.<br>
However meanwhile I was trying to figure out and achieved using below code:</p>
<pre><code class="lang-auto">class Autoencoder(nn.Module):
    def __init__(self):
        super(Autoencoder, self).__init__()
        self.encoder = nn.Sequential(
            nn.Linear(29, 14),
            nn.Tanh(),
            nn.Linear(14, 7),
            nn.LeakyReLU(),
            )
        
        self.decoder = nn.Sequential(
           nn.Linear(7, 7),
           nn.Tanh(),
           nn.Linear(7, 29),
           nn.LeakyReLU()
        )

    def forward(self, x):
        x = self.encoder(x)
        x = self.decoder(x)
        return x

model = Autoencoder()
</code></pre>
<p>========= another model  which will use trained weights of first model(Autoencoder one)========</p>
<pre><code class="lang-auto">class Latent(nn.Module):
    def __init__(self):
        super().__init__()
        #self.linear = nn.Linear(input_size, output_size)
        self.layer1 = nn.Linear(29, 14)
        self.layer1.weight=nn.Parameter(model.encoder[0].weight.data)
        self.layer2 = nn.Linear(14,7)
        self.layer2.weight=nn.Parameter(model.encoder[2].weight.data)
        
    def forward(self, xb):
        #out = self.linear(xb)
        xb = F.tanh(self.layer1(xb))
        out = F.relu(self.layer2(xb))
        return out

hidden = Latent()
</code></pre>
<p>Could you please have a look ,if this approach is right?</p>

Andrei_Cristea: <p>Hi Anuj,</p>
<p>This definitely could work, but just wanted to point out a few things.</p>
<p>First, note that you did match the weights, but not the biases, which might cause an issue if you’re not aware of it:</p>
<pre><code class="lang-auto">print(
    torch.allclose(model.encoder[0].weight, hidden.layer1.weight),
    torch.allclose(model.encoder[0].bias,   hidden.layer1.bias)
)
Output:
True False
</code></pre>
<p>Second, I think the two models are now sharing same weight tensors in those layers, which means that updating the values in one place will change them in the other place. This might be fine if it’s what you intended, but it’s important to be aware that it’s happening.</p>
<p>To illustrate, suppose we make a copy of the shared tensor, and we confirm that the copy preserves the values:</p>
<pre><code class="lang-auto">weight_initial = model.encoder[0].weight.detach().clone()
print(torch.allclose(model.encoder[0].weight, weight_initial))
Output:
True
</code></pre>
<p>Then we train the “Latent” model for a few steps:</p>
<pre><code class="lang-auto">from torch import optim
criterion = nn.MSELoss()
optimizer = optim.SGD(hidden.parameters(), lr=1e-3)
for _ in range(10):
    optimizer.zero_grad()
    input = torch.randn(1, 29)
    target = torch.randn(1, 7)
    output = hidden(input)  # explicitly updating "latent" model only, not "autoencoder"
    loss = criterion(output, target)
    loss.backward()
    optimizer.step()
</code></pre>
<p>Even though we never explicitly manipulated the Autoencoder, you can see its weights have changed, since they were shared with the Latent model:</p>
<pre><code class="lang-auto">print(torch.allclose(model.encoder[0].weight, weight_initial))  # "autoencoder" weights have changed
Output:
False
</code></pre>
<p>Relatedly, keep in mind that when you train the Latent model, all of its parameters are going to be changed, including the ones that you transferred over from the Autoencoder. If you don’t want this behavior, and you only want to train some parameters (the new ones) but not others (not the transferred ones), you can “freeze” the desired parameters, as described <a href="https://discuss.pytorch.org/t/how-the-pytorch-freeze-network-in-some-layers-only-the-rest-of-the-training/7088">here</a>.</p>
<p>Good luck!<br>
Andrei</p>

AlphaBetaGamma96: <aside class="quote no-group" data-username="anuj_singh1" data-post="3" data-topic="149243">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/anuj_singh1/48/44796_2.png" class="avatar"> anuj_singh1:</div>
<blockquote>
<pre><code class="lang-auto">        self.layer1.weight=nn.Parameter(model.encoder[0].weight.data)
</code></pre>
</blockquote>
</aside>
<p>Just also want to state here that you shouldn’t be using the <code>.data</code> attribute as it’s deprecated and can lead to undefined behavior. Try something like this instead,</p>
<pre><code class="lang-auto">self.layer1.weight.copy_(model.encoder[0].weight)
</code></pre>

anuj_singh1: <p>Thanks <a class="mention" href="/u/andrei_cristea">@Andrei_Cristea</a> for such a insightful response…</p>

anuj_singh1: <p>Thanks <a class="mention" href="/u/alphabetagamma96">@AlphaBetaGamma96</a> for highlighting issue.</p>