Fixing weights locally in a neural network

Benja: <p>I have trained a straighforward network and I have saved their weights. Then  I wanted to train an autoencoder using the weights of the previous trained model, training only the decoder part. I have specificied that I do not want gradients in the encoder part in order to not optimize their parameters. Also I have specified in the optimizer that I only want to optimize the parameters of the decoder part. I do not get any error, but it is not working well because when I compare the two dictionaries of weights of the encoder and the pretrained model, they are not the same. I have attached the models I am using for: the pretrained model is “Class1”, and the autoencoder is “Class2”(the encoder part is self.layers).This two classes are in a file named models.py.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/4/64a5a05fda596591fcdae3979462166a9bd65437.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/64a5a05fda596591fcdae3979462166a9bd65437" title="Class2"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/4/64a5a05fda596591fcdae3979462166a9bd65437_2_690x355.jpeg" alt="Class2" data-base62-sha1="emmBcNH8zQ0FIWjqKRZH63zYWXl" width="690" height="355" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/4/64a5a05fda596591fcdae3979462166a9bd65437_2_690x355.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/6/4/64a5a05fda596591fcdae3979462166a9bd65437_2_1035x532.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/6/4/64a5a05fda596591fcdae3979462166a9bd65437_2_1380x710.jpeg 2x" data-dominant-color="F9F9F9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Class2</span><span class="informations">1920×988 83.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/8/a8970a6f3795c8a8f51a1604107346d4b71f80fa.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/a8970a6f3795c8a8f51a1604107346d4b71f80fa" title="Class1"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/8/a8970a6f3795c8a8f51a1604107346d4b71f80fa_2_690x401.jpeg" alt="Class1" data-base62-sha1="o3pOJOyFJNBkPBCMkhQnVD1YLBU" width="690" height="401" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/8/a8970a6f3795c8a8f51a1604107346d4b71f80fa_2_690x401.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/a/8/a8970a6f3795c8a8f51a1604107346d4b71f80fa_2_1035x601.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/a/8/a8970a6f3795c8a8f51a1604107346d4b71f80fa_2_1380x802.jpeg 2x" data-dominant-color="FAFAFA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Class1</span><span class="informations">1920×1116 81.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/4/a49976c022fa96007f8eb4b09b89b12c3a87a9b2.png" data-download-href="https://discuss.pytorch.org/uploads/default/a49976c022fa96007f8eb4b09b89b12c3a87a9b2" title="Save_Model"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/4/a49976c022fa96007f8eb4b09b89b12c3a87a9b2_2_690x43.png" alt="Save_Model" data-base62-sha1="nu76cjmkMcNRpv6odvNXckhxTIC" width="690" height="43" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/4/a49976c022fa96007f8eb4b09b89b12c3a87a9b2_2_690x43.png, https://discuss.pytorch.org/uploads/default/original/3X/a/4/a49976c022fa96007f8eb4b09b89b12c3a87a9b2.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/a/4/a49976c022fa96007f8eb4b09b89b12c3a87a9b2.png 2x" data-dominant-color="DEDEDE"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Save_Model</span><span class="informations">848×54 34.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/e/7ef62b047d25fdace2e3a746828b327315f66706.png" data-download-href="https://discuss.pytorch.org/uploads/default/7ef62b047d25fdace2e3a746828b327315f66706" title="Load_Model"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/7/e/7ef62b047d25fdace2e3a746828b327315f66706_2_690x106.png" alt="Load_Model" data-base62-sha1="i79zzVQVUkr0rQRuSMtfZYioJnw" width="690" height="106" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/7/e/7ef62b047d25fdace2e3a746828b327315f66706_2_690x106.png, https://discuss.pytorch.org/uploads/default/optimized/3X/7/e/7ef62b047d25fdace2e3a746828b327315f66706_2_1035x159.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/7/e/7ef62b047d25fdace2e3a746828b327315f66706_2_1380x212.png 2x" data-dominant-color="EDECEC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Load_Model</span><span class="informations">1576×244 151 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Please do not pay attention to the comments in the code, I have tried so many things. Thanks in advance!!!</p>

ptrblck: <p>Could you post a minimal, executable code snippet reproducing this issue by wrapping it into three backticks ```, please?</p>

Benja: <p>My code is a bit long and I am using 70000 images as dataset, but the parts in the code in which I save and load the model is what I have attached before. For example if I print “model.layers.state_dict()[‘4.bias’]” in FFN() I get “tensor([-0.0336, -0.0421])”. But if I print the same in the Auto_encod_trained I get “tensor([0.0705, 0.0294])”, and they should be the same. I basically want to know if there is something more to take into account in order to fix the weights of the encoder part during the training with the weights of the pretained model (in my case FFN()). Thanks!!</p>

ptrblck: <p>No, nothing else should be needed and it works fine using:</p>
<pre><code class="lang-python">class Auto_encod_trained(nn.Module):
    def __init__(self):
        super().__init__()
        self.layers = nn.Sequential(
            nn.Linear(768, 100),
            nn.ReLU(),
            nn.Linear(100, 100),
            nn.ReLU(),
            nn.Linear(100, 2)
        )
        
class FFN(nn.Module):
    def __init__(self):
        super().__init__()
        self.olayer = nn.Linear(100, 2)
        self.layers = nn.Sequential(
            nn.Linear(768, 100),
            nn.ReLU(),
            nn.Linear(100, 100),
            nn.ReLU(),
            self.olayer
        )
        
model_a = Auto_encod_trained()
model_b = FFN()

print((model_a.layers[0].weight == model_b.layers[0].weight).all())
# tensor(False)

model_b.layers.load_state_dict(model_a.layers.state_dict())
# &lt;All keys matched successfully&gt;

print((model_a.layers[0].weight == model_b.layers[0].weight).all())
# tensor(True)
</code></pre>

Benja: <p>Okey, thanks!!. I think the problem is when I read the state_dict() of model a. Because I have the two models in different scripts I have saved “model a” using torch.save() as this:</p>
<p>model_path = ‘/home/ben/Desktop/model_trained_encoder.pth’<br>
torch.save(model.layers.state_dict(),model_path)</p>
<p>Then in the second script, when I load it like this:</p>
<p>model = models.Auto_encod_trained()<br>
model.layers.load_state_dict(torch.load(‘/home/ben/Desktop/model_trained_encoder.pth’))</p>
<p>,does not copy the weights correctly. Maybe is the format of the file in which I am saving the model, I don’t know. Anyway, thanks a lot!!</p>

ptrblck: <p>No,I don’t think it’s the format and PyTorch should raise an error is keys are missing or mismatching while loading the <code>state_dict</code>. Let me know if you have a code snippet to reproduce the issue.</p>

Benja: <p>Okey, in your example if you save “model a” as model_a.pth with torch.save, and then you load it as follows:</p>
<p>‘’’<br>
model = model_b()<br>
model.layers.load_state_dict(torch.load(‘model_a.pth’))</p>
<p>‘’’<br>
It works for you?. I mean both dictionaries, model_b.state_dict() and model_a.state_dict() match correctly?. Thanks!!</p>

ptrblck: <p>Yes, my previous code snippet still works for me by adding the <code>torch.save</code> and <code>torch.load</code> calls additionally.</p>