How to change the name of the weights to a new name when saving the model

isalirezag: <p>How I can change the name of the weights in a models when i want to save them?</p>
<p>Here is what i want to do:</p>
<p>I do  <code>torch.load</code>  to load the pretrained model and update the weights for<code>self.Conv1</code> (where<br>
<code>self.Conv1 = nn.Conv2d(3,3,kernel_size=1, stride=1, padding=0, bias=False)</code>)</p>
<p>after training my model i like to save the weights of <code>self.Conv1</code> as <code>self.Conv1_V2</code>, because i have the same layer called <code>self.Conv1_V2</code> in my new model that i like to be initialized with the weight of <code>self.Conv1</code>.</p>
<p>How can I do that? <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=6" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

isalirezag: <p>I found some way to deal with it but im not sure it that is correct…<br>
after loading my first model i can do<br>
<code>self.Conv1_V2 = self.Conv1</code></p>
<p>and then when i save this model i will have both <code>self.Conv1_V2</code> and <code>self.Conv1</code>, so i can use <code>self.Conv1_V2</code></p>
<p>Does this make sense?</p>

ptrblck: <p>Alternatively you could manipulate the names in your <code>state_dict</code> before saving it.<br>
This would avoid having “zombie” layers in your model, which are not used anymore.</p>

isalirezag: <p>That is a very useful way to manipulate names in the state_dict. However, im not sure how to do it…<br>
can you please explained a little bit more or provide an example on how to do that?</p>
<p>i also have another question.<br>
when i do the following code i will have:</p>
<pre><code class="lang-auto">load_name = os.path.join(input_dir, 'trfcn_1_1_52.pth')
checkpoint = torch.load(load_name)
checkpoint['model'].keys()
</code></pre>
<p>the output is:</p>
<blockquote>
<p>odict_keys([‘RCNN_rpn.RPN_Conv.weight’, ‘RCNN_rpn.RPN_Conv.bias’, ‘RCNN_rpn.RPN_cls_score.weight’, ‘RCNN_rpn.RPN_cls_score.bias’, ‘RCNN_rpn.RPN_bbox_pred.weight’, ‘RCNN_rpn.RPN_bbox_pred.bias’, ‘InsideNet.Conv1.weight’, ‘InsideNet.Conv3.weight’, ‘InsideNet.Conv4.weight’, ‘RCNN_bbox_base.weight’, ‘RCNN_cls_base.weight’])</p>
</blockquote>
<p>is there a way that i just keep the</p>
<blockquote>
<p>‘InsideNet.Conv1.weight’, ‘InsideNet.Conv3.weight’, ‘InsideNet.Conv4.weight’</p>
</blockquote>
<p>and delete the rest???</p>
<p>Thanks a lot</p>

ptrblck: <p>You code should rename the <code>self.conv1</code> layers:</p>
<pre><code class="lang-python">class ModelA(nn.Module):
    def __init__(self):
        super(ModelA, self).__init__()
        self.conv1 = nn.Conv2d(1, 6, 3, 1, 1)
        self.fc1 = nn.Linear(6*4*4, 2)
        
    def forward(self, x):
        x = F.relu(self.conv1(x))
        x = self.fc1(x)
        return x


class ModelB(nn.Module):
    def __init__(self):
        super(ModelB, self).__init__()
        self.conv1_v2 = nn.Conv2d(1, 6, 3, 1, 1)
        self.fc1 = nn.Linear(6*4*4, 2)
        
    def forward(self, x):
        x = F.relu(self.conv1_v2(x))
        x = self.fc1(x)
        return x

modelA = ModelA()
modelB = ModelB()
state_dict = modelA.state_dict()
state_dict_v2 = copy.deepcopy(state_dict)
for key in state_dict:
    if 'conv1' in key:
        pre, post = key.split('.')
        state_dict_v2[pre+'_v2'+'.'+post] = state_dict_v2.pop(key)

modelB.load_state_dict(state_dict_v2)
(modelB.conv1_v2.weight == modelA.conv1.weight).all()
</code></pre>
<p>If you want to filter out some parameters, you could adapt this dict comprehension with your condition:</p>
<pre><code class="lang-python">state_dict_filt = {k: v for k, v in state_dict.items() if 'fc1' in k}
</code></pre>

isalirezag: <p>hmmmm im a little confuse <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=6" title=":confused:" class="emoji" alt=":confused:"><br>
I mean when you rename <code>conv1</code> to <code>conv1_v2</code> why should <code>load_state_dict(state_dict_v2)</code> goes and update the weights of  conv1 with conv1_v2, they have the same name …<br>
Also im using <code>strict=False</code> (bc my two model are not exactly the same)  in my load_state_dict if that matters</p>

ptrblck: <p>I thought that was your use case, i.e. saving a model with certain layer names and load these parameters in another model using other layer names.<br>
Would that work for you? Alternatively, one could probably manipulate the layers using <code>getattr</code> and <code>setattr</code>, but I’m not a huge fan of it, since it might break other stuff (e.g. hooks), although I’m not sure about it.</p>

isalirezag: <p>Im sorry to coming back to this question<br>
Here is what im trying to do in a very simple example:<br>
I have this simple model:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import torch.nn.functional as F

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(1, 6, 3, 1, 1)
        self.fc1 = nn.Linear(6*4*4, 2)
        
    def forward(self, x):
        x = F.relu(self.conv1_v2(x))
        x = self.fc1(x)
        return x
    
model = Net()
print(model)

</code></pre>
<p>if i print it i will have</p>
<blockquote>
<p>Net(<br>
(conv1): Conv2d (1, 6, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))<br>
(fc1): Linear(in_features=96, out_features=2)<br>
)</p>
</blockquote>
<p>I want to make it to be</p>
<blockquote>
<p>Net(<br>
(<strong>conv1_V2</strong>): Conv2d (1, 6, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))<br>
(fc1): Linear(in_features=96, out_features=2)<br>
)</p>
</blockquote>
<p>and i want to do it before i start training it, so later when i save it i will no longer have conv1 and instead, i will have conv1_V2.<br>
and after saving and reloading this model, this conv1_V2 is going to be use in another network later. is it a way to do it?</p>
<p>so far i was using the zombie way to do it but i notice when i do that the conv1_V2 will not show up in the <code>model.named_parameters()</code> <img src="https://discuss.pytorch.org/images/emoji/apple/ok_man.png?v=12" title=":ok_man:" class="emoji" alt=":ok_man:" loading="lazy" width="20" height="20"><img src="https://discuss.pytorch.org/images/emoji/apple/man_shrugging/4.png?v=12" title=":man_shrugging:t4:" class="emoji" alt=":man_shrugging:t4:" loading="lazy" width="20" height="20"></p>
<p>P.S. by conv1_V2 not showing up in model.named_parameters() i mean:</p>
<pre><code class="lang-auto">
import torch
import torch.nn as nn
import torch.nn.functional as F

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(1, 6, 3, 1, 1)
        self.fc1 = nn.Linear(6*4*4, 2)
        self.conv1_v2 = self.conv1

        
    def forward(self, x):
        x = F.relu(self.conv1_v2(x))
        x = self.fc1(x)
        return x
    
model = Net()
print(model)
params=[]
for key, value in dict(model.named_parameters()).items():
    if value.requires_grad:
        print(key)
</code></pre>
<blockquote>
<p>Net(<br>
(conv1): Conv2d (1, 6, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))<br>
(fc1): Linear(in_features=96, out_features=2)<br>
(conv1_v2): Conv2d (1, 6, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))<br>
)<br>
conv1.weight<br>
conv1.bias<br>
fc1.weight<br>
fc1.bias</p>
</blockquote>

ptrblck: <p><code>conv1_v2</code> won’t show up in the parameters, as it’s basically just pointing to <code>conv1</code>.<br>
The underlying layer will be trained, since <code>conv1</code> is in the parameters.<br>
If you compare the gradients of <code>conv1</code> and <code>conv1_v2</code>, you’ll see that they are identical.</p>
<p>Would that work for your use case?<br>
If not, could you explain your use case a bit?</p>

isalirezag: <p>Yes, that actually works.<br>
your explanation made it more clear to understand what is going on,<br>
Thanks</p>

klory: <p>if I use <code>torch.save(model)</code>, and then later I have to change the name of the network from <code>ModelB</code> to <code>ModelBOthers</code>  in the class definition. Can I somehow reload the model using <code>torch.load(model_path)</code>?</p>

ptrblck: <p>I’m unfortunately not deeply familiar with saving and loading the model directly, as it might cause some side effects. <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=9" title=":confused:" class="emoji" alt=":confused:"></p>