Some detailed problem about torch.load_state_dict()

username: Zichun_Zhang
cooked: <pre><code class="lang-auto">      self.featureExtract = nn.Sequential(                               # 271
            nn.Conv2d(configs[0], configs[1] , kernel_size=11, stride=2),  # 131
            nn.BatchNorm2d(configs[1]),
            nn.MaxPool2d(kernel_size=3, stride=2),    #65
            nn.ReLU(inplace=True),
            nn.Conv2d(configs[1], configs[2], kernel_size=5),   #61
            nn.BatchNorm2d(configs[2]),
            nn.MaxPool2d(kernel_size=3, stride=2),    #30
            nn.ReLU(inplace=True),
            nn.Conv2d(configs[2], configs[3], kernel_size=3), #28
            nn.BatchNorm2d(configs[3]),
            nn.ReLU(inplace=True),
            nn.Conv2d(configs[3], configs[4], kernel_size=3), #26
            nn.BatchNorm2d(configs[4]),
            nn.ReLU(inplace=True),
            nn.Conv2d(configs[4], configs[5], kernel_size=3), #24
            nn.BatchNorm2d(configs[5]),
        )

        self.conv_r1 = nn.Conv2d(feat_in, feature_out*4*anchor, 3)
        self.conv_r2 = nn.Conv2d(feat_in, feature_out, 3)
        self.conv_cls1 = nn.Conv2d(feat_in, feature_out*2*anchor, 3)
        self.conv_cls2 = nn.Conv2d(feat_in, feature_out, 3)
        self.regress_adjust = nn.Conv2d(4*anchor, 4*anchor, 1)
</code></pre>
<p>Above is the nn module of my defined class, now I only want to use the pre-trained parameters of the self.featureExtract part but not the later part,<br>
so how should I do in order to load only partial parameters of the</p>
<pre><code class="lang-auto">net.load_state_dict(torch.load(net_file))
</code></pre>
<p>(This above is going to load all the params but I just want to load some specific part)</p>

username: Amrit_Das
cooked: <aside class="quote quote-modified" data-post="2" data-topic="1113">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/chenyuntc/48/299_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/how-to-load-part-of-pre-trained-model/1113/2">How to load part of pre trained model?</a> 
  </div>
  <blockquote>
    I’m afraid not 

The keys of state_dict must exactly match the keys returned by this module’s  state_dict() function. 

if name not in new model , it will raise  KeyError 
but I gusee this may work for you 
    def load_my_state_dict(self, state_dict):
 
        own_state = self.state_dict()
        for name, param in state_dict.items():
            if name not in own_state:
                 continue
            if isinstance(param, Parameter):
                # backwards compatibility for seria…
  </blockquote>
</aside>

<p>This is a similiar kind of answer from</p>
<aside class="quote" data-post="1" data-topic="1113">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/albertxavier001/48/1808_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/how-to-load-part-of-pre-trained-model/1113">How to load part of pre trained model?</a> 
  </div>
  <blockquote>
    How to load part of pre trained model? 
Will the unused parameters be auto deleted if I load the whole model but only use part of it?
  </blockquote>
</aside>


username: Zichun_Zhang
cooked: <p>thx a lot, BTW, do you know about what does “key” really mean?<br>
Is that kind of like hash value or?</p>

username: JuanFMontesinos
cooked: <p>when u have a python dictionary you acess to each element by doing dict[key]<br>
it’s an string or integer.</p>
<p>models are stored in ordered dictionaries whose keys are the names of each parameter</p>
<p>if you do statedict[model.submodel.weight] it will return the tensor value</p>

username: Zichun_Zhang
cooked: <p>Thx a lot Juan,<br>
But there is a great a lot parameters that have same names, eg. nn.Conv2d, really a lot.<br>
So how could it know which nn.Conv2d is the right one to choose, when use this dictionary to mapping.<br>
Thx~</p>

username: JuanFMontesinos
cooked: <p>They key is the name that you assign to the variable in the nn.Module, therefore</p>
<pre><code class="lang-auto">class test(torch.nn.Module):
    def __init__(self):
        super(test,self).__init__()
        self.conv1 = torch.nn.Conv2d(10,15,10)
        self.customconv = torch.nn.Conv2d(100,1000,10)

test()
Out[7]: 
test(
  (conv1): Conv2d(10, 15, kernel_size=(10, 10), stride=(1, 1))
  (customconv): Conv2d(100, 1000, kernel_size=(10, 10), stride=(1, 1))
)
</code></pre>
<p>if you assigned proper names<br>
you will get a tractable state dict<br>
in this case</p>
<pre><code class="lang-auto">q.keys()
Out[17]: odict_keys(['conv1.weight', 'conv1.bias', 'customconv.weight', 'customconv.bias'])
</code></pre>
<p>since I assigned different names i get different keys.<br>
if you always use conv1 you will get a wrong and not functional state dict</p>

username: Zichun_Zhang
cooked: <p>thx a lot!!!<img src="https://discuss.pytorch.org/images/emoji/apple/+1.png?v=6" title=":+1:" class="emoji" alt=":+1:"><img src="https://discuss.pytorch.org/images/emoji/apple/+1.png?v=6" title=":+1:" class="emoji" alt=":+1:"><img src="https://discuss.pytorch.org/images/emoji/apple/+1.png?v=6" title=":+1:" class="emoji" alt=":+1:"><img src="https://discuss.pytorch.org/images/emoji/apple/+1.png?v=6" title=":+1:" class="emoji" alt=":+1:"></p>

username: Jimmy_Xiaoke_Shen
cooked: <aside class="quote no-group" data-username="How to load part of pre trained model?" data-post="2" data-topic="1113">
<div class="title">
<div class="quote-controls"></div>
<a href="https://discuss.pytorch.org/t/how-to-load-part-of-pre-trained-model/1113/2">How to load part of pre trained model?</a></div>
<blockquote>
<p>load_my_state_dict</p>
</blockquote>
</aside>
<p>Thanks a lot. In “q.keys()”, what is the meaning of ‘q’. Thanks.</p>

username: JuanFMontesinos
cooked: <p>Dont remember, a local variable i used to store some dict for showing.</p>

username: Jimmy_Xiaoke_Shen
cooked: <p>Thanks for your quick reply. I am transferring from tensorflow to pytorch recently and I am still learning. q should be the state_dict of the model defined in the test. I will try to explore this part. Thanks Juan.</p>

username: Jimmy_Xiaoke_Shen
cooked: <p>I figured it out. It is interesting that the state_dict has no relation to the forward function. I thought the model is defined by the forward function.</p>
<pre><code class="lang-python">import torch
import torch.nn as nn
import torch.nn.functional as F
class test(torch.nn.Module):
    def __init__(self):
        super(test,self).__init__()
        self.conv1 = torch.nn.Conv2d(10,15,10)
        self.customconv = torch.nn.Conv2d(100,1000,10)
    """
    def forward(self, x):
        x = self.pool(F.relu(self.conv1(x)))
        x = self.pool(F.relu(self.suctomconv(x)))
        x = self.pool(F.relu(self.suctomconv(x)))
        return x
    """
model = test()
q = model.state_dict()
print(q.keys())
</code></pre>

username: JuanFMontesinos
cooked: <p>Hi,<br>
The model is “defined” in the forward function. However, pytorch assumes that if you define some weights it means you are gonna use them and saves them in the state_dict.</p>
<p>Forward define how those weights are connected</p>

username: Jimmy_Xiaoke_Shen
cooked: <aside class="quote no-group" data-username="Jimmy_Xiaoke_Shen" data-post="11" data-topic="31970">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/j/76d3ee/48.png" class="avatar"> Jimmy_Xiaoke_Shen:</div>
<blockquote>
<p>x = self.pool(F.relu(self.suctomconv(x))) x = self.pool(F.relu(self.suctomconv(x)))</p>
</blockquote>
</aside>
<p>Thanks Juan. It’s interesting to know this. Then I assume<br>
if we have the following forward, the graph defined here will go through customconv two times, it is something like the following:<br>
CONV1 →   CUSTOMCONV1 → CUSTOMCONV1(SAME as previous one)<br>
instead of<br>
CONV1 →   CUSTOMCONV1 → CUSTOMCONV1(DIFFERENT as previous one)</p>
<pre data-code-wrap="python"><code class="lang-python">def forward(self, x):
        x = self.pool(F.relu(self.conv1(x)))
        x = self.pool(F.relu(self.customconv(x)))
        x = self.pool(F.relu(self.customconv(x)))
        return x
</code></pre>
<p>If so, all parameters are defined in __init__ and in the forward, we just use them. It is possible that parameters used in forward is a subset  of the parameters defined in __init__.<br>
For those not used one, we did not train them, and the model will be finally saved as the initial random weights.<br>
I am not sure whether I am understanding this part correctly.</p>

username: JuanFMontesinos
cooked: <p>Yes that’s more or less the idea.<br>
So customconv will be updated given the  contribution of both runs.</p>

username: Jimmy_Xiaoke_Shen
cooked: <p>Thanks a lot. Your explanation makes me much clear about the relationship between the init and forward. Also I am pretty clear about how to define the model through the forward function.<br>
It is great. Thanks.</p>