Loading a few layers from a pretrained MDNet

Nimrod_Daniel: <p>Hi,</p>
<p>I’m using MDNet, which is a network with 3 conv layers,2 fc layers, plus branches in offline mode or fc layer in online mode (not pretrained, this layer is learned in the offline/online learning).<br>
Thing is, I changed fc5 to add another feature thus I want to load only the weights from the 3 conv layers (conv1-conv3) and the first fc layer (fc4) in online learning. I saw the two solutions mentioned in <a href="https://discuss.pytorch.org/t/how-to-load-part-of-pre-trained-model/1113" class="inline-onebox">How to load part of pre trained model?</a><br>
but still not sure  how to change load_model function so the net will load only conv1, conv2,  conv3, and fc4 from the .pth file no need to load the .mat file).<br>
Any idea what exactly should I do?</p>
<p>Plus, what exactly this line mean (‘shared_weights’ is not defined anywhere)?<br>
shared_layers = states[‘shared_layers’]</p>
<p>Thanks!</p>
<p>That’s the MDNet class (the forward also includes the new feature, it’s just not shown here, but it’s irrelevant) :</p>
<p>class MDNet(nn.Module):<br>
def <strong>init</strong>(self, model_path=None, K=1):<br>
super(MDNet, self).<strong>init</strong>()<br>
self.K = K<br>
self.layers = nn.Sequential(OrderedDict([<br>
(‘conv1’, nn.Sequential(nn.Conv2d(3, 96, kernel_size=7, stride=2),<br>
nn.ReLU(),<br>
LRN(),<br>
nn.MaxPool2d(kernel_size=3, stride=2))),<br>
(‘conv2’, nn.Sequential(nn.Conv2d(96, 256, kernel_size=5, stride=2),<br>
nn.ReLU(),<br>
LRN(),<br>
nn.MaxPool2d(kernel_size=3, stride=2))),<br>
(‘conv3’, nn.Sequential(nn.Conv2d(256, 512, kernel_size=3, stride=1),<br>
nn.ReLU())),<br>
(‘fc4’,   nn.Sequential(nn.Dropout(0.5),<br>
nn.Linear(512 * 3 * 3, 512),<br>
nn.ReLU())),<br>
(‘fc5’,   nn.Sequential(nn.Dropout(0.5),<br>
nn.Linear(512, 512),<br>
nn.ReLU()))]))</p>
<pre><code>    self.branches = nn.ModuleList([nn.Sequential(nn.Dropout(0.5), 
                                                 nn.Linear(512, 2)) for _ in range(K)])
    
    if model_path is not None:
        if os.path.splitext(model_path)[1] == '.pth':
            self.load_model(model_path)
        elif os.path.splitext(model_path)[1] == '.mat':
            self.load_mat_model(model_path)
        else:
            raise RuntimeError("Unkown model format: %s" % (model_path))
    self.build_param_dict()

def build_param_dict(self):
    self.params = OrderedDict()
    for name, module in self.layers.named_children():
        append_params(self.params, module, name)
    for k, module in enumerate(self.branches):
        append_params(self.params, module, 'fc6_%d'%(k))

def set_learnable_params(self, layers):
    for k, p in self.params.items():
        if any([k.startswith(l) for l in layers]):
            p.requires_grad = True
        else:
            p.requires_grad = False

def get_learnable_params(self):
    params = OrderedDict()
    for k, p in self.params.items():
        if p.requires_grad:
            params[k] = p
    return params

def forward(self, x, k=0, in_layer='conv1', out_layer='fc6'):
    #
    # forward model from in_layer to out_layer

    run = False
    for name, module in self.layers.named_children():
        if name == in_layer:
            run = True
        if run:
            x = module(x)
            if name == 'conv3':
                x = x.view(x.size(0),-1)
            if name == out_layer:
                return x
    
    x = self.branches[k](x)
    if out_layer=='fc6':
        return x
    elif out_layer=='fc6_softmax':
        return F.softmax(x)

def load_model(self, model_path):
    states = torch.load(model_path)
    shared_layers = states['shared_layers']
    self.layers.load_state_dict(shared_layers)

def load_mat_model(self, matfile):
    mat = scipy.io.loadmat(matfile)
    mat_layers = list(mat['layers'])[0]
    
    # copy conv weights
    for i in range(3):
        weight, bias = mat_layers[i*4]['weights'].item()[0]
        self.layers[i][0].weight.data = torch.from_numpy(np.transpose(weight, (3,2,0,1)))
        self.layers[i][0].bias.data = torch.from_numpy(bias[:,0])</code></pre>

LeviViana: <aside class="quote no-group" data-username="Nimrod_Daniel" data-post="1" data-topic="45221">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/nimrod_daniel/48/10282_2.png" class="avatar"> Nimrod_Daniel:</div>
<blockquote>
<p>Plus, what exactly this line mean (‘shared_weights’ is not defined anywhere)?<br>
shared_layers = states[‘shared_layers’]</p>
</blockquote>
</aside>
<p>I guess you are using <a href="https://github.com/HyeonseobNam/py-MDNet" rel="noopener nofollow ugc">this repo</a>. So, in this case, when you load a pth file, say <code>mdnet_vot-otb.pth</code>, it loads a dictionary having only one key: <code>'shared_layers'</code>. This corresponds to the layers trained offline.</p>
<p>Try it out:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
model_weights = torch.load("mdnet_vot-otb.pth")
print(type(model_weights))
for k in model_weights: print(k)
for k in model_weights['shared_layers']: print("Shared layer", k)
</code></pre>
<p>It prints:</p>
<pre data-code-wrap="bash"><code class="lang-bash">&lt;class 'dict'&gt;
shared_layers
Shared layer conv1.0.weight
Shared layer conv1.0.bias
Shared layer conv2.0.weight
Shared layer conv2.0.bias
Shared layer conv3.0.weight
Shared layer conv3.0.bias
Shared layer fc4.0.weight
Shared layer fc4.0.bias
Shared layer fc5.1.weight
Shared layer fc5.1.bias
</code></pre>
<aside class="quote no-group" data-username="Nimrod_Daniel" data-post="1" data-topic="45221">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/nimrod_daniel/48/10282_2.png" class="avatar"> Nimrod_Daniel:</div>
<blockquote>
<p>how to change load_model function so the net will load only conv1, conv2, conv3, and fc4 from the .pth file no need to load the .mat file).<br>
Any idea what exactly should I do?</p>
</blockquote>
</aside>
<p>Your MDNet object has a module called <code>layers</code>. I extracted its structure in the snippet below:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
import torch.nn as nn
from collections import OrderedDict

layers = nn.Sequential(OrderedDict([
                ('conv1', nn.Sequential(nn.Conv2d(3, 96, kernel_size=7, stride=2),
                                        nn.ReLU(inplace=True),
                                        nn.LocalResponseNorm(2),
                                        nn.MaxPool2d(kernel_size=3, stride=2))),
                ('conv2', nn.Sequential(nn.Conv2d(96, 256, kernel_size=5, stride=2),
                                        nn.ReLU(inplace=True),
                                        nn.LocalResponseNorm(2),
                                        nn.MaxPool2d(kernel_size=3, stride=2))),
                ('conv3', nn.Sequential(nn.Conv2d(256, 512, kernel_size=3, stride=1),
                                        nn.ReLU(inplace=True))),
                ('fc4',   nn.Sequential(nn.Linear(512 * 3 * 3, 512),
                                        nn.ReLU(inplace=True))),
                ('fc5',   nn.Sequential(nn.Dropout(0.5),
                                        nn.Linear(512, 512),
                                        nn.ReLU(inplace=True)))]))
for k in layers.state_dict(): print("Module Layer", k)
</code></pre>
<p>And it prints:</p>
<pre data-code-wrap="bash"><code class="lang-bash">Module Layer conv1.0.weight
Module Layer conv1.0.bias
Module Layer conv2.0.weight
Module Layer conv2.0.bias
Module Layer conv3.0.weight
Module Layer conv3.0.bias
Module Layer fc4.0.weight
Module Layer fc4.0.bias
Module Layer fc5.1.weight
Module Layer fc5.1.bias
</code></pre>
<p>It means then that the keys in the shared_layers match perfectly the keys in the module layer, that’s why <a href="https://github.com/HyeonseobNam/py-MDNet/blob/master/modules/model.py#L135" rel="noopener nofollow ugc">this works</a>.</p>
<p>Suppose I changed the <code>fc5</code> linear layer to <code>nn.Linear(512, 1024)</code>. Now If I tried to load the weights it wouldn’t work directly. Here’s a workaround:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
import torch.nn as nn
from collections import OrderedDict

layers = nn.Sequential(OrderedDict([
                ('conv1', nn.Sequential(nn.Conv2d(3, 96, kernel_size=7, stride=2),
                                        nn.ReLU(inplace=True),
                                        nn.LocalResponseNorm(2),
                                        nn.MaxPool2d(kernel_size=3, stride=2))),
                ('conv2', nn.Sequential(nn.Conv2d(96, 256, kernel_size=5, stride=2),
                                        nn.ReLU(inplace=True),
                                        nn.LocalResponseNorm(2),
                                        nn.MaxPool2d(kernel_size=3, stride=2))),
                ('conv3', nn.Sequential(nn.Conv2d(256, 512, kernel_size=3, stride=1),
                                        nn.ReLU(inplace=True))),
                ('fc4',   nn.Sequential(nn.Linear(512 * 3 * 3, 512),
                                        nn.ReLU(inplace=True))),
                ('fc5',   nn.Sequential(nn.Dropout(0.5),
                                        nn.Linear(512, 1024),
                                        nn.ReLU(inplace=True)))]))

model_weights = torch.load("mdnet_vot-otb.pth")

d = model_weights['shared_layers']
d['fc5.1.weight'] = torch.randn((1024, 512)) * 0.01
d['fc5.1.bias'] = torch.zeros(1024)
layers.load_state_dict(d)
</code></pre>

Nimrod_Daniel: <p>Interesting approach. Loading the weights, create an initialized modified layer weights, and then loading the initialized weights from that layer to the net.<br>
In my case I need to add a new input to fc5</p>
<pre><code>            ('fc4',   nn.Sequential(nn.Dropout(0.5),
                                    nn.Linear(512 * 3 * 3, 512),  
                                    nn.ReLU())),
            ('fc5',   nn.Sequential(nn.Dropout(0.5),
                                    nn.Linear(512+1, 512), 
                                    nn.ReLU()))]))
</code></pre>
<p>I changed the load_model function to:</p>
<pre><code>def load_model(self, model_path):
    model_weights = torch.load(model_path)
    d = model_weights['shared_layers']
    d['fc5.1.weight'] = torch.randn((512, 513)) * 0.01
    d['fc5.1.bias'] = torch.zeros(512)
    self.layers.load_state_dict(d)
</code></pre>
<p>And it seems to work. I’ll just take a look at  the weights of fc5.<br>
Thanks a lot for your elaborate answer!</p>

Nimrod_Daniel: <p>Which changes to load_model would be required if I’d want  to add a new initialized conv layer (let’s call it ‘conv4’)?</p>

LeviViana: <p>You would need to change <code>self.layers</code> too, by inserting your conv4 in the right place. You should add two new keys to the dictionary <code>d</code> (say that your conv4 is something like <code>torch.nn.Conv2d(512, 512, 3)</code>):</p>
<pre><code class="lang-python">d['conv4.0.weight'] = torch.randn((512, 512, 3, 3)) * 0.01
d['conv4.0.bias'] = torch.zeros(512)
</code></pre>
<p>Note: be sure to flag these parameters as <code>requires_grad = True</code>, otherwise it won’t learn.</p>

Nimrod_Daniel: <p>In that case I’ll need to change self.layers and the forward section, sure. I was just wondering in regard to changes in load_model.<br>
Good to know that the same trick would work even if a new layer is not defined in the pretrained model. So basically I can add whatever I want and assign it to the relevant weights. Things are clearer now. Thank you <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

Nimrod_Daniel: <p>Something very odd had happened After I switched from the semi-original code to  the original code. They’re pretty much the same with small changes and organized differently. I made relevant adjustments, but everything that relates to loading the file is the same (I even copied the MDNet class).</p>
<p>I get a very odd error message:<br>
RuntimeError: Error(s) in loading state_dict for Sequential:<br>
Missing key(s) in state_dict: “fc4.1.weight”, “fc4.1.bias”.<br>
Unexpected key(s) in state_dict: “fc4.0.weight”, “fc4.0.bias”.</p>
<p>The network is ok, looks like this:<br>
Module Layer conv1.0.weight<br>
Module Layer conv1.0.bias<br>
Module Layer conv2.0.weight<br>
Module Layer conv2.0.bias<br>
Module Layer conv3.0.weight<br>
Module Layer conv3.0.bias<br>
Module Layer fc4.1.weight<br>
Module Layer fc4.1.bias<br>
Module Layer fc5.1.weight<br>
Module Layer fc5.1.bias</p>
<p>But oddly, d looks like this:<br>
([‘conv1.0.weight’, ‘conv1.0.bias’, ‘conv2.0.weight’, ‘conv2.0.bias’, ‘conv3.0.weight’, ‘conv3.0.bias’, ‘fc4.0.weight’, ‘fc4.0.bias’, ‘fc5.1.weight’, ‘fc5.1.bias’])</p>
<p>Where did fc4.0.weight and fc4.0.bias come from? I load from the exact same .pth file, use the exact same class model, I really have no idea how is it even possible.  Any idea?</p>

LeviViana: <aside class="quote no-group" data-username="LeviViana" data-post="2" data-topic="45221">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/leviviana/48/9548_2.png" class="avatar"> LeviViana:</div>
<blockquote>
<p>Shared layer fc4.0.weight Shared layer fc4.0.bias</p>
</blockquote>
</aside>
<p>In my post above, you can see that these parameters exist in the loaded dictionary (i.e. the <code>mdnet_vot-otb.pth</code> file).</p>

Nimrod_Daniel: <p>Yeah, I see. Things is, the exact same code and same weights file result in different keys.<br>
In the older code the net has fc4.1.bias and fc4.1.weight, and in the .pth file there’s also  fc4.1.bias and fc4.1.weight in d.<br>
In the new code the net has fc4.1.bias and fc4.1.weight, but in the .pth file there’s  fc4.0.bias and fc4.0.weight in d.</p>
<p>Seeing that, I copied only to network and weight loader to a new file in order to see what would happen - the net has fc4.0.bias and fc4.0.weight, but in the file there’s  fc4.1.bias and fc4.1.weight.</p>
<p>Why there’s a difference if it’s exactly the same code?</p>

Ghsoun: <p>I am a beginner，and I want to change the base CNN model ,the conv1 conv2 conv3 in the MDNet, to the corresponding parts of Resnet-50).<br>
Could you please give me any idea that how can I do that?<br>
I really appreciate any help.<br>
Thanks</p>