Copying the weights from one model to another

username: Vishak_Raj
cooked: <p>Hi,</p>
<p>I am experiencing this situation,</p>
<p>I trained a model named <code>src_model</code> using resnet18, and I want to use the first four layer and its weight in another model <code>dest_model</code>, as it is.</p>
<p>I saved the <code>src_model</code> using <code>torch.save()</code></p>
<p>In the <code>dest_model</code>, I used to create by taking the first four layers from the <code>src_model </code> which is the resnet18 with fc layers at last.</p>
<pre><code class="lang-auto">    class dest_model(nn.Module):
        def __init__(self):
            super(dest_model, self).__init__()
            resnet = models.resnet18(pretrained=True)
            layers = list(resnet.children())
            self.features = nn.Sequential(*layers[:4])

        def forward(self, x):
            return self.features(x)
    print(dest_model)
</code></pre>
<blockquote>
<p>dest_model(<br>
(features): Sequential(<br>
(0): Conv2d(3, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False)<br>
(1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)<br>
(2): ReLU(inplace=True)<br>
(3): MaxPool2d(kernel_size=3, stride=2, padding=1, dilation=1, ceil_mode=False)<br>
…</p>
</blockquote>
<p>The first four layer from <code>src_model</code> and <code>dest_model</code> are same<br>
and transferred its weights using this format -</p>
<pre><code class="lang-auto">dest_model.features[0].weight.data=src_model.features[0].weight.data.to(device).type(torch.cuda.FloatTensor)
dest_model.features[1].weight.data=src_model.features[1].weight.data.to(device).type(torch.cuda.FloatTensor)
dest_model.features[1].bias.data=src_model.features[1].bias.data.to(device).type(torch.cuda.FloatTensor)
</code></pre>
<p>and I saved the <code>dest_model</code> using <code>torch.save()</code>, ever thing works smoothly</p>
<p>But, the results from <code>dest_model</code>  is different. so, I manually checked the outputs of the intermediate layers using forward hook by referring this <a href="https://discuss.pytorch.org/t/how-can-i-extract-intermediate-layer-output-from-loaded-cnn-model/77301/2">link</a>.</p>
<p>I gave the same input sample to the <code>src_model</code> and <code>dest_model</code> and took the first four layers output using the forward hook from both models, to get to know what is happening,</p>
<pre><code class="lang-auto">for src_key, dest_key in zip(src_activation, dest_activation):
    print(src_key, dest_key)
    print(torch.equal(src_activation[src_key], dest_activation[dest_key]))
    print(src_activation[src_key].shape, dest_activation[dest_key].shape)
</code></pre>
<p><code>src_activation</code>, <code>dest_activation</code> are the Ordered_dict output from forward hook</p>
<p>Here is the result -</p>
<blockquote>
<p>features.0 features.0<br>
True<br>
torch.Size([1, 64, 112, 112]) torch.Size([1, 64, 112, 112])<br>
features.1 features.1<br>
False<br>
torch.Size([1, 64, 112, 112]) torch.Size([1, 64, 112, 112])<br>
features.2 features.2<br>
False<br>
torch.Size([1, 64, 112, 112]) torch.Size([1, 64, 112, 112])<br>
features.3 features.3<br>
False<br>
torch.Size([1, 64, 56, 56]) torch.Size([1, 64, 56, 56])</p>
</blockquote>
<p>From this results, I get to know that the 1st layers has same output and the rest of the layers have different results even though they have same weights,</p>
<p>However, as I copied the weights, the results should be same in all the layers from both the models. But, the results are not same for same input.<br>
The result gets differ from the 2nd layer which is <code>BatchNorm2d</code>,</p>
<p>Could you please explain, What is happening in the <code>BatchNorm2d</code> layer, why, it behaves differently in both models even though, it has same parameters</p>
<p>Thanks</p>

username: ptrblck
cooked: <p>How large are the absolute errors in these layers?<br>
Since you are dealing with floating point numbers using <code>torch.equal</code> doesn’t sound like a good idea as you might run into small errors caused by the limited floating point precision.</p>

username: Vishak_Raj
cooked: <p>hello <a class="mention" href="/u/ptrblck">@ptrblck</a>, thanks for writting,</p>
<p>I calculated the mean absolute error between the output of layers from both models<br>
the result -</p>
<pre><code class="lang-auto">features.0 : mae - 0
features.1 : mae - 0.1092
features.2 : mae - 0.1092
features.3 : mae - 0.1259
</code></pre>
<p>I am wondering that this difference is not good, because due to this variation i get different results at the end of 4th layer</p>
<p>thanks</p>

username: ptrblck
cooked: <p>These errors look a bit high so could you post a minimal, executable code snippet to reproduce the issue, please?</p>

username: Vishak_Raj
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a>, Here is the code</p>
<pre><code class="lang-auto">class src_model1(nn.Module):
    def __init__(self):
        super(src_model1, self).__init__()
        resnet = models.resnet18(pretrained=True)
        layers = list(resnet.children())
        self.features = nn.Sequential(*layers[:4])

    def forward(self, x):
        return 

class dest_model1(nn.Module):
        def __init__(self):
            super(dest_model1, self).__init__()
            resnet = models.resnet18(pretrained=True)
            layers = list(resnet.children())
            self.features = nn.Sequential(*layers[:4])

        def forward(self, x):
            return self.features(x)

src_model = src_model1()
src_model = torch.load('src_model.pth')

dest_model = dest_model1()

dest_model.features[0].weight.data=src_model.features[0].weight.data.to(device).type(torch.cuda.FloatTensor)
dest_model.features[1].weight.data=src_model.features[1].weight.data.to(device).type(torch.cuda.FloatTensor)
dest_model.features[1].bias.data=src_model.features[1].bias.data.to(device).type(torch.cuda.FloatTensor)

activation = OrderedDict()
def get_activation(name):
    def hook(model, input, output):
        activation[name] = output.detach()
    return hook

for name, layer in src_model.named_modules():
    layer.register_forward_hook(get_activation(name))

output = src_model.forward(samples) # only one sample is given as input
 
activation1 = OrderedDict()
def get_activation1(name):
    def hook1(model, input, output):
        activation1[name] = output.detach()
    return hook1

for name, layer in dest_model.named_modules():
    layer.register_forward_hook(get_activation1(name))

output1 = dest_model.forward(samples) # only one sample is given as input which is same input given to src_model

mae = nn.L1Loss()

for key, key1 in zip(activation, activation1):
    print(key, key1)
    print(torch.equal(activation[key], activation1[key1]), mae(activation[key], activation1[key1]))
    print(activation[key].shape, activation1[key1].shape)

</code></pre>
<p>This is the flow, I used,</p>
<p>thanks</p>

username: ptrblck
cooked: <p>Thanks for the code.<br>
A few issues:</p>
<ul>
<li>your <code>src_model1</code> is not executing anything and returns <code>None</code>
</li>
<li>don’t use <code>model.forward</code> as it could skip hooks but call the model instead directly</li>
<li>don’t use the <code>.data</code> attribute as it’s deprecated and could easily break things</li>
<li>
<code>seamples</code> is undefined so I used a random tensor</li>
</ul>
<p>After fixing these issues, your code works fine:</p>
<pre><code class="lang-python">class src_model1(nn.Module):
    def __init__(self):
        super(src_model1, self).__init__()
        resnet = models.resnet18(pretrained=True)
        layers = list(resnet.children())
        self.features = nn.Sequential(*layers[:4])

    def forward(self, x):
        return self.features(x)

class dest_model1(nn.Module):
        def __init__(self):
            super(dest_model1, self).__init__()
            resnet = models.resnet18(pretrained=True)
            layers = list(resnet.children())
            self.features = nn.Sequential(*layers[:4])

        def forward(self, x):
            return self.features(x)

src_model = src_model1().cuda()
dest_model = dest_model1().cuda()

# not needed as you are using the same pretrained models
#with torch.no_grad():
#    dest_model.features[0].weight.copy_(src_model.features[0].weight)
#    dest_model.features[1].weight.copy_(src_model.features[1].weight)
#    dest_model.features[1].bias.copy_(src_model.features[1].bias)

activation = OrderedDict()
def get_activation(name):
    def hook(model, input, output):
        activation[name] = output.detach()
    return hook

for name, layer in src_model.named_modules():
    layer.register_forward_hook(get_activation(name))

samples = torch.randn(1, 3, 224, 224).cuda()
output = src_model(samples) 
 
activation1 = OrderedDict()
def get_activation1(name):
    def hook1(model, input, output):
        activation1[name] = output.detach()
    return hook1

for name, layer in dest_model.named_modules():
    layer.register_forward_hook(get_activation1(name))

output1 = dest_model(samples) 

mae = nn.L1Loss()

for key, key1 in zip(activation, activation1):
    print(key, key1)
    print(torch.equal(activation[key], activation1[key1]), mae(activation[key], activation1[key1]))
    print(activation[key].shape, activation1[key1].shape)
</code></pre>
<p>Output:</p>
<pre><code class="lang-python">features.0 features.0
True tensor(0., device='cuda:0')
torch.Size([1, 64, 112, 112]) torch.Size([1, 64, 112, 112])
features.1 features.1
True tensor(0., device='cuda:0')
torch.Size([1, 64, 112, 112]) torch.Size([1, 64, 112, 112])
features.2 features.2
True tensor(0., device='cuda:0')
torch.Size([1, 64, 112, 112]) torch.Size([1, 64, 112, 112])
features.3 features.3
True tensor(0., device='cuda:0')
torch.Size([1, 64, 56, 56]) torch.Size([1, 64, 56, 56])
features features
True tensor(0., device='cuda:0')
torch.Size([1, 64, 56, 56]) torch.Size([1, 64, 56, 56])
 
True tensor(0., device='cuda:0')
torch.Size([1, 64, 56, 56]) torch.Size([1, 64, 56, 56])
</code></pre>

username: Vishak_Raj
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a>, actually, the <code>src_model</code> is trained model, the weights from the <code>src_model</code> copied to the weights of the <code>dest_model</code> but only for the first four layers.</p>
<p>I missed this line in the above code, <code>src_model = torch.load('src_model.pth')</code></p>
<p>And, <code>samples</code> is the input image, I removed the <code>.data</code> attribute while transferring the weights and predict the <code>sample</code> using <code>model(sample)</code>, the same issue happens again…</p>

username: ptrblck
cooked: <p>OK, could you then update your code snippet (or post a new one) which I can use to reproduce the issue, please?</p>

username: Vishak_Raj
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a>, I already update the <a href="https://discuss.pytorch.org/t/copying-the-weights-from-one-model-to-another/153010/5">code</a>, I just missed this one line,</p>
<p><code>src_model = torch.load('src_model.pth')</code></p>
<p>the <code>src_model</code> is trained with different dataset, I just want to use the first four layers of <code>src_model</code>, in the <code>dest_model</code>, for that, I transferred the weights from <code>src_model</code> to <code>dest_model</code>. when I checked both models with same input image, the first four layers should return the same output, but in this flow, I get different results</p>
<p>could you please help me to solve this issue, thanks</p>