Getting different accuracy using similar models but why?

username: Umair_Javaid
cooked: <p><strong>model1:</strong></p>
<pre><code class="lang-auto">def forward(self, x, labels=None, return_cam=False):
        batch_size = x.shape[0]
        x = self.features(x)
        
        x1 = self.conv6(x)
        x1 = self.relu(x1)
        x1 = self.conv7(x1)
        x1 = self.relu(x1)
        
        x2 = self.conv8(x)
        x2 = self.relu(x2)
        x2 = self.conv9(x2)
        x2 = self.relu(x2)

        x3 = self.conv10(x)
        x3 = self.relu(x3)
        x3 = self.conv11(x3)
        x3 = self.relu(x3)

        x = x1 + x2 + x3
</code></pre>
<p><strong>model2:</strong></p>
<pre><code class="lang-auto">def forward(self, x, labels=None, return_cam=False):
        batch_size = x.shape[0]
        x1 = self.features(x)
        x1 = self.conv6(x1)
        x1 = self.relu(x1)
        x1 = self.conv7(x1)
        x1 = self.relu(x1)
        
        x2 = self.features(x)
        x2 = self.conv8(x2)
        x2 = self.relu(x2)
        x2 = self.conv9(x2)
        x2 = self.relu(x2)

        x3 = self.features(x)
        x3 = self.conv10(x3)
        x3 = self.relu(x3)
        x3 = self.conv11(x3)
        x3 = self.relu(x3)

        x = x1 + x2 + x3
</code></pre>
<p><code>self.features</code> contain pretrained vgg layers<br>
I get a little better accuracy using model2. Why is that?</p>
user_suspended: yes

username: ptrblck
cooked: <p>If <code>self.features</code> contains e.g. batchnorm layers, you would update the running statistics three times in <code>model2</code>, which might improve the training accuracy.<br>
Let me know, if that’s not the case and we can dig a bit deeper.</p>

username: Umair_Javaid
cooked: <p>There is no batch_normalization. But, after block 3, 4 and 5, I am taking sigmoid of the feature maps and then multiplying the resultant with the feature maps.</p>
<pre><code class="lang-auto">attention = torch.mean(input_, dim=1, keepdim=True)
importance_map = torch.sigmoid(attention)
return input_.mul(importance_map)
</code></pre>
<p>could that be the reason?<br>
Another question, why is <code>self.features</code> being updated three times in <code>model2</code> but only once in <code>model1</code>? Is the flow of gradients not the same for both the models?</p>
user_suspended: yes

username: ptrblck
cooked: <p>The gradient flow should be the same and I was only referring to the updates of the internal batchnorm statistics.</p>
<p>The additional operation using the <code>importance_map</code> might of course change the performance of the model.<br>
If I understand the use case correctly, you are using this attention op in the second approach, but not in the first one?</p>

username: Umair_Javaid
cooked: <p>No, I am using attention op in both the models. <code>self.features</code> is identical for both the approaches.<br>
Can you explain me the differences between both the models?</p>
user_suspended: yes

username: ptrblck
cooked: <p>Could you post the complete architecture, please?</p>

username: ptrblck
cooked: <p>Thanks for the code.<br>
I’ve added <code>myModel2</code> as:</p>
<pre><code class="lang-python">class myModel2(nn.Module):
    def __init__(self, features, num_classes=200, **kwargs):
        super(myModel2, self).__init__()
        self.features = features
        self.conv6 = nn.Conv2d(512,  1024, kernel_size=3, padding=1) 
        self.conv7 = nn.Conv2d(1024, num_classes, kernel_size=1)
        self.conv8 = nn.Conv2d(512,  1024, kernel_size=3, padding=1) 
        self.conv9 = nn.Conv2d(1024, num_classes, kernel_size=1)
        self.conv10 = nn.Conv2d(512,  1024, kernel_size=3, padding=1) 
        self.conv11 = nn.Conv2d(1024, num_classes, kernel_size=1)
        self.conv12 = nn.Conv2d(512,  1024, kernel_size=3, padding=1) 
        self.conv13 = nn.Conv2d(1024, num_classes, kernel_size=1)
        self.relu = nn.ReLU(inplace=False)
        self.avgpool = nn.AdaptiveAvgPool2d(1)
        #self.fc = nn.Linear(1024, num_classes)
        initialize_weights(self.modules(), init_mode='he')
    def forward(self, x, labels=None, return_cam=False):
        batch_size = x.shape[0]
        x = self.features(x)
        
        x1 = self.conv6(x)
        x1 = self.relu(x1)
        x1 = self.conv7(x1)
        x1 = self.relu(x1)
        #x2 = self.features(x)
        x2 = self.conv8(x)
        x2 = self.relu(x2)
        x2 = self.conv9(x2)
        x2 = self.relu(x2)
        #x3 = self.features(x)
        x3 = self.conv10(x)
        x3 = self.relu(x3)
        x3 = self.conv11(x3)
        x3 = self.relu(x3)
        #x4 = self.features(x)
        x4 = self.conv12(x)
        x4 = self.relu(x4)
        x4 = self.conv13(x4)
        x4 = self.relu(x4)
        x = x1 + x2 + x3 + x4
        if return_cam:
          normalized_feature_map = normalize_tensor(x.detach().clone())
          cams = normalized_feature_map[range(batch_size), labels]
          return cams

        
        x = self.avgpool(x)
        x = x.view(x.size(0), -1)
        return {'logits': x}
</code></pre>
<p>and get exactly the same results using this code:</p>
<pre><code class="lang-python">features = nn.Conv2d(1, 512, 1)
model1 = myModel7(features)
model2 = myModel2(features)
model2.load_state_dict(model1.state_dict())

x = torch.randn(1, 1, 4, 4)
out1 = model1(x)
out2 = model2(x)

print((out1['logits'] - out2['logits']).abs().max())
&gt; tensor(0., grad_fn=&lt;MaxBackward1&gt;)
</code></pre>

username: Umair_Javaid
cooked: <p>I am sure the output is the same.</p>
<p>But after training both the models on CUB dataset,  myModel6 gets  <strong>2.1%</strong>  increase in classification accuracy compared to the other model. I just want to know, why is that? are the weights being updated similarly in both the models? Are there any differences?</p>
<pre><code class="lang-auto">def forward(self, x, labels=None, return_cam=False):
        batch_size = x.shape[0]
        x1 = self.features(x)
        x1 = self.conv6(x1)
        x1 = self.relu(x1)
        x1 = self.conv7(x1)
        x1 = self.relu(x1)
        
        x2 = self.features(x)
        x2 = self.conv8(x2)
        x2 = self.relu(x2)
        x2 = self.conv9(x2)
        x2 = self.relu(x2)

        x3 = self.features(x)
        x3 = self.conv10(x3)
        x3 = self.relu(x3)
        x3 = self.conv11(x3)
        x3 = self.relu(x3)

        x = x1 + x2 + x3
</code></pre>
<p>(I am sorry if i am bothering you.)</p>
user_suspended: yes

username: ptrblck
cooked: <p>The updates should be equal, as I also get exactly the same gradients for all parameters in these models.<br>
How reproducible is the accuracy difference? I.e. are you always seeing this gap for e.g. 10 runs with different seeds?</p>

username: Umair_Javaid
cooked: <p>I will have to check that!</p>
user_suspended: yes

username: ptrblck
cooked: <p>The gap might be of course just due to “bad luck”, but if you are seeing a consistent difference, something else is going on I’m missing.</p>