Significant Differences in the Confidence scores across IMAGENET1K_V1 and IMAGENET1K_V2 ResNet50 during Inference

username: Kunal_7
cooked: <p>I performed a simple <strong>classification</strong> inference on a sample image using ResNet-50. I used both the <strong><code>IMAGENET1K_V1</code></strong> and <strong><code>IMAGENET1K_V2</code></strong>  versions of the model weights. I found that there was around <strong>70%</strong> jump in the final confidence scores (even after applying the appropriate transforms) in the <strong>V1</strong> version as opposed to the <strong>V2</strong>.</p>
<p>This is odd since the <strong>V2</strong> version was expected to give a better confidence score!</p>
<p>I have attached a <a href="https://colab.research.google.com/drive/12uMzcou5sMRgSDlskv406N7FQOt0VSJ9" rel="noopener nofollow ugc">Colab Notebook</a> as a reference.</p>
<p>The class with the highest confidence score on the same image across the <code>IMAGENET1K_V1</code> and <code>IMAGENET1K_V2</code> weights (even with the appropriate transforms)  are <strong><code>99.771%</code></strong> and <strong><code>58.404%</code></strong>. Something seems quite off!</p>
<ul>
<li>The custom transforms for <code>IMAGENET1K_V1</code> weights were:</li>
</ul>
<pre><code class="lang-auto">T.Compose([T.Resize(256),                    
                    T.CenterCrop(224),                
                    T.ToTensor(),                     
                    T.Normalize(                      
                       mean=[0.485, 0.456, 0.406],                
                       std=[0.229, 0.224, 0.225]                  
                 )])
</code></pre>
<ul>
<li>The custom transforms for <code>IMAGENET1K_V2</code> weights were:</li>
</ul>
<pre><code class="lang-auto">T.Compose([T.Resize(232),                    
                    T.CenterCrop(224),                
                    T.ToTensor(),                     
                    T.Normalize(                      
                       mean=[0.485, 0.456, 0.406],                
                       std=[0.229, 0.224, 0.225]                  
                 )])
</code></pre>
<p>Even the <code>ResNet50_Weights.IMAGENET1K_V1.transforms()</code> and <code>ResNet50_Weights.DEFAULT.transforms()</code> were used with literally no difference in the results!</p>
<p>The <strong>Torchvision</strong> documentation for <strong>ResNet50</strong> with <strong><code>IMAGENET1K_V2</code></strong> weights states:</p>
<blockquote>
<p>The inference transforms are available at <code>ResNet50_Weights.IMAGENET1K_V2.transforms</code> and perform the following preprocessing operations: Accepts <code>PIL.Image</code> , batched <code>(B, C, H, W)</code> and single <code>(C, H, W)</code> image <code>torch.Tensor</code> objects. The images are resized to <code>resize_size=[232]</code> using <code>interpolation=InterpolationMode.BILINEAR</code> , followed by a central crop of <code>crop_size=[224]</code> . Finally the values are first rescaled to <code>[0.0, 1.0]</code> and then normalized using <code>mean=[0.485, 0.456, 0.406]</code> and <code>std=[0.229, 0.224, 0.225]</code> .</p>
</blockquote>
<p>This is precisely what I have performed above.</p>
<p>Could someone possibly point out what the issue is here?</p>
<p><strong>PS:</strong> I have also raised an <a href="https://github.com/pytorch/vision/issues/7570" rel="noopener nofollow ugc">issue</a> for the same.</p>