Correct pre-processing for pretrained GoogLeNet

username: addisonklinke
cooked: <p>I would like to use <code>torchvision.models.googlenet</code> as a pre-trained backbone in a custom model. The inputs to my custom model (during training and evaluation) will already be cropped to 224 pixels. I need to understand how to apply the ImageNet mean and standard deviation values to my images before they are passed through the model.</p>
<p>Interestingly, the <code>GoogLeNet</code> class provides a <a href="https://github.com/pytorch/vision/blob/master/torchvision/models/googlenet.py#L130-L137" rel="noopener nofollow ugc"><code>_transform_input()</code></a> method which sounds like it should handle all the pre-processing for you. When initializing with <code>pretrained=True</code>, the <a href="https://github.com/pytorch/vision/blob/master/torchvision/models/googlenet.py#L38-L40" rel="noopener nofollow ugc">constructor</a> automatically sets <code>transform_input=True</code> as well. However, in the docs <a href="https://pytorch.org/hub/pytorch_vision_googlenet/" rel="noopener nofollow ugc">example</a>, a <code>torchvision.transforms</code> pipeline is used <em>in addition to</em> the model’s internal method. To me, this seems like we are performing the ImageNet transform on the input twice.</p>
<p>My questions are</p>
<ol>
<li>Why does the <code>GoogLeNet</code> implementation include an internal transform method when none of the other torchvision models (i.e. densenet, mobilenet, etc) have one?</li>
<li>In the internal transform method, why are there several 0.5 factors used in the formula to transform the image. From my understanding, ImageNet transforms should be calculated as <code>(pixel_value - mean) / std</code>, but the current implementation does not follow this</li>
<li>Is applying a “double transform” like the docs example really the correct way to preprocess input for this pretrained model?</li>
</ol>
<p>The script below tests several different pre-processing approaches</p>
<ol>
<li>Using <code>GoogLeNet._transform_inputs()</code><br>
a. No other additions<br>
b. Plus the docs <code>torchvision.transforms</code> pipeline<br>
c. Plus my manual implementation of ImageNet transforms</li>
<li>Without using <code>GoogLeNet._transform_inputs()</code><br>
a. No other additions<br>
b. Using the docs <code>torchvision.transforms</code> pipeline<br>
c. Using my manual implementation of ImageNet transforms</li>
</ol>
<pre><code class="lang-auto">import cv2
from PIL import Image
import torch
import torch.nn.functional as F
from torchvision.models import googlenet
from torchvision import transforms


def load_tensor(filename):
    rgb = cv2.cvtColor(cv2.imread(filename), cv2.COLOR_BGR2RGB)
    nchw = torch.tensor(rgb).permute(2, 0, 1).unsqueeze(0)
    normalized = nchw.float() / 255.0
    return normalized


def manual(filename):
    normalized = load_tensor(filename)
    return torch.stack((
        (normalized[:, 0, :, :] - 0.485) / 0.229,
        (normalized[:, 1, :, :] - 0.456) / 0.224,
        (normalized[:, 2, :, :] - 0.406) / 0.225), dim=1)


def torch_transforms(filename):
    input_image = Image.open(filename)
    preprocess = transforms.Compose([
        transforms.Resize(256),
        transforms.CenterCrop(224),
        transforms.ToTensor(),
        transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]),
    ])
    input_tensor = preprocess(input_image)
    input_batch = input_tensor.unsqueeze(0)
    return input_batch


idx2class = { ... }  # Fill in from https://gist.github.com/maraoz/388eddec39d60c6d52d4

# Run model with different pre-processing approaches
model = googlenet(pretrained=True)
model.eval()
filename = '/home/addison/Documents/openalpr/vehicle-classifier/dog-224.jpg'
model.transform_input = True
scores = {
    '1a. Internal': model(load_tensor(filename)),
    '1b. Internal + torch.transforms': model(torch_transforms(filename)),
    '1c. Internal + manual': model(manual(filename))}
model.transform_input = False
scores.update({
    '2a. No internal': model(load_tensor(filename)),
    '2b. No internal (torch.transforms)': model(torch_transforms(filename)),
    '2c. No internal (manual)': model(manual(filename))})

# Summarize results
max_str = max(len(k) for k in scores)
for method, tensor in scores.items():
    probs = F.softmax(tensor, dim=1)
    values, indices = torch.topk(probs, k=1)
    print(f'{method:&lt;{max_str}}: {idx2class[indices[0][0].item()]} ({values[0][0] * 100:.2f}%)')
</code></pre>
<p>I use the dog image from the docs example and pre-cropped it to 224 pixels in an external image editing program (see end of post). This produces different confidences for the top class prediction</p>
<pre><code class="lang-auto">1a. Internal                      : Samoyed, Samoyede (97.31%)
1b. Internal + torch.transforms   : Samoyed, Samoyede (91.27%)
1c. Internal + manual             : Samoyed, Samoyede (97.37%)
2a. No internal                   : Samoyed, Samoyede (96.38%)
2b. No internal (torch.transforms): Samoyed, Samoyede (33.95%)
2c. No internal (manual)          : Samoyed, Samoyede (83.95%)
</code></pre>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/0/7/073be8fc15b3919fd7f2d187e551a87125fcb1cd.jpeg" alt="dog-224" data-base62-sha1="11ZHlh2ncbLItQQulbTLonDdzrf" width="224" height="224"></p>

username: hydrochloricacid
cooked: <p>I encountered the same problem too when evaluating the model with Imagenet1k dataset.It seems that to achieve the  accuracy claimed in torchvision.models.googlenet, a “double pre-process” is needed. I guess the way transform_input deals with input data may be related to the googlenet model itself, and it’s really confusing to me that few people mentioned this.</p>

username: jkbv
cooked: <p>The internal transformation seems to revert the standard ImageNet-specific normalization and replace it by simply subtracting constant “mean” and dividing by constant “std”. Not sure what the last division by 0.5 stands for there. I found a commit where the contributor claimed that it was done to match the TensorFlow implementation. However, I don’t see this in the TF.</p>
<p>It is weird that 1b. is different to 1c. in your comparison. Probably the cropping…</p>