Pretrained model accuracy for Imagenet different from documentation

Raul_Murillo: <p>Hello all,</p>
<p>I am using a TorchVision model with pre-trained weights and checking its accuracy using the original ImageNet validation set (downloaded from <a href="https://www.kaggle.com/competitions/imagenet-object-localization-challenge/" rel="noopener nofollow ugc">Kaggle</a>). Also, I try to use the latest up-to-date API for reproducibility.<br>
The problem is I am not getting exactly the same accuracy as reported in the documentation (see <a href="https://pytorch.org/vision/stable/models.html#table-of-all-available-classification-weights" class="inline-onebox" rel="noopener nofollow ugc">Models and pre-trained weights — Torchvision main documentation</a> or <a href="https://pytorch.org/blog/introducing-torchvision-new-multi-weight-support-api/" class="inline-onebox" rel="noopener nofollow ugc">Introducing TorchVision’s New Multi-Weight Support API | PyTorch</a>)</p>
<p>The code is pretty straightforward:</p>
<pre><code class="lang-python">import os
from torch.utils.data import Dataset
from PIL import Image
import json

from torch.utils.data import DataLoader
from torchvision import transforms
import torch
import torchvision
from tqdm import tqdm


FOLDER = "/home/Documents/ImageNet"

class ImageNetKaggle(Dataset):
    def __init__(self, root, split, transform=None):
        self.samples = []
        self.targets = []
        self.transform = transform
        self.syn_to_class = {}
        with open(os.path.join(root, "imagenet_class_index.json"), "rb") as f:
                    json_file = json.load(f)
                    for class_id, v in json_file.items():
                        self.syn_to_class[v[0]] = int(class_id)
        with open(os.path.join(root, "ILSVRC2012_val_labels.json"), "rb") as f:
                    self.val_to_syn = json.load(f)
        samples_dir = os.path.join(root, "ILSVRC/Data/CLS-LOC", split)
        for entry in os.listdir(samples_dir):
            if split == "train":
                syn_id = entry
                target = self.syn_to_class[syn_id]
                syn_folder = os.path.join(samples_dir, syn_id)
                for sample in os.listdir(syn_folder):
                    sample_path = os.path.join(syn_folder, sample)
                    self.samples.append(sample_path)
                    self.targets.append(target)
            elif split == "val":
                syn_id = self.val_to_syn[entry]
                target = self.syn_to_class[syn_id]
                sample_path = os.path.join(samples_dir, entry)
                self.samples.append(sample_path)
                self.targets.append(target)
    def __len__(self):
            return len(self.samples)
    def __getitem__(self, idx):
            x = Image.open(self.samples[idx]).convert("RGB")
            if self.transform:
                x = self.transform(x)
            return x, self.targets[idx]


weights = torchvision.models.ResNet50_Weights.DEFAULT # or IMAGENET1K_V2
model = torchvision.models.resnet50(weights=weights)
model.eval().cuda()  # Needs CUDA, don't bother on CPUs

dataset = ImageNetKaggle(FOLDER, "val", weights.transforms())
dataloader = DataLoader(
            dataset,
            batch_size=64, # may need to reduce this depending on your GPU 
            num_workers=8, # may need to reduce this depending on your num of CPUs and RAM
            shuffle=False,
            drop_last=False,
            pin_memory=True
        )


# Run inference
top1 = 0
K=5
top5 = 0
total = 0
with torch.no_grad():
    for x, y in tqdm(dataloader):
        y_pred = model(x.cuda())
        _, y_pred_k = y_pred.topk(k=K, dim=1)
        y_pred_k = y_pred_k.t()

        top1 += (y_pred.argmax(axis=1) == y.cuda()).sum().item()

        correct = (y_pred_k == y.unsqueeze(dim=0).cuda()).expand_as(y_pred_k)
        top5 += (correct[:K].reshape(-1)).sum().item()
        total += len(y)

print(f"Acc@1: {100 * top1 / total:.3f}%  Acc@5: {100 * top5 / total:.3f}%")
</code></pre>
<p>However, the accuracies are <em>slightly different</em> from those indicated in the official <a href="https://pytorch.org/vision/stable/models.html#table-of-all-available-classification-weights" rel="noopener nofollow ugc">documentation</a>, which are  80.858 and 95.434 top-1 and top-5, respectively, for this ResNet50 model.</p>
<pre><code class="lang-bash">$ python inference.py
y 
100%|█████████████████████████████████████| 782/782 [01:51&lt;00:00,  7.02it/s]
Acc@1: 80.854%  Acc@5: 95.438%
</code></pre>
<p>Is this any kind of issue, or am I doing something wrong? Could this just depend on the specific hardware that runs the inference?<br>
I get the same accuracy values through multiple executions, and observed the behavior for other models such as GoogLeNet.<br>
I am using PyTorch version <code>1.13.1+cu116</code> and TorchVision <code>0.14.1+cu116</code>.</p>

zhulinchng: <aside class="quote no-group" data-username="Raul_Murillo" data-post="1" data-topic="173253">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/raul_murillo/48/56961_2.png" class="avatar"> Raul_Murillo:</div>
<blockquote>
<pre><code class="lang-auto">with torch.no_grad():
    for x, y in tqdm(dataloader):
        y_pred = model(x.cuda())
        _, y_pred_k = y_pred.topk(k=K, dim=1)
        y_pred_k = y_pred_k.t()

        top1 += (y_pred.argmax(axis=1) == y.cuda()).sum().item()

        correct = (y_pred_k == y.unsqueeze(dim=0).cuda()).expand_as(y_pred_k)
        top5 += (correct[:K].reshape(-1)).sum().item()
        total += len(y)

print(f"Acc@1: {100 * top1 / total:.3f}%  Acc@5: {100 * top5 / total:.3f}%")
</code></pre>
</blockquote>
</aside>
<p>Hi Raul, were you able to find out the difference? I am also facing the same reproducibility issue, but with larger difference, I got Acc@1 80.844 and Acc@5 95.430. So far I couldn’t find any answers, at this point I’m guessing it’s hardware issue.</p>
<p>Thanks.</p>

Raul_Murillo: <p>Hi Zhu! As of today, I have not gotten a clear answer, although if I had to bet, I would say it is due to hardware and parallelism. Depending on how the parallel computations are done, the results could be rounded one way or another, causing subtle differences in the final results.</p>

Met4physics: <p>Do you set this?</p>
<pre data-code-wrap="python"><code class="lang-python">torch.backends.cudnn.benchmark = False
torch.backends.cudnn.deterministic = True
</code></pre>

Raul_Murillo: <p>I didn’t know about the existence of those options.<br>
The result was a higher speedup, but with the exact accuracy results that I mentioned in the original comment.</p>