Classification is incoherent with test results on CIFAR10 and custom CNN

denishem: <p>Hello everyone,</p>
<p>I’m trying to run classification on the CIFAR10 dataset using a custom CNN which looks like this:</p>
<pre><code class="lang-auto">class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()

        self.conv = nn.Sequential(nn.Conv2d(3, 16, kernel_size=3, stride=2),
                                  nn.BatchNorm2d(16), nn.ReLU(inplace=True),
                                  nn.Conv2d(16, 32, kernel_size=3, stride=2),
                                  nn.BatchNorm2d(32), nn.ReLU(inplace=True),
                                  nn.Conv2d(32, 64, kernel_size=3, stride=2),
                                  nn.BatchNorm2d(64), nn.ReLU(inplace=True),
                                  nn.Conv2d(64, 10, kernel_size=3),
                                  nn.BatchNorm2d(10), nn.Flatten())

    def forward(self, x):
        x = self.conv(x)
        return x
</code></pre>
<p>The training and test goes well, I’m getting around 73% accuracy. Then I save the model using <code>torch.save(model.state_dict(), save_path)</code>.</p>
<p>From another script I load the model like this:</p>
<pre><code class="lang-auto">def load_model(path):
    model = Net()
    model.load_state_dict(torch.load(path))
    print(model)
    model.eval()
    return model
</code></pre>
<p>Thing is, I want to do detection on CIFAR10 but directly from images without using DataLoader or Dataset classes. I saved images from the DataLoader into <code>class_i.png</code> files and then load them like this:</p>
<pre><code class="lang-auto">def load_imgs(path):
    imgs = []
    labels = []
    img_files = os.listdir(path)
    for i in range(len(img_files)):
        label = re.search(r"\w*(?=_)", img_files[i]).group(0)
        img_path = os.path.join(path, img_files[i])
        img = cv2.imread(img_path, cv2.IMREAD_COLOR)
        img = np.asarray(img)
        imgs.append(img)
        labels.append(label)
    return imgs, labels
</code></pre>
<p>Finally, I’m running classification on these images by firstly converting them to tensors:</p>
<pre><code class="lang-auto">def classification(model, imgs):
    if torch.cuda.is_available():
        print(f"Using CUDA device {torch.cuda.get_device_name(0)}")
        device = torch.device("cuda:0")
    else:
        print("No CUDA device found, using CPU")
        device = torch.device("cpu")
    to_tensor = torchvision.transforms.ToTensor()
    tensor_imgs = [to_tensor(img).float() for img in imgs]
    pred = []
    with torch.no_grad():
        for img in tensor_imgs:
            img = img.to(device)
            model = model.to(device)
            output = model(img[None, ...])
            pred.append(output.argmax(dim=1, keepdim=True).cpu().squeeze().item())
    return pred
</code></pre>
<p>But when comparing predicted classes with the original labels, I’m getting less than 25% of accuracy.  My guess is that the problem is somewhere around the way I pass images for detection. Unfortunately, I am strictly limited to the OpenCV library for loading the images.</p>
<p>What would be the correct way to run classification on imported images without using data loaders?</p>

ptrblck: <p>Loading images directly without a <code>DataLoader</code> should work as long as you are applying the same processing (you wouldn’t need data augmentation).<br>
What kind of transformations did you use during the training, i.e. did you resize the images etc.?<br>
To isolate the issue to the data loading you could load some images during training and compare the predictions to the same images returned by the <code>DataLoader</code> (comparing the raw tensor values could also give you more information about the differences).</p>

denishem: <p>These are the transforms that I apply for training and validation:</p>
<pre><code class="lang-auto">transform_train = transforms.Compose([
    transforms.RandomAffine(5, translate=(0.1, 0.1)),
    transforms.ToTensor(),
    transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))
])

transform_test = transforms.Compose([
    transforms.ToTensor(),
    transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))
])
</code></pre>
<p>I realized I made a mistake for inference by not using the same transforms. Applying <code>transform_test</code> on images before inference now gives me around 50% for a run on 200 random images, which is the figure I got after the first epoch during training.</p>
<p>This is how my code looks right now:</p>
<pre><code class="lang-auto">def classification(model, imgs):
    if torch.cuda.is_available():
        print(f"Using CUDA device {torch.cuda.get_device_name(0)}")
        print(DIV)
        device = torch.device("cuda:0")
    else:
        print("No CUDA device found, using CPU")
        print(DIV)
        device = torch.device("cpu")

    tensor_imgs = [transform_test(img).unsqueeze_(0) for img in imgs]
    pred = []
    with torch.no_grad():
        for img in tensor_imgs:
            img = img.to(device)
            model = model.to(device)
            output = model(img)
            pred.append(output.data.cpu().numpy().argmax())
    return pred
</code></pre>
<p>I didn’t quite understand your suggestion, should I save the images that I use for training and run inference on them for comparison? For the training part I am using <code>DataLoader</code> for both training and test datasets.</p>

ptrblck: <aside class="quote no-group" data-username="denishem" data-post="3" data-topic="121752">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/denishem/48/38477_2.png" class="avatar"> denishem:</div>
<blockquote>
<p>I didn’t quite understand your suggestion, should I save the images that I use for training and run inference on them for comparison? For the training part I am using <code>DataLoader</code> for both training and test datasets.</p>
</blockquote>
</aside>
<p>This would be one possibility to further isolate it.<br>
Issues like these are often caused by either a different data loading and processing pipeline (majority of the issues) or by an invalid model loading (e.g. using <code>strict=False</code> as a “workaround” to load an invalid <code>state_dict</code>).<br>
To further isolate the root cause I would suggest to start with the data loading and verify that the training and test script create the “same” input data. Note that the tensors wouldn’t be exactly the same, if the training pipeline uses random transformations, so for debugging purposes you could disable them.<br>
The model can be checked by comparing the output for a static input (e.g. <code>torch.ones</code>) and make sure both scripts return an <code>allclose</code> output.</p>

denishem: <p>I managed to find my error. It was due to color channels order since OpenCV loads images as BGR and not RGB. Converting images uisng <code>cv2.cvtColor(img, cv2.BGR2RGB)</code> fixed it.</p>