Classifier does the wrong prediction on single images

eFroD: <p>Hello,</p>
<p>I am very new to pytorch and DNNs in general.<br>
I’ve created a classifier on top of a pretrained densenet161, to classify images of flowers, into the groups: daisy(0), dandelion(1), rose(2), sunflower(3) and tulip(4).</p>
<p>The training-process works fine and in order to test the model I went on implementing the example from chaper 5 of the tutorial <a href="https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html" rel="noopener nofollow ugc">from the PyTorch site</a>.</p>
<p>All this seems to work, the classifier predicts a defined set of images with an acceptable performance but the next thing I wanted to do, was to test it with a single image.<br>
The first step was to simply set the batch-size to one but at this point the output of my network is wrong.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/2X/5/5aececc206997aa9e77f2e91844370c23fb99cbb.png" alt="49" data-base62-sha1="cYmA386EXqHYNPtpWFEKL7kOJTB" width="473" height="185"><br>
This is the output tensor of the working example, the index of these classes is mapped like shown above.</p>
<p>After this worked, i tried to load a single image like this:</p>
<pre><code class="lang-auto">test_transforms = transforms.Compose([
                          transforms.Resize((224,224)), 
                          transforms.ToTensor(),
                          transforms.Normalize([0.485, 0.456, 0.406],
                             [0.229, 0.224, 0.225])])
def make_image(image):
    image_tensor = test_transforms(image)
    image_tensor.unsqueeze_(0)
    return image_tensor

img = make_image(Image.open("path-to-a-sunflower"))
output = torch.exp(model.forward(img)) # exp because of LogSoftMax as output layer 
print(output)
val, pos = out.topk(1,dim=1)
</code></pre>
<p>Via the topk I wanted to get the position of the highest value in order to display the class.<br>
The output of the model was:<br>
tensor([[0.1126, 0.2525, 0.1020, 0.0674, 0.4655]], grad_fn=)<br>
which means that it has classified the picture as a tulip.<br>
The strange thing is, that if I try the same images within a dataloader with identical transformation, the results are just right.</p>
<p>I have no idea where to start with this problem, so I wonder if there is a reason for this behavior and, in the best case a solution.</p>
<p>To provide more information I will post some of the Code here, I hope this helps.</p>
<h3><a name="classifier-criterion-and-optimizer-1" class="anchor" href="#classifier-criterion-and-optimizer-1"></a>Classifier, Criterion and Optimizer:</h3>
<pre><code class="lang-auto">   model.classifier = nn.Sequential(nn.Linear(model.classifier.in_features, 512),
                                     nn.ReLU(),
                                     nn.Dropout(0.2),
                                     nn.Linear(512, 265),
                                     nn.ReLU(),
                                     nn.Linear(265,5),
                                     nn.LogSoftmax(dim=1))
    criterion = nn.NLLLoss()
    optimizer = optim.Adam(model.classifier.parameters(), lr=0.003)
</code></pre>
<h3><a name="the-epoch-loop-2" class="anchor" href="#the-epoch-loop-2"></a>The Epoch Loop</h3>
<pre><code class="lang-auto">for epoch in range(epochs):
        for inputs, labels in tqdm_notebook(trainloader,desc="Training batches"):
            steps += 1
            inputs, labels = inputs.to(device), labels.to(device)
            optimizer.zero_grad()
            logps = model.forward(inputs)
            loss = criterion(logps, labels)
            loss.backward()
            optimizer.step()
            running_loss += loss.item()

            if steps % print_every == 0:
                test_loss = 0
                accuracy = 0
                model.eval()
                with torch.no_grad():
                    for inputs, labels in tqdm_notebook(testloader,desc="Testing batches",leave = False):
                        inputs, labels = inputs.to(device), labels.to(device)
                        logps = model.forward(inputs)
                        batch_loss = criterion(logps, labels)
                        test_loss += batch_loss.item()

                        ps = torch.exp(logps)
                        top_p, top_class = ps.topk(1, dim=1)
                        equals = top_class == labels.view(*top_class.shape)
                        accuracy += torch.mean(equals.type(torch.FloatTensor)).item()
                # plotting
                train_losses.append(running_loss / len(trainloader))
                test_losses.append(test_loss / len(testloader))
                print(f"Epoch {epoch + 1}/{epochs}.. "
                  f"Train loss: {running_loss / print_every:.3f}.. "
                  f"Test loss: {test_loss / len(testloader):.3f}.. "
                  f"Test accuracy: {accuracy / len(testloader):.3f}")
                running_loss = 0
                model.train()
</code></pre>
<p>I hope these informations could describe my problem and I appreciate any idea concerning this topic. Thank you very much in advance.</p>

ptrblck: <p>Did you also call <code>model.eval()</code> before testing the single image?</p>

eFroD: <p>That does the trick!<br>
In my first model I didn’t do that but maybe it had not that much of an impact than in the model here, since I use a dropout-layer.</p>
<p>Thank you very much for your answer!</p>

yuanzhoulvpi: <p>Many freshmen just started using pytorch and didn’t know to use model.train() and model.eval().  If you train the model, you can add a model.train() in front. If you test the model, you must add a model.eval().</p>

Aymen_RH: <p>Hey i’m having a similar problem even while using model.eval()</p>
<p>The inference results are good with 90% testing accuracy but when I try to predict with a single input it always outputs 0.<br>
i have two classes and i tried to output the results of each prediction but it is kind of weird because the 0th class output is always positive and 1st class output is negative</p>
<pre><code class="lang-auto">tensor([[ 687.3718, -580.9390]], device='cuda:0')
tensor([[ 578.3836, -362.3758]], device='cuda:0')
tensor([[ 507.2051, -494.3643]], device='cuda:0')
tensor([[ 623.5096, -523.1122]], device='cuda:0')
tensor([[ 610.4017, -603.3063]], device='cuda:0')
tensor([[ 711.8994, -506.7224]], device='cuda:0')
tensor([[ 766.9221, -555.6006]], device='cuda:0')
tensor([[ 1092.4576, -1050.7422]], device='cuda:0')
tensor([[1106.5111, -977.1776]], device='cuda:0')
tensor([[1168.5472, -978.1863]], device='cuda:0')
</code></pre>
<p>my inference function is</p>
<pre><code class="lang-auto">def inference(model, val_dl):
    confmat_fn = torchmetrics.ConfusionMatrix(task="binary", num_classes=N_CLASSES).to(device)
    acc_fn = BinaryAccuracy().to(device)
    precision_fn = BinaryPrecision().to(device)
    speci_fn = BinarySpecificity().to(device)
    f1_fn = BinaryF1Score().to(device)

    model.eval()
    with torch.inference_mode():
        for data in val_dl:
            inputs, labels = data[0].to(device), data[1].to(device)
            inputs_m, inputs_s = inputs.mean(), inputs.std()
            inputs = (inputs - inputs_m) / inputs_s
            outputs = model(inputs)
            _, prediction = torch.max(outputs, 1)

            acc_fn.update(prediction, labels)
            precision_fn.update(prediction, labels)
            speci_fn.update(prediction, labels)
            f1_fn.update(prediction, labels)
            confmat_fn.update(prediction, labels)
        
        acc = acc_fn.compute().item()
        precision = precision_fn.compute().item()
        speci = speci_fn.compute().item()
        f1 = f1_fn.compute().item()
        confmat = confmat_fn.compute()
        print(f'Accuracy: {acc:.2f}, Precision: {precision:.2f}, Specificity: {speci:.2f}, F1 Score: {f1:.2f}')
        cm_display = metrics.ConfusionMatrixDisplay(confmat.cpu().numpy(), display_labels=['Normal', 'Abnormal'])
        cm_display.plot()
</code></pre>
<p>and my predicting function for one signal is (Basically i have a group of inputs that belong to the same signal and im calculating the predictions for all of them to take the majority decision):</p>
<pre><code class="lang-auto">def predict(model, group):
    preds = []
    model.eval()
    with torch.inference_mode():
        for i in range(len(group)):
            audio_file = group.loc[i, 'relative_path']
            data = torch.load(audio_file)
            data.to(device)
            outputs = model(data.unsqueeze(dim=0))
            print(outputs)
            _, prediction = torch.max(outputs, 1)
            preds.append(prediction.item())
    return preds
</code></pre>

ptrblck: <p>Could you describe the class distribution and check if class0 is the majority class with approx. 90% of all samples?</p>

Aymen_RH: <p>Hey thanks for answering, The distribution of the classes for the different data splits is<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/9/89bad77bc6a070d10c7df667e189bda1136d5a5d.png" data-download-href="https://discuss.pytorch.org/uploads/default/89bad77bc6a070d10c7df667e189bda1136d5a5d" title="image_2023-05-03_173306161"><img src="https://discuss.pytorch.org/uploads/default/original/3X/8/9/89bad77bc6a070d10c7df667e189bda1136d5a5d.png" alt="image_2023-05-03_173306161" data-base62-sha1="jEpIDKdEyuFAwbCSAimbPsFxmUJ" width="690" height="89" data-dominant-color="ECECEC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image_2023-05-03_173306161</span><span class="informations">728×94 4.18 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
(Normal = 0/ Abnormal = 1)<br>
I understand that the class distribution is unbalanced, but I have also reviewed the confusion matrix for the testing:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/9/b/9b1a30fa060e29e674124d7450ffbf76bf6f1bf0.png" alt="image_2023-05-03_173014768" data-base62-sha1="m867B5SZN1H3DzROOGkhS3IPdoA" width="574" height="436"></p>

ptrblck: <p>Did you create this confusion matrix with a large batch size or the single samples? If both, could you show the difference between these matrices?</p>

Aymen_RH: <p>I generated the confusion matrix using the same batch size that was used during training. When testing the model on individual samples, all of the samples were predicted to be in the left side of the matrix (Normal) and there were no predictions on the right side (Abnormal). since the testing set is relatively small, I even printed out each prediction and it was all 0s.</p>

ptrblck: <p>Could you post the model definition which would reproduce the issue of changing the output during evaluation based on the batch size?</p>

Aymen_RH: <p>Sure here’s my model definition, it is a hybrid CNN LSTM model.</p>
<pre><code class="lang-auto">class CNN(nn.Module):
    def __init__(self):
        super(CNN, self).__init__()
        self.conv1 = nn.Sequential(
            nn.Conv2d(1, 16, 3), nn.ReLU(), nn.MaxPool2d(2), nn.BatchNorm2d(16)
        )
        self.conv2 = nn.Sequential(
            nn.Conv2d(16, 32, 3), nn.ReLU(), nn.MaxPool2d(2), nn.BatchNorm2d(32)
        )
        self.conv3 = nn.Sequential(
            nn.Conv2d(32, 64, 3), nn.ReLU(), nn.MaxPool2d(4), nn.BatchNorm2d(64)
        )
        self.flatten = nn.Flatten()

    def forward(self, x):
        out = self.conv1(x)
        out = self.conv2(out)
        out = self.conv3(out)
        out = self.flatten(out)
        return out


class RNN(nn.Module):
    def __init__(self, input_size, hidden_size=64, n_layers=1, device="cuda:0"):
        super(RNN, self).__init__()
        self.device = device
        self.n_layers = n_layers
        self.hidden_size = hidden_size

        self.lstm = nn.LSTM(input_size, hidden_size, n_layers, batch_first=True)
        self.flatten = nn.Flatten()

    def forward(self, x):
        # [batch_size, 1, n_mfcc, seq_length]
        out = x.squeeze(dim=1)
        # [batch_size, n_mfcc, seq_length]
        out = out.permute(0, 2, 1)
        hidden_states = torch.zeros(self.n_layers, out.size(0), self.hidden_size).to(
            self.device
        )
        cell_states = torch.zeros(self.n_layers, out.size(0), self.hidden_size).to(
            self.device
        )
        # [batch_size, seq_length, n_mfcc]
        out, _ = self.lstm(out, (hidden_states, cell_states))
        out = self.flatten(out[:, -1, :])
        return out

class CRNN(nn.Module):
    def __init__(
        self,
        input_size,
        n_classes,
        n_layers_rnn=64,
        fc_in=8576,
        device="cuda:0",
    ):
        super(CRNN, self).__init__()
        self.cnn = CNN()
        self.rnn = RNN(input_size, 64, n_layers_rnn, device=device)
        self.fc1 = nn.Linear(fc_in, 32)
        self.relu1 = nn.ReLU()
        self.dropout = nn.Dropout(0.5)
        self.fc2 = nn.Linear(32, n_classes)

    def forward(self, x):
        cnn_out = self.cnn(x)
        rnn_out = self.rnn(x)
        out = torch.cat([cnn_out, rnn_out], dim=1)
        out = self.fc1(out)
        out = self.relu1(out)
        out = self.dropout(out)
        out = self.fc2(out)
        return out
</code></pre>
<p>the args of the model are:</p>
<pre><code class="lang-auto">myModel = CRNN(
    input_size=32, n_classes=2, n_layers_rnn=64, fc_in=576, device=device
)
</code></pre>

ptrblck: <p>Thank for the code!<br>
I cannot reproduce the issue and get the same outputs with the expected small numerical mismatch caused by the limited floating point precision:</p>
<pre><code class="lang-python">x = torch.randn(100, 1, 32, 144)
myModel.eval()

# all samples
ref = myModel(x)

# single sample
out = []
for x_ in x:
    x_.unsqueeze_(0)
    out.append(myModel(x_))
out = torch.cat(out)

print((ref - out).abs().max())
# tensor(8.1956e-08, grad_fn=&lt;MaxBackward1&gt;)
</code></pre>

Aymen_RH: <p>Hey thanks for the reply!<br>
Did you train the model before running this code? because I tried it on an untrained model and it showed the same results you had. But I tried printing out the ref and out tensors and all the outputs  were similar maybe that’s why the difference was small.</p>
<pre><code class="lang-auto">tensor([[-0.1333, -0.1921],
        [-0.1357, -0.1879],
        [-0.1344, -0.1888],
        [-0.1349, -0.1890],
        [-0.1334, -0.1884],
        [-0.1332, -0.1926],
        [-0.1319, -0.1924],
        [-0.1320, -0.1903],
        [-0.1320, -0.1872],
        [-0.1311, -0.1926],
        [-0.1340, -0.1892],
        [-0.1347, -0.1902],
        [-0.1340, -0.1900],
        [-0.1336, -0.1910],
        [-0.1346, -0.1910],
        [-0.1348, -0.1894],
        [-0.1335, -0.1908],
        [-0.1360, -0.1912],
        [-0.1299, -0.1925],
        [-0.1349, -0.1893],
        [-0.1320, -0.1918],
        [-0.1334, -0.1886],
        [-0.1339, -0.1885],
        [-0.1354, -0.1866],
        [-0.1326, -0.1932],
        [-0.1321, -0.1884],
        [-0.1351, -0.1886],
        [-0.1353, -0.1907],
        [-0.1329, -0.1912],
        [-0.1330, -0.1915],
        [-0.1330, -0.1889],
        [-0.1332, -0.1888],
        [-0.1329, -0.1922],
        [-0.1341, -0.1918],
        [-0.1329, -0.1900],
        [-0.1323, -0.1910],
        [-0.1326, -0.1912],
        [-0.1320, -0.1899],
        [-0.1324, -0.1878],
        [-0.1328, -0.1899],
        [-0.1329, -0.1934],
        [-0.1335, -0.1895],
        [-0.1323, -0.1928],
        [-0.1339, -0.1885],
        [-0.1327, -0.1900],
        [-0.1326, -0.1897],
        [-0.1337, -0.1865],
        [-0.1331, -0.1888],
        [-0.1294, -0.1939],
        [-0.1330, -0.1866],
        [-0.1331, -0.1900],
        [-0.1329, -0.1887],
        [-0.1340, -0.1917],
        [-0.1359, -0.1861],
        [-0.1325, -0.1917],
        [-0.1344, -0.1857],
        [-0.1348, -0.1911],
        [-0.1327, -0.1914],
        [-0.1327, -0.1890],
        [-0.1337, -0.1905],
        [-0.1359, -0.1886],
        [-0.1348, -0.1908],
        [-0.1317, -0.1883],
        [-0.1339, -0.1901]], grad_fn=&lt;AddmmBackward0&gt;)
</code></pre>
<p>However when I tried it on my trained model it resulted in</p>
<pre><code class="lang-auto">tensor(0.4208, device='cuda:0', grad_fn=&lt;MaxBackward1&gt;)
</code></pre>
<p>which I think is pretty high.</p>

ptrblck: <p>No, I didn’t train the model but it also doesn’t change the output if I do:</p>
<pre><code class="lang-python">device = "cuda"
myModel = CRNN(
    input_size=32, n_classes=2, n_layers_rnn=64, fc_in=576, device=device
)
myModel.to(device)


x = torch.randn(100, 1, 32, 144).to(device)
optimizer = torch.optim.Adam(myModel.parameters(), lr=1e-3)
criterion = nn.CrossEntropyLoss()
target = torch.randint(0, 2,(100,)).to(device)

# train
for epoch in range(100):
    optimizer.zero_grad()
    output = myModel(x)
    loss = criterion(output, target)
    loss.backward()
    optimizer.step()
    print(f"epoch: {epoch}; loss: {loss:.3f}")

# epoch: 96; loss: 0.000
# epoch: 97; loss: 0.001
# epoch: 98; loss: 0.000
# epoch: 99; loss: 0.000

myModel.eval()

# all samples
ref = myModel(x)

# single sample
out = []
for x_ in x:
    x_.unsqueeze_(0)
    out.append(myModel(x_))
out = torch.cat(out)

print((ref - out).abs().max())
# tensor(6.6757e-06, device='cuda:0', grad_fn=&lt;MaxBackward1&gt;)
</code></pre>

Aymen_RH: <p>what could be the issue then?</p>

ptrblck: <p>I don’t know, but the model doesn’t seem to be the issue, so I would still need a minimal and executable code snippet to reproduce the issue.</p>

Aymen_RH: <p>you can find the full code in my <a href="https://github.com/iTzAymen/PCG_Classification/blob/main/main_preprocessed.ipynb" rel="noopener nofollow ugc">GitHub Repository</a>.<br>
Scroll down to section <strong>5. Evaluation</strong> in the <em>main_preprocessed.ipynb</em> file there is a comparison between batched signal predictions and unbatched predictions.</p>