Confusion matrix

evo: <p>Hello, I did FNN for 4 class classifications.<br>
How is it possible to calculate confusion matrix?</p>

ptrblck: <p>You could use the <a href="http://scikit-learn.org/stable/modules/generated/sklearn.metrics.confusion_matrix.html">scikit-learn implementation</a>. Just get your predictions and targets using <code>.numpy()</code>.</p>
<p>Alternatively you could of course calculate the conf matrix manually, if you don’t want to install/use scikit-learn.</p>

evo: <p>I applied scikit-learn implementation, but output from FNN tend to 0. As a result my confusion matrix looks weird.</p>
<pre><code class="lang-python">
for epoch in range (num_epochs):
    out = model(input_train).to(device)
    _, pred = out.max(1)
    total += target_train.size(0)
    correct += (pred == target_train).sum().item()
    print(input_train)
    print(pred)
    loss = loss_func(out,target_train)
    counter +=1
    print('loss train', "Epoch N", counter,loss.data[0])
    model.zero_grad()
    loss.backward()
    opt.step()
print('Accuracy of the network on train dataset: {} %'.format(100 * correct / total))

conf_matrix = metrics.confusion_matrix(pred, y)

[[530783 0 0 0]
[ 8097 0 0 0]
[ 20079 0 0 0]
[ 16682 0 0 0]]
</code></pre>
<p>Where can be an error?</p>

ptrblck: <p>It looks like your model does not learn anything useful.<br>
Since your classes are imbalanced, you could try to use a weighted loss function or the <a href="https://pytorch.org/docs/stable/data.html#torch.utils.data.sampler.WeightedRandomSampler">WeightedRandomSampler</a>.</p>

immuaz: <p>But I have to find confusion matrix for multi class image segmentation problem of high resolution images i.e. 1024x2048.<br>
Copying tensors from gpu to cpu i.e. numpy and then calculating confusion matrix is really time consuming.<br>
I found <a href="https://gist.github.com/the-bass/cae9f3976866776dea17a5049013258d" rel="nofollow noopener">this</a> but it is only of binary classification, not sure how to scale it to multi class.</p>

Neda: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> I did use the scikit-learn implementation to calculate the confusion matrix. The snippet is like this.</p>
<pre><code class="lang-auto">
    with torch.no_grad():
        for i, data in enumerate(test_loader, 0):
            # get the inputs
            t_image, mask = data
            t_image, mask = Variable(t_image.to(device)), Variable(mask.to(device))
        
            output = model(t_image)
            pred = torch.exp(output)
            conf_matrix = confusion_matrix(pred, mask)
            print(conf_matrix)
</code></pre>
<p>I am wondering why I am getting this error , what do you think? is it related to the shape of pred or input?</p>
<pre><code class="lang-auto">  File "C:\Users\Neda\Anaconda3\lib\site-packages\sklearn\metrics\classification.py", line 88, in _check_targets
    raise ValueError("{0} is not supported".format(y_type))
ValueError: unknown is not supported
</code></pre>

ptrblck: <p>Probably <code>scikit</code> doesn’t recognize the format of your inputs to <code>confusion_matrix</code>.<br>
Could you print the shapes and some values of <code>pred</code> and <code>mask</code>?<br>
I guess flattening both inputs should work.</p>

Neda: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> the shape of inputs are t_image:  <code>(1, 1, 240, 320), pred: (1, 2, 240, 320), mask: (1, 240, 320)</code>.</p>

ptrblck: <p>To calculate the confusion matrix you need the class predictions. Currently it looks like <code>pred</code> contains the logits or probabilities for two classes.<br>
Try to call <code>torch.argmax(pred, 1)</code> to get the predicted classes.<br>
Here is a small example:</p>
<pre><code class="lang-python">output = torch.randn(1, 2, 4, 4)
pred = torch.argmax(output, 1)
target = torch.empty(1, 4, 4, dtype=torch.long).random_(2)
confusion_matrix(pred.view(-1), target.view(-1))
</code></pre>

Neda: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Thanks a lot. yes it work <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=6" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

MariosOreo: <p>I have an idea but don’t know whether it works.</p>
<p>change <code>pred</code> and <code>target</code> into <code>one_hot</code> format on <code>gpus</code></p>
<p>TP = pred * target<br>
FP = pred * (1-target)<br>
FN = (1-pred) * target<br>
TN = (1-pred) * (1-target)</p>

Shubhankar: <p>How would you know when to do pred or (1 - pred), target or (1 - target)?</p>

Hrishkesh_Pattepu: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Hello, May I know how to create a confusion matrix for  YOLOv3.<br>
For a 3 class classification</p>

ptrblck: <p>I don’t know what kind of classification output YOLOv3 returns, but for a multi-class classification the linked <code>sklearn.metrics.confusion_matrix</code> should work, while for a multi-label classification you could use <a href="https://scikit-learn.org/stable/modules/generated/sklearn.metrics.multilabel_confusion_matrix.html"><code>sklearn.metrics.multilabel_confusion_matrix</code></a>.</p>

Hrishkesh_Pattepu: <p>def ap_per_class(tp, conf, pred_cls, target_cls):<br>
“”" Compute the average precision, given the recall and precision curves.<br>
Source: <a href="https://github.com/rafaelpadilla/Object-Detection-Metrics" rel="noopener nofollow ugc">https://github.com/rafaelpadilla/Object-Detection-Metrics</a>.<br>
# Arguments<br>
tp:    True positives (nparray, nx1 or nx10).<br>
conf:  Objectness value from 0-1 (nparray).<br>
pred_cls: Predicted object classes (nparray).<br>
target_cls: True object classes (nparray).<br>
# Returns<br>
The average precision as computed in py-faster-rcnn.<br>
“”"</p>
<pre><code># Sort by objectness
i = np.argsort(-conf)
tp, conf, pred_cls = tp[i], conf[i], pred_cls[i]


# Find unique classes
unique_classes = np.unique(target_cls)

# Create Precision-Recall curve and compute AP for each class
pr_score = 0.1  # score to evaluate P and R https://github.com/ultralytics/yolov3/issues/898
s = [unique_classes.shape[0], tp.shape[1]]  # number class, number iou thresholds (i.e. 10 for mAP0.5...0.95)
ap, p, r = np.zeros(s), np.zeros(s), np.zeros(s)

for ci, c in enumerate(unique_classes):
    
    fig, ax = plt.subplots(1, 1, figsize=(5, 5))
    i = pred_cls == c
    n_gt = (target_cls == c).sum()  # Number of ground truth objects
    n_p = i.sum()  # Number of predicted objects

    if n_p == 0 or n_gt == 0:
        continue
    else:
        # Accumulate FPs and TPs
        fpc = (1 - tp[i]).cumsum(0)
        tpc = tp[i].cumsum(0)
        # Recall
        recall = tpc / (n_gt + 1e-16)  # recall curve
        r[ci] = np.interp(-pr_score, -conf[i], recall[:, 0])  # r at pr_score, negative x, xp because xp decreases

        # Precision
        precision = tpc / (tpc + fpc)  # precision curve
        p[ci] = np.interp(-pr_score, -conf[i], precision[:, 0])  # p at pr_score

        # AP from recall-precision curve
        for j in range(tp.shape[1]):
            ap[ci, j] = compute_ap(recall[:, j], precision[:, j])

        # Plot

            ax.plot(recall, precision)
            ax.set_xlabel('Recall')
            ax.set_ylabel('Precision')
            ax.set_xlim(0, 1.01)
            ax.set_ylim(0, 1.01)
            fig.tight_layout()
        

# Compute F1 score (harmonic mean of precision and recall)
    fig.savefig(f'PR_curve_{c}.png', dpi=300)
f1 = 2 * p * r / (p + r + 1e-16)
print(ap)

return p, r, ap, f1, unique_classes.astype('int32')
</code></pre>
<p>Thank you for the response, but may I request to have a look at this snippet and can request you to say where the line of code multilabel_confusion_matrix fits in.<br>
<a class="mention" href="/u/ptrblck">@ptrblck</a></p>

ptrblck: <p><code>multilabel_confusion_matrix</code> expects the multi-label target as well as the predicted classes as the inputs.<br>
Based on your code snippet I guess <code>target_cls</code> and <code>pred_cls</code> would be the corresponding tensors.</p>

111404: <p>Hello，bro.Does Mr.ptrblck’s reply work？I have problem about Confusion-Matrix too.And I’m going to calculate yolo’s accuracy.Waiting for your reply.&lt;3</p>

Amin_Jun: <p>Hi everyone, I made a class for confusion matrix that supports <code>cuda</code>. I hope that it helps other people as well. The trick to make it work was to use <code>torch.bincount</code>.</p>
<pre><code class="lang-auto">import torch


class ConfusionMatrix:
    _device = 'cuda' if torch.cuda.is_available() else 'cpu'

    def __init__(self, n_classes: int = 10):
        self._matrix = torch.zeros(n_classes * n_classes).to(self._device)
        self._n = n_classes

    def cpu(self):
        self._matrix.cpu()

    def cuda(self):
        self._matrix.cuda()

    def to(self, device: str):
        self._matrix.to(device)

    def __add__(self, other):
        if isinstance(other, ConfusionMatrix):
            self._matrix.add_(other._matrix)
        elif isinstance(other, tuple):
            self.update(*other)
        else:
            raise NotImplemented
        return self

    def update(self, prediction: torch.tensor, label: torch.tensor):
        conf_data = prediction * self._n + label
        conf = conf_data.bincount(minlength=self._n * self._n)
        self._matrix.add_(conf)

    @property
    def value(self):
        return self._matrix.view(self._n, self._n).T


def main():
    label = torch.tensor([0, 0, 0, 1, 1, 1, 2, 2, 2])
    pred = torch.tensor([0, 1, 0, 0, 0, 1, 2, 2, 2])
    conf = ConfusionMatrix(3)
    conf += pred, label
    print(conf.value)

    conf2 = ConfusionMatrix(3)
    pred2 = torch.Tensor([0, 0, 0, 1, 1, 1, 2, 2, 2]).long()
    conf2.update(pred2, label)
    print(conf2.value)

    conf += conf2
    print(conf.value)


if __name__ == '__main__':
    main()
</code></pre>