Understanding different Metrics implementations (IoU)

mayool: <p>Hey!<br>
I am working on an evaluation script for my semantic segmentation model and was looking for some IoU implementations.</p>
<p>The first one I found was <a href="https://www.kaggle.com/iezepov/fast-iou-scoring-metric-in-pytorch-and-numpy" rel="nofollow noopener">this one:</a></p>
<pre><code class="lang-auto">EPS = 1e-6
#slightly modified
def get_IoU(outputs, labels):
    outputs = outputs.int()
    labels = labels.int()
    # Taken from: https://www.kaggle.com/iezepov/fast-iou-scoring-metric-in-pytorch-and-numpy
    intersection = (outputs &amp; labels).float().sum((1, 2))  # Will be zero if Truth=0 or Prediction=0
    union = (outputs | labels).float().sum((1, 2))  # Will be zero if both are 0

    iou = (intersection + EPS) / (union + EPS)  # We smooth our devision to avoid 0/0

    # thresholded = torch.clamp(20 * (iou - 0.5), 0, 10).ceil() / 10  # This is equal to comparing with thresolds
    # return thresholded.mean()  # Or thresholded.mean() if you are interested in average across the batch
    return iou.mean()
</code></pre>
<p>The second one was found <a href="https://github.com/kevinzakka/pytorch-goodies/blob/master/metrics.py" rel="nofollow noopener">here (Jaccard Index):</a></p>
<pre><code class="lang-auto"># computes confusion matrix
def _fast_hist(true, pred, num_classes):
    mask = (true &gt;= 0) &amp; (true &lt; num_classes)
    hist = torch.bincount(
        num_classes * true[mask] + pred[mask],
        minlength=num_classes ** 2,
    ).reshape(num_classes, num_classes).float()
    return hist

# computes IoU based on confusion matrix
def jaccard_index(hist):
    """Computes the Jaccard index, a.k.a the Intersection over Union (IoU).
    Args:
        hist: confusion matrix.
    Returns:
        avg_jacc: the average per-class jaccard index.
    """
    A_inter_B = torch.diag(hist)
    A = hist.sum(dim=1)
    B = hist.sum(dim=0)
    jaccard = A_inter_B / (A + B - A_inter_B + EPS)
    avg_jacc = nanmean(jaccard) #the mean of jaccard without NaNs
    return avg_jacc, jaccard
</code></pre>
<p>I wanted to cross check that both compute the same think:</p>
<pre><code class="lang-auto">true = torch.tensor([[
    [1, 0, 0],
    [1, 0, 0],
    [1, 0, 0],
]])
pred = torch.tensor([[
    [1, 0, 0],
    [1, 1, 0],
    [1, 0, 0],
]])
print(true.shape, pred.shape)
print(get_IoU(pred, true))
hist = _fast_hist(pred, true, num_classes=2)
print(jaccard_index(hist))
</code></pre>
<p>result is:</p>
<pre><code class="lang-auto">torch.Size([1, 3, 3]) torch.Size([1, 3, 3])
tensor(0.7500)
(tensor(0.7917), tensor([0.8333, 0.7500]))
</code></pre>
<p>I have 2 Questions:</p>
<ol>
<li>What is the correct IoU result? 0.75 / 0.83 / 0.79?</li>
<li>Can someone explain to me what the purpose of the EPS (I assume epsilone?) value is and why we need it?</li>
</ol>
<p>Thanks for any help!</p>

Antonio_Ossa: <p>Hi <a class="mention" href="/u/mayool">@mayool</a>,</p>
<p>I think that the answer is: it depends (as usual).</p>
<ul>
<li>The first code assumes you have one class: “1”. If you calculate the IoU score manually you have: 3 "1"s in the right position and 4 "1"s in the union of both matrices: <code>3/4 = 0.7500</code>.</li>
<li>If you consider that you have two classes: “1” and “0”. We know already that “1” has an IoU score of 0.7500. Considering only "0"s, you have 5 "0"s correctly predicted, and 6 “0s” in the union: <code>5/6 = 0.8333</code>.</li>
</ul>
<p>Also, your code seems to have mistaken the input order of <code>_fast_hist</code>, it should be <code>hist = _fast_hist(true, pred, num_classes=2)</code> (notice pred and true are not in the right order), but it gave the right answer anyway.</p>
<p>So, for question 1: <code>0.7500</code> is the IoU for class “1” and <code>0.8333</code> is the IoU for class “0”. <code>0.7971</code> is just the average of both.</p>
<p>The purpouse of <code>EPS</code> (yes, epsilon) is to avoid division by zero. I think that case is possible only if your class is not present in any of the matrices, but is not bad to have it. Also, the epsilon value is so small that it should not modify your values in a significant way.</p>

stark: <p>Hi <a class="mention" href="/u/antonio_ossa">@Antonio_Ossa</a>,</p>
<p>Thanks for this nice explanation.</p>
<p>I have a slightly different but still relevant question. Is there some standard practice in the research community for publishing results in journals/conferences?</p>
<p>Let’s consider a two class problem (a.k.a. <strong>binary segmentation</strong>) where <strong><code>c=0</code></strong> for background (healthy tissues) and <strong><code>c=1</code></strong> for foreground (diseased tissues).</p>
<p>What should be reported<br>
<strong>a.</strong> <code>IoU(c=1)</code><br>
<strong>b.</strong> <code>mean IoU = [IoU(c=0) + IoU(c=1)] / 2</code></p>

Antonio_Ossa: <p>Hi <a class="mention" href="/u/stark">@stark</a> ,</p>
<p>In that case, considering the context, I think it should be relevant to present both IoU (<code>IoU(c=0)</code> and <code>IoU(c=1)</code>). I’m assuming that there may be some class imbalance and also that both values have different importance. I’m not sure if there’s any standard practice but you should always try to provide the relevant insight considering the context. Personally, I would report the overall IoU in a table but in the analysis I would report both to confirm that the mean is (or not) hidding any insights.</p>