Calculating accuracy of the current minibatch?

username: deepcode
cooked: <p>How can I calculate the acuracy for  the current mini batch in my training? My trainining code is just:</p>
<pre><code>for epoch in range(args.epochs):
  for i, (images, captions, lengths) in enumerate(train_loader):
    targets = pack_padded_sequence(captions, lengths, batch_first=True)[0]
    features = encoder(images)
    outputs = decoder(features, captions, lengths)
    loss = criterion(outputs, targets)
</code></pre>
<p>how can I get how many are correct from that code?<br>
I tried several ways but cant get it to work:<br>
correct = (targets.eq(outputs)).sum()</p>
<p>Im sure there should be a generic way to do this. if criterion can calculate the loss without knowing the shapes, I think I should be able to calculate accuracy as well? Any idea on how to do this?</p>
<p>File “<a href="http://train.py" rel="nofollow noopener">train.py</a>”, line 182, in <br>
main(args)<br>
File “<a href="http://train.py" rel="nofollow noopener">train.py</a>”, line 139, in main<br>
correct = (ttargets.eq(toutputs)).sum()<br>
File “/home/joy/anaconda3/lib/python3.6/site-packages/torch/autograd/variable.py”, line 710, in eq<br>
return Eq()(self, other)<br>
File “/home/joy/anaconda3/lib/python3.6/site-packages/torch/autograd/_functions/compare.py”, line 14, in forward<br>
mask = getattr(tensor1, self.fn_name)(other)<br>
TypeError: eq received an invalid combination of arguments - got (torch.cuda.FloatTensor), but expected one of:</p>
<ul>
<li>(int value)<br>
didn’t match because some of the arguments have invalid types: (torch.cuda.FloatTensor)</li>
<li>(torch.cuda.LongTensor other)<br>
didn’t match because some of the arguments have invalid types: (torch.cuda.FloatTensor)</li>
</ul>

username: chenyuntc
cooked: <pre><code class="lang-auto">correct = (ttargets.eq(toutputs.long())).sum()
</code></pre>

username: deepcode
cooked: <p>Thanks ,that fixed the Long issue. I get another issue after that, the tensor sizes do not match.<br>
ttargets.size() is<br>
[torch.cuda.LongTensor of size 11 (GPU 0)]<br>
and toutputs.size() is:<br>
[torch.cuda.FloatTensor of size 11x13 (GPU 0)]</p>
<p>My question is how can I calculate accuracy generically for any tensor like how  " loss = criterion(outputs, targets)" can calculate the loss without knowing the details? I looked through the code for criterion (CrossEntropyLoss) and I am confused on how it is calculating loss.</p>

username: chenyuntc
cooked: <pre><code class="lang-auto">max_index = ttoutputs.max(dim = 1)[1]
(max_index == ttargets).sum()
</code></pre>
<p>the detail of <code>max</code> function can be found from docs of <code>torch.max</code></p>

username: Brando_Miranda
cooked: <p>Does the following not work:</p>
<pre><code class="lang-auto">def calc_accuracy(mdl,X,Y):
    # TODO: why can't we call .data.numpy() for train_acc as a whole?
    max_vals, max_indices = torch.max(mdl(X),1)
    train_acc = (max_indices == Y).sum().data.numpy()/max_indices.size()[0]
    return train_acc
</code></pre>
<p>??</p>

username: Brando_Miranda
cooked: <p>seems this works too:</p>
<pre><code class="lang-auto">def calc_accuracy(mdl,X,Y):
    max_vals, max_indices = torch.max(mdl(X),1)
    train_acc = (max_indices == Y).sum().item()/max_indices.size()[0]
    return train_acc
</code></pre>
<p>if you want the output to be a tensor:</p>
<pre><code class="lang-auto">def calc_accuracy(mdl,X,Y):
    """Calculates model accuracy
    
    Arguments:
        mdl {nn.model} -- nn model
        X {torch.Tensor} -- input data
        Y {torch.Tensor} -- labels/target values
    
    Returns:
        [torch.Tensor] -- accuracy
    """
    max_vals, max_indices = torch.max(mdl(X),1)
    n = max_indices.size(0) #index 0 for extracting the # of elements
    train_acc = (max_indices == Y).sum(dtype=torch.float32)/n
    return train_acc
</code></pre>

username: Ali_Amiri
cooked: <p>i use this usually for classification:</p>
<pre><code class="lang-auto">def accuracy(true,pred):
    acc = (true.argmax(-1) == pred.argmax(-1)).float().detach().numpy()
    return float(100 * acc.sum() / len(acc))
</code></pre>
<p>which true and pred are both a torch tensor</p>

username: sajastu
cooked: <p>I use the following snippet when having preds, and labels and mask:</p>
<p><code>acc=(((pred==labels)*mask).sum(dim=1))/mask.sum(dim=1)</code></p>

username: Brando_Miranda
cooked: <p>do you have a small snippet to test this that is self contained?</p>
<p><strong>Can you explain the negative index?</strong></p>

username: Brando_Miranda
cooked: <p>I know this is basic but it’s not fresh in my head. What is the mask suppose to stand for when they are predictions? is it 0-1 values? What is it suppose to mask off? Unwanted examples…?</p>
<hr>
<p><a class="mention" href="/u/ali_amiri">@Ali_Amiri</a></p>
<p>can you comment on the <code>true.argmax(-1)</code>? What does the <code>-1</code> do? ----- hopefully answer will be here someday: <a href="https://discuss.pytorch.org/t/argmax-with-pytorch/1528/10" class="inline-onebox">Argmax with PyTorch</a></p>

username: Brando_Miranda
cooked: <p>I think the simplest answer is the one from <a href="https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html" rel="nofollow noopener">the cifar10 tutorial</a>:</p>
<pre><code class="lang-auto">total = 0
with torch.no_grad():
    net.eval()
    for data in testloader:
        images, labels = data
        outputs = net(images)
        _, predicted = torch.max(outputs.data, 1)
        total += labels.size(0)
        correct += (predicted == labels).sum().item()

print('Accuracy of the network on the 10000 test images: %d %%' % (
    100 * correct / total))
</code></pre>
<p>so:</p>
<pre><code class="lang-auto">acc = (true == pred).sum().item()
</code></pre>
<p>If you have a counter don’t forget to eventually divide by the size of the data-set or analogous values.</p>
<p>I’ve used:</p>
<pre><code class="lang-auto">N = data.size(0) # since usually it's size (batch_size, D1, D2, ...)
correct += (1/N) * correct
</code></pre>
<hr>
<p>Self contained code:</p>
<pre><code class="lang-auto"># testing accuracy function
# https://discuss.pytorch.org/t/calculating-accuracy-of-the-current-minibatch/4308/11
# https://stackoverflow.com/questions/51503851/calculate-the-accuracy-every-epoch-in-pytorch

import torch
import torch.nn as nn

D = 1
true = torch.tensor([0,1,0,1,1]).reshape(5,1)
print(f'true.size() = {true.size()}')

batch_size = true.size(0)
print(f'batch_size = {batch_size}')
x = torch.randn(batch_size,D)
print(f'x = {x}')
print(f'x.size() = {x.size()}')

mdl = nn.Linear(D,1)
logit = mdl(x)
_, pred = torch.max(logit.data, 1)

print(f'logit = {logit}')

print(f'pred = {pred}')
print(f'true = {true}')

acc = (true == pred).sum().item()
print(f'acc = {acc}')
</code></pre>
<hr>
<p>Also, I find this code to be good reference:</p>
<pre><code class="lang-auto">def calc_accuracy(mdl, X, Y):
    # reduce/collapse the classification dimension according to max op
    # resulting in most likely label
    max_vals, max_indices = mdl(X).max(1)
    # assumes the first dimension is batch size
    n = max_indices.size(0)  # index 0 for extracting the # of elements
    # calulate acc (note .item() to do float division)
    acc = (max_indices == Y).sum().item() / n
    return acc
</code></pre>
<hr>
<p>Explaining <code>pred = mdl(x).max(1)</code>see this <a href="https://discuss.pytorch.org/t/how-does-one-get-the-predicted-classification-label-from-a-pytorch-model/91649" class="inline-onebox">How does one get the predicted classification label from a pytorch model?</a></p>
<p>the main thing is that you have to reduce/collapse the dimension where the classification raw value/logit is with a max and then select it with a <code>.indices</code>. Usually this is dimensions <code>1</code> since dim 0 has the batch size e.g. <code>[batch_size,D_classification]</code> where the raw data might of size <code>[batch_size,C,H,W]</code></p>
<p>A synthetic example with raw data in 1D as follows:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn

# data dimension [batch-size, D]
D, Dout = 1, 5
batch_size = 16
x = torch.randn(batch_size, D)
y = torch.randint(low=0,high=Dout,size=(batch_size,))

mdl = nn.Linear(D, Dout)
logits = mdl(x)
print(f'y.size() = {y.size()}')
# removes the 1th dimension with a max, which is the classification layer
# which means it returns the most likely label. Also, note you need to choose .indices since you want to return the
# position of where the most likely label is (not it's raw logit value)
pred = logits.max(1).indices
print(pred)

print('--- preds vs truth ---')
print(f'predictions = {pred}')
print(f'y = {y}')

acc = (pred == y).sum().item() / pred.size(0)
print(acc)
</code></pre>
<p>output:</p>
<pre><code class="lang-auto">
y.size() = torch.Size([16])
tensor([3, 1, 1, 3, 4, 1, 4, 3, 1, 1, 4, 4, 4, 4, 3, 1])
--- preds vs truth ---
predictions = tensor([3, 1, 1, 3, 4, 1, 4, 3, 1, 1, 4, 4, 4, 4, 3, 1])
y = tensor([3, 3, 3, 0, 3, 4, 0, 1, 1, 2, 1, 4, 4, 2, 0, 0])
0.25
</code></pre>
<hr>
<p>reference:</p>
<ul>
<li><a href="https://discuss.pytorch.org/t/calculating-accuracy-of-the-current-minibatch/4308/5" class="inline-onebox">Calculating accuracy of the current minibatch?</a></li>
<li><a href="https://discuss.pytorch.org/t/how-does-one-get-the-predicted-classification-label-from-a-pytorch-model/91649/3" class="inline-onebox">How does one get the predicted classification label from a pytorch model?</a></li>
<li>SO: <a href="https://stackoverflow.com/questions/51503851/calculate-the-accuracy-every-epoch-in-pytorch/63253642#63253642" rel="nofollow noopener">https://stackoverflow.com/questions/51503851/calculate-the-accuracy-every-epoch-in-pytorch/63253642#63253642</a>
</li>
</ul>