Jumpy validation graph

username: netaglazer
cooked: <p>hi, i have a very jumpy validation accuracy graph, and i dont know why.</p>
<p>the graph:</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/a/5/a506368ace63a01906c47bd14d44e76309369e6b.png" alt="image" data-base62-sha1="nxS5PsY54uvHztiMEzNdaWSsILp" width="364" height="281"></p>
<p>blue - train loss<br>
red - validation<br>
purple - train accuracy<br>
green - validation accuracy</p>
<p>it is a multi label problem, and i calculate the accuracy this way:</p>
<pre><code class="lang-auto">    for i in range(l):
        if(prediction[i] &gt; 0.5 and label[i] == 1):
            return True
</code></pre>
<p>the last layer of the model is softmax</p>

username: charan_Vjy
cooked: <p>It does smoothen out in the end which is encouraging.<br>
As far as the accuracy is concerned, could you show the way you calculate accuracy in more detail. For example, it is not clear to me as to what ‘l’ is. I am assuming label is the ground truth and ‘l’ is the number of classes. The code, <code>if(prediction[i] &gt; 0.5 and label[i] == 1)</code> is not an ideal way to compute accuracy. A better idea is to take the index of the maximum value of prediction and use that as the predicted label as shown below. This is done to ensure that you can compute the predicted labels independent of the ground truth labels which is not the case in your code.</p>
<pre><code class="lang-auto">import numpy as np
import torch
no_of_classes = 2
target = torch.from_numpy(np.random.randint(0,1,64))
predicted = torch.from_numpy(np.random.randn(64,no_of_classes))
### Accuracy is computed after this
no_correct_predictions = torch.sum((predicted.argmax(dim = 1) == target).float())
acc = no_correct_predictions/ predicted.shape[0]
</code></pre>

username: netaglazer
cooked: <p>this is the whole calculation:</p>
<pre><code class="lang-auto">def success(prediction, label):
    l = len(prediction)
    for i in range(l):
        if(prediction[i] &gt; 0.5 and label[i] == 1):
            return True
    return False


def accuracy_score(y_true, y_pred):
    acc = 0
    batch_size = len(y_true)
    for i in range(batch_size):
        if(success(y_pred[i],y_true[i])):
            acc = acc + 1
    return float(acc)/batch_size
</code></pre>

username: charan_Vjy
cooked: <p>Looks like my assumptions were right. Can you replace your method for computing accuracy with mine and obtain the curves? I don’t think my code will solve the jumpiness. However, it is advisable to use this method.<br>
Also, your validation accuracy is jumping corresponding to jumps in validation loss which means it is not the accuracy computation that is wrong. Since the scale of the validation loss is in the 1 - 10^-1, you see a smaller oscillation. This means there is something funny happening in the loss itself. I suspect it has got something to do with the validation data.</p>

username: netaglazer
cooked: <p>i have a cotume loss because my problem is not classic<br>
maybe it will help you to help me <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"><br>
this is my loss function:</p>
<pre><code class="lang-auto">def new_loss_function(p, y, b):
    eps = 0.000000000000000000000000001
    losses = 0
    k = len(y[0])
    ones = torch.ones(1, 1).expand(b, k).cuda()
    loss1 = -((ones-y)*(((ones-p)+(eps)).log())).sum(dim=1)
    prod = (ones-y)*k - y*((p+ eps).log())
    loss2 = torch.min(prod, dim=1)[0]
    losses = (loss1 + loss2).sum()
    return losses / b
</code></pre>

username: netaglazer
cooked: <p>did it change enything?</p>

username: charan_Vjy
cooked: <p>Again a lot of the variables that are not clear. Could you expand them?</p>

username: netaglazer
cooked: <p>Yes,<br>
P - model predictiona<br>
Y - labels<br>
B - batch size</p>
<p>It is a multi label problem, so k - the nimber of the classes</p>

username: charan_Vjy
cooked: <p>Did you check the validation data? They validation accuracy seems to oscillate between 0.4 and 1. Very weird behavior. I do not have a prima facie answer for this.</p>

username: netaglazer
cooked: <p>ok,<br>
thank you vary much</p>