My model has 0.7142 Accuracy in 99% of the time while my loss value jumps around

username: Patrick1
cooked: <p>I am using a lstm as a classifier for timeseries data for 3 classes. I print out my predictions and labels at the end with my loss and accuracy. No matter if my Loss is 0.1 or 1.0 my accuracy is almost always 0.7142.</p>
<p>I am wondering if I calculated loss and accuracy correctly in my model or if this is just a problem about hyperparamter tuning. I am under a lot of time pressure so if someone could help me I would be very thankful.</p>
<p>My validation loop looks as follows.<br>
EDIT: I am using CrossEntropyLoss() as criterion</p>
<pre><code class="lang-auto">for i, (inputs, labels) in enumerate(valloader):

            y_pred = model(inputs)                                                 
            _,prediction2 = torch.max(y_pred, 1)
            labels = labels.type_as(y_pred)
            labels = labels.long()
            val_loss = criterion(y_pred, labels)
            running_val_loss += val_loss.item()

            for x in range(len(labels)):
                y_true_list.append(labels[x].item())
            for x in range(len(prediction2)):
                y_pred_list.append(prediction2[x].item())
            
            total += labels.size(0)
            correct += (prediction2 == labels).sum().item()
            
            #getting output for metric calc correct
            softmax = nn.Softmax(dim=1)
            probability = softmax(y_pred)
           
            for x in range(len(probability)):
                probability_each = probability[x].detach().numpy()
                prob_softmax.append(probability_each)
           val_loss_mean = running_val_loss/len(valloader)
        validation_losses.append(val_loss_mean)

        accuracy = (correct/total)
        validation_accuracys.append(accuracy)
 
        print("Y True and Y Pred :",y_true_list, y_pred_list)
        print(f'Fold {fold}: For Epoche {epoch} the Loss is {val_loss_mean} and the Acc is {accuracy}')

</code></pre>

username: ptrblck
cooked: <p>I guess <code>0.7142</code> would correlate to the potential class imbalance in your <code>Dataset</code>?<br>
Could you check, if your model is predicting the same class for all samples?</p>

username: Patrick1
cooked: <p>That is what I thought as well but it does not. I am printing the true label list and the predicted label list at the end and my model does not always predict the same class.</p>
<p>Maybe I found one problem in my model which causes this. Maybe my model does not know when a sequence inside a batch ends and the next one starts and it is treating it like a whole sequence? I am having trouble understanding how to make my model effectively use batch training. Different sources say different things about implementing hidden state reset and none work for me.</p>
<aside class="quote quote-modified" data-post="1" data-topic="143698">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/patrick1/48/30075_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/how-do-i-tell-my-model-that-a-new-sequence-starts-when-batch-training-timeseries/143698">How do I tell my model that a new sequence starts when batch training timeseries?</a> 
  </div>
  <blockquote>
    I am training an LSTM model on timeseries data to build a classifier. So when I do batch training with for example batch size = 8. How does my model know when one sequence ends and another one starts? I have read a couple times that I would need to reset hidden and cell state inside my model, but I am not sure if this is correct or not, as I have seen some people not doing it. Either way I have tried it and the Problem I have is that when initializing h0 und c0 in my model it expects the size of…
  </blockquote>
</aside>

<p><a href="https://stackoverflow.com/questions/71053842/how-do-i-tell-pytorch-when-one-sequence-inside-a-batch-ends-and-the-next-one-sta" class="onebox" target="_blank" rel="noopener nofollow ugc">https://stackoverflow.com/questions/71053842/how-do-i-tell-pytorch-when-one-sequence-inside-a-batch-ends-and-the-next-one-sta</a></p>