Accuracy not changing after second training epoch

username: Taher_Anjary
cooked: <p>Hi there. Kinda new to PyTorch. I have implemented a fully connected neural network and am trying to compute accuracy on the training samples. After the first epoch, I get a certain accuracy, after the second epoch, i get a slightly better accuracy, great. But then accuracy doesn’t change. Can anyone point me to the bug please? Note that I am not yet testing. I am still debugging my training algorithm</p>
<pre><code class="lang-auto">#HINT: note that your training time should not take many days.

max_epoch = 100
train_batch = 4
test_batch = 500
learning_rate = 0.1
use_gpu = torch.cuda.is_available()

def main(): # you are free to change parameters
    
    xl_file = 'labels.xlsx'
    root_dir = 'C:\\Users\\t.anjary\\Desktop\\ML\\ML HW3\\images'
    train_set, val_set, test_set = get_dataset(xl_file, root_dir)
    
    train_loader = torch.utils.data.DataLoader(train_set, batch_size=train_batch, shuffle=True)
    val_loader = torch.utils.data.DataLoader(val_set, batch_size=test_batch, shuffle=False)
    test_loader = torch.utils.data.DataLoader(test_set, batch_size=test_batch, shuffle=False)
    
    model = FNet().float() #.cuda()
    criterion = torch.nn.MSELoss(reduction='sum')
    optimizer = torch.optim.SGD(model.parameters(), lr=learning_rate)#, momentum=0.9, weight_decay=5e-04) # you can play with momentum and weight_decay parameters as well
    
    for epoch in range(max_epoch):
        print('\n epoch '+ str(epoch))
        train(epoch, model, criterion, optimizer, train_loader)
    
''' Train your network for a one epoch '''
def train(epoch, model, criterion, optimizer, loader): # you are free to change parameters
    #print('train is called')
    accuracies =0
    model.train()
    correct = 0
    total = 0
    for batch_idx, (data, labels) in enumerate(loader):
        # TODO:
        # Implement training code for a one iteration
        
        #forward
        optimizer.zero_grad()
        data = data.float() #.cuda()
        labels = labels.float() #.cuda()
        y_pred = model(data)
        
        #loss
        loss = criterion(y_pred, labels)
        
        #update
        loss.backward()
        optimizer.step()
        
        #performance
        #y_pred = y_pred.cpu().numpy()
        y_pred = y_pred.flatten()
        total = labels.size(0)
        correct = (y_pred == labels).sum().item()
        accuracy = 100 * correct / total
        accuracies += accuracy 
        
        #info = 'iteration: ' + str(batch_idx) + '  loss: ' + str(loss.item()) + '  accuracy: ' + str(accuracy)
        #print(info)
        #sys.stdout.flush()
        sys.stdout.write('\r iteration: ' + str(batch_idx) + '  loss: ' + str(loss.item()) + '  accuracy: ' + str(accuracy))
        
        
    accuracies = accuracies/(batch_idx+1)
    print('\n training accuracy '+str(accuracies))

''' Test&amp;Validate your network '''
def test(model, loader,criterion): # you are free to change parameters

    model.eval()
    correct = 0
    total = 0
    with torch.no_grad():
        for batch_idx, (data, labels) in enumerate(loader):
            data = data.float() #.cuda()
            labels = labels.float() #.cuda()
            
            #forward only
            y_pred = model(data)

            #loss
            loss = criterion(y_pred, labels)
        
            #assessy_pred = y_pred.flatten()
            y_pred = y_pred.flatten()
            total = labels.size(0)
            correct = (y_pred == labels).sum().item()
            acc = 100 * correct / total
            
            print('validation accuracy '+str(acc))
            return acc
        
        
main()
</code></pre>

username: KFrank
cooked: <p>Hi Taher!</p>
<aside class="quote no-group" data-username="Taher_Anjary" data-post="1" data-topic="80405" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/taher_anjary/48/23973_2.png" class="avatar"> Taher_Anjary:</div>
<blockquote>
<p>But then accuracy doesn’t change.</p>
</blockquote>
</aside>
<p>The short answer is that this line:</p>
<aside class="quote no-group">
<blockquote>
<pre><code class="lang-auto">        correct = (y_pred == labels).sum().item()
</code></pre>
</blockquote>
</aside>
<p>is a mistake because it is performing an exact-equality test on<br>
floating-point numbers.  (In general, doing so is a programming<br>
bug except in certain special circumstances.)</p>
<p>(Note, this doesn’t affect your loss function, so your training<br>
could be working.)</p>
<p>You don’t mention what sort of problem you’re working on or<br>
what your data or model looks like, so we can’t really proceed<br>
without some speculation.  But if speculation is what you need,<br>
you’ve come to the right place!</p>
<aside class="quote no-group">
<blockquote>
<pre><code class="lang-auto">    root_dir = 'C:\\Users\\t.anjary\\Desktop\\ML\\ML HW3\\images'
</code></pre>
</blockquote>
</aside>
<p>I speculate that you are working with images.  Based on this, I<br>
speculate that you are working on an object classification or<br>
object detection problem.</p>
<aside class="quote no-group">
<blockquote>
<pre><code class="lang-auto">    model = FNet().float() #.cuda()
</code></pre>
</blockquote>
</aside>
<p>There is something floating around out there called “FNet” that is<br>
used for face detection.  So maybe you are working on object<br>
detection.</p>
<aside class="quote no-group">
<blockquote>
<pre><code class="lang-auto">    criterion = torch.nn.MSELoss(reduction='sum')
</code></pre>
</blockquote>
</aside>
<p>Your loss could be the mean-squared-error between the predicted<br>
locations of objects detected by your object detector, and their<br>
known locations as given in your annotated dataset.  This could<br>
make sense.</p>
<aside class="quote no-group">
<blockquote>
<pre><code class="lang-auto">        labels = labels.float() #.cuda()
        y_pred = model(data)
        
        #loss
        loss = criterion(y_pred, labels)
</code></pre>
</blockquote>
</aside>
<p>Based on this, both your <code>y_pred</code> and <code>labels</code> are floats.</p>
<aside class="quote no-group">
<blockquote>
<pre><code class="lang-auto">        correct = (y_pred == labels).sum().item()
        accuracy = 100 * correct / total
</code></pre>
</blockquote>
</aside>
<p>This is a problem.  Here you are performing an exact-equality test<br>
of floating-point numbers.  Doing so is (almost) always a mistake.</p>
<p>Even if your <code>labels</code> are nicely discrete values (say, pixel locations)<br>
such as <code>1.00</code> or <code>503.00</code>, for which an exact-equality test could<br>
make sense, your <code>y_pred</code> are not.  So the exact-equality test will<br>
almost always fail, and the <code>accuracy</code> you calculate will always be<br>
very low.</p>
<p>(Note, if you try to force your <code>y_pred</code> to be discrete, for example,<br>
by rounding them to the nearest integral pixel location, you will<br>
have performed a non-differentiable operation on them, so you<br>
won’t be able to back-propagate usefully.)</p>
<p>As to whether the rest of your code makes sense, it’s hard to<br>
say without knowing what you are trying to do.</p>
<p>Good luck.</p>
<p>K. Frank</p>

username: Taher_Anjary
cooked: <p>Hi there. My apologies, I will give more details. The task is to  perform binary classification on Ocular Disease Recognition, ODIR5k, dataset. Images are 128x256 rgb images. I process them into grayscale and flatten them in my custom dataset class:</p>
<pre><code class="lang-auto">class OcularDataset(Dataset):
    
    # TODO:
    # Define constructor for AnimalDataset class
    # HINT: You can pass processed data samples and their ground truth values as parameters 
    def __init__(self, xl_file, root_dir):
        #print('init was called')
        self.index_labels = pd.read_excel(xl_file)
        self.root_dir = root_dir
        self.n_samples = len(self.index_labels)
        
    '''This function should return sample count in the dataset'''
    def __len__(self):
        return self.n_samples

    '''This function should return a single sample and its ground truth value from the dataset corresponding to index parameter '''
    def __getitem__(self, index):
        img_path = os.path.join(self.root_dir,str(self.index_labels.iloc[index,0])+'.jpg')
        img = mpimg.imread(img_path)
        rgb_weights = [0.2989, 0.5870, 0.1140]
        img = np.dot(img, rgb_weights)/255
        img = torch.tensor(img)
        img = img.reshape([1,-1])
        y = torch.tensor(int(self.index_labels.iloc[index,1]))
        
        return img, y


def get_dataset(xl_file, root_dir):
    fdataset = OcularDataset(xl_file, root_dir)
    
    train_set, val_set, test_set = torch.utils.data.random_split(fdataset, [2500,500,500])
    
    return train_set, val_set, test_set
</code></pre>
<p>Sample original image:</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/7/c/7cdfc8fda82a177db341ccd342606df1f8c5fd74.jpeg" alt="406" data-base62-sha1="hOGF9r2OCaa3gic3n4TYusSTGtu" width="256" height="128"></p>
<p>After the custom dataset, I implement a neural network class:</p>
<pre><code class="lang-auto">class FNet(nn.Module):
    
    def __init__(self): 
        super(FNet, self).__init__()
        self.linear1 = torch.nn.Linear(128*256, 1024)
        self.linear2 = torch.nn.Linear(1024, 256)
        self.linear3 = torch.nn.Linear(256, 1)
     
    def forward(self, X): 
        z1 = torch.nn.functional.relu(self.linear1(X))
        z2 = torch.nn.functional.relu(self.linear2(z1))
        y = torch.sigmoid(self.linear3(z2))
        
        return y
</code></pre>
<p>And then comes the code I had attached earlier, but with a few changes based on <strong>K.Franks</strong> insight on doing equality comparisons between predictions and labels. If you’re wondering why I convert everything to float, it’s because it solved other errors I was getting.</p>
<pre><code class="lang-auto">#HINT: note that your training time should not take many days.

max_epoch = 100
train_batch = 25
test_batch = 500


learning_rate = 0.0001

use_gpu = torch.cuda.is_available()


def main(): # you are free to change parameters
    
    xl_file = 'labels.xlsx'
    root_dir = 'C:\\Users\\t.anjary\\Desktop\\ML\\ML HW3\\images'
    train_set, val_set, test_set = get_dataset(xl_file, root_dir)
    
    train_loader = torch.utils.data.DataLoader(train_set, batch_size=train_batch, shuffle=True)
    val_loader = torch.utils.data.DataLoader(val_set, batch_size=test_batch, shuffle=False)
    test_loader = torch.utils.data.DataLoader(test_set, batch_size=test_batch, shuffle=False)
    
    model = FNet().float() #.cuda()
    criterion = torch.nn.MSELoss(reduction='mean')
    optimizer = torch.optim.SGD(model.parameters(), lr=learning_rate, momentum=0.5, weight_decay=5e-04) # you can play with momentum and weight_decay parameters as well
    
    for epoch in range(max_epoch):
        print('\n epoch '+ str(epoch))
        train(epoch, model, criterion, optimizer, train_loader)
        test(model, test_loader,criterion)
    
''' Train your network for a one epoch '''
def train(epoch, model, criterion, optimizer, loader): # you are free to change parameters
    #print('train is called')
    accuracies = 0
    losses = 0
    model.train()
    correct = 0
    total = 0
    for batch_idx, (data, labels) in enumerate(loader):
        # TODO:
        # Implement training code for a one iteration
        
        #forward
        optimizer.zero_grad()
        data = data.float() #.cuda()
        labels = labels.float() #.cuda()
        y_pred = model(data)
        
        #loss
        loss = criterion(y_pred, labels)
        
        #update
        loss.backward()
        optimizer.step()
        
        #performance
        #y_pred = y_pred.cpu().numpy()
        y_pred = y_pred.flatten()
        total = len(y_pred)#labels.size(0)
        
        y_2 = torch.zeros(len(y_pred))
        y_2[y_pred&gt;=0.5] = 1
        y_2 = y_2.int()
        correct = 1*(y_2 == labels.int()).sum().item()
        correct = (y_pred == labels).sum().item()
        accuracy = correct / total
        accuracies += accuracy 
        losses += loss
        
        #info = 'iteration: ' + str(batch_idx) + '  loss: ' + str(loss.item()) + '  accuracy: ' + str(accuracy)
        #print(info)
        #sys.stdout.flush()
        #sys.stdout.write('\r iteration: ' + str(batch_idx) + '  loss: ' + str(loss.item()) + '  accuracy: ' + str(accuracy))
        
        
    accuracies = accuracies/(batch_idx+1)
    losses = losses/(batch_idx+1)
    print('training accuracy '+str(accuracies))
    print('training loss '+str(losses.item()))

''' Test&amp;Validate your network '''
def test(model, loader,criterion): # you are free to change parameters

    model.eval()
    correct = 0
    total = 0
    with torch.no_grad():
        for batch_idx, (data, labels) in enumerate(loader): # this loop will iterate once only
            data = data.float() #.cuda()
            labels = labels.float() #.cuda()
            
            #forward only
            y_pred = model(data)

            #loss
            loss = criterion(y_pred, labels)
        
            #assessy_pred = y_pred.flatten()
            y_pred = y_pred.flatten()
            y2 = torch.zeros(len(y_pred))
            y2[y_pred&gt;=0.5] = 1
            y2 = y2.int()
            total = labels.size(0)
            correct = (y2 == labels.int()).sum().item()
            acc = correct / total
            
            print('validation accuracy '+str(acc))
            print('validation loss '+str(loss.item()))
            #return acc
        
        
main()
</code></pre>
<p>In any case, running the code produces the following output:</p>
<pre><code class="lang-auto">epoch 0
training accuracy 0.0
training loss 0.24466806650161743
validation accuracy 0.674
validation loss 0.23942209780216217

 epoch 1
training accuracy 0.0
training loss 0.23551715910434723
validation accuracy 0.674
validation loss 0.23317639529705048

 epoch 2
training accuracy 0.0
training loss 0.2304917573928833
validation accuracy 0.674
validation loss 0.22967855632305145

 epoch 3
training accuracy 0.0
training loss 0.22782883048057556
validation accuracy 0.674
validation loss 0.22740887105464935

 epoch 4
training accuracy 0.0
training loss 0.22573482990264893
validation accuracy 0.674
validation loss 0.22585797309875488

 epoch 5
training accuracy 0.0
training loss 0.22470112144947052
validation accuracy 0.674
validation loss 0.22480493783950806

 epoch 6
training accuracy 0.0
training loss 0.22381442785263062
validation accuracy 0.674
validation loss 0.22404111921787262

 epoch 7
training accuracy 0.0
training loss 0.22296249866485596
validation accuracy 0.674
validation loss 0.22347119450569153

 epoch 8
training accuracy 0.0
training loss 0.222427099943161
validation accuracy 0.674
validation loss 0.2230590283870697

 epoch 9
training accuracy 0.0
training loss 0.22189371287822723
validation accuracy 0.674
validation loss 0.22275172173976898

 epoch 10
training accuracy 0.0
training loss 0.22178654372692108
validation accuracy 0.674
validation loss 0.2225237935781479

 epoch 11
training accuracy 0.0
training loss 0.221751868724823
validation accuracy 0.674
validation loss 0.22235801815986633

 epoch 12
training accuracy 0.0
training loss 0.22149959206581116
validation accuracy 0.674
validation loss 0.22224263846874237

 epoch 13
training accuracy 0.0
training loss 0.22099249064922333
validation accuracy 0.674
validation loss 0.22214549779891968
</code></pre>
<p>As you may observe, nothing  is changing. I really can’t seem to spot the flaw. I do suspect I may not be using zero_grad correctly but I’m really not sure. I have tried several learning rates: 0.1, 0.01, 0.00, 0.0000001 and diffrerent batch sizes: 4, 5, 10, 25 but nothing changes. Any advice would be much appreciated. Please help.</p>

username: chetan06
cooked: <p>You can use lr_scheduler.ReduceOnPlateau</p>
<p><a href="https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate</a></p>
<p>You can read more about it in the above link</p>

username: KFrank
cooked: <p>Hi Taher!</p>
<p>There are several issues here.  See my in-line comments, below.</p>
<aside class="quote no-group" data-username="Taher_Anjary" data-post="3" data-topic="80405" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/taher_anjary/48/23973_2.png" class="avatar"> Taher_Anjary:</div>
<blockquote>
<p>The task is to  perform binary classification on Ocular Disease Recognition, ODIR5k, dataset.</p>
</blockquote>
</aside>
<p>For binary classification, you should be using <code>BCEWithLogitsLoss</code><br>
and get rid of the <code>sigmoid()</code> at the end of your model.  (You could<br>
also keep the <code>sigmoid()</code> and use <code>BCELoss</code>, but with greater numerical<br>
instability.)  <code>MSELoss</code> isn’t the right choice for binary classification.</p>
<aside class="quote no-group">
<blockquote>
<p>Images are 128x256 rgb images. I process them into grayscale</p>
</blockquote>
</aside>
<p>As an aside, your rgb images have information in the color.  So it<br>
would make sense not to convert them to grayscale.  But to make<br>
use of the color information, you would have to create a model<br>
that did make use of the color – typically by having it accept inputs<br>
with a channel dimension for the three color channels.</p>
<aside class="quote no-group">
<blockquote>
<pre><code class="lang-auto">        y = torch.sigmoid(self.linear3(z2))
</code></pre>
</blockquote>
</aside>
<p>As mentioned above, get rid of the <code>sigmoid()</code>.</p>
<aside class="quote no-group">
<blockquote>
<pre><code class="lang-auto">    criterion = torch.nn.MSELoss(reduction='mean')
</code></pre>
</blockquote>
</aside>
<p>As mentioned above, you should use <code>BCEWithLogitsLoss</code>.</p>
<aside class="quote no-group">
<blockquote>
<pre><code class="lang-auto">        labels = labels.float() #.cuda()
</code></pre>
</blockquote>
</aside>
<p>Your <code>labels</code> should, indeed, be floats for use with <code>BCEWithLogitsLoss</code>.<br>
By the way, what type were they before you called <code>float()</code>?</p>
<p>Also, if you continue to have problems, you should check what values<br>
they take on.  They most likely take on values of <code>0.0</code> (the “class label”<br>
for class-“0”) and <code>1.0</code> (for class-“1”).  But they could be values<br>
(understood to be probabilities) that range over <code>[0.0, 1.0]</code>.  In any<br>
event, whether discrete or continuous, they should be floats.</p>
<aside class="quote no-group">
<blockquote>
<pre><code class="lang-auto">        y_2 = torch.zeros(len(y_pred))
        y_2[y_pred&gt;=0.5] = 1
        y_2 = y_2.int()
        correct = 1*(y_2 == labels.int()).sum().item()
</code></pre>
</blockquote>
</aside>
<p>This is a reasonable way to convert <code>y_pred</code> (the output of your model)<br>
to yes-no (0-1) predictions, suitable for computing your accuracy.</p>
<p>Note, this is true if <code>y_pred</code> in made up of <em>probabilities</em> that range<br>
over <code>[0.0, 1.0]</code>, as will be the case if you have the final <code>sigmoid()</code><br>
in your model.  However, if you get rid of the <code>sigmoid()</code> and use<br>
<code>BCEWithLogitsLoss</code> (as you should), your outputs will be raw-score<br>
<em>logits</em> that range over <code>(-inf, inf)</code>, and you should use a threshold<br>
of <code>0.0</code>, rather than <code>0.5</code>, that is:</p>
<pre data-code-wrap="python"><code class="lang-python"># y_2[y_pred&gt;=0.5] = 1   # change this
y_2[y_pred&gt;=0.0] = 1     # to this
</code></pre>
<p>Also, your accuracy code it somewhat roundabout, and could be<br>
tightened up.  But that is left as an exercise for the reader …</p>
<aside class="quote no-group">
<blockquote>
<pre><code class="lang-auto">        correct = (y_pred == labels).sum().item()
</code></pre>
</blockquote>
</aside>
<p>This, however, is a problem.  After calculating <code>correct</code> in a sensible<br>
way, you immediately overwrite it with the exact-equality-test version<br>
that is incorrect.  So your (train) accuracy will be wrong in the same<br>
way as it was in your first version.</p>
<aside class="quote no-group">
<blockquote>
<pre><code class="lang-auto">        accuracy = correct / total
</code></pre>
</blockquote>
</aside>
<p>I see that you’ve gotten rid of the factor of <code>100</code> you had in your first<br>
version.  I suppose that you are now calculating your <code>accuracy</code> as a<br>
“fraction” where it had been a “percentage” before.  No matter, but it<br>
makes comparing this version with the first more confusing.</p>
<aside class="quote no-group">
<blockquote>
<pre><code class="lang-auto">            y_pred = y_pred.flatten()
            y2 = torch.zeros(len(y_pred))
            y2[y_pred&gt;=0.5] = 1
            y2 = y2.int()
            total = labels.size(0)
            correct = (y2 == labels.int()).sum().item()
            acc = correct / total
</code></pre>
</blockquote>
</aside>
<p>As an aside, in <code>test()</code> you are calculating <code>accuracy</code> correctly, and<br>
not making the exact-equality-test mistake you kept in <code>train()</code>.<br>
This explains the difference between your <code>training accuracy</code> and<br>
<code>validation accuracy</code> in your results below.</p>
<aside class="quote no-group">
<blockquote>
<p>In any case, running the code produces the following output:<br>
…</p>
<p>As you may observe, nothing  is changing.</p>
</blockquote>
</aside>
<p>Redo this with <code>BCEWithLogitsLoss</code> and the correct accuracy<br>
calculation.</p>
<aside class="quote no-group">
<blockquote>
<p>I do suspect I may not be using zero_grad correctly but I’m really not sure.</p>
</blockquote>
</aside>
<p>Your use of <code>optimizer.zero_grad()</code> is correct.</p>
<aside class="quote no-group">
<blockquote>
<p>I have tried several learning rates: 0.1, 0.01, 0.00, 0.0000001</p>
</blockquote>
</aside>
<p>After you get your code more-or-less working, it does make sense<br>
to try different learning rates.  Note, that a learning rate of <code>0.00</code><br>
doesn’t make sense – it multiplies the gradient by <code>0.00</code>, in effect,<br>
doing nothing.  Also, <code>0.0000001</code> seems  awfully small.</p>
<aside class="quote no-group">
<blockquote>
<p>and diffrerent batch sizes: 4, 5, 10, 25</p>
</blockquote>
</aside>
<p>And experimenting with different batch sizes makes sense, as well.</p>
<p>Good luck.</p>
<p>K. Frank</p>

username: NomiMalik0207
cooked: <p>In my case, I was facing the same error. On my laptop without GPU the training was fine. When I tried on GPU the model didn’t change the accuracy and loss after the first epochs. I was using nn.CrossEntropyLoss() with Adam.<br>
Changing Adam with SGD worked for me.</p>