Loss and accuracy stuck, very low gradient

username: barthelemymp
cooked: <p>Hello,</p>
<p>I’m was making a neural network to to try a few option of pytorch, and then when I tried it on the classical Breast Cancer dataset my algo was just stuck. I simplified to the maximum my model, but accurracy seem to be blocked.</p>
<p>my model:</p>
<pre><code class="lang-auto">class Modela(torch.nn.Module):
    def __init__(self):
        super(Modela, self).__init__()
        self.l1 = torch.nn.Linear(30,16)
        self.l2 = torch.nn.Linear(16,4)
        self.l3 = torch.nn.Linear(4,2)
        
        self.sigmoid = torch.nn.Sigmoid()
        self.relu = torch.nn.ReLU()
        
    def forward(self, x):
        out1 = self.relu(self.l1(x)) 
        out2 = self.relu(self.l2(out1))
        y_pred = self.sigmoid(self.l3(out2))
        return y_pred
</code></pre>
<p>my training:</p>
<pre><code class="lang-auto">train_size = int(0.8 * len(full_dataset))
test_size = len(full_dataset) - train_size
train_dataset, test_dataset = torch.utils.data.random_split(full_dataset, [train_size, test_size])



model = Modela()
print(model)
#TRAIN
n_epoch = 50
criterion = torch.nn.CrossEntropyLoss()#torch.nn.BCELoss(reduction='sum')

        

        
def train(model,train_dataset,test_dataset,loss_fn,n_epoch=50):
    optimizer = torch.optim.Adam(model.parameters(), lr=0.0001)
    valid_losses = [] #list of the losses on valid
    valid_acc = [] #list of the accuracy on valid
    train_losses = [] #list of all the loss on training 
    train_loss_store = torch.zeros(1) #will store the current accuracy
    for e in range(n_epoch):
        print('n_epoch',e)
        for i in range(len(train_dataset)):

            #let's evaluate the performan,ce of my model
            if (i%100 == 0):

                score = torch.zeros(1)
                valid_loss_store = torch.zeros(1)
                
                with torch.no_grad():
                    for j in range(len(test_dataset)):
                        sample = test_dataset[j]
                        X, label = sample['X'], sample['label']
                        output = model(X)
                        output = torch.unsqueeze(output,0)
                        label = torch.unsqueeze(label,0)
                        loss = loss_fn(output, label)
                        valid_loss_store += loss
                        _, predicted = torch.max(output.data, 1)
                        score += predicted.eq(label).float()
                #let's print a few result
                score = score/len(test_dataset)
                valid_acc += [score.item()]
                valid_loss_store = valid_loss_store/len(test_dataset)
                valid_losses += [valid_loss_store.item()]
                train_loss_store = train_loss_store/100
                print('acc', score, 'valid loss :',valid_loss_store, 'train_loss_store', train_loss_store )
                train_losses += [train_loss_store.item()]
                train_loss_store = torch.zeros(1)
                    

   
            # here we train on the training dataset        
            sample = train_dataset[i]
            X, label = sample['X'], sample['label']
            optimizer.zero_grad()
            output = model(X)

            output = torch.unsqueeze(output,0)
            label = torch.unsqueeze(label,0)
            loss = loss_fn(output, label)
            train_loss_store += loss
            loss.backward()
#             for param in model.parameters():
#                 print(i,'gradient odg',param.grad.data.sum())
            optimizer.step()
    return train_losses, valid_losses, valid_acc
           
train_losses, valid_losses, valid_acc = train(model,train_dataset, test_dataset,criterion)
</code></pre>
<p>If I start this last part, sometimes I quickly grow to a 0.9 accurracy but most of the time I m just stuck to the exact first accuracy, with gradient close from zero. It looks like a local minimum maybe but in that case the fact of making a step for each data should enables it to escape this minimum. I don’t understand where it does come from.<br>
I’ve tried to change the learning rate, and it does not change much.</p>
<p>Best regards</p>
<p>b</p>

username: KFrank
cooked: <p>Hi Barthelemy!</p>
<aside class="quote no-group" data-username="barthelemymp" data-post="1" data-topic="50048" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/barthelemymp/48/14354_2.png" class="avatar"> barthelemymp:</div>
<blockquote>
<p>I’m was making a neural network to to try a few option of pytorch, and then when I tried it on the classical Breast Cancer dataset my algo was just stuck. I simplified to the maximum my model, but accurracy seem to be blocked.</p>
</blockquote>
</aside>
<p>Let me speculate that you are building a <em>binary classification</em><br>
model – i.e., some sample either is or is not breast cancer, based<br>
on some kind of input features taken from a breast-tissue sample.</p>
<aside class="quote no-group">
<blockquote>
<p>my model:</p>
<pre><code class="lang-auto">class Modela(torch.nn.Module):
    def __init__(self):
        super(Modela, self).__init__()
        self.l1 = torch.nn.Linear(30,16)
        self.l2 = torch.nn.Linear(16,4)
        self.l3 = torch.nn.Linear(4,2)
        
        self.sigmoid = torch.nn.Sigmoid()
        self.relu = torch.nn.ReLU()
</code></pre>
</blockquote>
</aside>
<p>(Your last layer, <code>self.l3</code>, has two outputs, suggesting that this<br>
may be binary.)</p>
<aside class="quote no-group">
<blockquote>
<pre><code class="lang-auto">    def forward(self, x):
        out1 = self.relu(self.l1(x)) 
        out2 = self.relu(self.l2(out1))
        y_pred = self.sigmoid(self.l3(out2))
        return y_pred
</code></pre>
</blockquote>
</aside>
<p>I think your main problem is that you are calling <code>self.sigmoid()</code><br>
on the output of your last layer.  Given that (from your code further<br>
down) you are using <code>nn.CrossEntropyLoss</code> as your loss function,<br>
you should be using the outputs of <code>l3</code> as your <code>y_pred</code> and pass<br>
them directly to <code>nn.CrossEntropyLoss</code>.  This is because (maybe<br>
a little bit inconsistently with its name) <code>CrossEntropyLoss</code> expects<br>
<em>logits</em> (numbers running from -infinity to  infinity) as its inputs, but<br>
your <code>sigmoid</code> maps the logits coming out of <code>l3</code> to (0, 1).</p>
<p>So try getting rid of the <code>sigmoid</code>, and see if that helps.</p>
<p>A couple more comments:</p>
<p>If this is a binary classification problem, you should consider<br>
having your last layer have a single output, and using<br>
<code>nn.BCEWithLogitsLoss</code> as your loss function.  This won’t<br>
change the actual math or results of your network, but will<br>
make it a little simpler and more efficient.</p>
<p>Also, it’s not clear that having three layers / two hidden layers<br>
makes your network better.  It could make it worse or harder<br>
to train.  You might try a single hidden layer with something<br>
like 16 hidden neurons (as you have in your first hidden layer).</p>
<p>So your code fragment (also getting rid of the <code>sigmoid</code> and<br>
changing to the binary version instead of the general multi-class<br>
version) might look something like:</p>
<pre data-code-wrap="python"><code class="lang-python">class Modela(torch.nn.Module):
    def __init__(self):
        super(Modela, self).__init__()
        self.l1 = torch.nn.Linear(30,16)
        self.l2 = torch.nn.Linear(16,1)

        self.relu = torch.nn.ReLU()
        
    def forward(self, x):
        hidden1 = self.relu(self.l1(x)) 
        y_pred = self.l2(hidden1)
        return y_pred

# and to match the single output of l2
criterion = torch.nn.BCEWithLogitsLoss()
</code></pre>
<p>Whether a single hidden layer will work better than two will be<br>
problem dependent, but it’s likely to, so it’s worth trying.  Also,<br>
the best number of hidden neurons will be problem dependent,<br>
so you should experiment some, but, given that you have 30<br>
inputs (and one output), something like 16 hidden neurons is<br>
a reasonable place to start.</p>
<p>Best.</p>
<p>K. Frank</p>