Accuracy, training, test loss doesn't change throughout all the epochs

username: Cassin_Thangam_Edwin
cooked: <p>I am working on a text classification problem with a binary output 0 or 1. The accuracy, train loss and test loss remains the same. The accuracy is exact the same for all the epochs. All the steps looks very correct. I had tried out several ways to figure out what is going wrong. Nothing actually worked. Please help.</p>
<pre><code class="lang-auto">class RNN(nn.Module):
    def __init__(self, num_layers, num_classes, input_size, hidden_size,vocab,dropout):
        super(RNN,self).__init__()
        self.num_layers = num_layers
        self.num_classes = num_classes
        self.input_size = input_size
        self.hidden_size = hidden_size
        self.sequence_len = sequence_len
        self.embedding = nn.Embedding(len(vocab),input_size)
        nn.init.xavier_normal_(self.embedding.weight)
        self.rnn = nn.RNN(input_size, hidden_size,num_layers,dropout = dropout, nonlinearity = 'tanh', batch_first=True, bias = True, bidirectional = False)
        self.linear = nn.Linear(hidden_size, 1)
        nn.init.xavier_normal_(self.linear.weight)
        self.sigmoid = nn.Sigmoid()
        
    def forward(self,x):
        lens = list(map(len, x))
        padded = pad_sequence(x, batch_first=True)
        output_embedding=self.embedding(padded)   

        packed = pack_padded_sequence(input = output_embedding,lengths = lens, batch_first=True, enforce_sorted=False)
        input_rnn = packed
        h011 = torch.zeros(1,32,3)
        output_11, hidden_11 = self.rnn(input_rnn,h011)
        output_padded, output_lengths = pad_packed_sequence(output_11, batch_first=True)
        final_output_11 = self.linear(hidden_11)        
        prob_11 = self.sigmoid(final_output_11)
        return output_padded, hidden_11, prob_11

</code></pre>
<pre><code class="lang-auto"> num_layers = 1
   num_classes = 2
   input_size = 5
   hidden_size = 3
   criterion = nn.BCELoss()
   sequence_len = 1
   dropout = 0.5
   rnn = RNN(num_layers, num_classes, input_size, hidden_size,vocab,dropout)
   epochs = 10
   lr = 0.01
   weight_decay=0.011
</code></pre>
<pre><code class="lang-auto">   def train_loop(model,criterion,optimizer,train_loader,valid_loader,epochs):
    train_losses= []
    valid_losses= []

    for epoch in range(epochs):
        train_loss=0
        for label,text in train_loader:
            
            output,hidden,prob = rnn.forward(text)
            prob = torch.tensor([item.item() for sublist in prob for item in sublist],dtype = torch.float32)
            label = torch.tensor(label, dtype = torch.float32)
            loss=criterion(prob,label)
            optimizer.zero_grad()
            loss.requires_grad = True
            
            loss.backward()
            optimizer.step()
            train_loss += loss.item()
        train_loss=train_loss/len(train_loader)
        
        
        valid_loss=0
        
        with torch.no_grad():
            correct=0
            total=0
            for label,text in valid_loader:
                output,hidden,prob = rnn.forward(text)
                prob = torch.tensor([item.item() for sublist in prob for item in sublist],dtype = torch.float32)
                label = torch.tensor(label, dtype = torch.float32)
                loss=criterion(prob,label)
                valid_loss += loss.item()
                p = torch.tensor([1 if i &gt; 0.5 else 0 for i in prob.data], dtype = torch.float32)
                #predicted = torch.max(p, 1)
                
                total += label.size(0)
                
                correct += (p == label).sum().item()
            valid_loss=valid_loss/len(valid_loader)
            accuracy = 100 * correct / total
            print(accuracy)
        scheduler.step(accuracy)
        train_losses.append(train_loss)
        valid_losses.append(valid_loss)
        
        print(f'Epoch {epoch+1:&lt;2d}/{epochs} --&gt; Train Loss: {train_loss:.4f} |  Valid Loss: {valid_loss:.4f}')

</code></pre>
<pre><code class="lang-auto">   from torch.optim.lr_scheduler import StepLR
   from torch.optim.lr_scheduler import ReduceLROnPlateau
   optimizer = torch.optim.Adam(rnn.parameters(), lr=0.01)
   scheduler = ReduceLROnPlateau(optimizer, mode='max', factor=0.4, patience=5, 
   verbose=True) # need to change scheduler.step

   train_loop(rnn,criterion,optimizer,train_loader,valid_loader,epochs)

</code></pre>

username: ptrblck
cooked: <p>Re-wrapping a tensor in another tensor as well as calling <code>item()</code> on it will detach it from the computation graph:</p>
<pre><code class="lang-python">prob = torch.tensor([item.item() for sublist in prob for item in sublist],dtype = torch.float32)
</code></pre>
<p>and the subsequent <code>backward()</code> call would raise an issue.<br>
However, since you’ve also assigned <code>loss.requires_grad = True</code>, the error is masked.</p>
<p>I’m unsure what shape <code>prob</code> has before and after, but you would need to use e.g. <code>tensor.view</code> in case you need to change its shape.</p>

username: Cassin_Thangam_Edwin
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> thank you very much for pointing it out.</p>
<pre><code class="lang-auto">prob = torch.tensor([item.item() for sublist in prob for item in sublist],dtype = torch.float32)
</code></pre>
<p>This above code didn’t let loss function calculate the gradients. Finally I used torch.flatten() to reshape the tensor.</p>
<p>But, now I have another question. Still my weights looks unupdated after calculating gradients and even after every epoch the weights remain the very same. I used this code to check it.</p>
<pre><code class="lang-auto">list(rnn.parameters())[0].clone()
</code></pre>
<p>Where I am I going wrong? Please check the screenshot.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/a/9adc76cdc14c236e124edcceb416e074d76158f7.png" data-download-href="https://discuss.pytorch.org/uploads/default/9adc76cdc14c236e124edcceb416e074d76158f7" title="Screen Shot 2021-07-22 at 2.00.38 AM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/a/9adc76cdc14c236e124edcceb416e074d76158f7_2_690x483.png" alt="Screen Shot 2021-07-22 at 2.00.38 AM" data-base62-sha1="m5XS66uDH0q5cDNYgUWx6jPqm6H" width="690" height="483" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/a/9adc76cdc14c236e124edcceb416e074d76158f7_2_690x483.png, https://discuss.pytorch.org/uploads/default/optimized/3X/9/a/9adc76cdc14c236e124edcceb416e074d76158f7_2_1035x724.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/9/a/9adc76cdc14c236e124edcceb416e074d76158f7.png 2x" data-dominant-color="EBEBEB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2021-07-22 at 2.00.38 AM</span><span class="informations">1376×964 95.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: ptrblck
cooked: <p>To make sure the computation graph is not detached anymore you could check the <code>.grad</code> attribute of model parameters before and after the first <code>loss.backward()</code> call.<br>
Before it, <code>.grad</code> should return <code>None</code> (as no gradients were computed yet), and afterwards it should show a valid tensor.</p>