Loss of training remains unchanged

pourya_farzi: <p>Hi,<br>
I have designed a regression model made up of CNN and LSTM. In the forward pass, first I put CNN and use its output for LSTM. Then I compute the score.<br>
The problem is that, after the third epoch, LOSS remains constant. I tested different loss functions(L1Loss.MSEloss, … ), and even optimizers such as AdamW, adam, …, but there is no difference.<br>
Moreover, I checked both output and the Last Hidden Layer of my  LSTM to compute the score. Again, I couldn’t see any significant changes.<br>
Herein lies my code.</p>
<pre><code class="lang-auto">criterion = torch.nn.MSELoss()
optimizer = torch.optim.Adam(model.parameters(), lr=0.01)
print("Training for %d epochs..." % N_EPOCHS)
for epoch in range(1, N_EPOCHS + 1):
     for i, (xA, xB, score) in enumerate(train_loader2, 1):
          predicted_score = model(xA, xB)
          optimizer.zero_grad()
          loss = criterion(predicted_score,  score)
          total_loss += loss.item()
          loss.backward()
          optimizer.step()
        
          if i % 20 == 0:
             print('[{}] Train Epoch: {} [{}/{} ({:.0f}%)]\tLoss: {:.2f}'.format(
                time_since(start), epoch,  i *
                len(xA), len(train_loader.dataset),
                100. * i * len(xA) / len(train_loader.dataset),
                total_loss / i * len(xA)))
</code></pre>
<p>Besides, my result :</p>
<pre><code class="lang-auto">[0m 3s] Train Epoch: 1 [1000/4500 (22%)]	Loss: 4.25
[0m 5s] Train Epoch: 1 [2000/4500 (44%)]	Loss: 2.97
[0m 6s] Train Epoch: 1 [3000/4500 (67%)]	Loss: 3.08
[0m 8s] Train Epoch: 1 [4000/4500 (89%)]	Loss: 2.67
[0m 11s] Train Epoch: 2 [1000/4500 (22%)]	Loss: 2.69
[0m 13s] Train Epoch: 2 [2000/4500 (44%)]	Loss: 2.19
[0m 15s] Train Epoch: 2 [3000/4500 (67%)]	Loss: 2.56
[0m 16s] Train Epoch: 2 [4000/4500 (89%)]	Loss: 2.28
.
.
[14m 0s] Train Epoch: 100 [1000/4500 (22%)]	Loss: 2.69
[14m 2s] Train Epoch: 100 [2000/4500 (44%)]	Loss: 2.19
[14m 4s] Train Epoch: 100 [3000/4500 (67%)]	Loss: 2.56
[14m 6s] Train Epoch: 100 [4000/4500 (89%)]	Loss: 2.28
</code></pre>
<p>I’d be grateful if someone could inform me why this might happen.</p>

vdw: <p>Can you post the code if your models with the <code>forward()</code> method, particularly the LSTM one?</p>

adamr: <p>Make sure  you’ve checked the gradients. If they are zero (or very close to it), that means you hit the local minimum. You may use this function:</p>
<pre><code class="lang-python">def pprint_step(model, lr: float = 0.01):
    all_parameters = dict((k, str(v.detach().numpy()) + ("&lt;no grad&gt;" if v.grad is None else (
        "↑" if all(v.grad &gt; 0) else ("↓" if all(v.grad &lt; 0) else "")) + ", Δ: " + str(v.grad.numpy() * lr))) for k, v in
                          model.named_parameters())
    pp = pprint.PrettyPrinter()
    s = pp.pformat(all_parameters)
    print(s)
    return s
</code></pre>

pourya_farzi: <p>this is the forward function which returns 50 scores. I have BATCH_SIZE = 50</p>
<pre><code class="lang-auto"> def forward(self, xA, xB):
        ## input  (batch,seq_len)
        b_sizeA = xA.size(0)
        b_sizeB = xB.size(0)
        
        xA=xA.t()
        xB=xB.t()
        embeddedA = self.embedding(xA) 
        embeddedB = self.embedding(xB) # S*B x I (embedding size)
        # Again b*s*i
        embeddedA = embeddedA.permute(1,0,2) 
        embeddedB = embeddedB.permute(1,0,2) 
    
        # 3D -&gt; 4D
        embeddedA = embeddedA.unsqueeze(1)
        embeddedB = embeddedB.unsqueeze(1)
        
        xA = self.activation(self.convA1(embeddedA))   
        xB = self.activation(self.convB1(embeddedB))

        xA=embeddedA * xA.expand_as(embeddedA)
        xB=embeddedB * xB.expand_as(embeddedB)

        xA=torch.cat((embeddedA,xA),dim=3)
        xB=torch.cat((embeddedB,xB),dim=3)
        
        xA = xA.squeeze(1) # batch, seq-length, embed size 
        xB = xB.squeeze(1) 

        hA = Variable(torch.ones(self.num_layers * self.number_dir, 
                            b_sizeA, self.hidden_sizeA))
        cA = Variable(torch.ones(self.num_layers * self.number_dir, 
                             b_sizeA, self.hidden_sizeA))
        
        hB = Variable(torch.ones(self.num_layers * self.number_dir, 
                            b_sizeB, self.hidden_sizeB))
        cB = Variable(torch.ones(self.num_layers * self.number_dir, 
                            b_sizeB, self.hidden_sizeB))

        outputA, (hn_A, cn_A) = self.rnnA(xA, (hA, cA)) 
        outputB, (hn_B, cn_B) = self.rnnB(xB, (hB, cB)) 

        cos_sim_list=self.cosinesim(hn_A,hn_B)

        return cos_sim_list.float()
</code></pre>

vdw: <p>Given the code that I see I can only try to give some pointers that might help. Given that you posted in NLP and use embeddings, I assume that you work with text.</p>
<ul>
<li>
<p>You’re unsqueezing your embeddings from 3d to 4d, so I assume that you use a <code>nn.Conv2d</code>. What are the kernel sizes you choose. Note that you could also use <code>nn.Conv1d</code> which would make the unsqueezing obsolete</p>
</li>
<li>
<p>To be honest, I have no idea what the <code>expand()</code> methods <em>really</em> does or why you need it. But any method that is needed to “fix” the shape of tensors needs to be applied with great care.</p>
</li>
<li>
<p>Your setup looks a bit like a Siamese Network. However, you use 2 different CNN layers and 2 different RNN layers – only 1 embedding layer though. In a Siamese setting, you would push both inputs through the same layer.</p>
</li>
<li>
<p>I trust that you define your LSTMs with <code>batch_first=True</code> <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
</li>
<li>
<p>Why do you transpose <code>xA</code> and <code>xB</code> and permute the first 2 dimensions after the embedding layer. I’m pretty sure you can simply remove both <code>.t()</code> and both <code>permute()</code> operations.</p>
</li>
</ul>

pourya_farzi: <p>hi again Chris,<br>
First of all, thanks a bunch for your tips.</p>
<ul>
<li>As for your first hint, I used nn.Conv2d since I decided to add more channels for the next test.<br>
*OMG, it’s a typo. EXPAND method was just a trial, I shouldn’t have sent it.<br>
*Yeah, you are right. and also batch_first=True.</li>
<li>In respect of permuting and transposing, you’re definitely right. I wanted to exactly prepare the same shape as the torch example for the embedding layer. (I had thought that  it might be the problem)<br>
However, having said that, I don’t know what is wrong. One more thing, I have a varied size of sentences so I used pad sequences to fix the size of each input (It was the same thing as previous papers). I couldn’t use pack_padded due to network producing higher Loss. perhaps it’s because of the differences in the length of two sentences as one input each time.  <img src="https://discuss.pytorch.org/images/emoji/apple/slightly_frowning_face.png?v=9" title=":slightly_frowning_face:" class="emoji" alt=":slightly_frowning_face:">
</li>
</ul>

pourya_farzi: <p>Hi again Chris,<br>
First of all, thanks a bunch for your tips.</p>
<ul>
<li>As for your first hint, I used nn.Conv2d since I decided to add more channels for the next test.</li>
<li>OMG, it’s a typo. EXPAND method was just a trial, I shouldn’t have sent it.</li>
<li>Yeah, you are right. and also batch_first=True.</li>
<li>In respect of permuting and transposing, you’re definitely right. I wanted to exactly prepare the same shape as the torch example for the embedding layer <img src="https://discuss.pytorch.org/images/emoji/apple/smile.png?v=9" title=":smile:" class="emoji" alt=":smile:">. (I had thought that  it might be the problem)<br>
However, having said that, I don’t know what is wrong. One more thing, I have a varied size of sentences so I used pad sequences to fix the size of each input (It was the same thing as previous papers). I couldn’t use pack_padded due to network producing higher Loss. perhaps it’s because of the differences in the length of two sentences as one input each time.  <img src="https://discuss.pytorch.org/images/emoji/apple/slightly_frowning_face.png?v=9" title=":slightly_frowning_face:" class="emoji" alt=":slightly_frowning_face:">
</li>
</ul>

pourya_farzi: <p>Hi, Thanks a million for your help.<br>
I checked it and a part of my result is like this.</p>
<pre><code class="lang-auto">    'rnnB.weight_hh_l0':
        '[[ 0.04917971  0.06434657 -0.09789566 ... -0.08545043  0.06401704\n  '
        '-0.0960645 ]\n [ 0.04035911  0.03888848  0.00772235 ...  0.01090766  '
        '0.07309137\n   0.0360844 ]\n [-0.06821513  0.04650425  0.00499327 '
        '...  0.03282529  0.07157829\n  -0.0139528 ]\n ...\n [-0.0370195  '
        '-0.09515273  0.00834717 ... -0.07280935 -0.07355522\n   0.03910803]\n '
        '[-0.05833184  0.0991596   0.02914995 ... -0.06485003 -0.08581042\n   '
        '0.08041064]\n [ 0.05045321 -0.01596119  0.08300199 ...  0.05857178  '
        '0.06642925\n   0.059383  ]]&lt;no grad&gt;',
</code></pre>
<p>And after a while, this appeared.<br>
RuntimeError: Boolean value of Tensor with more than one value is ambiguous.<br>
I don’t know why that happens and  which boolean it does mean.</p>

vdw: <aside class="quote no-group" data-username="pourya_farzi" data-post="7" data-topic="92980">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/pourya_farzi/48/27808_2.png" class="avatar"> pourya_farzi:</div>
<blockquote>
<p>I used nn.Conv2d since I decided to add more channels for the next test</p>
</blockquote>
</aside>
<p>The choice of the type of convolution layer does not depend on the number of channels. Can you post the definitions of <code>sefl.convA1</code> and <code>self.convB1</code>?</p>
<p>Sorry, I don’t know what this RuntimeError means.</p>

pourya_farzi: <p>Great!<br>
Here it is :</p>
<pre><code class="lang-auto">self.convA1 = nn.Conv2d(1, 1, kernel_size=(5,100), padding=(2,0))

self.convB1 = nn.Conv2d(1, 1, kernel_size=(5,100), padding=(2,0))
</code></pre>
<p>padding is (2,0) for maintaining the height. Besides, each time I considered five rows(words). I’ve changed the shape of it for example (2,100), (28,1),… and even symmetric forms (5,5), … . I wanna check another dataset, simpler than what I have now. Any suggestion is welcomed. <img src="https://discuss.pytorch.org/images/emoji/apple/slightly_smiling_face.png?v=9" title=":slightly_smiling_face:" class="emoji" alt=":slightly_smiling_face:"></p>

vdw: <p>OK, that was I was looking for. If you use <code>nn.Conv2d</code> over sentences, e.g., sequences of word vectors, then there are constraints on the kernel size: if your embedding vectors have as size of <code>embed_size</code>, then <code>kernel_size=(x,embed_size)</code> where <code>x</code> is the only value you can adjust. The issue is that it makes no semantic sense to convolve over the embedding dimension; see this <a href="https://discuss.pytorch.org/t/problem-in-making-embedding-layer-for-a-cnn-document-classification/67313/8">older post of mine</a>.</p>
<p>So if your 100 is the size of your embedding vectors, then <code>kernel_size=(5,100)</code> is valid while <code>kernel_size=(5,5)</code> is not.</p>
<p>By the way, what happens when you give the embeddings directly to the RNN(s), thus skipping the CNN part. I can’t intuitively tell what the CNN layer(s) are suppose to do.</p>