Loss value not decreasing and predictions are constant for regression model

Jay_Timbadia: <p>I am doing a transfer learning using USE to train quora question pairs for similarity using a Siamese network.<br>
When I am training I am getting almost constant loss, not decreasing nor increase, also when I predict after train, all predictions value are same.<br>
Can someone please help? I am stuck on this issue since last 2 days!</p>
<p>Here is the colab notebook link.</p><aside class="onebox allowlistedgeneric">
  <header class="source">
      <img src="https://colab.research.google.com/img/favicon.ico?vrz=colab-20201230-085609-RC00_349558424" class="site-icon" width="16" height="16">
      <a href="https://colab.research.google.com/drive/18EGMELMpbuDFuFS50wSDigse96vcpBVV?usp=sharing" target="_blank" rel="noopener nofollow ugc">colab.research.google.com</a>
  </header>
  <article class="onebox-body">
    <img src="https://colab.research.google.com/img/colab_favicon_256px.png" class="thumbnail onebox-avatar" width="117" height="117">

<h3><a href="https://colab.research.google.com/drive/18EGMELMpbuDFuFS50wSDigse96vcpBVV?usp=sharing" target="_blank" rel="noopener nofollow ugc">Google Colaboratory</a></h3>



  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


ptrblck: <p>Your notebook requires a permission to access it. Could you post an “open” link so that we could check it for errors?</p>

Jay_Timbadia: <p>Here.</p><aside class="onebox allowlistedgeneric">
  <header class="source">
      <img src="https://colab.research.google.com/img/favicon.ico?vrz=colab-20210120-085604-RC00_352789880" class="site-icon" width="16" height="16">
      <a href="https://colab.research.google.com/drive/18EGMELMpbuDFuFS50wSDigse96vcpBVV?usp=sharing" target="_blank" rel="noopener nofollow ugc">colab.research.google.com</a>
  </header>
  <article class="onebox-body">
    <img src="https://colab.research.google.com/img/colab_favicon_256px.png" class="thumbnail onebox-avatar" width="60" height="60">

<h3><a href="https://colab.research.google.com/drive/18EGMELMpbuDFuFS50wSDigse96vcpBVV?usp=sharing" target="_blank" rel="noopener nofollow ugc">Google Colaboratory</a></h3>



  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


Jay_Timbadia: <p>Let me know if you can access it?</p>

ptrblck: <p>Yes, I can access it, thanks.<br>
I don’t see any obvious errors and the model is generally able to learn random data:</p>
<pre><code class="lang-python">model = torch_siamese()
data1 = torch.randn(64, 512)
data2 = torch.randn(64, 512)
target = torch.randn(64, 1)
optimizer = torch.optim.Adam(model.parameters(), lr=1e-2)
nb_epochs = 100

for epoch in range(nb_epochs):
    optimizer.zero_grad()
    output = model(data1, data2)
    loss = loss_func(output, target)
    loss.backward()
    optimizer.step()
    print('epoch {}, loss {}'.format(epoch, loss.item()))
</code></pre>
<p>I would recommend to check the shapes of the <code>output</code> and <code>target</code> tensors and make sure they are equal. Sometimes unwanted broadcasting could be applied if e.g. the <code>target</code> is missing a dimension, which would result in this bad training performance.</p>