Model performance not chaging even if grad values changes

alit: <p>Hello,</p>
<p>I defined a sentiment analysis model using lstm i usually do it with torchtext this this time i wanted to try it without. I used glove as embeddings and a 2 layers LSTM connected with a fc layer.</p>
<p>My problem is even if my gradients are changing the loss doesn’t change and I don’t understand why?<br>
Model below:</p>
<aside class="quote no-group">
<blockquote>
<p>class ReccurentModel(nn.Module):<br>
def <strong>init</strong>(self,glove_matrix, n_layers, hidden_dim, bidirectional, dropout,output_dim):<br>
super().<strong>init</strong>()<br>
self.emb = nn.Embedding.from_pretrained(glove_matrix)<br>
self.lstm = nn.LSTM(300,hidden_dim, n_layers, batch_first=True)<br>
self.fc1 = nn.Linear(hidden_dim * 2 if bidirectional else hidden_dim, output_dim)<br>
self.dropout = nn.Dropout(dropout)<br>
def forward(self, x):<br>
x_emb = self.emb(x)<br>
output, (hidden, cell) = self.lstm(x_emb)<br>
hidden = self.dropout(hidden[-1,:,:])<br>
x_res = self.fc1(hidden)<br>
return x_res</p>
</blockquote>
</aside>
<p>glove_mat is a matrix with size vocabulary*300(size of an embedding)</p>
<aside class="quote no-group">
<blockquote>
<p>loss = nn.BCEWithLogitsLoss()<br>
optimizer = optim.SGD(model.parameters(), lr=1e-4)</p>
</blockquote>
</aside>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/2/a22728a7e675f949c9c03892e3ab15bb5c4c144e.png" data-download-href="https://discuss.pytorch.org/uploads/default/a22728a7e675f949c9c03892e3ab15bb5c4c144e" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/a/2/a22728a7e675f949c9c03892e3ab15bb5c4c144e.png" alt="image" data-base62-sha1="n8tflHlANJaOlqFkBXK8xmKQEAS" width="690" height="494" data-dominant-color="3D3D3D"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">885×634 13 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>If you have any idea to change this ?</p>

vdw: <p>How does you training loop for an epoch/batch look like, i.e. where you have to call something like <code>model.zero_grad()</code> and <code>optimizer.step()</code>.</p>
<p>Also, in the definition of your LSTM:</p>
<pre><code>self.lstm = nn.LSTM(300,hidden_dim, n_layers, batch_first=True)
</code></pre>
<p>you probably want to add user <code>bidirectional</code> parameter, Otherwise you LSTM is always unidirectional and might not match <code>self.fc1</code> if <code>bidirectional=True</code>.</p>

alit: <p>Hey Chris,</p>
<p>Thanks for your help. I managed to make the network work by changing the optimizer i was using a SGD but the loss was not changing when i switched to an Adam optimizer after 10 epochs the loss start decreasing.</p>
<blockquote>
<p>final_optimizer = optim.Adam(model.parameters(),<br>
lr =LEARNING_RATE, # args.learning_rate - default is 5e-5, our notebook had 2e-5<br>
eps = 1e-8 # args.adam_epsilon  - default is 1e-8.<br>
)<br>
initial_optimizer = optim.SGD(model.parameters(), lr=1e-3, weight_decay=1e-6)</p>
</blockquote>
<p>I still don’t understand the problem with SGD as SGD’s fluctuation, on the one hand, enables it to jump to new and potentially better local minima.</p>