Hierarchical Model not training well

username: Rohan_Kumar
cooked: <p>I wrote this code for hierarchical RNN, which is not training at all. The model takes in 4 dimensional input for text based dialogue conversations,</p>
<p>[batch size, num of sentences, num of words, word embedding]</p>
<p>then i manipulate it in a way to resemble time distributed layer of keras</p>
<ul>
<li>[batch size* num of sentences, num of words, word embedding] → rnn<br>
 → [batch size,  num of sentences, hidden size (sentence embedding)]<br>
then try to generate the next sentence in the in the conversation.</li>
</ul>
<p>but what is happening is the loss is decreasing in the first epoch and then it stays constant the reason behind it is, when generating the words of the sentence I have the output dimension as the vocab size. but the the model is heavily predicting the 0th vocab as the predicted word everytime.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/c/7c4992795ec8b27cff6237bef4f835610c0a8609.png" data-download-href="https://discuss.pytorch.org/uploads/default/7c4992795ec8b27cff6237bef4f835610c0a8609" title="Screenshot from 2020-03-20 07-11-20"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/7/c/7c4992795ec8b27cff6237bef4f835610c0a8609_2_689x388.png" alt="Screenshot from 2020-03-20 07-11-20" data-base62-sha1="hJuPHMENCiw8P4YmeyioUrvLjaV" width="689" height="388" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/7/c/7c4992795ec8b27cff6237bef4f835610c0a8609_2_689x388.png, https://discuss.pytorch.org/uploads/default/optimized/3X/7/c/7c4992795ec8b27cff6237bef4f835610c0a8609_2_1033x582.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/7/c/7c4992795ec8b27cff6237bef4f835610c0a8609.png 2x" data-dominant-color="B9B9BA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2020-03-20 07-11-20</span><span class="informations">1344×756 35.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div> <div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/8/18b076dec36c771ddbbb2b7d074011f779ac15b4.png" data-download-href="https://discuss.pytorch.org/uploads/default/18b076dec36c771ddbbb2b7d074011f779ac15b4" title="Screenshot from 2020-03-20 07-11-06"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/8/18b076dec36c771ddbbb2b7d074011f779ac15b4_2_689x388.png" alt="Screenshot from 2020-03-20 07-11-06" data-base62-sha1="3wpx0v2d1pQjLqw4aEpdBzTUE04" width="689" height="388" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/8/18b076dec36c771ddbbb2b7d074011f779ac15b4_2_689x388.png, https://discuss.pytorch.org/uploads/default/optimized/3X/1/8/18b076dec36c771ddbbb2b7d074011f779ac15b4_2_1033x582.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/1/8/18b076dec36c771ddbbb2b7d074011f779ac15b4.png 2x" data-dominant-color="3C3C3C"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2020-03-20 07-11-06</span><span class="informations">1344×757 57.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>the figures are loss and val curve,</p>
<p>the outputs are - the output to the network and the target (after manipulating it in trg.view(-1, v.shape[-1]) for cross entropy loss )</p>
<p>can anyone help me with why is it not working?</p>