Teacher forcing with model.eval()

Dean_Sumner: <p>I have a seq2seq LSTM model with 100% teacher forcing. The loss decreases nicely as time progresses. However, when I place my model into model.eval() mode  &amp; without teacher forcing, the eval loss diverges (increases) away from the training loss (decreases) as time progresses.</p>
<p>I have tried using learning_rate schedulers during training e.g. exponential probability, or 0.5 probability of teacher forcing at a given time step, this resulted in no change to the eval-train trend.</p>
<p>I then tried the following:</p>
<ul>
<li>train data == eval data,      results: same trend</li>
<li>remove batch shuffling,      results : same trend</li>
<li>vary batch size,                  results: same trend</li>
<li>vary data set size,              results: same trend</li>
</ul>
<p>Note:<br>
I do not have batch normalization in my model.<br>
Putting teacher forcing in my eval function gives same eval loss as training loss.</p>
<p>Is there anything that could explain such results aside from coding errors (I have unit tests at each stage).</p>
<p>Thanks.</p>

ptrblck: <p>Based on your description it seems that teacher forcing gives your model a strong signal, so that the “underlying” distribution wasn’t learned from previously predicted samples.<br>
How does the training loss look when you don’t use teacher forcing or with a low probability?</p>

Dean_Sumner: <p>Terrible, converges to very high loss (start: 4, converges:2). I have also tried to force the model to over-fit by increasing the layers and hidden units. Yet without teacher forcing the model doesn’t improve. I’m at a bit of a loss, the only other thing I can consider is:</p>
<p><strong>Change internals:</strong><br>
Deviate from the tutorial, instead of feeding only the final GRU/LSTM h_state to the decoder. I could try feeding in each encoder layer h_state to the corresponding layers in the decoder.</p>
<p><strong>Change model paradigm:</strong><br>
Alternatively, I would like to experiment  with feeding the entire input-target sequences into the model and then retrospectively make another char-level decoder and transfer the weights of this decoder from the full model.</p>

Dean_Sumner: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/b/7bb84d39985b95797edc6d504da8d8390c6c85fe.png" data-download-href="https://discuss.pytorch.org/uploads/default/7bb84d39985b95797edc6d504da8d8390c6c85fe" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/7/b/7bb84d39985b95797edc6d504da8d8390c6c85fe_2_689x375.png" alt="image" data-base62-sha1="hEtAMQ7JyuRH3ThRGu5uFuqw8jA" width="689" height="375" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/7/b/7bb84d39985b95797edc6d504da8d8390c6c85fe_2_689x375.png, https://discuss.pytorch.org/uploads/default/optimized/3X/7/b/7bb84d39985b95797edc6d504da8d8390c6c85fe_2_1033x562.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/7/b/7bb84d39985b95797edc6d504da8d8390c6c85fe.png 2x" data-dominant-color="F9F9FA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1148×625 78.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

Dean_Sumner: <p>I also noticed  that when following the tutorial, and using a batch_size &gt; 1 the loss converges very early, this potentially explains the results I am getting from my first question.</p>
<p>Is there a bug in the tutorial code?</p>
<p><strong>Loss observed when varying batch_size in range [1, 2]</strong><br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/5/a/5af185ac96933708ff10c83db886a6de15deac06.png" alt="image" data-base62-sha1="cYwqJv5VD8AMSNP1TPchL4ZYRmK" width="449" height="308"></p>
<p>Here is a graph showing what happens to the loss when I move from a batch_size of 1 (with increasing max_sequence _lengths (purple:30, blue:50, orange:80, brown:100). Then increasing to a batch_size of 2, with increasing max_sequence _lengths(red: 30, green:50).</p>