Training a generative model without teacher forcing

Messiah: <p>I understand this is more a theoretical question, but would there be any reason to train an RNN or transformer architecture without using teacher forcing, that is without using the target to guide predictions for such generative models during training? Wouldn’t doing so bypass the train-test discrepency known as exposure bias or covariate/distribution shift, where at test time the model outputs are autoregressively fed back in as input as opposed to the targets seen during training? A couple of obvious downsides to this approach is the serial processing requirement which slows down training considerably, and that the inputs to the model are changing at each epoch which may affect its ability to converge.</p>
<p>Training a model without teacher forcing also creates an additional relationship between the model’s parameters and its ultimate prediction. Should we be backpropagating through these predictions made by the model? Assume in this case that the outputs of the sequence model are real-valued/continuous (MSE loss) and that the network is fully differentiable.</p>
<p>Some related dicussions:<br>
<a href="https://www.evernote.com/shard/s189/client/snv?isnewsnv=true&amp;noteGuid=c9ac2e3f-a150-4d0c-9a44-16657e5d42cd&amp;noteKey=5eb49d50695c903ca1b4a04934e63363&amp;sn=https%3A%2F%2Fwww.evernote.com%2Fshard%2Fs189%2Fsh%2Fc9ac2e3f-a150-4d0c-9a44-16657e5d42cd%2F5eb49d50695c903ca1b4a04934e63363&amp;title=Notes%2Bon%2BScheduled%2BSampling%2Bfor%2BSequence%2BPrediction%2Bwith%2BRecurrent%2BNeural%2BNetworks" rel="noopener nofollow ugc">Notes on Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks</a><br>
<a href="https://discuss.pytorch.org/t/correct-way-to-train-without-teacher-forcing/15508/2">Correct way to train without teacher forcing</a><br>
<a href="https://discuss.pytorch.org/t/train-transformer-without-teacher-forcing/132938">Train transformer without teacher forcing</a></p>

_Bustion: <p>I have no practical experience with RNN architectures, but I can speak for Transformer architectures. Simply speaking, <strong>it is possible to train a good model without teacher forcing method</strong>, and it may lead to some interesting results because the model <em>kinda</em> learns on its mistakes. <strong>However, it will come with costs</strong>. Specifically, convergence in the beginning of training will be awful that will lead to longer training time. And this is why <strong>you should use the scheduled teacher forcing method</strong>, where in the beginning of the training the model is heavily teacher forced but closer to the end not so much.</p>
<p><a href="https://discuss.pytorch.org/t/transformers-with-scheduled-sampling-implementation/114713">Transformers with scheduled sampling implementation</a> (PyTorch discussions)<br>
<a href="https://aclanthology.org/P19-2049/" rel="noopener nofollow ugc">The Paper</a> (Scheduled Sampling For Transformers)</p>
<p>Hopefully this helps.</p>

dreidizzle: <p>There is also a tutorial on attention in the PyTorch tutorials page and there you can see that teacher forcing is used 50% of the time (randomly). So yes, you can do it. The in that tutorial a GRU is used but you can use any LM (GPT etc). I would think per batch/data pair the model learns more via teacher forcing, but I might circle back on this after looking into the links. I would think that yes at the start it’s best to use it, and later you can phase it out.</p>