RNN outperforming Transformer for dialogue task

username: jafioti
cooked: <p>I have been experimenting with different model architectures for dialogue modeling, and I am currently working with the Transformer. I am seeing a big difference between an RNN Seq2Seq model’s performance vs. the Transformer’s performance in terms of both test_ppl and test_wer. Both models use roughly the same number of parameters (~70 million). Is this to be expected or strange? I am training on the Cornell Movie dialogue dataset just for experimentation, I will be moving to a larger dataset once I am satisfied my setup works properly.</p>
<p>Here are the graphs of each metric, the red is the RNN and the purple is the transformer:</p>
<p>Train PPL:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/c/6c9f713918b6d8af3c4a4ace49ca9b6644409637.png" data-download-href="https://discuss.pytorch.org/uploads/default/6c9f713918b6d8af3c4a4ace49ca9b6644409637" title="ppl"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/c/6c9f713918b6d8af3c4a4ace49ca9b6644409637_2_690x239.png" alt="ppl" data-base62-sha1="fuVayGf84VsljgQHktxA9QwPoMf" width="690" height="239" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/c/6c9f713918b6d8af3c4a4ace49ca9b6644409637_2_690x239.png, https://discuss.pytorch.org/uploads/default/optimized/3X/6/c/6c9f713918b6d8af3c4a4ace49ca9b6644409637_2_1035x358.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/6/c/6c9f713918b6d8af3c4a4ace49ca9b6644409637_2_1380x478.png 2x" data-dominant-color="1A1B1C"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">ppl</span><span class="informations">1739×603 67 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Test PPL:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/6/c6417cd42b001671160bedf0059047d54586e816.png" data-download-href="https://discuss.pytorch.org/uploads/default/c6417cd42b001671160bedf0059047d54586e816" title="test_ppl"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/6/c6417cd42b001671160bedf0059047d54586e816_2_690x243.png" alt="test_ppl" data-base62-sha1="shQQqUIwJjULBXBrhU1G6e6eCfI" width="690" height="243" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/6/c6417cd42b001671160bedf0059047d54586e816_2_690x243.png, https://discuss.pytorch.org/uploads/default/optimized/3X/c/6/c6417cd42b001671160bedf0059047d54586e816_2_1035x364.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/c/6/c6417cd42b001671160bedf0059047d54586e816_2_1380x486.png 2x" data-dominant-color="191B1C"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">test_ppl</span><span class="informations">1746×615 30.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Test WER:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/5/55edb902a460b9a981f46eac92df67ca003b0c46.png" data-download-href="https://discuss.pytorch.org/uploads/default/55edb902a460b9a981f46eac92df67ca003b0c46" title="test_wer"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/5/55edb902a460b9a981f46eac92df67ca003b0c46_2_690x240.png" alt="test_wer" data-base62-sha1="cg9U17iZvi4jnrbK0pgO3vXCDye" width="690" height="240" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/5/55edb902a460b9a981f46eac92df67ca003b0c46_2_690x240.png, https://discuss.pytorch.org/uploads/default/optimized/3X/5/5/55edb902a460b9a981f46eac92df67ca003b0c46_2_1035x360.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/5/5/55edb902a460b9a981f46eac92df67ca003b0c46_2_1380x480.png 2x" data-dominant-color="191B1C"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">test_wer</span><span class="informations">1743×607 33.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>