Feedback on my Transformer for LM

andcarnivorous: <p>Hi everyone!</p>
<p>I have a project where I have to build, train and analyze a simple transformer for language modeling.</p>
<p>I built my model following the tutorial in the pytorch examples:<br>
<a href="https://pytorch.org/tutorials/beginner/transformer_tutorial.html" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/tutorials/beginner/transformer_tutorial.html</a></p>
<p>Here is my jupyter notebook: <a href="https://github.com/andcarnivorous/TestingTransformers/blob/master/transformertest.ipynb" rel="nofollow noopener">https://github.com/andcarnivorous/TestingTransformers/blob/master/transformertest.ipynb</a></p>
<p>I have tried to implement the input and target masks and so I started training on the wikitext dataset different versions with more/less encoders, decoders and heads. What I have noticed is that even when I use only two heads and two encoders/decoders the model manages to get to quite low validation loss.</p>
<p>Any feedback on how I set this up would be really appreciated.</p>

zhangguanheng66: <p>You need an attention mask.</p>

andcarnivorous: <p>Thanks! That would be the memory_mask variable to pass to the transformer block, right? I guess this is why the model was reaching such low loss!</p>

zhangguanheng66: <p>actually <code>src_mask</code>. Without mask, the model will be able to see the tokens it’s going to predict.</p>

andcarnivorous: <p>Oh but I am already applying a mask in the forward pass:</p>
<pre><code class="lang-auto">x = self.transformer(x, tgt, tgt_mask=generate_square_subsequent_mask(x.shape[0]), src_mask=generate_square_subsequent_mask(x.shape[0]))
</code></pre>
<p>From my understanding that generate_subsequent_mask() function hides the next token.</p>

zhangguanheng66: <p>You can use your model to generate some texts and see if the generated tokens are same. If same, it means the src/tgt are not masked properly.</p>

andcarnivorous: <p>Yeah, I already tested and it does produce gibberish even when the loss is low, on training data, so I think the mask is working.</p>

zhangguanheng66: <p>If it returns some random tokens, it means the mask is working.</p>
<p>Could you just use TransformerEncoder for a test? Encode/decoder is not necessary for a WLM task, IMO. And we have a tutorial on our website to set up a TransformerEncoder for WLM <a href="https://pytorch.org/tutorials/beginner/transformer_tutorial.html" rel="nofollow noopener">link</a>.</p>

andcarnivorous: <p>That is the tutorial I am currently following! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"> Because of this project requirements, I have to use the decoder part as well, since I will also have to probably train it on translation tasks.<br>
What I would like to do now is plot heatmaps of the attention matrix activation like this:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://i.imgur.com/383J5O5.png" title="" rel="nofollow noopener"><img src="https://i.imgur.com/383J5O5.png" alt width="685" height="500"><div class="meta">
<svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use xlink:href="#far-image"></use></svg><span class="filename"></span><span class="informations">2680×1956</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use xlink:href="#discourse-expand"></use></svg>
</div></a></div></p>
<p>I am accessing the self_attention and multihead like so:</p>
<pre><code class="lang-auto">attn_activation = model.transformer.encoder.layers[1].self_attn.out_proj(emb)
</code></pre>
<p>But I am a little lost on how that heatmap is mapped… what is the value of every point ij in the map?</p>