How to get memory_mask for nn.TransformerDecoder

username: xdwang0726
cooked: <p>The generate_square_subsequent_mask function in nn.Transformer can only generate square masks, but memory_mask requires the dimension (T, S). I am wondering is there a built in function in transformer?? Thank you!</p>

username: zhangguanheng66
cooked: <p>If you take a look at the source code of <code>generate_square_subsequent_mask</code> function, you will see how it work. It’s pretty simple.</p>

username: xdwang0726
cooked: <p>Hi, Thank you for your reply! I am wondering the <code>generate_square_masks</code> can only generate square mask, but according to the nn.Transformer documentation, <code>memory_mask</code> requres shape as (T, S). So I am wondering do I need a customized function to generate it? Thank you!!</p>

username: suhara
cooked: <p>I don’t think so. You don’t need to use <code>memory_mask</code> unless you want to prevent the decoder from attending some tokens in the input sequence, and the original Transformer didn’t use it in the first place because the decoder should be aware of the entire input sequence for any token in the output sequence. The same thing can be said to the input sequence (i.e., <code>src_mask</code>.)</p>
<p>In the PyTorch language, the original Transformer settings are <code>src_mask=None</code> and <code>memory_mask=None</code>, and for <code>tgt_mask=generate_square_subsequent_mask(T)</code>.</p>
<p>Again, <code>memory_mask</code> is used only when you don’t want to let the decoder attend certain tokens in the input sequence. That is why the input shape is <code>(S, T)</code> (where <code>S</code> is input sequence lenegth and <code>T</code> is output sequence length.)</p>
<p>If you still want to create a mask, let’s say, so that the decoder does not attend the future positions in the encoder, I’d consider using <code>torch.concat()</code> to create such a mask. For example,</p>
<pre><code class="lang-auto"># if S &gt; T

&gt;&gt;&gt; torch.cat([model.generate_square_subsequent_mask(T),
               torch.zeros(T - S, T)], 0)
			   
tensor([[0., -inf, -inf, -inf, -inf],
        [0., 0., -inf, -inf, -inf],
        [0., 0., 0., -inf, -inf],
        [0., 0., 0., 0., -inf],
        [0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0.],				
        [0., 0., 0., 0., 0.]])

# if T &gt; S
&gt;&gt;&gt; torch.cat([model.generate_square_subsequent_mask(S),
               torch.zeros(S, T - S)], 1)
			   
tensor([[0., -inf, -inf, -inf, -inf, 0., 0.],
        [0., 0., -inf, -inf, -inf, 0., 0.],
        [0., 0., 0., -inf, -inf, 0., 0.],
        [0., 0., 0., 0., -inf, 0., 0.],
        [0., 0., 0., 0., 0., 0., 0.]])		
</code></pre>
<p>Hope it helps.</p>

username: xdwang0726
cooked: <p>Thank you for your reply! For my understanding, there are two masks in the MultiHeadAttention, one is <code>attn_mask</code> and the other is <code>key_pad_mask</code>. In the decoder, the first Multi-head self-attention takes <code>tgt_mask</code> as <code>attn_mask</code> which is used to prevent decoder see its subsequent tokens, and <code>key_pad_mak</code> is the padding mask for target sequence. The second multi-head attention is contextual attention, which leverage the output from encoder and decoder self-attention. I don’t quite understand  <code>attn_mask</code> and <code>key_pad_mask</code> in the contextual attention, and wondering whether could you explain those two masks? Thank you!</p>

username: suhara
cooked: <blockquote>
<p>The second multi-head attention is contextual attention, which leverage the output from encoder and decoder self-attention. I don’t quite understand attn_mask and key_pad_mask in the contextual attention, and wondering whether could you explain those two masks? Thank you!</p>
</blockquote>
<p>If “the contextual attention” means the encoder-decoder attention of the Transformer, they (<code>attn_mask</code> and <code>key_pad_mask</code>) are <code>memory_mask</code> and <code>memory_key_padding_mask</code>.</p>

username: xdwang0726
cooked: <p>Thanks for your reply! Yes, I understand in the encoder-decoder attention, <code>attn_mask</code> is <code>memory_mask</code> and <code>key_pad_mask</code> is <code>memory_key_padding_mask</code>, but I didnt quite get how these two mask looks like and how they work. Also, I am confused are theses two masks mandatory  when we use <code>nn.TransformerDecoder</code>? (In the pytorch documentation, it shows optional). I am also wondering when inference process, are these masks needed?</p>

username: suhara
cooked: <p>They are optional as their default values are <code>None</code>.</p>
<p>It might be helpful for you to understand that there are three types of attention in the Transformer:</p>
<ul>
<li>
<ol>
<li>encoder self-attention (no mask needed)</li>
</ol>
</li>
<li>
<ol start="2">
<li>encoder-decoder attention (no mask needed)</li>
</ol>
</li>
<li>
<ol start="3">
<li>decoder self-attention (mask needed)</li>
</ol>
</li>
</ul>
<p>In some sense, attention is a way to calculate a vector representation of a token using its context. (2) encoder-decoder attention calculates a vector representation of each token in the (generated) output sequence based on all tokens in the input sequence. Thus, it’s called encoder-decoder attention.</p>
<p>Unless you don’t want to use some certain token(s) in the input sequence for some output token(s), there is no reason to mask the information from the input sequence. Therefore, the default mask is <code>None</code>.</p>
<p>The following article visually explains how encoder/decoder self-attention and encoder-decoder attention work in the Transformer.<br>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <a href="http://jalammar.github.io/illustrated-transformer/" target="_blank" rel="nofollow noopener">jalammar.github.io</a>
  </header>
  <article class="onebox-body">
    <img src="" class="thumbnail">

<h3><a href="http://jalammar.github.io/illustrated-transformer/" target="_blank" rel="nofollow noopener">The Illustrated Transformer</a></h3>

<p>Discussions:
Hacker News (65 points, 4 comments), Reddit r/MachineLearning (29 points, 3 comments)


Translations: Chinese (Simplified), Korean

Watch: MIT’s Deep Learning State of the Art lecture referencing this post

In the previous post, we...</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>

username: xdwang0726
cooked: <p>Thank you! Your explaination is very clear and is very helpful! I still have a contains that I followed your suggestion and trained a summarization model using <code>nn.TransformerDecoder</code> (I used BERT as an encoder), the training seems go well (training can generate meaningful sentences), but the inference process only generate the same token (repeatly). I am wondering does the training and inference different in settings? (The only difference I did for training and inference is that in the inference, I send one token at the very begining and add the generated token to the decoder for the next prediction.)</p>

username: agnes
cooked: <p>Did you find a solution ?</p>

username: zhangguanheng66
cooked: <p>If you see a repeating token from transformer during inference, you may need a mask during training. Your model are not effectively learning anything.</p>

username: mathematicsofpaul
cooked: <p>What could be possible reasons why my model is delivering repeated tokens during training and effectively at inference?</p>

username: shamoons
cooked: <p>Same problem - would love any input or ideas?</p>