Understanding mask size in Transformer Example

mapa17: <p>Hi,</p>
<p>i am trying to understand the Transformer architecture, following one of the pytorch examples at (<a href="https://pytorch.org/tutorials/beginner/transformer_tutorial.html#load-and-batch-data" class="inline-onebox" rel="noopener nofollow ugc">Language Modeling with nn.Transformer and TorchText — PyTorch Tutorials 1.11.0+cu102 documentation</a>)</p>
<p>I have troubles thought to understand the dimension/shape of the mask that is used to limit the self-attention to sequence elements before the “current” token.</p>
<p>In the example, the mask size is [batch_size, batch_size]. I would have thought it would be something like [sequence_length, sequence_length]. So for each position in the sequence, there is a separate mask that indicates what other tokens the self-attention mechanism can “access”.</p>
<p>Running the code, I see that the additiative mask has the shape [BS, BS] with the content</p>
<pre><code class="lang-auto">tensor([[0., -inf, -inf, -inf],
        [0., 0., -inf, -inf],
        [0., 0., 0., -inf],
        [0., 0., 0., 0.]])
</code></pre>
<p>using a BS of 4.</p>
<p>Can someone maybe clarify this for me? Are different masks applied for the different batches? And even more general, if I have a BS of 4, and sequences of length 6 for example, during training is the model not supposed to learn the probability of the N+1 sequence token based on all previous N tokens? So for each input token position there exists its own mask vector?</p>
<p>thank you for the clarification,<br>
Manuel</p>

Mauricio_Maroto: <p>Hi. Let me share my insights. Please correct me if I’m wrong. Long but hopefully useful post coming.</p>
<p>Let’s start with PyTorch’s TransformerEncoder. According to the docs, it says <code>forward(src, mask=None, src_key_padding_mask=None)</code>. Also it says that the mask’s shape is (S,S), the source sequence length, and that it is <em>additive</em>.</p>
<p>Note: The (S,S) and additive parts are found in the Transformer class docs, not in the TransformerEncoder class docs. Similar thing happens with the Decoder docs.</p>
<p>So, in this case, the <code>src_mask</code> is a square matrix (S,S) that adds right after the <code>Q*Kt</code> part of the attention mechanism (before the Values (<code>V</code>) multiplication and <code>softmax</code> operations). This way, if we add zeros, there is no effect over the subsequent Values (V) multiplication and/or softmax operations. However, if we add a -inf, then most likely the softmax operation will result in zero, whatsoever the V values are. Why would we want to do this? If we do this, this effect will carry on to the top of the TransformerEncoder effectively asking it to predict only by seeing past info (tokens). As an interesting note, this is exactly what the Transformer’s static method called <code>generate_square_subsequent_mask</code> does, i.e. generate a square matrix with zeros and -inf. Lastly, the <code>src_key_padding_mask</code> does something utterly similar but in order to mask the padding values which have (clearly) no semantic meaning, i.e. please do not pay attention (attend) to the bunch of zeros that were appended to the right of the different length sentences.</p>
<p>Having understood the previous ideas, the following is easier to understand.</p>
<p>Let’s now take the TransformerDecoder class. According to the docs, it says <code>forward(tgt, memory, tgt_mask=None, memory_mask=None, tgt_key_padding_mask=None, memory_key_padding_mask=None)</code>.</p>
<p>Let’s take slowly it by parts.</p>
<p>First, the <code>tgt_mask</code> behaves <em>identically</em> to the <code>src_mask</code> with shape (T,T) (which is simply the target sequence length). So, now you must know that its purpose is to mask some desired tokens of the target sentences inside the attention mechanism, and of course, in our scenario, to mask all the way the future target tokens.</p>
<p>Second, the <code>memory_mask</code> is interesting. Although it’s also another masking mechanism as we know, what’s interesting is its shape. According to docs, its shape is (T,S). Since we now know that S is the source sequence length and T is the target sequence length, this means that the masking mechanism takes place right on top of the encoder’s outputs which are feed into each of the decoder’s inputs. This mask need not be square, for example if we use different sequence lengths between source and target sentences (other words when if S!=T). Why would we want to use this one? I have always regarded it as a memory bandwidth. Suppose you apply a lot of -inf, then a lot of -inf will be fed into the decoder’s encoder-decoder attention, effectively making it to not pay attention to a lot of tokens!</p>
<p>Thirdly, and somewhat easy to understand, we also once again have padding masking arguments: <code>tgt_key_padding_mask</code> and <code>memory_key_padding_mask=None</code>, which you must know that they allow to make the transformer’s life and grads easier by telling it to not attend to zeros both padded to the right in the memory and target sequences.</p>
<p>Lastly, as per your question, the official example is not the best in terms of self-explanations. I have to read countless other resources in order to reach this level of understanding, where sometimes I often confuse myself still! According to the docs, src, tgt and memory masks have no dimension as the batch’s dimension. Only the padding ones, which can be confusing. About this, all I can say is my opinion: I think that Pytorch is designed to disregard the batch size, so for example the result of some operation over a batch with shape (10,10) will be the same as (1,1,1,1,10,10) as it’ll disregard whatever the batches dimension are and will only work with <em>what is has to be worked on</em>, i.e. in the example the last two dimensions. I can only say this after much experimentation and practice.</p>
<p>Hope this helps a bit with your question.</p>