[Transformer] Difference between src_mask and src_key_padding_mask

sahiluppal2k: <p>I am having a difficult time in understanding transformers. Everything is getting clear bit by bit but one thing that makes my head scratch is<br>
what is the difference between src_mask and src_key_padding_mask which is passed as an argument in forward function in both encoder layer and decoder layer.</p>
<p><a href="https://pytorch.org/docs/master/_modules/torch/nn/modules/transformer.html#Transformer" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/docs/master/_modules/torch/nn/modules/transformer.html#Transformer</a></p>

h-shad: <p>Hi.</p>
<p>Based on the PyTorch implementation source code (look at <a href="https://github.com/pytorch/pytorch/blob/4bf90558e0cbafbf03fa7e4285367f12658bde54/torch/nn/modules/transformer.py#L296" rel="noopener nofollow ugc">here</a>) <strong>src_mask</strong> is what is called <strong>attn_mask</strong> in a MultiheadAttention module and <strong>src_key_padding_mask</strong> is equivalent to <strong>key_padding_mask</strong> in a MultiheadAttention module.</p>
<p><strong>src_mask</strong> or <strong>attn_mask</strong> is a matrix used to represent which parts of the input sequence are allowed to be attended to (relative to the sequence itself). For example, in an autoregressive NN this would constitute a triangular matrix. Assuming the input sequence if of size N, this matrix would be <strong>of size N*N</strong>.</p>
<p><strong>src_key_padding_mask</strong> or <strong>key_padding_mask</strong> is a matrix that is supposed to mark the padding areas that the layer should not attend to. It is <strong>of size batch_size*N</strong>.</p>

Brando_Miranda: <p>found the the exact same question on SO:</p>
<aside class="onebox stackexchange" data-onebox-src="https://stackoverflow.com/questions/62170439/difference-between-src-mask-and-src-key-padding-mask">
  <header class="source">

      <a href="https://stackoverflow.com/questions/62170439/difference-between-src-mask-and-src-key-padding-mask" target="_blank" rel="noopener nofollow ugc">stackoverflow.com</a>
  </header>

  <article class="onebox-body">
      <a href="https://stackoverflow.com/users/13093875/leo" target="_blank" rel="noopener nofollow ugc">
    <img alt="Leo" src="https://www.gravatar.com/avatar/6447c807f3e2364f56f9337ad1367855?s=128&amp;d=identicon&amp;r=PG&amp;f=1" class="thumbnail onebox-avatar" width="128" height="128">
  </a>

<h4>
  <a href="https://stackoverflow.com/questions/62170439/difference-between-src-mask-and-src-key-padding-mask" target="_blank" rel="noopener nofollow ugc">Difference between src_mask and src_key_padding_mask</a>
</h4>

<div class="tags">
  <strong>pytorch, transformer</strong>
</div>

<div class="date">
  asked by
  
  <a href="https://stackoverflow.com/users/13093875/leo" target="_blank" rel="noopener nofollow ugc">
    Leo
  </a>
  on <a href="https://stackoverflow.com/questions/62170439/difference-between-src-mask-and-src-key-padding-mask" target="_blank" rel="noopener nofollow ugc">10:18AM - 03 Jun 20 UTC</a>
</div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>more related links:</p>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://www.reddit.com/r/pytorch/comments/okfh2k/transformer_difference_between_src_mask_and_src/">
  <header class="source">
      <img src="https://www.redditstatic.com/desktop2x/img/favicon/android-icon-192x192.png" class="site-icon" width="192" height="192">

      <a href="https://www.reddit.com/r/pytorch/comments/okfh2k/transformer_difference_between_src_mask_and_src/" target="_blank" rel="noopener nofollow ugc">reddit</a>
  </header>

  <article class="onebox-body">
    <img src="https://external-preview.redd.it/ptTnDZUx3X9A3hi4wxKik9qFkPv8iZAiRYu9fRZmU0U.jpg?auto=webp&amp;s=25578284f23558a4b75f01ba8e9eeac86b51ff7b" class="thumbnail onebox-avatar" width="500" height="500">

<h3><a href="https://www.reddit.com/r/pytorch/comments/okfh2k/transformer_difference_between_src_mask_and_src/" target="_blank" rel="noopener nofollow ugc">r/pytorch - [Transformer] Difference between src_mask and src_key_padding_mask</a></h3>

  <p>0 votes and 0 comments so far on Reddit</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<aside class="onebox stackexchange" data-onebox-src="https://stackoverflow.com/questions/62399243/transformerencoder-with-a-padding-mask">
  <header class="source">

      <a href="https://stackoverflow.com/questions/62399243/transformerencoder-with-a-padding-mask" target="_blank" rel="noopener nofollow ugc">stackoverflow.com</a>
  </header>

  <article class="onebox-body">
      <a href="https://stackoverflow.com/users/9330252/pourya-vakilipourtakalou" target="_blank" rel="noopener nofollow ugc">
    <img alt="Pourya Vakilipourtakalou" src="https://lh3.googleusercontent.com/-q06y56cN3Zw/AAAAAAAAAAI/AAAAAAAAACg/0V3LtrvvoJA/photo.jpg?sz=128" class="thumbnail onebox-avatar" width="128" height="128">
  </a>

<h4>
  <a href="https://stackoverflow.com/questions/62399243/transformerencoder-with-a-padding-mask" target="_blank" rel="noopener nofollow ugc">TransformerEncoder with a padding mask</a>
</h4>

<div class="tags">
  <strong>pytorch, transformer, attention-model</strong>
</div>

<div class="date">
  asked by
  
  <a href="https://stackoverflow.com/users/9330252/pourya-vakilipourtakalou" target="_blank" rel="noopener nofollow ugc">
    Pourya Vakilipourtakalou
  </a>
  on <a href="https://stackoverflow.com/questions/62399243/transformerencoder-with-a-padding-mask" target="_blank" rel="noopener nofollow ugc">12:43AM - 16 Jun 20 UTC</a>
</div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<hr>
<p>perhaps reading the docs for MHA is the best option:</p>
<p><a href="https://pytorch.org/docs/master/generated/torch.nn.MultiheadAttention.html#torch.nn.MultiheadAttention" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/master/generated/torch.nn.MultiheadAttention.html#torch.nn.MultiheadAttention</a></p>

Brando_Miranda: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> sorry for straight up pinging you…but do you mind explaining the different btw <code>src_mask</code> vs <code>src_key_padding_mask </code> (or more generally <code>_mask</code> vs <code>_key_padding_mask</code>)?</p>
<p>My understanding is that everything wrt masking can be done with <code>_mask</code> e.g. if the length of the sequence is <code>T</code> then I can always just mark the padding with a <code>T * T</code> matrix…is that not right?</p>
<p>What are use cases for one vs the other?</p>

Brando_Miranda: <p>I found this to be the most useful answer:</p>
<aside class="onebox stackexchange" data-onebox-src="https://stackoverflow.com/questions/62170439/difference-between-src-mask-and-src-key-padding-mask/68396781#68396781">
  <header class="source">

      <a href="https://stackoverflow.com/questions/62170439/difference-between-src-mask-and-src-key-padding-mask/68396781#68396781" target="_blank" rel="noopener nofollow ugc">stackoverflow.com</a>
  </header>

  <article class="onebox-body">
      <a href="https://stackoverflow.com/users/1601580/charlie-parker" target="_blank" rel="noopener nofollow ugc">
    <img alt="Charlie Parker" src="https://i.stack.imgur.com/2B6rV.png?s=128&amp;g=1" class="thumbnail onebox-avatar" width="128" height="128">
  </a>

<h4>
  <a href="https://stackoverflow.com/questions/62170439/difference-between-src-mask-and-src-key-padding-mask/68396781#68396781" target="_blank" rel="noopener nofollow ugc">Difference between src_mask and src_key_padding_mask</a>
</h4>

<div class="tags">
  <strong>pytorch, transformer</strong>
</div>

<div class="date">
  
  answered by
  <a href="https://stackoverflow.com/users/1601580/charlie-parker" target="_blank" rel="noopener nofollow ugc">
    Charlie Parker
  </a>
  on <a href="https://stackoverflow.com/questions/62170439/difference-between-src-mask-and-src-key-padding-mask/68396781#68396781" target="_blank" rel="noopener nofollow ugc">03:36PM - 15 Jul 21 UTC</a>
</div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


Zheng_Han: <p>Still not clear to me. The comment/document claims they both are to block part of the data which sounds like redundant parameters. For whatever <code>src_key_padding_mask</code> is doing, <code>src_mask</code> can do as well. On the other hand, if they are different why the <a href="https://pytorch.org/docs/stable/generated/torch.nn.TransformerEncoderLayer.html" rel="noopener nofollow ugc">documentation</a> says:</p>
<p><code>at most one of src_mask and src_key_padding_mask is passed</code></p>
<p>If they are different, why we can’t pass both <code>src_mask</code> and <code>src_key_padding_mask</code> to <code>forward()</code> ?</p>

Ty4Reading: <p><a class="mention" href="/u/zheng_han">@Zheng_Han</a> I did some further research that might be helpful for you as well because I had the same question.</p>
<p>For your first point, in the documentation where it says:</p>
<blockquote>
<p>at most one of src_mask and src_key_padding_mask is passed</p>
</blockquote>
<p>That part of the documentation is referring specifically to the ‘fast path’ of the model execution at inference time. It is not saying that you can’t pass both of them, it is saying that you can’t pass both of them AND expect ‘fast path’ to be executed for the model.</p>
<p>Also furthermore, if anyone is interested in the real difference between ‘src_mask’ and ‘src_key_padding_mask’.</p>
<p>The main difference is that ‘src_key_padding_mask’ looks at masks applied to entire tokens. So for example, when you set a value in the mask Tensor to ‘True’, you are essentially saying that the token is a ‘pad token’ and should not be attended by any other tokens.</p>
<p>However, if you wanted to do the same thing with ‘src_mask’, you would have to create a larger matrix and you would have to mask out ALL the connections between the ‘pad token’ and all other tokens.</p>
<p>So yes, you can accomplish ‘src_key_padding_mask’ with just ‘src_mask’ alone. However, it would require more work and would be more complicated. The src_key_padding_mask is a simple way to shortcut that.</p>