Relation between `is_causal` and `src_mask`?

Cyber_punk: <p>In several Transformer components and functions, the two arguments <code>is_causal</code> and <code>src_mask</code> exist. But it seems the definitions are different and confusing.<br>
<strong>In my test with <code>nn.MultiheadAttention</code>, whether <code>is_causal</code> is set to be <code>True</code> or <code>False</code> (default), there is no difference in the output. Why?</strong> BTW the attention mask I used is not causal, but only a mask for test purpose:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/4/c445a76789e01c861439747022dcccf7aa10ded7.png" data-download-href="https://discuss.pytorch.org/uploads/default/c445a76789e01c861439747022dcccf7aa10ded7" title="Screenshot from 2023-07-02 17-35-33"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/4/c445a76789e01c861439747022dcccf7aa10ded7_2_460x500.png" alt="Screenshot from 2023-07-02 17-35-33" data-base62-sha1="s0iOEK9UZEZ4uqij4Ui3Kb2rbIr" width="460" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/4/c445a76789e01c861439747022dcccf7aa10ded7_2_460x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/c/4/c445a76789e01c861439747022dcccf7aa10ded7_2_690x750.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/c/4/c445a76789e01c861439747022dcccf7aa10ded7.png 2x" data-dominant-color="F3F3F3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2023-07-02 17-35-33</span><span class="informations">694×754 71 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>In <a href="https://pytorch.org/docs/stable/generated/torch.nn.TransformerEncoder.html?highlight=transformer+encoder#torch.nn.TransformerEncoder" rel="noopener nofollow ugc"><code>torch.nn.TransformerEncoder</code></a>:</p>
<ul>
<li><strong>mask</strong> (<a href="https://docs.python.org/3/library/typing.html#typing.Optional" rel="noopener nofollow ugc"><em>Optional</em></a><em>[</em> <a href="https://pytorch.org/docs/stable/tensors.html#torch.Tensor" rel="noopener nofollow ugc"><em>Tensor</em></a><em>]</em>) – the mask for the src sequence (optional).</li>
<li><strong>is_causal</strong> (<a href="https://docs.python.org/3/library/typing.html#typing.Optional" rel="noopener nofollow ugc"><em>Optional</em></a><em>[</em> <a href="https://docs.python.org/3/library/functions.html#bool" rel="noopener nofollow ugc"><em>bool</em></a><em>]</em>) – If specified, applies a causal mask as mask (optional) and ignores attn_mask for computing scaled dot product attention. Default: <code>False</code>.</li>
</ul>
<p>In <a href="https://pytorch.org/docs/stable/generated/torch.nn.TransformerEncoderLayer.html" rel="noopener nofollow ugc"><code>torch.nn.TransformerEncoderLayer</code></a>:</p>
<ul>
<li><strong>src_mask</strong> (<a href="https://docs.python.org/3/library/typing.html#typing.Optional" rel="noopener nofollow ugc"><em>Optional</em></a><em>[</em> <a href="https://pytorch.org/docs/stable/tensors.html#torch.Tensor" rel="noopener nofollow ugc"><em>Tensor</em></a><em>]</em>) – the mask for the src sequence (optional).</li>
<li><strong>is_causal</strong> (<a href="https://docs.python.org/3/library/functions.html#bool" rel="noopener nofollow ugc"><em>bool</em></a>) – If specified, applies a causal mask as src_mask. Default: <code>False</code>.</li>
</ul>
<p>(1) <strong>Does the above two cases mean if <code>is_causal</code> is True, a new upper triangular causal mask is created internally to replace the provided <code>src_mask</code>? What is the difference if <code>is_causal</code> is False and <code>src_mask</code> is also provided?</strong></p>
<p>In <a href="https://pytorch.org/docs/stable/generated/torch.nn.MultiheadAttention.html" rel="noopener nofollow ugc">torch.nn.MultiheadAttention</a>:</p>
<ul>
<li><strong>is_causal</strong> (<a href="https://docs.python.org/3/library/functions.html#bool" rel="noopener nofollow ugc"><em>bool</em></a>) – If specified, applies a causal mask as attention mask. Default: <code>False</code>. Warning: <code>is_causal</code> provides a hint that <code>attn_mask</code> is the causal mask. Providing incorrect hints can result in incorrect execution, including forward and backward compatibility.</li>
</ul>
<p>In  <a href="https://github.com/pytorch/pytorch/blob/main/torch/nn/functional.py#L5085" rel="noopener nofollow ugc">nn.functional.multi_head_attention_forward</a>:</p>
<pre><code>    is_causal: If specified, applies a causal mask as attention mask, and ignores
        attn_mask for computing scaled dot product attention.
        Default: ``False``.
        .. warning::
            is_causal is provides a hint that the attn_mask is the
            causal mask.Providing incorrect hints can result in
            incorrect execution, including forward and backward
            compatibility.
</code></pre>
<pre><code class="lang-auto">    if is_causal and attn_mask is None:
        raise RuntimeError(
            "Need attn_mask if specifying the is_causal hint. "
            "You may use the Transformer module method "
            "`generate_square_subsequent_mask` to create this mask."
        )
</code></pre>
<p>(2) <strong>If <code>is_causal</code> is True means we can ignore the <code>attn_mask</code>, why raising this error?</strong></p>
<p>In <a href="https://pytorch.org/docs/stable/generated/torch.nn.functional.scaled_dot_product_attention.html#torch.nn.functional.scaled_dot_product_attention" rel="noopener nofollow ugc"><code>torch.nn.functional.scaled_dot_product_attention</code></a>:</p>
<ul>
<li><strong>attn_mask</strong> (<em>optional Tensor</em>) – Attention mask; shape (N,…,L,S). Two types of masks are supported. A boolean mask where a value of True indicates that the element <em>should</em> take part in attention. A float mask of the same type as query, key, value that is added to the attention score.</li>
<li><strong>dropout_p</strong> (<a href="https://docs.python.org/3/library/functions.html#float" rel="noopener nofollow ugc"><em>float</em></a>) – Dropout probability; if greater than 0.0, dropout is applied</li>
<li><strong>is_causal</strong> (<a href="https://docs.python.org/3/library/functions.html#bool" rel="noopener nofollow ugc"><em>bool</em></a>) – If true, assumes causal attention masking and errors if both attn_mask and is_causal are set.</li>
</ul>
<p>(3)<br>
<strong>“errors if both attn_mask and is_causal are set” - this function does not allow both attn_mask and is_causal to be set together.</strong> Are they designed to be different?</p>

Cyber_punk: <p>What’s the purpose of the <code>is_causal</code> flag?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/2/f26da53b60c3b1133c00eac8210cd2bd9d6898ef.png" data-download-href="https://discuss.pytorch.org/uploads/default/f26da53b60c3b1133c00eac8210cd2bd9d6898ef" title="Screenshot from 2023-07-03 15-29-00"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/2/f26da53b60c3b1133c00eac8210cd2bd9d6898ef_2_546x500.png" alt="Screenshot from 2023-07-03 15-29-00" data-base62-sha1="yACsEmReZDvYcqhLX0gpN2kfFx5" width="546" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/2/f26da53b60c3b1133c00eac8210cd2bd9d6898ef_2_546x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/f/2/f26da53b60c3b1133c00eac8210cd2bd9d6898ef_2_819x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/f/2/f26da53b60c3b1133c00eac8210cd2bd9d6898ef_2_1092x1000.png 2x" data-dominant-color="241E1C"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2023-07-03 15-29-00</span><span class="informations">1156×1058 119 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>