nn.Transformer src/tgt/memory masks fail to work

haoxing: <p>nn.TransformerEncoderLayer produces exactly the <strong>same output with same src</strong>,no matter what <strong>src_key_padding_mask</strong> or <strong>src_mask</strong> is.</p>
<p>Likewise,nn.TransformerDecoderLayer output is <strong>not affected</strong> by <strong>any one</strong> of <em>tgt_mask,memory_mask,tgt_key_padding_mask or memory_key_padding_mask</em>.</p>
<p>Does anyone know what’s going wrong?How could I make the masks work in the right way?Thanks a lot.</p>
<blockquote>
<p>[Input0]:<br>
import torch<br>
import torch.nn as nn<br>
encoder_layer=nn.TransformerEncoderLayer(d_model=6,nhead=2)<br>
encoder_layer.eval()<br>
src=torch.ones((4,3,6))<br>
encoder_layer(src)</p>
<p>[Output0]:<br>
tensor([[[ 0.9927,  0.2251, -1.5199,  1.2508, -1.0397,  0.0910],<br>
[ 0.9927,  0.2251, -1.5199,  1.2508, -1.0397,  0.0910],<br>
[ 0.9927,  0.2251, -1.5199,  1.2508, -1.0397,  0.0910]],</p>
<pre><code>    [[ 0.9927,  0.2251, -1.5199,  1.2508, -1.0397,  0.0910],
     [ 0.9927,  0.2251, -1.5199,  1.2508, -1.0397,  0.0910],
     [ 0.9927,  0.2251, -1.5199,  1.2508, -1.0397,  0.0910]],

    [[ 0.9927,  0.2251, -1.5199,  1.2508, -1.0397,  0.0910],
     [ 0.9927,  0.2251, -1.5199,  1.2508, -1.0397,  0.0910],
     [ 0.9927,  0.2251, -1.5199,  1.2508, -1.0397,  0.0910]],

    [[ 0.9927,  0.2251, -1.5199,  1.2508, -1.0397,  0.0910],
     [ 0.9927,  0.2251, -1.5199,  1.2508, -1.0397,  0.0910],
     [ 0.9927,  0.2251, -1.5199,  1.2508, -1.0397,  0.0910]]],
   grad_fn=&lt;NativeLayerNormBackward0&gt;)
</code></pre>
</blockquote>
<blockquote>
<p>print(encoder_layer(src,src_mask=torch.zeros((4,4)).bool()))<br>
print(encoder_layer(src,src_mask=torch.tensor(<br>
[[0,1,1,1],<br>
[0,0,1,1],<br>
[0,0,0,1],<br>
[0,0,0,0]]<br>
).bool()))<br>
print(encoder_layer(src,src_mask=torch.tensor(<br>
[[0,1,0,1],<br>
[1,0,1,1],<br>
[0,1,0,1],<br>
[0,1,1,1]]<br>
).bool()))</p>
<p>[Output2]:<br>
tensor([[[ 0.9927,  0.2251, -1.5199,  1.2508, -1.0397,  0.0910],<br>
[ 0.9927,  0.2251, -1.5199,  1.2508, -1.0397,  0.0910],<br>
[ 0.9927,  0.2251, -1.5199,  1.2508, -1.0397,  0.0910]],</p>
<pre><code>    [[ 0.9927,  0.2251, -1.5199,  1.2508, -1.0397,  0.0910],
     [ 0.9927,  0.2251, -1.5199,  1.2508, -1.0397,  0.0910],
     [ 0.9927,  0.2251, -1.5199,  1.2508, -1.0397,  0.0910]],

    [[ 0.9927,  0.2251, -1.5199,  1.2508, -1.0397,  0.0910],
     [ 0.9927,  0.2251, -1.5199,  1.2508, -1.0397,  0.0910],
     [ 0.9927,  0.2251, -1.5199,  1.2508, -1.0397,  0.0910]],

    [[ 0.9927,  0.2251, -1.5199,  1.2508, -1.0397,  0.0910],
     [ 0.9927,  0.2251, -1.5199,  1.2508, -1.0397,  0.0910],
     [ 0.9927,  0.2251, -1.5199,  1.2508, -1.0397,  0.0910]]],
   grad_fn=&lt;NativeLayerNormBackward0&gt;)
</code></pre>
<p>tensor([[[ 0.9927,  0.2251, -1.5199,  1.2508, -1.0397,  0.0910],<br>
[ 0.9927,  0.2251, -1.5199,  1.2508, -1.0397,  0.0910],<br>
[ 0.9927,  0.2251, -1.5199,  1.2508, -1.0397,  0.0910]],</p>
<pre><code>    [[ 0.9927,  0.2251, -1.5199,  1.2508, -1.0397,  0.0910],
     [ 0.9927,  0.2251, -1.5199,  1.2508, -1.0397,  0.0910],
     [ 0.9927,  0.2251, -1.5199,  1.2508, -1.0397,  0.0910]],

    [[ 0.9927,  0.2251, -1.5199,  1.2508, -1.0397,  0.0910],
     [ 0.9927,  0.2251, -1.5199,  1.2508, -1.0397,  0.0910],
     [ 0.9927,  0.2251, -1.5199,  1.2508, -1.0397,  0.0910]],

    [[ 0.9927,  0.2251, -1.5199,  1.2508, -1.0397,  0.0910],
     [ 0.9927,  0.2251, -1.5199,  1.2508, -1.0397,  0.0910],
     [ 0.9927,  0.2251, -1.5199,  1.2508, -1.0397,  0.0910]]],
   grad_fn=&lt;NativeLayerNormBackward0&gt;)
</code></pre>
<p>tensor([[[ 0.9927,  0.2251, -1.5199,  1.2508, -1.0397,  0.0910],<br>
[ 0.9927,  0.2251, -1.5199,  1.2508, -1.0397,  0.0910],<br>
[ 0.9927,  0.2251, -1.5199,  1.2508, -1.0397,  0.0910]],</p>
<pre><code>    [[ 0.9927,  0.2251, -1.5199,  1.2508, -1.0397,  0.0910],
     [ 0.9927,  0.2251, -1.5199,  1.2508, -1.0397,  0.0910],
     [ 0.9927,  0.2251, -1.5199,  1.2508, -1.0397,  0.0910]],

    [[ 0.9927,  0.2251, -1.5199,  1.2508, -1.0397,  0.0910],
     [ 0.9927,  0.2251, -1.5199,  1.2508, -1.0397,  0.0910],
     [ 0.9927,  0.2251, -1.5199,  1.2508, -1.0397,  0.0910]],

    [[ 0.9927,  0.2251, -1.5199,  1.2508, -1.0397,  0.0910],
     [ 0.9927,  0.2251, -1.5199,  1.2508, -1.0397,  0.0910],
     [ 0.9927,  0.2251, -1.5199,  1.2508, -1.0397,  0.0910]]]
</code></pre>
</blockquote>
<p>All above produces the same result!What’s going wrong?</p>

haoxing: <p>Problem solved! It’s a special instance causing by making input filled with all ones.Despite attn_weights differs,attn_weights.sum(dim=-1) always equals to 1,and different masks does not affect the value_matrix,which contains equaled line vectors.Thus when dim -1 sumed to 1 attn_weights multiplies value_matrix,the result is the same.And the output could only be the same after a dnn layer.</p>