In TransformerEncoder, is src_key_padding_mask enough for proper backprop?

ado_sar: <p>I want to use a transformer encoder for classification/regression over sequence of vectors, <strong>not necessarily text</strong>.</p>
<p>However, the problem is that these sequences have different length, meaning I need to pad in order to process efficiently with batches (unfortunately, nested tensors are not supported yet in <code>TransformerEncoderLayer</code>).</p>
<p>My question is whether the <code>src_key_padding_mask</code> is enough for proper backpropagation, so the padding doesn’t influece the gradient/update of parameters during training.</p>
<p><strong>Masking is only applied during the attention operation, but how does this ensure that all layers of the transformer encoder, e.g. feedforward, layernorm etc, are not influenced by the padded inputs?</strong> Is the masking of loss what effectively ignores them?</p>
<p>Lets assume the following architecture. A sequence of vectors is first non-linearly projected, then processed by the encoder. Finally, the whole sequence is converted to a global feature vector and processed by a task-specific head. In code:</p>
<pre data-code-wrap="python"><code class="lang-python">class Model(nn.Module):
    def __init__(self, in_dim):
        def super().__init__()

        # Example of custom embedding.
        self.embedding = nn.Sequential(
                nn.Linear(in_dim, ...),
                nn.LayerNorm(...),
                nn.ReLU(),
                )
        self.encoder = nn.TransformerEncoder(...)
        self.head = nn.Linear(...)

        def forward(self, x, mask):
            x = self.embedding(x)
            x = self.encoder(x, src_key_padding_mask=mask)

            # Map each sequence to a global feature vector.
            value = -torch.inf
            x = x.masked_fill(mask, value)  # Ignore padding during selection.
            x = torch.max(...)
            
            return self.head(x)
</code></pre>
<p>We need to ensure that all model parameters are not affected by padding. Since the padding vectors are never selected by <code>torch.max</code>, this effectively eliminates the effect of padding to gradients (since the gradient of an element not selected by max is 0). Is my understanding correct?</p>

Chenze_W: <p>Personal understanding here, might be helpful.</p>
<p>In transformer, the feed forward NN is applied <em>embedding-wise</em>.</p>
<p>If we have input shape (batch, seq, dmodel), then the weight of the first linear layer will be of shape (dmodel, hidden), projecting input embeddings into the dimension of hidden layer.</p>
<p>This means, the padding tokens, once disconnected by attention mask, will not affect other tokens during the feed forward NN.</p>
<p>As for the layer norm, I am not very sure. In my case, I never treated padding tokens differently, and the results weren’t bad. It seems that masking padding tokens is not necessary. Transformer can adapt to their presence itself.</p>

ado_sar: <p><a class="mention" href="/u/chenze_w">@Chenze_W</a> Appreciate the answer!</p>
<p>It is true that other tokens affect other tokens only during attention since other layers transform tokens identically and independently. However, if we don’t mask the contribution of padding to gradients, these dummy tokens would affect the gradients and as such the parameters’ update.</p>
<blockquote>
<p>I never treated padding tokens differently, and the results weren’t bad. It seems that masking padding tokens is not necessary. Transformer can adapt to their presence itself.</p>
</blockquote>
<p>Do you mask the loss during training?</p>

Chenze_W: <p>No, I’ve never tried to mask the loss and it didn’t seem to affect the performance. I should try maksing the padding tokens.</p>
<p>I guess this can be done by detaching the padding tokens from the computational graph between each transformer layer.</p>
<p>I’ll share the results when I actually manage to do this.</p>