Padding mask when 0 is an actual value

bogdan_ka: <p>Hello,</p>
<p>I have a transformer model where a 0 is an actual value in an input sequence and the sequence values go from 0 to 49 (sort of like dictionary size =50). So the sequence can look like this s = [0,1,3,5,8,20] The input to the embedding layer has input_dim=50.</p>
<p>I pad the sequences with 0 to make sure they are all the same length. The attention layer requires the padding_mask to be specified. And I do it as in code below. Problem is that this way valid 0 values are ignored and then the error pops up in the training loop because predictions are nan. I can’t pad with -1 because the input_dim=50 won’t work anymore.</p>
<p>I can’t seem to find a solution to this, even though this seems like it should be a common problem in lagnuage modeling. Please help!</p>
<pre><code class="lang-auto">nopeek_mask = torch.triu(torch.from_numpy(np.ones((query.shape[1], query.shape[1]))), diagonal=1)
        nopeek_mask[nopeek_mask == 1] = -float("Inf")
        nopeek_mask=torch.tensor(nopeek_mask)

        # https://discuss.pytorch.org/t/how-to-add-padding-mask-to-nn-transformerencoder-module/63390/2
        pad_token_index = 0
        pad_mask = (x == pad_token_index) 

        query_att,_ = self.atten_head(query, key, values , attn_mask=nopeek_mask , key_padding_mask=pad_mask )
        
</code></pre>

AbdulsalamBande: <p>A solution that I can think of is to change the embedding dictionary so that 0 represents the  token. This is what is usually done in the literature.</p>

bogdan_ka: <p>Thanks for the response <a class="mention" href="/u/abdulsalambande">@AbdulsalamBande</a> . I think that’s what it’s currently doing. So using the example where the sequence values can be 0-49, the embedding layer is  <code>embeddings = torch.nn.Embedding(50, self.emb_size)</code> . So 0s are being embedded, but then the pad masking, masks the 0s that are due to padding as well as 0s early in the sequence that are valid cases.</p>
<p>Eg. sequence can be s = [0,0,0,1,1,2,3, 0,0,0] , where last 3 zeros are padding, but first 3 are actual values.</p>

AbdulsalamBande: <p>Dont forget that the padding_mask is a binary mask that has 1 for actual tokens and 0 for padding tokens. So in the case of  [0,0,0,1,1,2,3, 0,0,0] , the padding_mask is [1,1,1,1,1,1,1, 0,0,0]</p>

brighteningeyes: <p>for padding as an index, you can have zero as your padding value, and i + 1 can be the value. for example, 0 can be encoded as 1, 1 as 2, and so on.<br>
in this case, you will have 51 indices instead of 50 where padding mask is 0</p>

bogdan_ka: <p><a class="mention" href="/u/abdulsalambande">@AbdulsalamBande</a> that makes sense but there is no way for me to know in advance where in the sequense 0 will appear, hence for each batch this generates the mast <code>pad_mask = (x == pad_token_index)</code> , which then masks any 0.</p>
<p><a class="mention" href="/u/brighteningeyes">@brighteningeyes</a> thank you, this seems like a reasonable solution to me. I think, this just means that in the output when I get the probability for each of the values in the sequence I have to discard the first one, since I’ll have 51 predicted values instead of 50.</p>