XLNet / sentence padding
Author: cherepanovic
Pull request: no
Labels: name: wontfix

My samples have different lengths and I want to apply the padding to bring them to the same length, because my purpose is to create sentence embeddings batchwise. For that all sentences must have the same length, otherwise it is not possible to create a tensor. 

How does padding work in use of XLNet model? 

the snippet below shows my first try to do it with XLNet, I apply maxpooling on the model output. 

```
class MaxPoolingChannel(torch.nn.AdaptiveMaxPool1d):
    def forward(self, input):
        input = input[0]
        input = input.transpose(1,2)
        result =  torch.nn.AdaptiveMaxPool1d(1)(input)
        return result.transpose(2,1)

tokenizer = XLNetTokenizer.from_pretrained('xlnet-base-cased')
model = XLNetModel.from_pretrained('xlnet-base-cased')
model.eval()

input_ids = torch.tensor(tokenizer.encode("Hello, my dog is cute")).unsqueeze(0)  # Batch size 1
input_ids_2 = torch.tensor(tokenizer.encode("Hello, my dog is cute <pad>")).unsqueeze(0)
input_ids_3 = torch.tensor(tokenizer.encode("Hello, my dog is cute <pad> <pad>")).unsqueeze(0)


with torch.no_grad():
    model = torch.nn.Sequential(model, MaxPoolingChannel(1))
    res = model(input_ids)
    res_2 = model(input_ids_2)
    res_3 = model(input_ids_3)

print(cosine_similarity(res.detach().numpy()[0][0],res_2.detach().numpy()[0][0]))
print(cosine_similarity(res.detach().numpy()[0][0],res_3.detach().numpy()[0][0]))
```

There is a thread #790 (about document embeddings), however the point in regard to padding in XLNet has not been touched.  

Thanks!

Comments:
Hi!

By concatenating the `<pad>` value to the end of your sentences you are successfully padding them. It can be observed by identifying the encoded sentence, which shows that a `5` value (which is the padding index in the tokenizer dictionary) is appended to the end of your token sequences.

Once you have padded your sentences, you can tell the model to ignore the padded values by specifying an `attention_mask` or an `input_mask`, as described in [the documentation.](https://huggingface.co/pytorch-transformers/model_doc/xlnet.html#xlnetmodel)

I did a comparison in all dimensions between the outputs, they are different 

```
input_ids = torch.tensor(tokenizer.encode("Hello, my dog is cute")).unsqueeze(0)  # Batch size 1
input_ids_2 = torch.tensor(tokenizer.encode("Hello, my dog is cute <pad>")).unsqueeze(0)
mask_2 = torch.ones((1, input_ids_2.shape[1], input_ids_2.shape[1]), dtype=torch.float)
mask_2[:, :, -1] = 0.0
input_ids_3 = torch.tensor(tokenizer.encode("Hello, my dog is cute <pad> <pad>")).unsqueeze(0)
mask_3 = torch.zeros((1, input_ids_3.shape[1], input_ids_3.shape[1]), dtype=torch.float)
mask_3[:, :, 0:-2] = 1

for i in range(1):
    with torch.no_grad():

        outputs = model(input_ids)
        res = MaxPoolingChannel(1)(outputs)
        outputs_2 = model(input_ids_2, attention_mask=mask_2[:, 0])
        res_2 = MaxPoolingChannel(1)(outputs_2)
        outputs_3 = model(input_ids_3, attention_mask=mask_3[:, 0])
        res_3 = MaxPoolingChannel(1)(outputs_3)

for i in range(outputs[0][0,:].shape[0]):
    print("Hello, my dog is cute/Hello, my dog is cute <pad> dim#:", i,cosine_similarity(outputs[0][0,i].numpy(),outputs_2[0][0,i].numpy()))

print('-------------------')
for i in range(outputs[0][0,:].shape[0]):
    print("Hello, my dog is cute/Hello, my dog is cute <pad> <pad> dim#:", i,cosine_similarity(outputs[0][0,i].numpy(),outputs_3[0][0,i].numpy()))

print('-------------------')

for i in range(outputs_2[0][0,:].shape[0]):
    print("Hello, my dog is cute <pad>/Hello, my dog is cute <pad> <pad> dim#:", i,cosine_similarity(outputs_2[0][0,i].numpy(),outputs_3[0][0,i].numpy()))
```

here are outputs

```
Hello, my dog is cute/Hello, my dog is cute <pad> dim#: 0 0.9999999413703398
Hello, my dog is cute/Hello, my dog is cute <pad> dim#: 1 1.0000000465438699
Hello, my dog is cute/Hello, my dog is cute <pad> dim#: 2 1.000000000000007
Hello, my dog is cute/Hello, my dog is cute <pad> dim#: 3 0.9999999620304815
Hello, my dog is cute/Hello, my dog is cute <pad> dim#: 4 1.0000000000015001
Hello, my dog is cute/Hello, my dog is cute <pad> dim#: 5 0.9999999502016026
Hello, my dog is cute/Hello, my dog is cute <pad> dim#: 6 1.000000047706968
-------------------
Hello, my dog is cute/Hello, my dog is cute <pad> <pad> dim#: 0 1.0000000000000617
Hello, my dog is cute/Hello, my dog is cute <pad> <pad> dim#: 1 0.9999999534561627
Hello, my dog is cute/Hello, my dog is cute <pad> <pad> dim#: 2 1.0000000000001106
Hello, my dog is cute/Hello, my dog is cute <pad> <pad> dim#: 3 1.0000000000000115
Hello, my dog is cute/Hello, my dog is cute <pad> <pad> dim#: 4 0.9999999518847271
Hello, my dog is cute/Hello, my dog is cute <pad> <pad> dim#: 5 1.0000000000003175
Hello, my dog is cute/Hello, my dog is cute <pad> <pad> dim#: 6 1.0000000954140886
-------------------
Hello, my dog is cute <pad>/Hello, my dog is cute <pad> <pad> dim#: 0 0.999999941370278
Hello, my dog is cute <pad>/Hello, my dog is cute <pad> <pad> dim#: 1 0.999999906912401
Hello, my dog is cute <pad>/Hello, my dog is cute <pad> <pad> dim#: 2 1.000000000000062
Hello, my dog is cute <pad>/Hello, my dog is cute <pad> <pad> dim#: 3 1.000000037969543
Hello, my dog is cute <pad>/Hello, my dog is cute <pad> <pad> dim#: 4 1.00000004811622
Hello, my dog is cute <pad>/Hello, my dog is cute <pad> <pad> dim#: 5 0.9999999502025548
Hello, my dog is cute <pad>/Hello, my dog is cute <pad> <pad> dim#: 6 1.0000000477071729
Hello, my dog is cute <pad>/Hello, my dog is cute <pad> <pad> dim#: 7 1.00000000000002

```

the values are not the same, they are a slightly different (an exact consistency is not possible?)

```
-------------------
0 -0.7767028212547302
1 0.15364784002304077
2 -0.5269558429718018
3 -0.04860188066959381
4 0.14985302090644836
5 -0.6860541105270386
6 -1.598402738571167
-------------------
0 -0.7766993641853333
1 0.15364792943000793
2 -0.5269524455070496
3 -0.04859305918216705
4 0.1498618721961975
5 -0.6860424280166626
6 -1.5983952283859253
7 -0.921322226524353
8 -0.6499249935150146
```

would be the result of some picked dimension of an unpadded and a padded sentence. 

```
for i in range(outputs[0][0,:].shape[0]):
    print(i, outputs[0][0][i, 0].item())
print('-------------------')
for i in range(outputs_3[0][0,:].shape[0]):
    print(i, outputs_3[0][0][i, 0].item())
```

dropout inplace is false on all layers in the model

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


> Hi!
> 
> By concatenating the `<pad>` value to the end of your sentences you are successfully padding them. It can be observed by identifying the encoded sentence, which shows that a `5` value (which is the padding index in the tokenizer dictionary) is appended to the end of your token sequences.
> 
> Once you have padded your sentences, you can tell the model to ignore the padded values by specifying an `attention_mask` or an `input_mask`, as described in [the documentation.](https://huggingface.co/pytorch-transformers/model_doc/xlnet.html#xlnetmodel)



> Hi!
> 
> By concatenating the `<pad>` value to the end of your sentences you are successfully padding them. It can be observed by identifying the encoded sentence, which shows that a `5` value (which is the padding index in the tokenizer dictionary) is appended to the end of your token sequences.
> 
> Once you have padded your sentences, you can tell the model to ignore the padded values by specifying an `attention_mask` or an `input_mask`, as described in [the documentation.](https://huggingface.co/pytorch-transformers/model_doc/xlnet.html#xlnetmodel)

I'd like to highlight that I encountered the same problem. the position IDs of padding tokens are set to 5 instead of 0. This value may either remain unchanged or undergo modifications in the future.