Batched BertForNextSentencePrediction with variable length sentences
Author: murdo25
Pull request: no

## ❓ Questions & Help

What's the proper way to pad a batch of variable length sentences for the BertForNextSentencePrediction model? 


I want to batch a list of sentences, and each sentence can have any length < max_seq_len. To fit them into a token tensor I assume I will need some form of padding? 

Here's an example with 2 candidate sentences where the first sentence has no padding, and the second has 2 padded 0s.

```python
import torch
from pytorch_transformers import BertTokenizer, BertModel, BertForMaskedLM,BertForNextSentencePrediction

# Load pre-trained model tokenizer (vocabulary)
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')

# Tokenized inputs
text1 = "[CLS] Who was Jim ? [SEP] Jim Henson was a puppeteer [SEP]"
text2 = "[CLS] Who was Jim ? [SEP] Jim Henson was a puppeteer [SEP] [PAD] [PAD]"
tokenized_text1 = tokenizer.tokenize(text1)
tokenized_text2 = tokenizer.tokenize(text2)

# Convert token to vocabulary indices
indexed_tokens1 = tokenizer.convert_tokens_to_ids(tokenized_text1)
indexed_tokens2 = tokenizer.convert_tokens_to_ids(tokenized_text2)

# Define sentence A and B indices associated to 1st and 2nd sentences 
segments_ids1 = [0, 0, 0, 0, 0,  0, 1, 1, 1, 1, 1, 1, 1]
segments_ids2 = [0, 0, 0, 0, 0,  0, 1, 1, 1, 1, 1, 1, 1, 0, 0]

# Convert inputs to PyTorch tensors
tokens_tensor1 = torch.tensor([indexed_tokens1])
tokens_tensor2 = torch.tensor([indexed_tokens2])
segments_tensors1 = torch.tensor([segments_ids1])
segments_tensors2 = torch.tensor([segments_ids2])

# Load pre-trained model (weights)
model = BertForNextSentencePrediction.from_pretrained('bert-base-uncased')
model.eval()

# Predict is Next Sentence ?
predictions1 = model(tokens_tensor1, segments_tensors1 )
predictions2 = model(tokens_tensor2, segments_tensors2 )

print(predictions1)  #(tensor([[ 5.6165, -5.2786]], grad_fn=<AddmmBackward>),)
print(predictions2) #(tensor([[ 5.0919, -4.4939]], grad_fn=<AddmmBackward>),)

```


As the number padding 0s increases the the confidence of the model continues to decline. We haven't been able to find any documentation for setting up the padding sequences, especially for the segment ids. Any idea how we can set this up? 
Thanks!

Comments:
Hello! 

1 - Indeed, if you want to have several sequences of variable length in a single batch, you should pad the shorter sequences.


2 - In the [`BertForNextSentencePrediction ` documentation](https://huggingface.co/transformers/model_doc/bert.html#bertfornextsentenceprediction) is written the following: 

`attention_mask`: Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: `1` for tokens that are NOT MASKED, `0` for MASKED tokens. 

If using the `attention_mask`, there should be no difference between a model's predictions of a sequence and its padded counterpart. 


3 - The segment ids padding indices can change according to the model. I believe it is `0` for most models, but `4` in the case of XLNet. You seem to be padding with `0` in your example, which is the way to go!

Thank you!!! This is exactly what I was missing. 

```python3
import torch
from pytorch_transformers import BertTokenizer, BertModel, BertForMaskedLM,BertForNextSentencePrediction

# Load pre-trained model tokenizer (vocabulary)
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')

# Tokenized inputs
text1 = "[CLS] Who was Jim ? [SEP] Jim Henson was a puppeteer [SEP]"
text2 = "[CLS] Who was Jim ? [SEP] Jim Henson was a puppeteer [SEP] [PAD] [PAD]"
tokenized_text1 = tokenizer.tokenize(text1)
tokenized_text2 = tokenizer.tokenize(text2)

# Convert token to vocabulary indices
indexed_tokens1 = tokenizer.convert_tokens_to_ids(tokenized_text1)
indexed_tokens2 = tokenizer.convert_tokens_to_ids(tokenized_text2)

# Define sentence A and B indices associated to 1st and 2nd sentences 
segments_ids1 = [0, 0, 0, 0, 0,  0, 1, 1, 1, 1, 1, 1, 1]
segments_ids2 = [0, 0, 0, 0, 0,  0, 1, 1, 1, 1, 1, 1, 1, 0, 0]

#Attention Mask [1] over tokens, [0] over padding
attention_mask = torch.FloatTensor([[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0]])

# Convert inputs to PyTorch tensors
tokens_tensor1 = torch.tensor([indexed_tokens1])
tokens_tensor2 = torch.tensor([indexed_tokens2])
segments_tensors1 = torch.tensor([segments_ids1])
segments_tensors2 = torch.tensor([segments_ids2])

# Load pre-trained model (weights)
model = BertForNextSentencePrediction.from_pretrained('bert-base-uncased')
model.eval()

# Predict is Next Sentence ?
predictions1 = model(tokens_tensor1, segments_tensors1 )
predictions2 = model(tokens_tensor2, segments_tensors2, attention_mask=attention_mask )

print(predictions1)  #(tensor([[ 5.6165, -5.2786]], grad_fn=<AddmmBackward>),)
print(predictions2) #(tensor([[ 5.6165, -5.2786]], grad_fn=<AddmmBackward>),)
```

With the attention mask now over tokens I get the same output without degradation. 
:)