batch_encode_plus model output is different from tokenizer.encode model's output
Author: monk1337
Pull request: no
Labels: name: Core: Tokenization

I am trying to encode multiple sentences with BertTokenizer. I tried batch_encode_plus but I am getting different output when I am feeding BertTokenizer's output vs batch_encode_plus's output to model.

```
single_sentence = 'checking single sentences'
sentences = ['checking single sentences', 'many sentences encoding together', 'hello world how and', 'All model checkpoint weights were used when initializing BertModel.']



# loading models
from transformers import BertModel, BertConfig, BertTokenizer
import torch
tokenizer = BertTokenizer.from_pretrained('bert-large-uncased')
model     = BertModel.from_pretrained('bert-large-uncased')


# single encoding and hidden layer weights

def single_query(sentence):
    single_input_id = torch.tensor(tokenizer.encode(sentence)).unsqueeze(0)  # Batch size 1
    outputs         = model(single_input_id)
    features        = outputs[0][:,0,:].detach().numpy()
    
    return features

def batch_encoding(sentences):
    
    input_ids   = torch.tensor(tokenizer.batch_encode_plus(sentences, pad_to_max_length=True)['input_ids'])
    outputs     = model(all_f)
    features = outputs[0][:,0,:].detach().numpy()
    
    return features
```

single_query(single_sentence) return :
```
array([[-0.39814326, -0.4902882 ,  0.02681825, ..., -0.28256905,
        -1.0546892 ,  0.1055279 ]], dtype=float32)

```
while batch_enc = batch_encoding(sentences)[0] return : 



```
array([ 0.1909762 ,  0.05580305,  0.221862  , ...,  0.16220105,
       -0.88524836,  0.12994497], dtype=float32)
```

Why there is difference in the model's output? 

Is it because of padding?

Comments:
Indeed, if you are using padding you need to provide the attention masks to your model otherwise it doesn't know which tokens it should not attend to.

Here is the correct version of `batch_encoding` which will give the same output as the non batched version:
```python
def batch_encoding(sentences):
    
    inputs   = tokenizer(sentences, padding=True, return_tensors='pt')
    print(inputs)
    outputs     = model(**inputs)
    features = outputs[0][:,0,:].detach().numpy()
    
    return features
```
I've also updated it to the new tokenizers API on which you can learn a lot more in the tutorial here: https://huggingface.co/transformers/preprocessing.html

@thomwolf I am using the pre-trained distilbert model.
encoded_batch = tokenizer.batch_encode_plus(["hello", "there you"], add_special_tokens=True, return_tensors="tf", padding=True, truncation=True)
`{'input_ids': <tf.Tensor: shape=(2, 4), dtype=int32, numpy=
array([[ 101, 7592,  102,    0],
       [ 101, 2045, 2017,  102]], dtype=int32)>, 'attention_mask': <tf.Tensor: shape=(2, 4), dtype=int32, numpy=
array([[1, 1, 1, 0],
       [1, 1, 1, 1]], dtype=int32)>}`

encoded_batch[0][0] -> pointing to hidden states of "hello"
<tf.Tensor: shape=(4, 768), dtype=float32, numpy=
array([[-0.20557329, -0.18245512,  0.0950693 , ..., -0.06398913,
         0.16745588,  0.37530535],
       [-0.48316494, -0.13651992,  0.3210112 , ...,  0.0130196 ,
         0.27123356,  0.15390822],
       [ 0.8976611 ,  0.14261621, -0.40148023, ...,  0.31185225,
        -0.68173647, -0.2594604 ],
       [-0.0716978 , -0.18830499,  0.35636497, ...,  0.09993267,
        -0.05575091,  0.14572877]], dtype=float32)>


and if i encode "hello" alone, 
<tf.Tensor: shape=(3, 768), dtype=float32, numpy=
array([[-0.20557335, -0.18245521,  0.09506968, ..., -0.06398894,
         0.16745585,  0.37530527],
       [-0.48316532, -0.13651986,  0.32101193, ...,  0.0130194 ,
         0.27123365,  0.15390885],
       [ 0.89766103,  0.14261642, -0.40148014, ...,  0.31185254,
        -0.68173563, -0.25946063]], dtype=float32)>

so here, if you see, 
there is one row extra due to padding in the above instance,
and there is no way to figure out if that is SEP embeds or padded embeds
and that can lead to discrepancies.

Is there some way to send inputs to the model in a batch, keeping the embeddings intact?


Thanks in advance