BERT outputs are different with the same input in training mode
Author: lamthuy
Pull request: no

When the training mode is enabled, BERT model returns different outputs even for the same input, is there any idea on why this happens?

```python
from transformers import BertModel, BertTokenizer
model = BertModel.from_pretrained("bert-base-uncased")
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
a = tokenizer.encode("Hello how are you?", return_tebsors='pt')
model.train()

torch.mean(model(a)[0])
# tensor(-0.0167, grad_fn=<MeanBackward0>)
torch.mean(model(a)[0])
# tensor(-0.0162, grad_fn=<MeanBackward0>)
torch.mean(model(a)[0])
# tensor(-0.0156, grad_fn=<MeanBackward0>)


```

Comments:
Hi there! The [forum](https://discuss.huggingface.co/) is a better place for those kinds of general questions, as we keep the issues for bugs and feature requests only.

To answer your question, this is because most Deep Learning models (including BERT) use a technique called [dropout](https://www.cs.toronto.edu/~hinton/absps/JMLRdropout.pdf) to generalize better, which randomly zeros some activations during training. This randomness is the reason you are getting different results for the same inputs.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

Hi @lamthuy were you able to investigate and find out the cause?

Same question here!

I am also facing the same error. 
@sgugger I thought the dropout is deactivated once I call model.eval()



Not sure what your code sample is @lava18 . The code sample above (fixed like below) always returns the same value once the line `model.train()` is removed.
```py
import torch
from transformers import BertModel, BertTokenizer
model = BertModel.from_pretrained("bert-base-uncased")
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
a = tokenizer.encode("Hello how are you?", return_tensors='pt')

torch.mean(model(a).to_tuple()[0])
```

You need to call `model.eval()` after training (or before inference). That should deactivate the dropouts and you will always get the same output for the same input. 

I'm facing the similar issue,

When I try to get embedding for a single seuqence ["apple tv 4k"] vs batch_sequnces = ["apple tv 4k", "whole milk"]

The embeddings generated for ["apple tv 4k"] are different beyond a certain decimal numbers.

Can someone help me understand why?

Code Snippet to reproduce:

```
# Set up maximum sequence length
max_seq_length = 32

tokenizer = AutoTokenizer.from_pretrained("./model_hub/bert-base-cased")
model = BertModel.from_pretrained("./model_hub/bert-base-cased")
model.eval()
# Set up maximum sequence length
max_seq_length = 32

def predict(input_seqs):
    tokenized_input = tokenizer(input_seqs, add_special_tokens=True, max_length=max_seq_length, padding="max_length", return_tensors="pt")
    print("Tokenzied Input Keys: ", tokenized_input.keys())
    print("Tokenized Input Ids Shape: ", tokenized_input['input_ids'].shape)
    print("Tokens: ", tokenized_input["input_ids"][0][:10])
    print("Tokens: ", tokenized_input["attention_mask"][0][:10])
    # Generate last hidden states using pre-trained BERT model
    with torch.no_grad():
        outputs = model(**tokenized_input)
        print("Output Keys: ", outputs.keys())
        print("Pooler Shape: ", outputs["pooler_output"].shape)
        print("Pooler: ", outputs["pooler_output"][0][:5])
        last_hidden_states = outputs.last_hidden_state
    return last_hidden_states

# Tokenize & predict for input sequence
input_sequence_1 = ["Apple tv 4k"]
last_hidden_states_1 = predict(input_seqs=input_sequence_1)
print("last_hidden_states shape: ", last_hidden_states_1.shape)
print("===========================================================================")

# Tokenize & predict for input sequence
input_sequence_2 = ["Apple tv 4k"]
last_hidden_states_2 = predict(input_seqs=input_sequence_2)
print("last_hidden_states shape: ", last_hidden_states_2.shape)
print("===========================================================================")

# Tokenize & predict for input sequence
input_sequence_3 = ["Apple tv 4k", "Whole Milk"]
last_hidden_states_3 = predict(input_seqs=input_sequence_3)
print("last_hidden_states shape: ", last_hidden_states_3.shape)
print("===========================================================================")

print(last_hidden_states_1[0][0][:5].tolist())
print(last_hidden_states_2[0][0][:5].tolist())
print(last_hidden_states_3[0][0][:5].tolist())
```

Output:

```
Tokenzied Input Keys:  dict_keys(['input_ids', 'token_type_ids', 'attention_mask'])
Tokenized Input Ids Shape:  torch.Size([1, 32])
Tokens:  tensor([ 101, 7302,  189, 1964,  125, 1377,  102,    0,    0,    0])
Tokens:  tensor([1, 1, 1, 1, 1, 1, 1, 0, 0, 0])
Output Keys:  odict_keys(['last_hidden_state', 'pooler_output'])
Pooler Shape:  torch.Size([1, 768])
Pooler:  tensor([-0.7582,  0.5700,  0.9999, -0.9942,  0.9642])
last_hidden_states shape:  torch.Size([1, 32, 768])
===========================================================================
Tokenzied Input Keys:  dict_keys(['input_ids', 'token_type_ids', 'attention_mask'])
Tokenized Input Ids Shape:  torch.Size([1, 32])
Tokens:  tensor([ 101, 7302,  189, 1964,  125, 1377,  102,    0,    0,    0])
Tokens:  tensor([1, 1, 1, 1, 1, 1, 1, 0, 0, 0])
Output Keys:  odict_keys(['last_hidden_state', 'pooler_output'])
Pooler Shape:  torch.Size([1, 768])
Pooler:  tensor([-0.7582,  0.5700,  0.9999, -0.9942,  0.9642])
last_hidden_states shape:  torch.Size([1, 32, 768])
===========================================================================
Tokenzied Input Keys:  dict_keys(['input_ids', 'token_type_ids', 'attention_mask'])
Tokenized Input Ids Shape:  torch.Size([2, 32])
Tokens:  tensor([ 101, 7302,  189, 1964,  125, 1377,  102,    0,    0,    0])
Tokens:  tensor([1, 1, 1, 1, 1, 1, 1, 0, 0, 0])
Output Keys:  odict_keys(['last_hidden_state', 'pooler_output'])
Pooler Shape:  torch.Size([2, 768])
Pooler:  tensor([-0.7582,  0.5700,  0.9999, -0.9942,  0.9642])
last_hidden_states shape:  torch.Size([2, 32, 768])
===========================================================================
[0.4850187599658966, 0.004770198836922646, 0.16018253564834595, -0.550728976726532, -0.24116773903369904]
[0.4850187599658966, 0.004770198836922646, 0.16018253564834595, -0.550728976726532, -0.24116773903369904]
[0.48501819372177124, 0.004770228639245033, 0.16018278896808624, -0.550728976726532, -0.24116836488246918]
```