BERT output is different even in an eval() mode for the same input string in single vs batch mode. 
Author: darshan2203
Pull request: no

I'm facing the issue where BERT output is different even in an eval() mode for the same input string in single vs batch mode.

When I try to get embedding for a string ["apple tv 4k"] in a single mode vs batch mode where input is ["apple tv 4k", "whole milk"]

Here, the embeddings generated for ["apple tv 4k"] are different beyond a certain decimal numbers.

Can someone help me understand why?

Code Snippet to reproduce:

```
# Set up maximum sequence length
max_seq_length = 32

tokenizer = AutoTokenizer.from_pretrained("./model_hub/bert-base-cased")
model = BertModel.from_pretrained("./model_hub/bert-base-cased")
model.eval()
# Set up maximum sequence length
max_seq_length = 32

def predict(input_seqs):
    tokenized_input = tokenizer(input_seqs, add_special_tokens=True, max_length=max_seq_length, padding="max_length", return_tensors="pt")
    print("Tokenzied Input Keys: ", tokenized_input.keys())
    print("Tokenized Input Ids Shape: ", tokenized_input['input_ids'].shape)
    print("Tokens: ", tokenized_input["input_ids"][0][:10])
    print("Tokens: ", tokenized_input["attention_mask"][0][:10])
    # Generate last hidden states using pre-trained BERT model
    with torch.no_grad():
        outputs = model(**tokenized_input)
        print("Output Keys: ", outputs.keys())
        print("Pooler Shape: ", outputs["pooler_output"].shape)
        print("Pooler: ", outputs["pooler_output"][0][:5])
        last_hidden_states = outputs.last_hidden_state
    return last_hidden_states

# Tokenize & predict for input sequence
input_sequence_1 = ["Apple tv 4k"]
last_hidden_states_1 = predict(input_seqs=input_sequence_1)
print("last_hidden_states shape: ", last_hidden_states_1.shape)
print("===========================================================================")

# Tokenize & predict for input sequence
input_sequence_2 = ["Apple tv 4k"]
last_hidden_states_2 = predict(input_seqs=input_sequence_2)
print("last_hidden_states shape: ", last_hidden_states_2.shape)
print("===========================================================================")

# Tokenize & predict for input sequence
input_sequence_3 = ["Apple tv 4k", "Whole Milk"]
last_hidden_states_3 = predict(input_seqs=input_sequence_3)
print("last_hidden_states shape: ", last_hidden_states_3.shape)
print("===========================================================================")

print(last_hidden_states_1[0][0][:5].tolist())
print(last_hidden_states_2[0][0][:5].tolist())
print(last_hidden_states_3[0][0][:5].tolist())
```

Output:

```
Tokenzied Input Keys:  dict_keys(['input_ids', 'token_type_ids', 'attention_mask'])
Tokenized Input Ids Shape:  torch.Size([1, 32])
Tokens:  tensor([ 101, 7302,  189, 1964,  125, 1377,  102,    0,    0,    0])
Tokens:  tensor([1, 1, 1, 1, 1, 1, 1, 0, 0, 0])
Output Keys:  odict_keys(['last_hidden_state', 'pooler_output'])
Pooler Shape:  torch.Size([1, 768])
Pooler:  tensor([-0.7582,  0.5700,  0.9999, -0.9942,  0.9642])
last_hidden_states shape:  torch.Size([1, 32, 768])
===========================================================================
Tokenzied Input Keys:  dict_keys(['input_ids', 'token_type_ids', 'attention_mask'])
Tokenized Input Ids Shape:  torch.Size([1, 32])
Tokens:  tensor([ 101, 7302,  189, 1964,  125, 1377,  102,    0,    0,    0])
Tokens:  tensor([1, 1, 1, 1, 1, 1, 1, 0, 0, 0])
Output Keys:  odict_keys(['last_hidden_state', 'pooler_output'])
Pooler Shape:  torch.Size([1, 768])
Pooler:  tensor([-0.7582,  0.5700,  0.9999, -0.9942,  0.9642])
last_hidden_states shape:  torch.Size([1, 32, 768])
===========================================================================
Tokenzied Input Keys:  dict_keys(['input_ids', 'token_type_ids', 'attention_mask'])
Tokenized Input Ids Shape:  torch.Size([2, 32])
Tokens:  tensor([ 101, 7302,  189, 1964,  125, 1377,  102,    0,    0,    0])
Tokens:  tensor([1, 1, 1, 1, 1, 1, 1, 0, 0, 0])
Output Keys:  odict_keys(['last_hidden_state', 'pooler_output'])
Pooler Shape:  torch.Size([2, 768])
Pooler:  tensor([-0.7582,  0.5700,  0.9999, -0.9942,  0.9642])
last_hidden_states shape:  torch.Size([2, 32, 768])
===========================================================================
[0.4850187599658966, 0.004770198836922646, 0.16018253564834595, -0.550728976726532, -0.24116773903369904]
[0.4850187599658966, 0.004770198836922646, 0.16018253564834595, -0.550728976726532, -0.24116773903369904]
[0.48501819372177124, 0.004770228639245033, 0.16018278896808624, -0.550728976726532, -0.24116836488246918]
```

_Originally posted by @darshan2203 in https://github.com/huggingface/transformers/issues/9039#issuecomment-2041339845_
            

Comments:
On your script, converting the model to double-precision using ```model.double()``` resolves this, interestingly increasing max_seq_len to 64 also prevented this issue from arising. I believe this is attributable to floating point error. Otherwise, even with model.eval() active and other deterministic helpers such as

```
torch.backends.cudnn.benchmark = False
torch.backends.cudnn.deterministic = True
torch.manual_seed(0)
```
the minor discrepancies you described are still observed.

Examining the output tensors prior to softmax shows that there are minute differences between some elements within the output tensor, e.g. a value of 0.4111 in single mode versus 0.41109 in batch mode. The tanh grad_fn used by Bert is very sensitive to small variances, especially around its center, and floating point ops are not perfectly associative or distributive, so at a very deep level minute changes in how certain calculations are performed on batched inputs (e.g. vectorisation and change of operation order for parallelized processes) may have tiny effects.

Tiny differences like this shouldn't have any effect on the majority of tasks, and to some extent a small level of noise has a helpful regularising effect. If you do require such a high level of precision however, you could consider using fp64, but in my opinion there are very few scenarios where such a high level of precision has a meaningful beneficial effect on model outputs, even before considering the massively increased computational cost. 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.