auto model encodings for a text snippet returns different floating values across different batch sizes
Author: murali1996
Pull request: no

## Environment info
- `transformers` version: 4.4.1
- Platform: Linux-4.19.112+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.10
- PyTorch version (GPU?): 1.8.0+cu101 (True)
- Tensorflow version (GPU?): 2.4.1 (True)
- Using GPU in script?: yes (but the bug issue is irrespective of it)
- Using distributed or parallel set-up in script?: no

### Who can help
@LysandreJik, @patrickvonplaten

## Information

Model I am using : `bert-base-cased` and `sentence-transformers/distilbert-base-nli-stsb-mean-tokens`

Consider the following code:
```python
#  pip install transformers

import torch
device = "cuda" if torch.cuda.is_available() else "cpu"
print(device)

import transformers
from transformers import AutoModel, AutoTokenizer
name = "sentence-transformers/distilbert-base-nli-stsb-mean-tokens"
model = AutoModel.from_pretrained(name)
tokenizer = AutoTokenizer.from_pretrained(name)
model.to(device)
model.eval()

from tqdm.autonotebook import trange
for ntimes in trange(1, 200, 1, desc="ntimes", disable=False):
    s = ['This framework generates embeddings for each input sentence' for _ in range(ntimes)]
    f = tokenizer(s, padding=True, truncation='longest_first', return_tensors="pt", max_length=128)
    f = f.to(device)
    with torch.no_grad():
        out = model(**f, return_dict=False)
        t = out[0]   # token_embedding
        print(str(ntimes).zfill(4), t[0][0][:5].tolist())
```

The testing setup is as follows: For every batch size considered in 1 to 200, the model output (last layer's output) for first sentence is taken and compared. Ideally, it is expected to be same but depending on the batch size, the output varies. Although the differences are after several decimal places, it still creates an issue when rounding off or when used for exact-text-match tasks. An example when comparing first 5 values of CLS token's positional representation is printed below:

```
batch_size first_5_values
0001 [-0.4345831274986267, 0.19430403411388397, -0.008721709251403809, 0.16533663868904114, -0.21307958662509918]
0002 [-0.4345831274986267, 0.19430403411388397, -0.008721709251403809, 0.16533663868904114, -0.21307958662509918]
0003 [-0.4345839023590088, 0.19430400431156158, -0.008721785619854927, 0.16533628106117249, -0.21307939291000366]
0004 [-0.4345839023590088, 0.19430400431156158, -0.008721785619854927, 0.16533628106117249, -0.21307939291000366]
0005 [-0.4345839023590088, 0.19430400431156158, -0.008721785619854927, 0.16533628106117249, -0.21307939291000366]
0006 [-0.4345828890800476, 0.19430409371852875, -0.0087218526750803, 0.1653369963169098, -0.2130797803401947]
0007 [-0.43458378314971924, 0.19430388510227203, -0.008721890859305859, 0.16533657908439636, -0.21307970583438873]
0008 [-0.43458378314971924, 0.19430388510227203, -0.008721890859305859, 0.16533657908439636, -0.21307970583438873]
0009 [-0.43458378314971924, 0.19430388510227203, -0.008721890859305859, 0.16533657908439636, -0.21307970583438873]
0010 [-0.43458378314971924, 0.19430388510227203, -0.008721890859305859, 0.16533657908439636, -0.21307970583438873]
0011 [-0.43458303809165955, 0.19430424273014069, -0.0087218526750803, 0.16533637046813965, -0.21307967603206635]
0012 [-0.43458303809165955, 0.19430424273014069, -0.0087218526750803, 0.16533637046813965, -0.21307967603206635]
0013 [-0.4345836043357849, 0.19430403411388397, -0.008721555583178997, 0.16533656418323517, -0.21307919919490814]
0014 [-0.43458399176597595, 0.19430390000343323, -0.008721986785531044, 0.16533610224723816, -0.2130793035030365]
0015 [-0.43458399176597595, 0.19430390000343323, -0.008721986785531044, 0.16533610224723816, -0.2130793035030365]
0016 [-0.43458399176597595, 0.19430390000343323, -0.008721986785531044, 0.16533610224723816, -0.2130793035030365]
0017 [-0.43458399176597595, 0.19430390000343323, -0.008721986785531044, 0.16533610224723816, -0.2130793035030365]
0018 [-0.43458399176597595, 0.19430390000343323, -0.008721986785531044, 0.16533610224723816, -0.2130793035030365]
0019 [-0.43458399176597595, 0.19430390000343323, -0.008721986785531044, 0.16533610224723816, -0.2130793035030365]
0020 [-0.43458399176597595, 0.19430390000343323, -0.008721986785531044, 0.16533610224723816, -0.2130793035030365]
0021 [-0.4345839321613312, 0.19430312514305115, -0.008722005411982536, 0.16533663868904114, -0.21307897567749023]
0022 [-0.43458428978919983, 0.1943034529685974, -0.008722092024981976, 0.16533707082271576, -0.21307975053787231]
0023 [-0.43458428978919983, 0.1943034529685974, -0.008722092024981976, 0.16533707082271576, -0.21307975053787231]
0024 [-0.4345826506614685, 0.19430312514305115, -0.008722330443561077, 0.16533666849136353, -0.2130790799856186]
0025 [-0.43458348512649536, 0.19430328905582428, -0.008722656406462193, 0.16533656418323517, -0.21307960152626038]
0026 [-0.43458348512649536, 0.19430328905582428, -0.008722656406462193, 0.16533656418323517, -0.21307960152626038]
0027 [-0.43458327651023865, 0.19430407881736755, -0.008721843361854553, 0.16533705592155457, -0.2130795419216156]
0028 [-0.43458327651023865, 0.19430407881736755, -0.008721843361854553, 0.16533705592155457, -0.2130795419216156]
0029 [-0.43458327651023865, 0.19430407881736755, -0.008721843361854553, 0.16533705592155457, -0.2130795419216156]
0030 [-0.43458327651023865, 0.19430407881736755, -0.008721843361854553, 0.16533705592155457, -0.2130795419216156]
0031 [-0.43458327651023865, 0.19430407881736755, -0.008721843361854553, 0.16533705592155457, -0.2130795419216156]
0032 [-0.43458327651023865, 0.19430407881736755, -0.008721843361854553, 0.16533705592155457, -0.2130795419216156]
0033 [-0.43458327651023865, 0.19430407881736755, -0.008721843361854553, 0.16533705592155457, -0.2130795419216156]
0034 [-0.43458327651023865, 0.19430407881736755, -0.008721843361854553, 0.16533705592155457, -0.2130795419216156]
0035 [-0.4345836639404297, 0.19430400431156158, -0.008721727877855301, 0.16533680260181427, -0.2130795419216156]
0036 [-0.4345836639404297, 0.19430400431156158, -0.008721727877855301, 0.16533680260181427, -0.2130795419216156]
0037 [-0.4345836639404297, 0.19430400431156158, -0.008721727877855301, 0.16533680260181427, -0.2130795419216156]
0038 [-0.4345836639404297, 0.19430400431156158, -0.008721727877855301, 0.16533680260181427, -0.2130795419216156]
0039 [-0.4345836639404297, 0.19430400431156158, -0.008721727877855301, 0.16533680260181427, -0.2130795419216156]
0040 [-0.4345836639404297, 0.19430400431156158, -0.008721727877855301, 0.16533680260181427, -0.2130795419216156]
0041 [-0.43458354473114014, 0.1943034529685974, -0.008721929043531418, 0.1653369963169098, -0.2130795270204544]
0042 [-0.43458354473114014, 0.1943034529685974, -0.008721929043531418, 0.1653369963169098, -0.2130795270204544]
0043 [-0.43458354473114014, 0.1943034529685974, -0.008721929043531418, 0.1653369963169098, -0.2130795270204544]
0044 [-0.43458375334739685, 0.1943032294511795, -0.008721861988306046, 0.1653372347354889, -0.21307994425296783]
0045 [-0.43458375334739685, 0.1943032294511795, -0.008721861988306046, 0.1653372347354889, -0.21307994425296783]
0046 [-0.43458375334739685, 0.1943032294511795, -0.008721861988306046, 0.1653372347354889, -0.21307994425296783]
0047 [-0.4345839321613312, 0.19430294632911682, -0.008722168393433094, 0.16533693671226501, -0.21308031678199768]
0048 [-0.4345839321613312, 0.19430294632911682, -0.008722168393433094, 0.16533693671226501, -0.21308031678199768]
0049 [-0.4345839321613312, 0.19430294632911682, -0.008722168393433094, 0.16533693671226501, -0.21308031678199768]
0050 [-0.43458375334739685, 0.19430378079414368, -0.008721306920051575, 0.1653364896774292, -0.21308039128780365]
0051 [-0.43458375334739685, 0.19430378079414368, -0.008721306920051575, 0.1653364896774292, -0.21308039128780365]
0052 [-0.43458375334739685, 0.19430378079414368, -0.008721306920051575, 0.1653364896774292, -0.21308039128780365]
0053 [-0.4345836043357849, 0.1943032443523407, -0.008722072467207909, 0.16533635556697845, -0.21307991445064545]
0054 [-0.4345836043357849, 0.1943032443523407, -0.008722072467207909, 0.16533635556697845, -0.21307991445064545]
0055 [-0.4345836043357849, 0.1943032443523407, -0.008722072467207909, 0.16533635556697845, -0.21307991445064545]
0056 [-0.43458402156829834, 0.19430415332317352, -0.008722043596208096, 0.16533638536930084, -0.2130793184041977]
0057 [-0.43458402156829834, 0.19430415332317352, -0.008722043596208096, 0.16533638536930084, -0.2130793184041977]
0058 [-0.43458348512649536, 0.19430400431156158, -0.008722235448658466, 0.16533590853214264, -0.2130792737007141]
0059 [-0.43458348512649536, 0.19430400431156158, -0.008722235448658466, 0.16533590853214264, -0.2130792737007141]
0060 [-0.43458348512649536, 0.19430400431156158, -0.008722235448658466, 0.16533590853214264, -0.2130792737007141]
0061 [-0.43458348512649536, 0.19430400431156158, -0.008722235448658466, 0.16533590853214264, -0.2130792737007141]
0062 [-0.4345838725566864, 0.19430425763130188, -0.008721861988306046, 0.16533733904361725, -0.21307975053787231]
0063 [-0.4345838725566864, 0.19430425763130188, -0.008721861988306046, 0.16533733904361725, -0.21307975053787231]
0064 [-0.4345833957195282, 0.19430355727672577, -0.008721498772501945, 0.16533666849136353, -0.21307994425296783]
0065 [-0.4345833957195282, 0.19430355727672577, -0.008721498772501945, 0.16533666849136353, -0.21307994425296783]
0066 [-0.4345833957195282, 0.19430355727672577, -0.008721498772501945, 0.16533666849136353, -0.21307994425296783]
0067 [-0.4345833957195282, 0.19430355727672577, -0.008721498772501945, 0.16533666849136353, -0.21307994425296783]
0068 [-0.4345833957195282, 0.19430355727672577, -0.008721498772501945, 0.16533666849136353, -0.21307994425296783]
0069 [-0.4345833957195282, 0.19430355727672577, -0.008721498772501945, 0.16533666849136353, -0.21307994425296783]
0070 [-0.4345833957195282, 0.19430355727672577, -0.008721498772501945, 0.16533666849136353, -0.21307994425296783]
0071 [-0.4345833957195282, 0.19430355727672577, -0.008721498772501945, 0.16533666849136353, -0.21307994425296783]
0072 [-0.4345833957195282, 0.19430355727672577, -0.008721498772501945, 0.16533666849136353, -0.21307994425296783]
0073 [-0.4345824718475342, 0.1943037360906601, -0.00872176606208086, 0.16533659398555756, -0.2130804806947708]
0074 [-0.4345824718475342, 0.1943037360906601, -0.00872176606208086, 0.16533659398555756, -0.2130804806947708]
0075 [-0.43458399176597595, 0.19430390000343323, -0.008721986785531044, 0.16533610224723816, -0.2130793035030365]
0076 [-0.43458399176597595, 0.19430390000343323, -0.008721986785531044, 0.16533610224723816, -0.2130793035030365]
0077 [-0.43458399176597595, 0.19430390000343323, -0.008721986785531044, 0.16533610224723816, -0.2130793035030365]
0078 [-0.43458399176597595, 0.19430390000343323, -0.008721986785531044, 0.16533610224723816, -0.2130793035030365]
0079 [-0.43458399176597595, 0.19430390000343323, -0.008721986785531044, 0.16533610224723816, -0.2130793035030365]
0080 [-0.43458354473114014, 0.19430452585220337, -0.008721709251403809, 0.16533659398555756, -0.21307997405529022]
0081 [-0.43458354473114014, 0.19430452585220337, -0.008721709251403809, 0.16533659398555756, -0.21307997405529022]
0082 [-0.43458354473114014, 0.19430452585220337, -0.008721709251403809, 0.16533659398555756, -0.21307997405529022]
0083 [-0.43458354473114014, 0.19430243968963623, -0.008721593767404556, 0.16533765196800232, -0.21307995915412903]
0084 [-0.43458354473114014, 0.19430243968963623, -0.008721593767404556, 0.16533765196800232, -0.21307995915412903]
0085 [-0.43458354473114014, 0.19430243968963623, -0.008721593767404556, 0.16533765196800232, -0.21307995915412903]
0086 [-0.43458354473114014, 0.19430243968963623, -0.008721593767404556, 0.16533765196800232, -0.21307995915412903]
0087 [-0.43458354473114014, 0.19430243968963623, -0.008721593767404556, 0.16533765196800232, -0.21307995915412903]
0088 [-0.43458354473114014, 0.19430243968963623, -0.008721593767404556, 0.16533765196800232, -0.21307995915412903]
0089 [-0.43458354473114014, 0.19430243968963623, -0.008721593767404556, 0.16533765196800232, -0.21307995915412903]
0090 [-0.43458354473114014, 0.19430243968963623, -0.008721593767404556, 0.16533765196800232, -0.21307995915412903]
0091 [-0.43458354473114014, 0.19430243968963623, -0.008721593767404556, 0.16533765196800232, -0.21307995915412903]
0092 [-0.43458354473114014, 0.19430243968963623, -0.008721593767404556, 0.16533765196800232, -0.21307995915412903]
0093 [-0.43458354473114014, 0.19430243968963623, -0.008721593767404556, 0.16533765196800232, -0.21307995915412903]
0094 [-0.43458354473114014, 0.19430243968963623, -0.008721593767404556, 0.16533765196800232, -0.21307995915412903]
0095 [-0.43458354473114014, 0.19430243968963623, -0.008721593767404556, 0.16533765196800232, -0.21307995915412903]
0096 [-0.43458354473114014, 0.19430243968963623, -0.008721593767404556, 0.16533765196800232, -0.21307995915412903]
0097 [-0.43458354473114014, 0.19430243968963623, -0.008721593767404556, 0.16533765196800232, -0.21307995915412903]
0098 [-0.43458354473114014, 0.19430243968963623, -0.008721593767404556, 0.16533765196800232, -0.21307995915412903]
0099 [-0.43458354473114014, 0.19430243968963623, -0.008721593767404556, 0.16533765196800232, -0.21307995915412903]
0100 [-0.43458354473114014, 0.19430243968963623, -0.008721593767404556, 0.16533765196800232, -0.21307995915412903]
0101 [-0.43458354473114014, 0.19430243968963623, -0.008721593767404556, 0.16533765196800232, -0.21307995915412903]
0102 [-0.43458354473114014, 0.19430243968963623, -0.008721593767404556, 0.16533765196800232, -0.21307995915412903]
0103 [-0.43458354473114014, 0.19430243968963623, -0.008721593767404556, 0.16533765196800232, -0.21307995915412903]
0104 [-0.43458354473114014, 0.19430243968963623, -0.008721593767404556, 0.16533765196800232, -0.21307995915412903]
0105 [-0.43458354473114014, 0.19430243968963623, -0.008721593767404556, 0.16533765196800232, -0.21307995915412903]
0106 [-0.43458354473114014, 0.19430243968963623, -0.008721593767404556, 0.16533765196800232, -0.21307995915412903]
0107 [-0.43458354473114014, 0.19430243968963623, -0.008721593767404556, 0.16533765196800232, -0.21307995915412903]
0108 [-0.43458354473114014, 0.19430243968963623, -0.008721593767404556, 0.16533765196800232, -0.21307995915412903]
0109 [-0.43458354473114014, 0.19430243968963623, -0.008721593767404556, 0.16533765196800232, -0.21307995915412903]
0110 [-0.43458354473114014, 0.19430243968963623, -0.008721593767404556, 0.16533765196800232, -0.21307995915412903]
0111 [-0.43458354473114014, 0.19430243968963623, -0.008721593767404556, 0.16533765196800232, -0.21307995915412903]
0112 [-0.43458354473114014, 0.19430243968963623, -0.008721593767404556, 0.16533765196800232, -0.21307995915412903]
0113 [-0.43458354473114014, 0.19430243968963623, -0.008721593767404556, 0.16533765196800232, -0.21307995915412903]
0114 [-0.43458354473114014, 0.19430243968963623, -0.008721593767404556, 0.16533765196800232, -0.21307995915412903]
0115 [-0.43458354473114014, 0.19430243968963623, -0.008721593767404556, 0.16533765196800232, -0.21307995915412903]
0116 [-0.43458354473114014, 0.19430243968963623, -0.008721593767404556, 0.16533765196800232, -0.21307995915412903]
0117 [-0.43458354473114014, 0.19430243968963623, -0.008721593767404556, 0.16533765196800232, -0.21307995915412903]
0118 [-0.43458354473114014, 0.19430243968963623, -0.008721593767404556, 0.16533765196800232, -0.21307995915412903]
0119 [-0.43458354473114014, 0.19430243968963623, -0.008721593767404556, 0.16533765196800232, -0.21307995915412903]
0120 [-0.43458354473114014, 0.19430243968963623, -0.008721593767404556, 0.16533765196800232, -0.21307995915412903]
0121 [-0.43458354473114014, 0.19430243968963623, -0.008721593767404556, 0.16533765196800232, -0.21307995915412903]
0122 [-0.43458354473114014, 0.19430243968963623, -0.008721593767404556, 0.16533765196800232, -0.21307995915412903]
0123 [-0.43458354473114014, 0.19430243968963623, -0.008721593767404556, 0.16533765196800232, -0.21307995915412903]
0124 [-0.43458354473114014, 0.19430243968963623, -0.008721593767404556, 0.16533765196800232, -0.21307995915412903]
0125 [-0.43458354473114014, 0.19430243968963623, -0.008721593767404556, 0.16533765196800232, -0.21307995915412903]
0126 [-0.43458354473114014, 0.19430243968963623, -0.008721593767404556, 0.16533765196800232, -0.21307995915412903]
0127 [-0.43458354473114014, 0.19430243968963623, -0.008721593767404556, 0.16533765196800232, -0.21307995915412903]
0128 [-0.43458354473114014, 0.19430243968963623, -0.008721593767404556, 0.16533765196800232, -0.21307995915412903]
0129 [-0.43458354473114014, 0.19430243968963623, -0.008721593767404556, 0.16533765196800232, -0.21307995915412903]
0130 [-0.43458354473114014, 0.19430243968963623, -0.008721593767404556, 0.16533765196800232, -0.21307995915412903]
0131 [-0.43458354473114014, 0.19430243968963623, -0.008721593767404556, 0.16533765196800232, -0.21307995915412903]
0132 [-0.43458354473114014, 0.19430243968963623, -0.008721593767404556, 0.16533765196800232, -0.21307995915412903]
0133 [-0.43458354473114014, 0.19430243968963623, -0.008721593767404556, 0.16533765196800232, -0.21307995915412903]
0134 [-0.43458354473114014, 0.19430243968963623, -0.008721593767404556, 0.16533765196800232, -0.21307995915412903]
0135 [-0.43458354473114014, 0.19430243968963623, -0.008721593767404556, 0.16533765196800232, -0.21307995915412903]
0136 [-0.43458354473114014, 0.19430243968963623, -0.008721593767404556, 0.16533765196800232, -0.21307995915412903]
0137 [-0.43458354473114014, 0.19430243968963623, -0.008721593767404556, 0.16533765196800232, -0.21307995915412903]
0138 [-0.43458354473114014, 0.19430243968963623, -0.008721593767404556, 0.16533765196800232, -0.21307995915412903]
0139 [-0.43458354473114014, 0.19430243968963623, -0.008721593767404556, 0.16533765196800232, -0.21307995915412903]
0140 [-0.43458354473114014, 0.19430243968963623, -0.008721593767404556, 0.16533765196800232, -0.21307995915412903]
0141 [-0.43458354473114014, 0.19430243968963623, -0.008721593767404556, 0.16533765196800232, -0.21307995915412903]
0142 [-0.43458354473114014, 0.19430243968963623, -0.008721593767404556, 0.16533765196800232, -0.21307995915412903]
0143 [-0.43458354473114014, 0.19430243968963623, -0.008721593767404556, 0.16533765196800232, -0.21307995915412903]
0144 [-0.43458354473114014, 0.19430243968963623, -0.008721593767404556, 0.16533765196800232, -0.21307995915412903]
0145 [-0.43458354473114014, 0.19430243968963623, -0.008721593767404556, 0.16533765196800232, -0.21307995915412903]
0146 [-0.43458354473114014, 0.19430243968963623, -0.008721593767404556, 0.16533765196800232, -0.21307995915412903]
0147 [-0.4345836937427521, 0.1943046748638153, -0.008721747435629368, 0.1653372347354889, -0.2130795568227768]
0148 [-0.4345836937427521, 0.1943046748638153, -0.008721747435629368, 0.1653372347354889, -0.2130795568227768]
0149 [-0.4345836937427521, 0.1943046748638153, -0.008721747435629368, 0.1653372347354889, -0.2130795568227768]
0150 [-0.4345836937427521, 0.1943046748638153, -0.008721747435629368, 0.1653372347354889, -0.2130795568227768]
0151 [-0.4345836937427521, 0.1943046748638153, -0.008721747435629368, 0.1653372347354889, -0.2130795568227768]
0152 [-0.4345836937427521, 0.1943046748638153, -0.008721747435629368, 0.1653372347354889, -0.2130795568227768]
0153 [-0.4345836937427521, 0.1943046748638153, -0.008721747435629368, 0.1653372347354889, -0.2130795568227768]
0154 [-0.4345836937427521, 0.1943046748638153, -0.008721747435629368, 0.1653372347354889, -0.2130795568227768]
0155 [-0.4345836937427521, 0.1943046748638153, -0.008721747435629368, 0.1653372347354889, -0.2130795568227768]
0156 [-0.4345836937427521, 0.1943046748638153, -0.008721747435629368, 0.1653372347354889, -0.2130795568227768]
0157 [-0.4345836937427521, 0.1943046748638153, -0.008721747435629368, 0.1653372347354889, -0.2130795568227768]
0158 [-0.4345836937427521, 0.1943046748638153, -0.008721747435629368, 0.1653372347354889, -0.2130795568227768]
0159 [-0.4345836937427521, 0.1943046748638153, -0.008721747435629368, 0.1653372347354889, -0.2130795568227768]
0160 [-0.4345836937427521, 0.1943046748638153, -0.008721747435629368, 0.1653372347354889, -0.2130795568227768]
0161 [-0.4345836937427521, 0.1943046748638153, -0.008721747435629368, 0.1653372347354889, -0.2130795568227768]
0162 [-0.4345826506614685, 0.19430312514305115, -0.008722330443561077, 0.16533666849136353, -0.2130790799856186]
0163 [-0.4345826506614685, 0.19430312514305115, -0.008722330443561077, 0.16533666849136353, -0.2130790799856186]
0164 [-0.4345826506614685, 0.19430312514305115, -0.008722330443561077, 0.16533666849136353, -0.2130790799856186]
0165 [-0.4345826506614685, 0.19430312514305115, -0.008722330443561077, 0.16533666849136353, -0.2130790799856186]
0166 [-0.4345826506614685, 0.19430312514305115, -0.008722330443561077, 0.16533666849136353, -0.2130790799856186]
0167 [-0.4345826506614685, 0.19430312514305115, -0.008722330443561077, 0.16533666849136353, -0.2130790799856186]
0168 [-0.4345826506614685, 0.19430312514305115, -0.008722330443561077, 0.16533666849136353, -0.2130790799856186]
0169 [-0.4345826506614685, 0.19430312514305115, -0.008722330443561077, 0.16533666849136353, -0.2130790799856186]
0170 [-0.4345826506614685, 0.19430312514305115, -0.008722330443561077, 0.16533666849136353, -0.2130790799856186]
0171 [-0.4345826506614685, 0.19430312514305115, -0.008722330443561077, 0.16533666849136353, -0.2130790799856186]
0172 [-0.4345826506614685, 0.19430312514305115, -0.008722330443561077, 0.16533666849136353, -0.2130790799856186]
0173 [-0.4345826506614685, 0.19430312514305115, -0.008722330443561077, 0.16533666849136353, -0.2130790799856186]
0174 [-0.4345826506614685, 0.19430312514305115, -0.008722330443561077, 0.16533666849136353, -0.2130790799856186]
0175 [-0.4345826506614685, 0.19430312514305115, -0.008722330443561077, 0.16533666849136353, -0.2130790799856186]
0176 [-0.4345826506614685, 0.19430312514305115, -0.008722330443561077, 0.16533666849136353, -0.2130790799856186]
0177 [-0.4345826506614685, 0.19430312514305115, -0.008722330443561077, 0.16533666849136353, -0.2130790799856186]
0178 [-0.4345839321613312, 0.19430294632911682, -0.008722168393433094, 0.16533693671226501, -0.21308031678199768]
0179 [-0.4345839321613312, 0.19430294632911682, -0.008722168393433094, 0.16533693671226501, -0.21308031678199768]
0180 [-0.4345839321613312, 0.19430294632911682, -0.008722168393433094, 0.16533693671226501, -0.21308031678199768]
0181 [-0.4345839321613312, 0.19430294632911682, -0.008722168393433094, 0.16533693671226501, -0.21308031678199768]
0182 [-0.4345839321613312, 0.19430294632911682, -0.008722168393433094, 0.16533693671226501, -0.21308031678199768]
0183 [-0.4345839321613312, 0.19430294632911682, -0.008722168393433094, 0.16533693671226501, -0.21308031678199768]
0184 [-0.4345839321613312, 0.19430294632911682, -0.008722168393433094, 0.16533693671226501, -0.21308031678199768]
0185 [-0.4345839321613312, 0.19430294632911682, -0.008722168393433094, 0.16533693671226501, -0.21308031678199768]
0186 [-0.4345839321613312, 0.19430294632911682, -0.008722168393433094, 0.16533693671226501, -0.21308031678199768]
0187 [-0.4345839321613312, 0.19430294632911682, -0.008722168393433094, 0.16533693671226501, -0.21308031678199768]
0188 [-0.4345839321613312, 0.19430294632911682, -0.008722168393433094, 0.16533693671226501, -0.21308031678199768]
0189 [-0.4345839321613312, 0.19430294632911682, -0.008722168393433094, 0.16533693671226501, -0.21308031678199768]
0190 [-0.4345839321613312, 0.19430294632911682, -0.008722168393433094, 0.16533693671226501, -0.21308031678199768]
0191 [-0.4345839321613312, 0.19430294632911682, -0.008722168393433094, 0.16533693671226501, -0.21308031678199768]
0192 [-0.4345839321613312, 0.19430294632911682, -0.008722168393433094, 0.16533693671226501, -0.21308031678199768]
0193 [-0.4345839321613312, 0.19430294632911682, -0.008722168393433094, 0.16533693671226501, -0.21308031678199768]
0194 [-0.4345839321613312, 0.19430294632911682, -0.008722168393433094, 0.16533693671226501, -0.21308031678199768]
0195 [-0.4345839321613312, 0.19430294632911682, -0.008722168393433094, 0.16533693671226501, -0.21308031678199768]
0196 [-0.4345839321613312, 0.19430294632911682, -0.008722168393433094, 0.16533693671226501, -0.21308031678199768]
0197 [-0.4345839321613312, 0.19430294632911682, -0.008722168393433094, 0.16533693671226501, -0.21308031678199768]
0198 [-0.4345839321613312, 0.19430294632911682, -0.008722168393433094, 0.16533693671226501, -0.21308031678199768]
0199 [-0.4345839321613312, 0.19430294632911682, -0.008722168393433094, 0.16533693671226501, -0.21308031678199768]
```


## To reproduce

Run the snippet of code provided above

Detailed snippets are available at this [colab notebook](https://colab.research.google.com/drive/19yXek9nx4E2pZTqk8JsS-tAhYjgZ5yGG?usp=sharing)


## Expected behavior

It is expected that an input has same representation irrespective of the batch size used to obtain it. 



Comments:
Hello! Thank you for your report. One question:
- Do you still observe this when you're not using padding? Padding can influence values because of the padding tokens, even with attention masks.

Also, you're using a `_batch_to_device` method, but you should just be able to cast the batch to the device :)
```py
    f = tokenizer(s, padding=True, truncation='longest_first', return_tensors="pt", max_length=128)
    f = f.to(device)
```

Hi @LysandreJik , thanks for `.to(device)` thingy. Regarding the bug, no i am not using padding tokens. For example, a batch two in above experimental setup looks like the following:

```
{'input_ids': tensor([[  101,  2023,  7705, 19421,  7861,  8270,  4667,  2015,  2005,  2169, 7953,  6251,   102],
					  [  101,  2023,  7705, 19421,  7861,  8270,  4667,  2015,  2005,  2169, 7953,  6251,   102]], device='cuda:0'), 
 'attention_mask': tensor([[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1],
						   [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]], device='cuda:0')}
```

I get this issue irrespective of whether I use `padding=True` or `padding=False` 

Okay, I see, thank you! Second question: do you obtain the same if you're running on CPU? I'm currently on a CPU setup and tried running your code, I have exactly the same values for each.

GPUs are known for numerical instabilities, so I wouldn't be surprised if this was the source of the issue!

@LysandreJik I tried on a CPU. I see different values for batch size 1 vs. greater than 1. For latter, all are exactly same. But still i see the following differences:


```
cpu
0001 [-0.43458425998687744, 0.19430384039878845, -0.008721470832824707, 0.16533654928207397, -0.2130793333053589]
0002 [-0.4345836639404297, 0.1943041831254959, -0.008721746504306793, 0.16533654928207397, -0.2130793035030365]
0003 [-0.4345836639404297, 0.1943041831254959, -0.008721746504306793, 0.16533654928207397, -0.2130793035030365]
0004 [-0.4345836639404297, 0.1943041831254959, -0.008721746504306793, 0.16533654928207397, -0.2130793035030365]
0005 [-0.4345828890800476, 0.19430366158485413, -0.008721619844436646, 0.16533659398555756, -0.21308015286922455]
0006 [-0.4345828890800476, 0.19430366158485413, -0.008721619844436646, 0.16533659398555756, -0.21308015286922455]
0007 [-0.4345828890800476, 0.19430366158485413, -0.008721619844436646, 0.16533659398555756, -0.21308015286922455]
0008 [-0.4345828890800476, 0.19430366158485413, -0.008721619844436646, 0.16533659398555756, -0.21308015286922455]
0009 [-0.4345828890800476, 0.19430366158485413, -0.008721619844436646, 0.16533659398555756, -0.21308015286922455]
0010 [-0.4345828890800476, 0.19430366158485413, -0.008721619844436646, 0.16533659398555756, -0.21308015286922455]
0011 [-0.4345828890800476, 0.19430366158485413, -0.008721619844436646, 0.16533659398555756, -0.21308015286922455]
0012 [-0.4345828890800476, 0.19430366158485413, -0.008721619844436646, 0.16533659398555756, -0.21308015286922455]
0013 [-0.4345828890800476, 0.19430366158485413, -0.008721619844436646, 0.16533659398555756, -0.21308015286922455]
0014 [-0.4345828890800476, 0.19430366158485413, -0.008721619844436646, 0.16533659398555756, -0.21308015286922455]
0015 [-0.4345828890800476, 0.19430366158485413, -0.008721619844436646, 0.16533659398555756, -0.21308015286922455]
0016 [-0.4345828890800476, 0.19430366158485413, -0.008721619844436646, 0.16533659398555756, -0.21308015286922455]
0017 [-0.4345828890800476, 0.19430366158485413, -0.008721619844436646, 0.16533659398555756, -0.21308015286922455]
0018 [-0.4345828890800476, 0.19430366158485413, -0.008721619844436646, 0.16533659398555756, -0.21308015286922455]
0019 [-0.4345828890800476, 0.19430366158485413, -0.008721619844436646, 0.16533659398555756, -0.21308015286922455]
```
This is obtained on following system:
- `transformers` version: 4.3.2
- Platform: Darwin-20.3.0-x86_64-i386-64bit
- Python version: 3.7.9
- PyTorch version (GPU?): 1.7.1 (False)
- Tensorflow version (GPU?): not installed (NA)


```
cpu
0001 [-0.43458291888237, 0.19430391490459442, -0.00872180424630642, 0.1653362363576889, -0.21307975053787231]
0002 [-0.4345830976963043, 0.19430406391620636, -0.008721794933080673, 0.16533644497394562, -0.21307967603206635]
0003 [-0.4345830976963043, 0.19430406391620636, -0.008721794933080673, 0.16533644497394562, -0.21307967603206635]
0004 [-0.4345830976963043, 0.19430406391620636, -0.008721794933080673, 0.16533644497394562, -0.21307967603206635]
0005 [-0.4345830976963043, 0.19430406391620636, -0.008721794933080673, 0.16533644497394562, -0.21307967603206635]
0006 [-0.4345830976963043, 0.19430406391620636, -0.008721794933080673, 0.16533644497394562, -0.21307967603206635]
0007 [-0.4345830976963043, 0.19430406391620636, -0.008721794933080673, 0.16533644497394562, -0.21307967603206635]
0008 [-0.4345830976963043, 0.19430406391620636, -0.008721794933080673, 0.16533644497394562, -0.21307967603206635]
0009 [-0.4345830976963043, 0.19430406391620636, -0.008721794933080673, 0.16533644497394562, -0.21307967603206635]
0010 [-0.4345830976963043, 0.19430406391620636, -0.008721794933080673, 0.16533644497394562, -0.21307967603206635]
0011 [-0.4345830976963043, 0.19430406391620636, -0.008721794933080673, 0.16533644497394562, -0.21307967603206635]
0012 [-0.4345830976963043, 0.19430406391620636, -0.008721794933080673, 0.16533644497394562, -0.21307967603206635]
0013 [-0.4345830976963043, 0.19430406391620636, -0.008721794933080673, 0.16533644497394562, -0.21307967603206635]
0014 [-0.4345830976963043, 0.19430406391620636, -0.008721794933080673, 0.16533644497394562, -0.21307967603206635]
0015 [-0.4345830976963043, 0.19430406391620636, -0.008721794933080673, 0.16533644497394562, -0.21307967603206635]
0016 [-0.4345830976963043, 0.19430406391620636, -0.008721794933080673, 0.16533644497394562, -0.21307967603206635]
0017 [-0.4345830976963043, 0.19430406391620636, -0.008721794933080673, 0.16533644497394562, -0.21307967603206635]
0018 [-0.4345830976963043, 0.19430406391620636, -0.008721794933080673, 0.16533644497394562, -0.21307967603206635]
0019 [-0.4345830976963043, 0.19430406391620636, -0.008721794933080673, 0.16533644497394562, -0.21307967603206635]
0020 [-0.4345830976963043, 0.19430406391620636, -0.008721794933080673, 0.16533644497394562, -0.21307967603206635]
0021 [-0.4345830976963043, 0.19430406391620636, -0.008721794933080673, 0.16533644497394562, -0.21307967603206635]
0022 [-0.4345830976963043, 0.19430406391620636, -0.008721794933080673, 0.16533644497394562, -0.21307967603206635]
0023 [-0.4345830976963043, 0.19430406391620636, -0.008721794933080673, 0.16533644497394562, -0.21307967603206635]
```
This is obtained on following system:
- `transformers` version: 4.4.1
- Platform: Linux-4.19.112+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.10
- PyTorch version (GPU?): 1.8.0+cu101 (False)
- Tensorflow version (GPU?): 2.4.1 (False)



Are you getting similar results or are you ending up getting exact same values irrespective of batch size equal or greater than 1 ?

Yes, you're right, the difference is between batch size  == 1 and batch size > 1! Talking about it with team members, we guess it's because the kernels used to compute the results differ according to the dimensions, as they're optimized differently. 

For batch size = 1, the model input would essentially be in one dimension (the vector of tokens), while for batch size > 1, the model input would essentially be in two dimension (an array of tokens).

Imo this is more of a PyTorch issue (if it's an issue in the first place) than a `transformers` issue!

Thanks for the information! 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.