FeatureExtractionPipeline for Causal model gives unexpected results
Author: tonifuc3m
Pull request: no

Causal model embeddings should not depend on right tokens. 

For example, the word `man` should have the same GPT2 embedding in these two sentences: `The man is successfull`, `The man has for sons and takes care of the house` because the left context is the same: `The`. However, when using the `FeatureExtractionPipeline` there are small differences in the embedding values.

### System Info

- `transformers` version: 4.24.0
- Platform: Linux-5.15.90.1-microsoft-standard-WSL2-x86_64-with-glibc2.35
- Python version: 3.10.6
- Huggingface_hub version: 0.10.1
- PyTorch version (GPU?): 1.13.0+cu117 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: NO
- Using distributed or parallel set-up in script?: NO

### Who can help?

- @Narsil 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

```
from transformers import AutoTokenizer, pipeline

# Load pipeline and tokenizer for GPT2
feature_extraction = pipeline('feature-extraction', model="gpt2")
tokenizer = AutoTokenizer.from_pretrained("gpt2")

# Define test sentences
sentences_en = ["The man is successfull", 
        "The man is not successfull",
        "The man has four sons and takes care of the house",
        "The man has a brilliant professional career"]

# Get contextual embeddings of all subtokens of all sentences
embeddings = feature_extraction(sentences_en)

# Get embeddings of word "man" -> in our examples is always in position 1 for GPT2
man_embeddings = []
for emb in embeddings:
  man_embeddings.append(emb[0][1])

# Check if embeddings are equal
print("1st and 2nd embedding are equal?", man_embeddings[0] == man_embeddings[1])
print("1st and 3rd embedding are equal?", man_embeddings[0] == man_embeddings[2])
print("1st embedding, first values:", man_embeddings[0][0:3])
print("3rd embedding, first values:", man_embeddings[2][0:3])
```

Results:
`1st and 2nd embedding are equal? True`
`1st and 3rd embedding are equal? False`
`1st embedding, first values: [0.06501030921936035, 0.2574709951877594, -0.8581657409667969]`
`3rd embedding, first values: [0.065010204911232, 0.25747090578079224, -0.8581656217575073]`

[Colab](https://colab.research.google.com/drive/16QrjVWy65mfZ5MprV26kz_9AcB-YLSCe?pli=1#scrollTo=N3Xyo9XeG_EY), to reproduce the results. 




### Expected behavior

I would expect that the four embeddings of the word `man` are exactly equal:

`1st and 2nd embedding are equal? True`
`1st and 3rd embedding are equal? True`

Comments:
This is entirely possible, and totally not necessarily a bug.
kernels can vary based on tensor shapes for efficiency leading to small but existing differences.
You can just adding padding for longer and longer sequences you will observe the same.

Just changing torch version can cause this too.

Just use higher tolerations to ignore that error.