batched feature extraction pipeline for GPT-style models
Author: davidegraff
Pull request: no

### Feature request

it would be nice to support feature extraction of batched input for GPT-style models using `Pipeline`s

### Motivation

I'm currently trying to generate encodings of a large number of sentences using LLMs. I.e.,:
```python
classification_token_idx: int
fe = pipeline("feature-extraction", model="some-LLM", framework="pt", return_tensors=True)
inputs = [...]
output = fe(inputs)
H = torch.stack([x[0, classification_token_idx, :] for x in output])
```
where depending on whether I'm using a BERT-style or GPT-style model, `classification_token_idx` will be either 0 or -1, respectively. My use-case can greatly benefit from batching, but the adapted snippet no longer works for GPT-style models:
```python
...
output = fe(inputs, batch_size=BATCH_SIZE)
H = torch.stack([x[0, classification_token_idx, :] for x in output])
```
In a batch of sequences, the 0th index of a sequence will always be the `[CLS]` token regardless of padding. However, the last index of a sequence in a padded batch of sequences will most likely be a `[PAD]` token rather than the true last token of the sequence. Using the `Pipeline` interface with a GPT-style model makes it non-trivial to extract features *and* take advantage of input batching, leaving users with three options:
1. do not batch. Possibly much slower, but reliable
2. do not use a `Pipeline`. More robust, but fairly cumbersome to implement and will likely be repeated across most users
3. implement a custom `Pipeline`. The most "elegant" solution (IMO), but one that should arguably be in the huggingface library (and is the point of this feature request.)

### Your contribution

I doubt I'm the person for the job.

Comments:
cc @Narsil 

This one is tricky.
In general GPT-like models should pad on the left, not on the right, meaning your snippet **should** work.
However, there's no telling if everything is properly configured simply. (The pipeline does whatever the config is set to do, it doesn't try to do reasoning on it).

In theory, everything should be quite transparent if the `padding_side` is properly set on the tokenizer.
Would that solve your issue ?
If you're unsure, maybe setting `classification_token_ids = 0 if fe.tokenizer.padding_side="right" else -1` could do the trick.

Maybe if you have a specific model where it doesn't work properly I could take a look ?

Note: I wasn't aware GPT-like models were good for document embedding.

Hi, @davidegraff this might be helpful, 

```
import torch
from transformers import pipeline

ipts = ["Hi I am human.", "The sky", "hello there"]
fe = pipeline(task="feature-extraction",
              model="gpt2",
              framework="pt",
              return_tensors=True)

# Since gpt2 doesn't have a pad_token
if not fe.tokenizer.special_tokens_map.get("pad_token"):
    pad_token = {"pad_token":"
"}
    fe.tokenizer.add_special_tokens(pad_token)
    fe.model.resize_token_embeddings(len(fe.tokenizer))

# Make sure the padding_side is 'left' (if you open gpt2tokenizer you will find that by default
# the padding_side is 'right')
fe.tokenizer.padding_size = "left" #For BERT like models use "right"

# get the outputs
opts = fe(ipts, batch_size=3)

classification_token_idx = -1 #For BERT like models use 0(if you want to use the embeddings of [CLS] token)
H = torch.stack([x[0, classification_token_idx, :] for x in opts])

```

To see if batch_size if working or not I ran,

```
%%timeit -n 100
opts = fe(ipts)
```
>> 143 ms ± 10.8 ms per loop (mean ± std. dev. of 7 runs, 100 loops each)

```
%%timeit -n 100
opts = fe(ipts, batch_size=3) 
```
>> 86.6 ms ± 10.8 ms per loop (mean ± std. dev. of 7 runs, 100 loops each)

I hope it helps.

I see! Thanks for the example @susnato!

re @Narsil:
My use-case is a little specialized- I'm looking specifically at LLMs trained on chemical corpora. Frey et al. [[1]] show that you can use the "embeddings" from a GPT model trained on molecules in an unsupervised fashion as molecular representations (Fig. 7). They made their model available on the HF hub [[2]], but it seems like there might be an issue with padding based on your explanation:
```python
>>> featurizer = pipeline(
    "feature-extraction",
    model="ncfrey/ChemGPT-1.2B",
    framework="pt",
    return_tensors=True
)
>>> featurizer.tokenizer.padding_side
"right"
```
The original data preparation code also makes no mention of left-wise padding [[3]]. Loading the individual tokenizer (via `AutoTokenizer`) results in the same thing. Does this mean it was trained incorrectly? Or is this just something I have to be aware of when loading `Tokenizer`s (by adding `tokenizer.padding_side = "left"` for GPT-style models)?

[1]: https://doi.org/10.26434/chemrxiv-2022-3s512
[2]: https://huggingface.co/ncfrey/ChemGPT-1.2B
[3]: https://github.com/ncfrey/litmatter/blob/main/lit_data/lm_data.py#L31

> Does this mean it was trained incorrectly?

I'm not sure how we train "correctly" so it'd be hard to train "incorrectly". Joke aside, a model is trained a certain way, it's up to the inference to understand how it was done and was it acceptable or not within the framework of how it was trained.

Padding side in training shouldn't matter in inference, especially for causal LM since, they're supposed to ignore the padding anyway. 
Now that's theory, in practice I would definitely run some tests to make sure what's supposed to be actually is correct.

But, you could always just override the `padding_side` and see how are the results compared to the non batched, non overridden ones on a subset of examples you know the answer for. That would be my first step at least.

To override.

```python
pipe = pipeline("feature-extraction", model="cfrey/ChemGPT-1.2B",
    framework="pt",
    return_tensors=True)
pipe.tokenizer.padding_side = "left"

for out in pipe(...):
     print(out)
```

For instance.

Thanks for taking the time to explain all this, I really appreciate it!

The original paper is sparse on details, so I'm not really sure what the authors are doing when they (1) generate encodings and (2) how (or if) they pad during inference. In the absence of these details, I guess I'm trying to take a principled approach to generate these encodings:
1) a sanity check to make sure no funny business is going on
```python
featurizer = pipeline(
    "feature-extraction", model="ncfrey/ChemGPT-1.2B", framework="pt", return_tensors=True, 
)
featurizer.tokenizer.add_special_tokens({'pad_token': '[PAD]'})
sfs = [
    '[C][C][N][=C][Branch1_1][O][N][C][C][C][C][C][C][C][Ring1][Branch1_3][S][C][Expl=Ring1][=N][C][Branch1_2][C][=O][O-expl]',
    '[C][N][Branch1_1][Branch2_2][C][C][C][C][C][C][Ring1][Branch1_2][S][Branch1_2][C][=O][Branch1_2][C][=O][C][=C][C][=C][Branch2_1][Ring1][Branch2_2][N][C][Branch1_2][C][=O][C][C][N][C][Branch1_2][C][=O][C@Hexpl][C][C][=C][C][C@@Hexpl][Ring1][Branch1_2][C][Ring1][Branch2_3][=O][C][=C][Ring2][Ring1][Branch1_2]',
    '[C][C@Hexpl][C][C][C@Hexpl][Branch2_1][Ring1][Branch1_3][NH+expl][C][C][C][C@Hexpl][Branch1_1][=N][C@Hexpl][Branch1_1][C][O][C][=N][C][=C][N][Ring1][Branch1_1][C][C][Ring1][=C][C][Ring2][Ring1][Ring1]',
    '[N][/C][Branch2_1][Ring1][Ring2][C][N][C][Branch1_2][C][=O][C@@Hexpl][C][C][=C][C][=C][C][=C][Ring1][Branch1_2][S][Ring1][Branch2_2][=N][\\O]'
]
featurizer.tokenizer.padding_side = 'right'
X_unpadded_r = torch.stack([H[0, -1, :] for H in featurizer(sfs)])
featurizer.tokenizer.padding_side = 'left'
X_unpadded_l = torch.stack([H[0, -1, :] for H in featurizer(sfs)])
torch.allclose(X_unpadded_r, X_unpadded_l)
# True
```
2) now seeing the effects of batching
```python
featurizer.tokenizer.padding_side = 'right'
X_padded_r = torch.stack([H[0, -1, :] for H in featurizer(sfs, batch_size=4)])
featurizer.tokenizer.padding_side = 'left'
X_padded_l = torch.stack([H[0, -1, :] for H in featurizer(sfs, batch_size=4)])
torch.allclose(X_padded_r, X_padded_l)
# False
torch.allclose(X_unpadded_r, X_padded_l), torch.allclose(X_unpadded_r, X_padded_r)
# (False, False)
```
so while it's expected that left vs. right padding produces different results if we take the same embedding (i.e., the last token), it's is surprising that _neither_ of these is the same as the unpadded results. The simple answer in this situation is likely just "then don't batch," but there are significant performance gains to be had when utilizing batching. Do you have any advice here? Thanks again!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.