How to get the last 4 Hidden states from the feature extraction pipeline
Author: Luke-4
Pull request: no

I have defined a pipeline for Feature extraction
```
# Create the pipeline
p = pipeline(
    task="feature-extraction",
    tokenizer="microsoft/biogpt",
    model="microsoft/biogpt",
    framework="pt",
    device=0
)
bio_gpt = AutoModel.from_pretrained("microsoft/biogpt", output_hidden_states= True)
bio_gpt = bio_gpt.to(device)
```

and I want to extract the embeddings of the last token of the last hidden state, and the Average Pooling of the last 4 layers using the pipeline approach I am doing it like this

_Last token of the last hidden state:_

```
def extract_last_token(last_hidden_states):
    last_hidden_states = np.array(last_hidden_states)
    return last_hidden_states[:,-1,:]

# Process the data using the pipeline
results = p([row["text"] for _, row in df2.iterrows()])

# Extract the last token of the last hidden state
embeddings = [extract_last_token(hidden_state) for hidden_state in results]

# Create a DataFrame to store the results
df2["embeddings2"] = embeddings
```
_Average pooling of the last 4 layers:_
```
def mean_pooling(last_hidden_states, ):
    last_4_layers = last_hidden_states[-4:]  # Consider the last 4 layers
    return np.mean(last_4_layers, axis=1)

# Process the data using the pipeline
results = p([row["text"] for _, row in df2.iterrows()])

features = np.squeeze(results)

print(features.shape)
# Perform mean pooling on the last hidden states
embeddings = [mean_pooling(hidden_state) for hidden_state in results]

# Create a DataFrame to store the results
df2["embeddings4"] = embeddings
```
The issues are:

1. When I extract the embeddings of the 4 last layers or the 12 last layers the embeddings are always the same

![image](https://github.com/huggingface/transformers/assets/138615931/70c265aa-4182-4265-bb22-ddc197388c03)

2. The embeddings of the last token of the last hidden state are different from the same embeddings using the "manual" method

![image](https://github.com/huggingface/transformers/assets/138615931/a7b9b629-9c65-4b76-b669-89d9e64103de)

Weardly in the above picture the 2 of the embeddings are the same but opposite row ids, this indicates another problem I don't see it if you can spot this I appreciate it.

Here is the code of how I did the manual version
```
output = bio_gpt(**model_inputs)

# Get the last state
last_state = output.last_hidden_state

cls_embeddings = last_state[:, -1, :]

# Print the last state
print(cls_embeddings)

# Assign cls_embeddings to "embeddings4" column in df2
df2["embeddings_manual"] = [cls_embeddings[i].cpu().detach().numpy() for i in range(len(df2))]
```

Comments:
Hi, could you also provide the data `df2` (or another version of it if privacy is concerned).

Thanks.

> Hi, could you also provide the data `df2` (or another version of it if privacy is concerned).
> 
> Thanks.

Sure, its just a text and the label
![image](https://github.com/huggingface/transformers/assets/138615931/85f312c0-5112-4b7b-8c29-36eec2545a34)






@Luke-4 

Not as an image please. Make it something that can be used to run the code snippet directly 🙏 


> @Luke-4
> 
> Not as an image please. Make it something that can be used to run the code snippet directly 🙏

here hope this works:
https://drive.google.com/drive/folders/186rEP0ZMYc3tjR_EKBYNhYEx9sUtmSTj?usp=sharing






I haven't look in full details. However, your input to `mean_pooling` (i.e. each element of the output from `p`) seems to have `[batch_dim, seq_len, hideen_dim]`. The `batch_dim` here is just `1`

When you do `last_hidden_states[-4:]` inside `mean_pooling`, it is actually the same element as `last_hidden_states`, as your are taking the last 4 elements along batch dimension (and not the different layers!). When you do `np.mean(..., axis=1)`, it's actually mean along the sequence dimension, and get a shape of `[batch_dim=1, hidden_dim]`.

This doesn't corresponds to what you describe that you want to get mean along the last 4 layers.
However, I am not sure if the `feature extraction pipeline` allow to get all hidden states (from all layers) - probably yes.

Could you verify first, please?



```python
r = p(["I love dog", "I love cat too", "I love cat that meow meow a lot"])

import numpy as np

print(len(a))

print(len(a[0]))
print(len(a[1]))
print(len(a[2]))

print(len(a[0][0]))
print(len(a[1][0]))
print(len(a[2][0]))

print(np.array(a[0][0]).shape)
print(np.array(a[1][0]).shape)
print(np.array(a[2][0]).shape)
```
gives
```bash
3
1
1
1
5
5
6
(5, 1024)
(5, 1024)
(12, 1024)
```

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.