Using pooler/hidden states output of an AutoModel vs. XForSequenceClassification for training another classifier?
Author: workworkwc3
Pull request: no

I have a finetuned RobertaForSequenceClassification model that I have already trained and saved to disk. I now want to load it, and instead of using it for classification tasks, extract the embeddings it generates and outputs, or "pooled/pooler output". From my understanding, I can load the model using X.fromPretrained() with "output_hidden_states=True". If I load the model using:

`model = AutoModel.from_pretrained(model_dir, num_labels=2, output_hidden_states=True)`

I get the warning about weights not being initialized: "roberta.pooler.dense.weight", "roberta.pooler.dense.bias", and understand they are randomly initialized on loading the model. Also from my understanding, I can still use this model to generate what I believe to be the pooler output by using something like:

`pooler_output = model(input_ids, attention_mask=attention_mask)`

Since the AutoModel does not load the weights/bias from the saved model, it leads to random results that I don't want.

I've now read two closed issues [[1](https://github.com/huggingface/transformers/issues/1328#issuecomment-534956703), [2](https://github.com/huggingface/transformers/issues/7540#issuecomment-704155218)] that gave me some insight on how to generate this pooler output from XForSequenceClassification models. I don't understand that from the first issue, the poster  "concatenates the last four layers" by using the indices -4 to -1 of the output. In my mind this means the last index of the hidden state's output of an XForSequenceClassification model is what is needed to retrieve pooler output. However, in the second link, and in the [RobertaClassificationHead code](https://github.com/huggingface/transformers/blob/master/src/transformers/models/roberta/modeling_roberta.py#L1445), they use the hidden representation of the [CLS] token only, something like:

```
outputs = model(input_ids, attention_mask=attention_mask)
last_hidden_state = outputs[0]
cls_representation = last_hidden_state[:,0,:]
```

However, when I do this by loading and using a RobertaForSequenceClassification model and instead using outputs[1] (since I do not use labels for the forward function), all rows in the resulting Tensor are equal. I am not sure if this is intended, but if so, then how is this useful for classification if I want to use this as input into another classifier as mentioned by the poster of the second link and the RobertaClassificationHead code? I am not sure if I missed something or there's something wrong with my trained model, but from what the first poster explains and how they take the last four layers, do I want to take outputs[-1][:,0,:] instead, or is outputs[0] actually the first embedding layer in XForSequenceClassification models rather than the final layer?

Comments:
Hello, thanks for opening an issue! We try to keep the github issues for bugs/feature requests.
Could you ask your question on the [forum](https://discuss.huggingface.co) instead?

Thanks!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.