Maybe the way SequenceClassification Model calculates the last non-pad token is not reasonable.
Author: liangxuZhang
Pull request: no
Labels: name: bug

### System Info

None

### Who can help?

@ArthurZucker 

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

SequenceClassification Model  finds the last token that is not a padding token in each row  by cal the first position of pad token when a `pad_token_id` is defined in the configuration. 
the code in https://github.com/huggingface/transformers/blob/main/src/transformers/models/llama/modeling_llama.py#L951

```
sequence_lengths = torch.eq(input_ids, self.config.pad_token_id).int().argmax(-1) - 1
```

When pad_token is eos token, what is calculated using this method will be the position of the first eos token. However, in some LLM templates, eos is often added at the end of the prompt. Also, when a model such llama does not have a default pad token, using eos as a pad token is a common practice.

For example, in llama, eos token is 


```
from transformers import AutoModelForSequenceClassification, AutoTokenizer
model = AutoModelForSequenceClassification.from_pretrained("meta-llama/Llama-3.2-1B-Instruct", num_labels=1)
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-3.2-1B-Instruct")
tokenizer.eos_token, model.config.pad_token_id, tokenizer.pad_token
# ('
', None, None)
```
Using chat template to encode QA pairs we can get

```
prompt = "what is 1+1?"
response1 = "1+1=2"
response2 = "1+1=3"

conv1 = [{"role": "user", "content": prompt}, {"role": "assistant", "content": response1}]
conv2 = [{"role": "user", "content": prompt}, {"role": "assistant", "content": response2}]

conv1_tokenized = tokenizer.apply_chat_template(conv1, tokenize=True, return_tensors="pt")
conv2_tokenized = tokenizer.apply_chat_template(conv2, tokenize=True, return_tensors="pt")

# conv1_tokenized(input_ids)
# (tensor([[128000, 128006,    882, 128007,    271,  12840,    374,    220,     16,
#             10,     16,     30, 128009, 128006,  78191, 128007,    271,     16,
#              10,     16,     28,     17, 128009, 128006,  78191, 128007,    271]]),
# conv1_tokenized(tokens)
# '

user
\n\nwhat is 1+1?

assistant
\n\n1+1=2

assistant
\n\n')
```
In Llama chat template, eos_token is added on the end of prompt. If we do not pad the input_ids and don't set pad_token_id, the score is correct. 

```
with torch.no_grad():
    score1 = model(conv1_tokenized).logits[0][0].item()
    score2 = model(conv2_tokenized).logits[0][0].item()
print(f"Score for response 1: {score1}")
print(f"Score for response 2: {score2}")
# Score for response 1: 1.7297050952911377
# Score for response 2: 1.43972647190094
```

If we set pad_token_id = eos_token_id, we get the same score for different QA pairs with the same prompt

```
model.config.pad_token_id = tokenizer.eos_token_id
with torch.no_grad():
    score1 = model(conv1_tokenized).logits[0][0].item()
    score2 = model(conv2_tokenized).logits[0][0].item()
print(f"Score for response 1: {score1}")
print(f"Score for response 2: {score2}")
# Score for response 1: -1.857212781906128
# Score for response 2: -1.857212781906128
```

This is because the score of SequenceClassification Model is the last non-pad token's logits，and the last non-pad token is the token before the first eos token. This is incorrect especially when training reward models with preference pairs that have the same prompt.

The complete recurrence script is as follows

```
from transformers import AutoModelForSequenceClassification, AutoTokenizer
model = AutoModelForSequenceClassification.from_pretrained("meta-llama/Llama-3.2-1B-Instruct", num_labels=1)
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-3.2-1B-Instruct")
prompt = "what is 1+1?"
response1 = "1+1=2"
response2 = "1+1=3"

conv1 = [{"role": "user", "content": prompt}, {"role": "assistant", "content": response1}]
conv2 = [{"role": "user", "content": prompt}, {"role": "assistant", "content": response2}]

conv1_tokenized1 = tokenizer.apply_chat_template(conv1, tokenize=True, return_tensors="pt")
conv2_tokenized1 = tokenizer.apply_chat_template(conv2, tokenize=True, return_tensors="pt")
model.config.pad_token_id = tokenizer.eos_token_id
with torch.no_grad():
    score1 = model(conv1_tokenized1).logits[0][0].item()
    score2 = model(conv2_tokenized1).logits[0][0].item()
print(f"Score for response 1: {score1}")
print(f"Score for response 2: {score2}")
```

### Expected behavior


Maybe we should prioritize using attention mask to calculate  the  position of the last non-pad token.

Comments:
This bug makes sense to me! I think very few people are finetuning chat models for `SequenceClassification`, but that doesn't mean we shouldn't support it. I agree that indexing on the EOS token should only be used when no attention mask is available.

Would you be willing to make a PR to modify the `SequenceClassification` behaviour?

@Rocketknight1  Currently, there is a common scenario that  finetunes chat models for `SequenceClassification`. Since the score layer of SequenceClassification is very convenient for generating sentence rewards, TRL uses `SequenceClassification` when training the reward model(`https://github.com/huggingface/trl/blob/main/examples/scripts/reward_modeling.py`). 

Sure, I would like to provide a PR to fix this, but there are so many models with SequenceClassification, should I modify all of them? This seems like a lot of work.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Hi @liangxuZhang I apologize for the delay over the holidays! We have a PR open to fix this issue now at https://github.com/huggingface/transformers/pull/35911