when model.generate with num_beams=2 and num_return_sequences=2,the output seqs are different from input_ids of stopping_criteria
Author: DavideHe
Pull request: no
Labels: name: WIP

name: bug

name: Generation

### System Info

- `transformers` version: 4.45.2
- Platform: Linux-5.10.134-13.an8.x86_64-x86_64-with-glibc2.35
- Python version: 3.10.12
- Huggingface_hub version: 0.24.6
- Safetensors version: 0.4.4
- Accelerate version: 1.0.0
- Accelerate config: 	not found
- PyTorch version (GPU?): 2.5.0a0+872d972e41.nv24.08 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: <fill in>
- Using GPU in script?: <fill in>
- GPU type: NVIDIA H800

### Who can help?

?

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

code:
```python
from transformers import AutoConfig, AutoModel,AutoModelForSequenceClassification,AutoModelForCausalLM,AutoTokenizer
import sys 
import torch 
import json

from transformers import StoppingCriteria, StoppingCriteriaList

token_ids = []
class StopOnToken(StoppingCriteria):
    def __init__(self, stop_token_ids):
        self.stop_token_ids = stop_token_ids

    def __call__(self, input_ids, scores, **kwargs):
        token_ids.append(input_ids[:,-1])
        return any([inp[-1].item() in self.stop_token_ids for inp in input_ids])
    
model_name_or_path = "Qwen/Qwen2.5-3B-Instruct"
model = AutoModelForCausalLM.from_pretrained(model_name_or_path,trust_remote_code=True,device_map="cuda")
tokenizer = AutoTokenizer.from_pretrained(model_name_or_path,trust_remote_code=True, use_fast=False)
tokenizer.padding_side = "left"

with torch.no_grad():
    text = "A regular polygon has exterior angles each measuring 15 degrees. How many sides does the polygon have?Think step by step:" 
    text = [tokenizer.apply_chat_template([{"role":"user","content":text}],tokenize=False,add_generation_prompt=True)]
    print("========>text:",text)
    tokenizerd = tokenizer(text,return_tensors="pt",padding=True,add_special_tokens=False).to(device="cuda")
    stopping_criteria = StoppingCriteriaList([StopOnToken([tokenizer.eos_token_id])])
    output = model.generate(**tokenizerd,num_beams=2,max_new_tokens=20,num_return_sequences=2,stopping_criteria=stopping_criteria)

    output = output[:,tokenizerd["input_ids"].shape[1]:]
    print(output)
    ans = tokenizer.batch_decode(output, skip_special_tokens=False)
    print("=================ans======================")
    for i,ans_i in enumerate(ans):
        print(f"ans [{i}]:",json.dumps(ans_i,ensure_ascii=False))

print("output ids:",output)
token_ids = torch.stack(token_ids,dim=1)
print("stopping_criteria output ids:",token_ids)
ans = tokenizer.batch_decode(token_ids, skip_special_tokens=False)
print("=================stopping_criteria ans======================")
for i,ans_i in enumerate(ans):
    print(f"ans [{i}]:",json.dumps(ans_i,ensure_ascii=False))
```

### Expected behavior

logs
```
========>text: ['
system\nYou are Qwen, created by Alibaba Cloud. You are a helpful assistant.
\n
user\nA regular polygon has exterior angles each measuring 15 degrees. How many sides does the polygon have?Think step by step:
\n
assistant\n']
Starting from v4.46, the `logits` model output will have the same type as the model (except at train time, where it will always be FP32)
tensor([[ 1249,  8253,   279,  1372,   315, 11067,   315,   264,  5792, 29372,
          2661,   429,  1817, 27263,  9210, 10953,   220,    16,    20, 12348],
        [ 1249,  8253,   279,  1372,   315, 11067,   315,   264,  5792, 29372,
          1380,  1817, 27263,  9210, 10953,   220,    16,    20, 12348,    11]],
       device='cuda:0')
=================ans======================
ans [0]: "To determine the number of sides of a regular polygon given that each exterior angle measures 15 degrees"
ans [1]: "To determine the number of sides of a regular polygon where each exterior angle measures 15 degrees,"
output ids: tensor([[ 1249,  8253,   279,  1372,   315, 11067,   315,   264,  5792, 29372,
          2661,   429,  1817, 27263,  9210, 10953,   220,    16,    20, 12348],
        [ 1249,  8253,   279,  1372,   315, 11067,   315,   264,  5792, 29372,
          1380,  1817, 27263,  9210, 10953,   220,    16,    20, 12348,    11]],
       device='cuda:0')
stopping_criteria output ids: tensor([[ 1249,  8253,   279,  1372,   315, 11067,   315,   264,  5792, 29372,
          2661,   429,  1817, 27263,  9210, 10953,   220,    16,    20, 12348],
        [39814,  1477,   279,  1372,   315, 11067,   315,   264,  5792, 29372,
          1380,  1817, 27263,  9210, 10953,   220,    16,    20, 12348,    11]],
       device='cuda:0')
=================stopping_criteria ans======================
ans [0]: "To determine the number of sides of a regular polygon given that each exterior angle measures 15 degrees"
ans [1]: "Sure find the number of sides of a regular polygon where each exterior angle measures 15 degrees,"
```
we can find the output tokens of generate return 2nd seq  are different from tokens of StopOnToken get from input_ids 2nd seq .
why? or bugs? 

Comments:
You can set `text='return 0.'` and you will get:
```
=================ans======================
ans [0]: "0.

















"
ans [1]: "I'm sorry, but I'm Qwen, not Alibaba Cloud. I don't have the capability"
...
=================stopping_criteria ans======================
ans [0]: "0. sorry, but I can'twen, not a Cloud. I don't have the capability"
ans [1]: "I'm0 for I as'm Q provide the a Alibaba language, As'm not perform a ability"
```
That means using tokens in `StoppingCriteria` for decoding is meaningless.

I used `Qwen2.5-0.5B-Instruct`. You can set `text = "return letter a." ` for reproduciton if using `Qwen2.5-3B-Instruct`.

> I used `Qwen2.5-0.5B-Instruct`. You can set `text = "return letter a." ` for reproduciton if using `Qwen2.5-3B-Instruct`.

I find `StoppingCriteria ` return input_ids is the same as generation return when `num_beams=1`. but it doesn't work when `num_beams>1`. And when `num_beams>1 `, what does pass to `stopping_criteria `? If it is random, `stopping_criteria output ids` decode is not messy code.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

cc @gante @zucchini-nlp 

Same as https://github.com/huggingface/transformers/issues/34843

So, the issue is that beam generation currently doesn't check any other stopping criteria passed except for the model's eos token id from config. We realize that this is sub-optimal and the beam search needs major refactoring. The general plan to refactor generation related code including beam search is in https://github.com/huggingface/transformers/issues/30810. @gante can share more about the future plans when he's back from vacation :)

In the meanwhile adding WIP tag so the issue doesn't get closed

Hi there 👋 

#35802 not only fixes it, but also speeds up beam search 🤗