[Bug] RuntimeError: dtype mismatch in _group_beam_search with bfloat16/fp16 models
Author: anmorgunov
Pull request: no
Labels: name: bug

### System Info

python 3.12, transformers >4.45

### Who can help?

@Cyrilvallez @gante @zucchini-nlp 

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

a minimal reproduction repository has been created here: [anmorgunov/huggingface-group-beam-search-bug](https://github.com/anmorgunov/huggingface-group-beam-search-bug)

the issue was introduced in `v4.45.0` and persists. to reproduce, run the `bfloat-gen.py` script from the repository.

1.  **this will FAIL**
```bash
uv run --with transformers==4.45.0 --with torch --with accelerate bfloat-gen.py
```

2.  **this will WORK**
```bash
uv run --with transformers==4.44.2 --with torch --with accelerate bfloat-gen.py
```

the crash reliably occurs when using group beam search (`num_beam_groups > 1`) with `output_scores=True` on a model loaded in a half-precision format like `bfloat16`.

### Expected behavior

the generation process should complete successfully without raising a `RuntimeError`. the code should handle the internal upcasting and downcasting of dtypes correctly, returning the generated sequences and scores.

it shouldn't crash.

## The Root Cause

The underlying issue is a `dtype` mismatch during an in-place tensor assignment.

1.  the generation code starting from v4.45.0 upcasts model logits from `bfloat16` to `float32` before a `log_softmax` operation.
2.  this `float32` `dtype` propagates through the score calculation.
3.  the code then attempts to assign these `float32` scores back into a `bfloat16` tensor (`processed_score`) without downcasting first.
4.  this triggers the `RuntimeError: Index put requires the source and destination types match...` because pytorch's `index_put_` kernel does not perform implicit casting.

### Evolution of code

the bug was unintentionally introduced in v4.45.0 and can be traced through the following changes:

- v4.44.2 (and earlier): No upcasting.

the logits were used in their native dtype.
```py
# file: src/transformers/generation/utils.py
# line 3742
next_token_logits = outputs.logits[batch_group_indices, -1, :]
```

- v4.45.0: Upcasting is introduced.

.float() is added to the logit calculation for precision, creating the float32 tensor that eventually causes the conflict.
```py
# file: src/transformers/generation/utils.py
# line 3762
# .float() is needed to retain precision for later logits manipulations
next_token_logits = outputs.logits[:, -1, :].clone().float()
```

- v4.51.0: Code is refactored.
the logic is cleaned up from .float() to a more explicit .to() call, but the intentional upcasting to float32 remains.
```py
(3673) next_token_logits = outputs.logits[batch_group_indices, -1, :].to(
                    dtype=torch.float32, device=input_ids.device
                )
```

Comments:
I was about to make a PR suggesting a fix, which is really a one-liner, but it seems like group beam search will be deprecated in the next release. I created a PR on the community repo.

https://huggingface.co/transformers-community/group-beam-search/discussions/1

@anmorgunov thank you for the fix, merged the hub PR 🤗