num_assistant_tokens not configured in assistant model
Author: geoffrey-young
Pull request: no
Labels: name: WIP

name: bug

### System Info

`num_assistant_tokens` and other parameters intended for the assistant model are not properly passed down, leaving the assistant model with default values.

```python
outputs = model.generate(**inputs, assistant_model=assistant_model, num_assistant_tokens=5)
print(f'num_assistant_tokens: {assistant_model.generation_config.num_assistant_tokens}')
```

This behavior is present both in the official 4.56.0 as well as main (currently 4.57.0.dev0/bb45d36)

Fix in PR https://github.com/huggingface/transformers/pull/40740

- `transformers` version: 4.57.0.dev0
- Platform: macOS-15.6.1-arm64-arm-64bit
- Python version: 3.12.11
- Huggingface_hub version: 0.34.4
- Safetensors version: 0.6.2
- Accelerate version: 1.10.1
- Accelerate config: 	not found
- DeepSpeed version: not installed
- PyTorch version (accelerator?): 2.8.0 (NA)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?:?

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```python
outputs = model.generate(**inputs, assistant_model=assistant_model, num_assistant_tokens=5)
print(f'num_assistant_tokens: {assistant_model.generation_config.num_assistant_tokens}')
```

see also this Collab: https://colab.research.google.com/drive/1BIY6yklrsarrPmXWrciV1chAqV5I9-wo?usp=sharing

### Expected behavior

`num_assistant_tokens` should be `5` not the default `20`

Comments:
Hi @geoffrey-young,

This PR addresses #40739 / #40740.

Ensures num_assistant_tokens and related assistant parameters are propagated correctly into assistant_model.generation_config.

Adds regression coverage for strict overlay behavior, including concurrency probes to confirm no config drift.

Hardens _prepare_special_tokens against meta tensors by avoiding .item()/.cpu().numpy() on meta and rebuilding safe tensors on the target device.

All new and existing tests pass locally and in CI (Python 3.10 / 3.12).

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

the reported issue ([40739](https://github.com/huggingface/transformers/issues/40739)) is still outstanding, and is still demonstrable in the Collab notebook.

the PR ([40740](https://github.com/huggingface/transformers/pull/40740)) has been rebased.  if there is anything more I need to do please let me know - I'm happy to help move it along.

cc @gante 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

cc @McPatate @remi-or since you're working on continuous batching / generation now that @gante is leaving! if you can't take it maybe @zucchini-nlp knows this area, and if none of you know then I can take it and maybe learn something 😅 

Yep, making it configurable with kwargs is necessary imo. Taking a look at the PR 👀 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Next step after https://github.com/huggingface/transformers/pull/42702