[Continuous Batching] Snapshot generation outputs without mutating request state
Author: Incheonkirin
Pull request: yes

# What does this PR do?

This fixes `RequestState.to_generation_output()` in continuous batching so that
building a `GenerationOutput` does not mutate the active request state or return
live aliases for growing output buffers.

Streaming continuous batching can call `to_generation_output()` after every
generated token. On current `main`, that method returns the same list objects
held by the active `RequestState` for `generated_tokens`, `logprobs`, and
`timestamps`. It also mutates `self.generated_tokens` and `self.initial_tokens`
in the `_true_initial_tokens` soft-reset path.

That creates two observable issues:

- a previously returned streaming output can change after later tokens are
  generated
- after soft reset, repeated output conversion can fold already-generated
  prompt-suffix tokens back into `state.generated_tokens`, so the request can
  terminate with fewer generated tokens than requested

This PR makes output conversion a snapshot operation. It keeps `prompt_ids` by
reference on the normal path to avoid copying the prompt for every streamed
token, copies only the growing buffers, and computes the soft-reset
prompt/generated split locally without changing `RequestState`.

I added two CPU-only regression tests:

- `test_generation_outputs_are_snapshots`
- `test_streaming_output_after_soft_reset_does_not_shorten_generation`

These fail on the previous implementation and pass with this change.

As an additional check, I ran a CUDA end-to-end run with real continuous
batching streaming requests, `use_cuda_graph=True`, forced soft reset
(`offload_calls=7`, `soft_reset_calls=7`), `do_sample=False`, and
`max_new_tokens=30`. On the unpatched source, several requests stopped at 21-23
generated tokens and some previously returned first chunks later had length
20/22/8. With this patch, all 12 requests reached 30 generated tokens and all
first chunks remained length 1. I can provide a small repro harness if useful.

## Code Agent Policy

- [x] I confirm that this is not a pure code agent PR.

## Before submitting

- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


Comments:
cc @remi-or @mcpatate for CB

Thanks, addressed both nits. I kept the list copies for the mutable output fields so the snapshot behavior stays intact.

**CI Dashboard:** [View test results in Grafana](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=46670)

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_46670). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Thanks @remi-or! Those nits were good calls, and the code reads cleaner now. Appreciate the careful review and merge.