Add assistant prefill for chat templates and TextGenerationPipeline
Author: Rocketknight1
Pull request: yes

Something that's been requested several times both internally and on Github is assistant prefill: The ability to begin the model's response for it and let it continue.

We use a slightly hacky solution suggested by @Narsil, but which I think will work in all cases that I know of: When we do assistant prefill, we simply truncate the formatted chat to the end of the final message text, removing any tokens that indicate the end of an assistant message, which will cause the model to simply continue from wherever the final message ends.

This PR also updates `TextGenerationPipeline` so that if you pass a chat where the final message is an assistant message, it will assume that you want to treat the final message as an assistant prefill. Before this PR, it would start a new message if you did this, which would generally result in malformed output because most models were not trained with multiple consecutive assistant messages.

Fixes #33096

Fixes #32213

Comments:
Let me know when you'd like a review @Rocketknight1!

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_33198). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@LysandreJik should be ready for review now! I had to tweak a couple of other tests that ended in assistant messages - the output was garbage in those cases anyway, and the tests were just checking that it wasn't changing unexpectedly.

For visibility:
cc @philschmid who requested this. Cc @Narsil for TGI and @xenova for `transformers.js`, though note that this doesn't change Jinja behaviour at all, it just adds a flag that trims the output string in Python! Also cc @gante for generation.

See also #33096 and #32213 where this was requested

Instead of `assistant_prefill`, may just add a new parameter called `prefill_last_message` to `apply_chat_template`. I feel like there's no need to limit prefill to the assistant role. This way, it's compatible with the current needs and can also adapt to potential future needs. Plus, the parameter name and the corresponding changes are simple and clear enough that there won't be any misunderstandings.

@DIYer22 I'm not sure about that - the "assistant" role is universal across all chat models that I know of. Although some users might want the model to continue a user or system message, I think that's more of a fun/research thing than a common use-case, and I think it's okay to expect people to construct their own prompts/templates for those cases.

There are other models in private that don't use standard `assistant` / `user` roles, and will likely be forced to by this type of change, or will just use something other than chat_templates for formatting their prompts (hurting standardization even more).

I don't think using a model through the standard `chat/completion` style endpoint for auto-complete for a users' message is a far off use case. It is just something that OpenAI doesn't support with their endpoints, so tooling hasn't been built to support it yet. That was one of the main points I was trying to enable (along with assistant prefill).

Not to mention "chat_templates" are not just for chat models, there are so many ways that the API can be used if things aren't artificially limited.

Edit: I am good with the approach in general though (not needing to modify the templates themselves is great).

Hmn, okay, there's more demand for it than I thought - you were right @DIYer22!

@Tostino @DIYer22 I've replaced `assistant_prefill` with `continue_final_message`. It has the same behaviour as `assistant_prefill`, but will no longer raise an error if the final message is not an `assistant` message.

I've kept the `TextGenerationPipeline` behaviour the same, however. I'll think about how I could possibly add support there without disrupting the UX for simple use-cases.

Update: Pipeline now correctly handles the `continue_final_message` kwarg, which overrides the default behaviour of continuing `assistant` messages only.

@LysandreJik sorry for the delay, should be ready for actual review now!

@LysandreJik all comments addressed/merged, and moved all the repos to `hf-internal-testing`. Also updated a couple of other cases where I was using personal repos for tests that were already merged to `main`. I also added a section to the chat templating docs about this.

Let me know if you're happy to merge, or if you want to re-review any of that!

Thank you very much for the work on this @Rocketknight1. I'm really happy you came up with a solution that helps solve all the needs brought up.

Thanks @Tostino! Please let me know if you encounter any issues while using it - you can try it out right now by installing from `main`.

@Rocketknight1 There seems to be an issue...

I was testing it the other day and just couldn't get it working...and then I needed to continue on my roadtrip. Just got back to testing and realized there is a discrepancy between the code and the documentation.

Documentation says [`continue_last_message`](https://huggingface.co/docs/transformers/main/en/chat_templating#what-does-continuelastmessage-do)  and code is actually `continue_final_message`.

@Tostino I have no idea how I overlooked that! I think I just changed the name of the argument quite late in the PR, opening a fix now.