feat(tokenization): add encode_message to tokenize messages one by one
Author: pco111
Pull request: yes
Labels: name: Chat Template

What does this PR do?
This PR introduces a new method, tokenizer.encode_message, to the base tokenizer class. This method allows for tokenizing a single chat message at a time while correctly handling the conversational context provided by conversation_history. This is particularly useful for token-by-token streaming applications where re-tokenizing the entire conversation history for each new token is inefficient.
The new method works by applying the chat template to the full conversation (history + new message) and then programmatically isolating the tokens that correspond to the new message. This ensures that all special tokens, roles, and formatting are applied correctly according to the model's chat template, maintaining consistency with apply_chat_template.

Fixes #39417
Before submitting
[x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
[x] Did you read the contributor guideline,
Pull Request section?
[x] Was this discussed/approved via a Github issue or the forum? Please add a link
to it if that's the case.
[x] Did you make sure to update the documentation with your changes? Here are the
documentation guidelines, and
here are tips on formatting docstrings.
[x] Did you write any new necessary tests?

Who can review?
@ArthurZucker @Rocketknight1

Comments:
I like this! cc @Rocketknight1 if you can have a look!

Hi @pco111, this is a cool idea, but I'm not sure about some of the details! In particular, the interaction with `add_generation_prompt` is awkward. If we set that to `True`, then a common scenario is that the `conversation_history` will be tokenized like this, where the "generation prompt" is the last line:

```
<im_start>user
message<im_end>
<im_start>assistant
```

But in this case, `encode_message()` will treat `<im_start>assistant` as part of the history, and remove it from the encoded message, and then the encoded message will be incomplete. I'm not sure what the best solution is - maybe always set `add_generation_prompt` to `False`?

> Hi @pco111, this is a cool idea, but I'm not sure about some of the details! In particular, the interaction with `add_generation_prompt` is awkward. If we set that to `True`, then a common scenario is that the `conversation_history` will be tokenized like this, where the "generation prompt" is the last line:
> 
> ```
> <im_start>user
> message<im_end>
> <im_start>assistant
> ```
> 
> But in this case, `encode_message()` will treat `<im_start>assistant` as part of the history, and remove it from the encoded message, and then the encoded message will be incomplete. I'm not sure what the best solution is - maybe always set `add_generation_prompt` to `False`?



> Hi @Rocketknight1 ,
>
> Thank you so much for your insightful feedback! You've pointed out a very important edge case with add_generation_prompt that I had overlooked.
>
> Following your thoughts, I've opted for a clearer and more robust approach:
>
> 1. Explicitly Disallowed add_generation_prompt: The encode_message method now raises a ValueError if add_generation_prompt is passed. This prevents any ambiguity.
> 2. Updated Documentation: The docstring for encode_message now clearly states that it does not handle the generation prompt and advises users on how to add it separately if needed.
> 3. Updated Tests: The tests have been updated to reflect this new design. There is now a test to ensure that the ValueError is raised correctly.
>
> Thank you again for guiding me toward a better solution! I've pushed the new changes for your review.

> Made some comments! Also, check the CI on Github - you may need to run `make fixup` to get the style tests to pass.

> Hi @Rocketknight1,
>
> Thank you for the detailed and helpful feedback! I've updated the PR according to all your suggestions:
>
> * The _encode_message helper has been folded into the main encode_message function.
> * An optimization has been added to handle empty conversation history directly.
> * The "Tensor Cores" typo has been corrected.
> * The new tests have been moved into the existing TokenizerUtilsTest class.
>
> All local checks (make fixup and pytest) are passing. The code should be in much better shape now. Thanks again for your guidance!

Hi @pco111, I think Copilot (or whatever code agent you're using) is making a lot of unrelated changes to the docstring in `get_chat_template`. Can you clean that up and then I'll do a final review?

> Hi @pco111, I think Copilot (or whatever code agent you're using) is making a lot of unrelated changes to the docstring in `get_chat_template`. Can you clean that up and then I'll do a final review?

> Hi @Rocketknight1,
>
> My apologies for the unintended changes in the previous commit. It seems my formatting agent was a bit too aggressive.
>
> I've now cleaned up the docstrings and reverted all unrelated changes. 
>
> Thank you for your patience. Look forward to your final review!

> > Hi @pco111, I think Copilot (or whatever code agent you're using) is making a lot of unrelated changes to the docstring in `get_chat_template`. Can you clean that up and then I'll do a final review?
> 
> > Hi @Rocketknight1,
> > My apologies for the unintended changes in the previous commit. It seems my formatting agent was a bit too aggressive.
> > I've now cleaned up the docstrings and reverted all unrelated changes.
> > Thank you for your patience. Look forward to your final review!

Hi @Rocketknight1 , sorry asking again, but just a gentle reminder 😊

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_39507). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Hi @Rocketknight1 and @ArthurZucker , thank you for your timely reply. I have changed the function name as suggested by @ArthurZucker . Now there are 2 workflows that need to be approved. Please have a look. Thank you !

Hi @ArthurZucker and @Rocketknight1 , thank you for approving the changes. I'm running into an issue where the main branch is updated frequently. By the time the required checks (which need approval) are complete, my branch is already out of date again. In this way, it seems to fall into a dead cycle where the merge can never happen.

1. Could you approve workflow & merge for me, in 1 shot ?
2. Otherwise, could we possibly coordinate a time to merge? I can update the branch and you could approve the workflows right after, so we can get it merged quickly once the checks pass. Please let me know what time works best for you.

Hey! Don't worry you are fine! We can merge now! 🤗 merging