Separate chat templates into a single file
Author: Rocketknight1
Pull request: yes

We have several issues with chat templates because they're stored as single lines in the JSON config files:

- Impossible to review diffs
- Very hard to edit in the web UI (or in general)
- Differences between `processor` templates in `chat_template.json` and `tokenizer` templates in `tokenizer_config.json` causing confusion
- Some models use multiple templates, requiring a template dict, but we're trying to discourage that in future and move those models to single templates with conditional behaviour instead

The solution:

- Just move chat templates to a single `chat_template.jinja` file in the repo
- If multiple templates are required, then they should still be stored in the JSON file. This is not supported for `Processor` classes, so processors should always be able to save their template as a raw Jinja file. In general, we'll be gently deprecating multiple templates in future.
- If a `chat_template.jinja` file is present, it overrides the JSON files. If a tokenizer is loaded with both Jinja and JSON chat templates and resaved, it should save only the Jinja file, and not have any `chat_template` entry in `tokenizer_config.json`.

For now, we continue saving in the old format by default. I'll probably keep it this way for several versions before making the new format the default, to ensure that most users are able to load the new format before it becomes common. Until then, the new format should mostly be used for testing, to make sure it's ready for deployment when we do the switch.

Extremely draft PR for now, since it'll probably break lots of things!

TODO:
- [X] Add loading/saving single files in tokenizers
- [x] Add loading/saving single files in processors
- [x] Add tokenizer tests for saving + loading + push_to_hub working correctly
- [x] Add processor tests for saving + loading + push_to_hub working correctly
- [x] Ensure processors + tokenizers don't clash when both saving chat templates

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_33957). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Interesting! I think we would need to be extra careful as this can influence quite a lot of other libs, to which we need to open PRs before merging this one! 

FYI @LysandreJik and @Narsil 

Also cc @zucchini-nlp 

Oh, wow, this will affect a lot of other projects. Also keep in mind that the HF API exposes the chat templates through the `config.tokenizer_config` entry in `ModelInfo`.

I will release a chat template editor (Gradio space) soon that lets you easily view, modify, test and submit PRs (just to address a couple of OP bullet-points), so thanks to @Rocketknight1 for pointing me towards this PR, I will need to keep a close eye on changes. :)

@CISC yes, I'm aware that this could be disruptive! When this PR is ready we'll discuss it internally and with the community, and maybe we might end up aborting it at that point (and I'll try not to let that delay the inverse templates PR too much either way)

This is ready for review! There's one major question: Right now I'm saving `processor_chat_template.jinja` and `chat_template.jinja` as separate files to ensure no conflicts, even though most repos should only have one.

We could consider always using `chat_template.jinja` for both, however in that case we would need to override the loading behaviour on the `tokenizer`, if it's the child of a `processor`, because if the `processor` saves a chat template file then the `tokenizer` will also load that and use it to set its own chat template attribute as well, which might have strange side effects.

Other than that, this should be ready for review! (cc some possibly affected people @zucchini-nlp @Narsil @xenova @LysandreJik)

Failing tests in UDOP seem to be unrelated and I can't reproduce them locally

@zucchini-nlp - agreed, but hopefully we can deprecate the others! I hope when we're finally finished, we just have `chat_template.jinja` and `processor_chat_template.jinja`. I was considering merging both into `chat_template.jinja` but that creates lots of weird side effects, and I decided against it.

Update: This should be ready for final review now! The failing UDOP tests are not related to this PR, and I have another PR open to fix them here #34180

To be clear, **we will not move the chat templates when this PR is merged. This PR just adds support for saving/loading to the new location, but we will not save to the new file locations by default. This will give the many other frameworks that load chat templates time to adapt, and give users time to update to versions that support the new locations.**

cc affected people: @Narsil @xenova @Wauplin 

Thanks for the ping @Rocketknight1! I'll check what it implies server-side to make the parsing of the chat template available in the API.

High level feedback: to avoid any confusion in the future, shouldn't the long-term files be called `tokenizer_chat_template.jinja` and `processor_chat_template.jinja`?
Otherwise some model repos will end up with `chat_template.json`/`processor_chat_template.jinja` for processors and `tokenizer_config.json`/`chat_template.jinja` for tokenizers (for backward compatibility). Meaning that `chat_completion.jinja` and `chat_completion.json` won't have the same meaning.

Having a longer and more explicit final name would avoid confusion in my opinion. 

I think that makes sense, yes! It'll be a bit longer but very explicit about which files affect which classes. I'll make that change if other reviewers agree as well.

> I was considering merging both into chat_template.jinja but that creates lots of weird side effects, and I decided against it.

Hi @Rocketknight1 @zucchini-nlp Getting back to this topic as I'd like to understand more the reasoning behind this file structure. I have a few questions:
1. What are the side effects in play if we set a single jinja file for both tokenizers and processors?
2. Apart from the text-only vs text+image(+<any modality?>) difference, is there another fundamental difference between a tokenizer and a processor? Can it happen that a VLM has 1 template for text-only and 1 template for text+image? 
3. (maybe a stupid question) if we have extra information to store about a jinja template, could it make sense to define a header structure that could be added to jinja templates (as jinja comments) with high level metadata? typically to indicate if the template is suitable for tokenizers and/or processors (an alternative could be to use a ~yaml~ toml config file instead of pure jinja)
4. what are the main libraries/tools that have to be updated when we update this standard? At least transformers/transformers.js/TGI for sure. Any other one to think about? 

Without knowing all the technical details, I'd advocate for a truly single file chat template if possible. If that's the case, we would support things differently server-side, typically having a single `model_info.config.chat_template` field instead of `model_info.config.tokenizer_config.chat_template` and `model_info.config.processor_config.chat_template`. Discussed it further with @julien-c who advocated to support only the (expected) final version on the Hub instead of the intermediate `chat_template.json`/... alternatives, to nudge model authors to use the "standard" asap (meaning reverting the PR I merged yesterday on moon-landing).

And sorry if all of this has already been discussed elsewhere! 

> Discussed it further with @julien-c who advocated to support only the (expected) final version on the Hub instead of the intermediate chat_template.json/... alternatives, to nudge model authors to use the "standard" asap

100%

Hi @wauplin, 1. is a good question! The main reason I wanted separate templates is because I thought the side effects would be unpredictable. For example, many processors have `processor.chat_template` but `tokenizer.chat_template` is `None`. However, if we put all chat templates in `chat_template.json`, then when the `Tokenizer` is loaded for that model, it will also set `processor.tokenizer.chat_template`. This is **probably** safe, but it might break some stuff that I can't predict!

I've published the initial version of [Chat Template Editor](https://huggingface.co/spaces/CISCai/chat-template-editor), ~~however the full functionality is currently blocked by [the gr.LoginButton not working](https://github.com/gradio-app/gradio/issues/9820)~~.

I've started work related to this and the inverse template PR, but it's currently disabled/hidden until finalized.

~~I have a small question I hope someone can provide an answer to however; this PR mentions processor chat templates, and I've tried locating repos containing them to see how they work and possibly add support for them, but I can't seem to find any?~~

Found [community Pixtral](https://huggingface.co/mistral-community/pixtral-12b) which was very helpful, added support for `chat_template.json` and multimodal messages.

I got waylaid by a couple of urgent tasks, but this should finally be ready! The summary is:

- The only raw chat template file is `chat_template.jinja`, for both processors and tokenizers. This means that processors and tokenizers saved in the same repo cannot have different chat templates.
- For now, we continue saving in the old format by default, but we support loading the new format. We keep it like that for a couple of versions so that when we start saving the new format it doesn't break everyone's experience.
- When a tokenizer has multiple templates, it cannot be saved in the new format. This probably won't be changed - we want to gently deprecate multiple templates.

PR description updated, and merging!

Thanks @Rocketknight1 ! Is there a first repo example on the Hub to check that? Asking since we'll have to support this new (and definitive!) file in the server-side parsing 

cc @Wauplin not yet, but I'll make one in `hf-internal-testing` today!