Add inverse chat templating
Author: Rocketknight1
Pull request: yes

Very experimental PR for now! This PR adds **inverse chat templating**, where we convert a formatted chat back into message dicts in a universal format. This has been requested several times, and it's critical to allow seamless tool use in pipelines, without requiring users to manually write parsers for each model.

Inverse templating requires the ability to extract text from a large input string. After testing several templates, I found that we can generally handle this by exposing two functions in the inverse templating environment: a slightly modified `re.finditer()` and a totally unmodified `json.loads()`

TODO:
- [x] Make some model PRs to test with this, but don't merge yet! (PRs not open, but templates written)
- [x] Add tests for loading/saving of inverse templates
- [x] Add tests for inverse template function
- [x] Test recovery of tools as well as messages
- [x] Make sure I don't need any extra functions
- [x] Refactor chat template tests out of `tokenization_common` so they're not run for every model
- [x] Add chat template tests to CircleCI
- [x] ~Make sure extraction works correctly with generation, and add some tests that use (static!) generation outputs~
- [x] ~Add tool use to pipelines~ (will put this in a separate PR)

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_33321). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Thanks @Rocketknight1! I'll review ASAP

@LysandreJik Don't worry, it's not quite ready! I'm still working on the little details of how this interacts with generation / incomplete inputs, etc.

Update @LysandreJik, this is now ready for review! The last issue I was testing was how the inverse templates would handle **incomplete prompts** - for example, if the user called `model.generate()`, but only passed the new tokens to the inverse template, instead of the entire prompt + output.

My conclusion was that this is just very messy and impossible to handle correctly - in particular, the prompt often contains the assistant header, so it's extremely hard for the inverse template to figure out what's going on if you only pass the generation output. Therefore, I think we should only support applying inverse templates to entire chats, and not single generation outputs, and so there's no need to test it with generation outputs only.

Awesome, will take a look ASAP

Shouldn't this support multiple (`tool_use`/`rag`) templates just like chat templates?

In the cases where it makes sense not to make a unified chat template I'd imagine it won't make sense to make a unified inverse template as well...

We might consider adding that in future! In general, we're trying to move away from having multiple chat templates, in favour of a single template that just uses conditions like `{% if tools %}` to switch its behaviour, although we'll continue supporting multiple (non-inverse) templates indefinitely.

Looks like a simple enough PR for me; would you like to check with others tha are using chat templates extensively like @xenova or @Narsil to get a review?

Sure, I'll ping them! cc @lewtun @xenova @narsil and @zucchini-nlp, since they've been working with templates (or have requested this feature).

From VLM side I dont think this feature will be very useful, as I don't see and easy way to implement that if the input text has already been processed (`processor.call()`) and then detokenized back. Saying because I guess it will be used to format back to dict after generating text 

But I think it's okay as VLMs are one step behind and don't support full potential of chat templates yet

@Rocketknight1 BTW, really looking forward to this feature, it will be super useful.

I'm working on a chat template editor (Gradio space) which will make it super easy to view, modify, test and create PRs for chat templates on HF. I'm hoping to add support for inverse templates as well, but it will require the HF API to return the inverse template in the `config.tokenizer_config` entry in ModelInfo, who can ensure the API is updated once this feature is added?

Hi @CISC - in that case, I'd pay attention to this PR too! #33957

We're planning a medium-term refactor because of how much chat templates have blown up, with the goal of eventually moving them to a single file in most model repos.

(I may delay this PR until we settle on a spec in #33957, to avoid putting inverse templates somewhere and then immediately having to move them again)

> (I may delay this PR until we settle on a spec in #33957, to avoid putting inverse templates somewhere and then immediately having to move them again)

That PR seems like it needs to be very gradually phased in though, it will impact a lot of projects that need to catch up, does it really make sense to delay this one until that is done?

@CISC good point, but we'll probably merge that PR relatively soon so that Transformers starts to support the new format, even though we won't actually transition repos until later!

@Rocketknight1 What's the status on this one? Still looking forward to this feature. :)

Still happening, just delayed! 

Closing now that https://github.com/huggingface/transformers/pull/40894 is merged instead