Multimodal chat sample
Author: Wovchena
Pull request: no
Labels: name: Feature request

### Feature request

Add a sample covering chat scenario including images, videos or audio.

### Motivation

`AutoModelForCausalLM`'s `use_cache` is barely documented.
Describe a pattern handling the following cases
1. Tokenizer replaces tokens that are already in kv cache with a different token. For example, the model generated 2 tokens with string representations: `a` and `b` and the tokenizer replaces them with a single `a b` token on the next iteration invalidating a part of kv cache
2. Reuse embeddings computed earlier for non text modalities

There's https://github.com/huggingface/transformers/blob/a7f29523361b2cc12e51c1f5133d95f122f6f45c/src/transformers/cli/chat.py but it doesn't cover non text modalities.

### Your contribution

I'm fine to submit a PR. That will help me to learn along the way. But I need guidance how to resolve the issues I described in the motivation section.

Comments:
Hi @Wovchena, this sounds like a great addition since `cli/chat.py` is currently text-only.

Regarding your concerns: standard generation pipelines usually accept the trade-off where new tokens don't merge with the history to keep the KV cache valid, so you probably don't need to implement complex token healing for a sample script. I'd suggest starting a PR that targets a specific model like LLaVA or Qwen-VL to get the interaction loop working first, rather than trying to solve generic embedding reuse immediately.

> you probably don't need to implement complex token healing for a sample script.

I thought samples should guide how to implement things the correct way. If you insist on ignoring token healing for `chat.py`, I should reformulate my feature request. I don't really care about the state of `transformers` samples. My concern is how to implement the chat correctly

My bad @Wovchena, I totally misread your goal.

To fix the `a` + `b` -> `ab` split, the standard "correct" way is a **cache rollback**: check if the last history token merges with the new input, and if so, slice the last entry off `past_key_values` before feeding the merged token.

A PR showing this pattern would be super valuable.

Tell me if i again made a mistake in understanding you req?

Yes, that's correct. I suspect that the hard part is to mix cache rollback and image embeddings caching. Sometimes kv cache part that's based on image embeddings should be recomputed and sometimes not.

By the way, I'm also interested in linear attention states but I assume there's no way to rollback the cache unless the intermediate states are saved. Let me know if you have thoughts on this topic.

What form should my contribution take: a sample, `chat.py` or something else?

You'd technically need to maintain a "state checkpoint" from the previous turn to rollback, which is heavy.

For the vision embedding interactions, usually, the tokenizer won't merge across the `<image>` special token boundary, so the "healing" mostly applies to the text suffix *after* the image features. As long as the rollback doesn't cut into the image tokens themselves, we can often treat the visual KV/embeddings as a frozen prefix and only recompute the text tail.

**Regarding the contribution:**
I’d suggest starting with a **standalone sample script** (e.g., `examples/generation/multimodal_chat_sample.py`) rather than modifying `cli/chat.py` directly. 

The current CLI is a bit rigid. A dedicated sample gives you the freedom to implement these advanced patterns (cache rollback, state checkpointing) cleanly without worrying about legacy CLI arguments. Once the pattern is proven there, we can discuss porting it to the core tool.

Sounds good?

Here's the initial draft: https://github.com/huggingface/transformers/pull/43049 Let's move the conversation there

Hey @Wovchena ,

We don't yet support multimodal chat CLI, it is on the roadmap and we will work on it soon. If you want to apply token healing for the prompt, you can simply pass in `token_healing=True` in generation call (see [example](https://github.com/huggingface/transformers-research-projects/tree/main/token-healing)). That is usually enough and works with multimodal models as well

1. Is there a snippet covering multiple iterations with `heal_tokens()`?
2. Should I still work on the sample in my PR?

We don't have an example snippet with multiple iterations and token healing. I don't think the PR will be needed because it is adding an example script which we have moved to a separate repo. If you think it's still useful for the community, feel free to extend the example script in `research-projects` repository shared above. I believe it will be straightforward to implement