Setting chat_template when creating a processor does not change the chat template
Author: NohTow
Pull request: no
Labels: name: bug

### System Info

Hello,
With `transformers == 4.55.3` (and below, it seems), passing `chat_template` as an args when creating the processor does not seems to change the chat template accordingly.
Few lines to reproduce:
```python
from transformers import AutoProcessor
processor = AutoProcessor.from_pretrained("Qwen/Qwen2.5-VL-3B-Instruct", chat_template="test")
print(processor.chat_template)
# Print the default chat template
```
When tracking the root cause, it seems that this line is [overriding the user args by what's resolved](https://github.com/huggingface/transformers/blob/88ba0f107eb8c96b34bbe664f17f07ce0c8c57b5/src/transformers/processing_utils.py#L1086) (kwargs["chat_template"] is correct until this line).
It is a bit odd because I tried to dig into the past and it seems to have been like that for a while.
Am I missing something? I tried digging as much as I could on my own but I do not really understand how the user template could not be overriden.

### Who can help?

@Rocketknight1 @zucchini-nlp 

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

Install transformers

Run
```python
from transformers import AutoProcessor
processor = AutoProcessor.from_pretrained("Qwen/Qwen2.5-VL-3B-Instruct", chat_template="test")
print(processor.chat_template)
# Print the default chat template
```


### Expected behavior

The processor.chat_template should be set to the one passed by the user when creating the processor

Comments:
Hmm, prob no-one ever needed to override chat template when loading from pretrained. I though we already tested this. Would you like to dog deeper and submit a PR? We definitely want it to be changeable when loading

Imho it would be as simple as checking whether kwargs["chat_template"] is defined before overriding it but I am probably missing something.
It is a bit odd because IIRC I was able to set the chat template doing this at some point earlier this year.
Is there any other (better) way to change the chat template of a pre-trained model (e.g, anyone wanting to change system prompt). Or is `processor.tokenizer.chat_template = XXX` the usual way?

> It is a bit odd because IIRC I was able to set the chat template doing this at some point earlier this year

Could have been a regression as well since we did a lot of updates on chat template format when saving/loading. Things get lost in the way sometimes 😄 

> Is there any other (better) way to change the chat template of a pre-trained model (e.g, anyone wanting to change system prompt). Or is processor.tokenizer.chat_template = XXX the usual way?

That is the recommended way, as you are already doing it. Usually the chat templates are fixed since the model was trained with it which is why it's a rare case when we want to override it

For system prompts, it depends a lot on how the system prompt is injected by model owners. For qwen-vl you can use your own system prompt by just passing it in the conversation history like:

```python
chat = [{"role": "system", "content": ["type": "text", "text": "My new system prompt here"]}]
```

> Could have been a regression as well since we did a lot of updates on chat template format when saving/loading. Things get lost in the way sometimes 😄

So this is what I supposed, but what puzzles me is that I traced back this overriding line to commits that are very very far, so I am pretty sure this was already included in the version I used back then. But maybe there was an issue with the loading of the chat template of this particular model and so it was not overridden and now it's fixed so it overrides.

> That is the recommended way, as you are already doing it.

You mean setting from_pretrained or setting the tokenizer value?

> Usually the chat templates are fixed since the model was trained with it which is why it's a rare case when we want to override it

I get that, but we are actually using HF modeling to train models (within Composer), so we need to be able to change it (more importantly remove it altogether for certain tasks)

> For system prompts, it depends a lot on how the system prompt is injected by model owners. For qwen-vl you can use your own system prompt by just passing it in the conversation history like:

Yes, exactly. I checked and this would work for a few models but essentially, we need a bit more control over it. For example again, you can change the system prompt for Qwen/Mistral, but you could not remove it, if it is not within the messages, it adds the default one.


> You mean setting from_pretrained or setting the tokenizer value?

Using `from_pretrained` which was supposed to work lol. I can submit a fix and track the possible regression later, a bit busy with other tasks rn


Setting as `self.processor.chat_template = XXX` can be a workaround for now :)

I've created a fix for this issue in my fork and submitted a PR. The issue is that user-provided chat_template parameters are being overwritten by default templates from model files. 

The fix ensures that user-provided chat_template values take precedence over model defaults by checking if 'chat_template' is already in kwargs before setting it.

You can view the PR here: https://github.com/qizwiz/transformers/tree/fix-chat-template-override

Can you open as a PR @qizwiz and tag me there? That makes it easier to review and leave comments

Hi @zucchini-nlp, 

I've created a PR that addresses this issue: https://github.com/huggingface/transformers/pull/40954

The PR fixes the bug where user-provided `chat_template` parameters were being overwritten by model defaults when creating a processor. The fix ensures that user-provided values take precedence over model defaults.

I've also added a test case to verify the fix works correctly. Please let me know if you have any feedback or suggestions!

Thanks for your guidance on this!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.