apply_chat_template return_assistant_tokens_mask not work for Qwen2.5
Author: DogeWatch
Pull request: no
Labels: name: bug

name: Chat Template

### System Info

huggingface-hub-0.25.2 tokenizers-0.20.1 transformers-4.45.2

### Who can help?

@ArthurZucker @itazap 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

```python
from transformers import AutoTokenizer
tk = AutoTokenizer.from_pretrained("Qwen2___5-7B-Instruct", trust_remote_code=True)
new_chat_template = '{%- if tools %}\n    {{- \'
system\\n\' }}\n    {%- if messages[0][\'role\'] == \'system\' %}\n        {{- messages[0][\'content\'] }}\n    {%- else %}\n        {{- \'You are Qwen, created by Alibaba Cloud. You are a helpful assistant.\' }}\n    {%- endif %}\n    {{- "\\n\\n# Tools\\n\\nYou may call one or more functions to assist with the user query.\\n\\nYou are provided with function signatures within <tools></tools> XML tags:\\n<tools>" }}\n    {%- for tool in tools %}\n        {{- "\\n" }}\n        {{- tool | tojson }}\n    {%- endfor %}\n    {{- "\\n</tools>\\n\\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\\n<tool_call>\\n{\\"name\\": <function-name>, \\"arguments\\": <args-json-object>}\\n</tool_call>
\\n" }}\n{%- else %}\n    {%- if messages[0][\'role\'] == \'system\' %}\n        {{- \'
system\\n\' + messages[0][\'content\'] + \'
\\n\' }}\n    {%- else %}\n        {{- \'
system\\nYou are Qwen, created by Alibaba Cloud. You are a helpful assistant.
\\n\' }}\n    {%- endif %}\n{%- endif %}\n{%- for message in messages %}\n    {%- if (message.role == "user") or (message.role == "system" and not loop.first) or (message.role == "assistant" and not message.tool_calls) %}\n        {{- \'
\' + message.role + \'\\n\' + message.content + \'
\' + \'\\n\' }}\n    {%- elif message.role == "assistant" %}\n        {% generation %} {{- \'
\' + message.role }}\n        {%- if message.content %}\n            {{- \'\\n\' + message.content }}\n        {%- endif %}\n        {%- for tool_call in message.tool_calls %}\n            {%- if tool_call.function is defined %}\n                {%- set tool_call = tool_call.function %}\n            {%- endif %}\n            {{- \'\\n<tool_call>\\n{"name": "\' }}\n            {{- tool_call.name }}\n            {{- \'", "arguments": \' }}\n            {{- tool_call.arguments | tojson }}\n            {{- \'}\\n</tool_call>\' }}\n        {%- endfor %}\n        {{- \'
\\n\' }} {% endgeneration %}\n    {%- elif message.role == "tool" %}\n        {%- if (loop.index0 == 0) or (messages[loop.index0 - 1].role != "tool") %}\n            {{- \'
user\' }}\n        {%- endif %}\n        {{- \'\\n<tool_response>\\n\' }}\n        {{- message.content }}\n        {{- \'\\n</tool_response>\' }}\n        {%- if loop.last or (messages[loop.index0 + 1].role != "tool") %}\n            {{- \'
\\n\' }}\n        {%- endif %}\n    {%- endif %}\n{%- endfor %}\n{%- if add_generation_prompt %}\n    {{- \'
assistant\\n\' }}\n{%- endif %}\n'
msg = [{'role': 'user', 'content': ' \n\n我想起十年前，我在大学里学习生物医学。我很熟悉书本知识，但我发现学术资格并不总是足够的。我需要实际的经验 – 现场工作，才能真正了解行业和工作流程。因此，我决定在学习期间参加志愿工作和实习项目。我参加了医院志愿者活动，这让我有机会与医疗专业人员和病人沟通，也在一定程度上了解了他们的需要。我还在当地的药房实习，这帮助我更深入地了解了医药企业的商业模式和销售策略。最终，我毕业后找到一份医药销售代表的工作，因为我既有学术背景，又有现场工作经验，能够灵活处理各种情况，更好地满足客户需求。这份工作最终让我在职业生涯中获得了巨大成功。\n\n\n基于以上这段文本内容回答： 你参加的志愿者活动和实习项目可以带给你哪些实际的经验和收获？ \n\n'}, {'role': 'assistant', 'content': '参加医院志愿者活动和实习项目让我有机会与医疗专业人员和病人沟通，了解他们的需要。在当地的药房实习让我更深入地了解了医药企业的商业模式和销售策略。这些实际的经验和收获帮助我更好地满足客户需求，并在职业生涯中取得了成功。'}]
output = tk.apply_chat_template(
    msg,
    chat_template=new_chat_template,
    tokenize=True,
    add_generation_prompt=True,
    padding=True,
    max_length=2048,
    truncation=True,
    return_dict=True,
    return_assistant_tokens_mask=True,
)
print(output['assistant_masks'])
```
I modified the chat template to add the {% generation %} and {% endgeneration %}
the new_chat_template looks like 
```
{%- if tools %}
    {{- '
system\n' }}
    {%- if messages[0]['role'] == 'system' %}
        {{- messages[0]['content'] }}
    {%- else %}
        {{- 'You are Qwen, created by Alibaba Cloud. You are a helpful assistant.' }}
    {%- endif %}
    {{- "\n\n# Tools\n\nYou may call one or more functions to assist with the user query.\n\nYou are provided with function signatures within <tools></tools> XML tags:\n<tools>" }}
    {%- for tool in tools %}
        {{- "\n" }}
        {{- tool | tojson }}
    {%- endfor %}
    {{- "\n</tools>\n\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\n<tool_call>\n{\"name\": <function-name>, \"arguments\": <args-json-object>}\n</tool_call>
\n" }}
{%- else %}
    {%- if messages[0]['role'] == 'system' %}
        {{- '
system\n' + messages[0]['content'] + '
\n' }}
    {%- else %}
        {{- '
system\nYou are Qwen, created by Alibaba Cloud. You are a helpful assistant.
\n' }}
    {%- endif %}
{%- endif %}
{%- for message in messages %}
    {%- if (message.role == "user") or (message.role == "system" and not loop.first) or (message.role == "assistant" and not message.tool_calls) %}
        {{- '
' + message.role + '\n' + message.content + '
' + '\n' }}
    {%- elif message.role == "assistant" %}
        {% generation %} {{- '
' + message.role }}
        {%- if message.content %}
            {{- '\n' + message.content }}
        {%- endif %}
        {%- for tool_call in message.tool_calls %}
            {%- if tool_call.function is defined %}
                {%- set tool_call = tool_call.function %}
            {%- endif %}
            {{- '\n<tool_call>\n{"name": "' }}
            {{- tool_call.name }}
            {{- '", "arguments": ' }}
            {{- tool_call.arguments | tojson }}
            {{- '}\n</tool_call>' }}
        {%- endfor %}
        {{- '
\n' }} {% endgeneration %}
    {%- elif message.role == "tool" %}
        {%- if (loop.index0 == 0) or (messages[loop.index0 - 1].role != "tool") %}
            {{- '
user' }}
        {%- endif %}
        {{- '\n<tool_response>\n' }}
        {{- message.content }}
        {{- '\n</tool_response>' }}
        {%- if loop.last or (messages[loop.index0 + 1].role != "tool") %}
            {{- '
\n' }}
        {%- endif %}
    {%- endif %}
{%- endfor %}
{%- if add_generation_prompt %}
    {{- '
assistant\n' }}
{%- endif %}
```

### Expected behavior

the output is
```
{'input_ids': [151644, 8948, 198, 2610, 525, 1207, 16948, 11, 3465, 553, 54364, 14817, 13, 1446, 525, 264, 10950, 17847, 13, 151645, 198, 151644, 872, 198, 4710, 104100, 71618, 117476, 3837, 104786, 99562, 69249, 100134, 100206, 104316, 1773, 106922, 102364, 90286, 21894, 100032, 3837, 105984, 99879, 104380, 104303, 100684, 104014, 103170, 1773, 35946, 85106, 99912, 106187, 1365, 10236, 236, 108, 82224, 99257, 3837, 101901, 100690, 99794, 99717, 33108, 99257, 102054, 1773, 101886, 3837, 35946, 103930, 18493, 100134, 101072, 101061, 101411, 99257, 33108, 102774, 73345, 1773, 35946, 106057, 100634, 104907, 99600, 3837, 43288, 104029, 106211, 57218, 100182, 99878, 99653, 33108, 104693, 104063, 3837, 104477, 106931, 17447, 99794, 34187, 104056, 85106, 1773, 35946, 104241, 109233, 99471, 99218, 102774, 3837, 43288, 100364, 35946, 33126, 100403, 29490, 99794, 34187, 101356, 104385, 108555, 33108, 100352, 104238, 1773, 103941, 3837, 35946, 109981, 101958, 104191, 101356, 100352, 99661, 104066, 3837, 106811, 107203, 104380, 102193, 3837, 105320, 100647, 111930, 3837, 100006, 105128, 54542, 100646, 99559, 3837, 105344, 101929, 116932, 1773, 106039, 99257, 103941, 104029, 18493, 111978, 15946, 105067, 102334, 19108, 1773, 1406, 104210, 70589, 107083, 108704, 43815, 102104, 5122, 220, 56568, 101061, 9370, 104907, 99600, 33108, 102774, 73345, 73670, 99278, 104314, 102224, 99912, 106187, 33108, 104619, 11319, 4710, 151645, 198, 151644, 77091, 198, 101061, 100634, 104907, 99600, 33108, 102774, 73345, 104029, 106211, 57218, 100182, 99878, 99653, 33108, 104693, 104063, 3837, 99794, 104056, 85106, 1773, 18493, 109233, 99471, 99218, 102774, 104029, 33126, 100403, 29490, 99794, 34187, 101356, 104385, 108555, 33108, 100352, 104238, 1773, 100001, 99912, 106187, 33108, 104619, 100364, 35946, 105344, 101929, 116932, 90395, 18493, 111978, 15946, 104847, 19108, 1773, 151645, 198, 151644, 77091, 198], 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], 'assistant_masks': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}
```
I the assistant_masks is all zero, it is not correct

Comments:
cc @Rocketknight1!

@yonigottesman do you have any idea what's going on here? I can take a look if not!

Hey @Rocketknight1 , I tested this out with the default `chat_template` and think maybe the update to `generation_indices` was missed here https://github.com/huggingface/transformers/pull/32684 in the function below:

https://github.com/huggingface/transformers/blob/9240137897096698f5292c7dd38d0651c8a33dc8/src/transformers/utils/chat_template_utils.py#L334-L349

LMK what you think!

Hi @DogeWatch it looks like you put the `generation` keyword in the wrong place.
in your template there is a condition:
`{%- if (message.role == "user") or (message.role == "system" and not loop.first) or (message.role == "assistant" and not message.tool_calls) %}` which is true as long as you dont use tools so its entering that path and not the path you put the `generation` token in.
try using:
```
{%- for message in messages %}
    {%- if (message.role == "user") or (message.role == "system" and not loop.first) %}
        {{- '
' + message.role + '\n' + message.content + '
' + '\n' }}
    {%- elif (message.role == "assistant" and not message.tool_calls) %}
    {% generation %}    {{- '
' + message.role + '\n' + message.content + '
' + '\n' }}    {% endgeneration %}
    {%- elif message.role == "assistant" %}
```
This works for me.
What do you think?

Also, please check the spaces and `\n` are as expected by qwen. And finally, i think you can remove the `
assistant\n` string from within the assistant generation block as it is added by the tokenizer at inference (probably doesn't matter )


@DogeWatch @Rocketknight1 if this works can we close the issue?

Yes, I'm happy to close it, and we can reopen it if that solution doesn't work.

Can you provide a valided chat template for `return_assistant_tokens_mask`?

I have used the following code, but got an jinja compiling error.

```py
from transformers import AutoTokenizer, PreTrainedTokenizer
tokenizer: PreTrainedTokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-0.5B-Instruct")
messages = [
    {"role": "user", "content": "你好，世界！"},
    {"role": "assistant", "content": "你好，世界"},
]

qwen25_chat_templace = """{%- if tools %}
    {{- '
system\n' }}
    {%- if messages[0]['role'] == 'system' %}
        {{- messages[0]['content'] }}
    {%- else %}
        {{- 'You are Qwen, created by Alibaba Cloud. You are a helpful assistant.' }}
    {%- endif %}
    {{- "\n\n# Tools\n\nYou may call one or more functions to assist with the user query.\n\nYou are provided with function signatures within <tools></tools> XML tags:\n<tools>" }}
    {%- for tool in tools %}
        {{- "\n" }}
        {{- tool | tojson }}
    {%- endfor %}
    {{- "\n</tools>\n\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\n<tool_call>\n{\"name\": <function-name>, \"arguments\": <args-json-object>}\n</tool_call>
\n" }}
{%- else %}
    {%- if messages[0]['role'] == 'system' %}
        {{- '
system\n' + messages[0]['content'] + '
\n' }}
    {%- else %}
        {{- '
system\nYou are Qwen, created by Alibaba Cloud. You are a helpful assistant.
\n' }}
    {%- endif %}
{%- endif %}
{%- for message in messages %}
    {%- if (message.role == "user") or (message.role == "system" and not loop.first) %}
        {{- '
' + message.role + '\n' + message.content + '
' + '\n' }}
    {%- elif (message.role == "assistant" and not message.tool_calls) %}
    {% generation %}    {{- '
' + message.role + '\n' + message.content + '
' + '\n' }}    {% endgeneration %}
    {%- elif message.role == "assistant" %}
        {{- '
' + message.role }}
        {%- if message.content %}
            {{- '\n' + message.content }}
        {%- endif %}
        {%- for tool_call in message.tool_calls %}
            {%- if tool_call.function is defined %}
                {%- set tool_call = tool_call.function %}
            {%- endif %}
            {{- '\n<tool_call>\n{"name": "' }}
            {{- tool_call.name }}
            {{- '", "arguments": ' }}
            {{- tool_call.arguments | tojson }}
            {{- '}\n</tool_call>' }}
        {%- endfor %}
        {{- '
\n' }}
    {%- elif message.role == "tool" %}
        {%- if (loop.index0 == 0) or (messages[loop.index0 - 1].role != "tool") %}
            {{- '
user' }}
        {%- endif %}
        {{- '\n<tool_response>\n' }}
        {{- message.content }}
        {{- '\n</tool_response>' }}
        {%- if loop.last or (messages[loop.index0 + 1].role != "tool") %}
            {{- '
\n' }}
        {%- endif %}
    {%- endif %}
{%- endfor %}
{%- if add_generation_prompt %}
    {{- '
assistant\n' }}
{%- endif %}
"""

tokenizer.chat_template = qwen25_chat_templace
tokenizer.apply_chat_template(messages, return_assistant_tokens_mask=True, return_dict=True, return_tensors="pt")
```

error as following:

```py
---------------------------------------------------------------------------
TemplateSyntaxError                       Traceback (most recent call last)
Cell In[18], line 65
      6 qwen25_chat_templace = """{%- if tools %}
      7     {{- '
system\n' }}
      8     {%- if messages[0]['role'] == 'system' %}
   (...)     61 {%- endif %}
     62 """
     64 tokenizer.chat_template = qwen25_chat_templace
---> 65 tokenizer.apply_chat_template(messages, return_assistant_tokens_mask=True, return_dict=True, return_tensors="pt")

File /data/workspace/coderepo/llm-compose/.venv/lib/python3.11/site-packages/transformers/tokenization_utils_base.py:1637, in PreTrainedTokenizerBase.apply_chat_template(self, conversation, tools, documents, chat_template, add_generation_prompt, continue_final_message, tokenize, padding, truncation, max_length, return_tensors, return_dict, return_assistant_tokens_mask, tokenizer_kwargs, **kwargs)
   1632     logger.warning_once(
   1633         "return_assistant_tokens_mask==True but chat template does not contain `{% generation %}` keyword."
   1634     )
   1636 # Compilation function uses a cache to avoid recompiling the same template
-> 1637 compiled_template = _compile_jinja_template(chat_template)
   1639 if isinstance(conversation, (list, tuple)) and (
   1640     isinstance(conversation[0], (list, tuple)) or hasattr(conversation[0], "messages")
   1641 ):
   1642     conversations = conversation

File /data/workspace/coderepo/llm-compose/.venv/lib/python3.11/site-packages/transformers/utils/chat_template_utils.py:435, in _compile_jinja_template(chat_template)
    433 jinja_env.globals["raise_exception"] = raise_exception
    434 jinja_env.globals["strftime_now"] = strftime_now
--> 435 return jinja_env.from_string(chat_template)

File /data/workspace/coderepo/llm-compose/.venv/lib/python3.11/site-packages/jinja2/environment.py:1111, in Environment.from_string(self, source, globals, template_class)
   1109 gs = self.make_globals(globals)
   1110 cls = template_class or self.template_class
-> 1111 return cls.from_code(self, self.compile(source), gs, None)

File /data/workspace/coderepo/llm-compose/.venv/lib/python3.11/site-packages/jinja2/environment.py:771, in Environment.compile(self, source, name, filename, raw, defer_init)
    769     return self._compile(source, filename)
    770 except TemplateSyntaxError:
--> 771     self.handle_exception(source=source_hint)

File /data/workspace/coderepo/llm-compose/.venv/lib/python3.11/site-packages/jinja2/environment.py:942, in Environment.handle_exception(self, source)
    937 """Exception handling helper.  This is used internally to either raise
    938 rewritten exceptions or return a rendered traceback for the template.
    939 """
    940 from .debug import rewrite_traceback_stack
--> 942 raise rewrite_traceback_stack(source=source)

File <unknown>:27, in template()

TemplateSyntaxError: expected token 'end of print statement', got 'name'
```

When I deleted all the template syntax related to tool, the template could be compiled normally.

```py
{%- for message in messages %}
    {%- if (message.role == "user") or (message.role == "system" and not loop.first) %}
        {{- '
' + message.role + '\\n' + message.content + '
' + '\\n' }}
    {%- elif (message.role == "assistant") %}
        {% generation %}    {{- '
' + message.role + '\\n' + message.content + '
' + '\\n' }}    {% endgeneration %}
    {%- endif %}
{%- endfor %}
{%- if add_generation_prompt %}
    {{- '
assistant\\n' }}
{%- endif %}
```

I used the following template and it worked. 


```python
    modified_template = """
{%- if tools %}
    {{- '
system\\n' }}
    {%- if messages[0]['role'] == 'system' %}
        {{- messages[0]['content'] }}
    {%- else %}
        {{- 'You are Qwen, created by Alibaba Cloud. You are a helpful assistant.' }}
    {%- endif %}
    {{- "\\n\\n# Tools\\n\\nYou may call one or more functions to assist with the user query.\\n\\nYou are provided with function signatures within <tools></tools> XML tags:\\n<tools>" }}
    {%- for tool in tools %}
        {{- "\\n" }}
        {{- tool | tojson }}
    {%- endfor %}
    {{- "\\n</tools>\\n\\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\\n<tool_call>\\n{\\\"name\\\": <function-name>, \\\"arguments\\\": <args-json-object>}\\n</tool_call>
\\n" }}
{%- else %}
    {%- if messages[0]['role'] == 'system' %}
        {{- '
system\\n' + messages[0]['content'] + '
\\n' }}
    {%- else %}
        {{- '
system\\nYou are Qwen, created by Alibaba Cloud. You are a helpful assistant.
\\n' }}
    {%- endif %}
{%- endif %}
{%- for message in messages %}
    {%- if (message.role == "user") or (message.role == "system" and not loop.first) %}
        {{- '
' + message.role + '\\n' + message.content + '
' + '\\n' }}
    {%- elif (message.role == "assistant" and not message.tool_calls) %}
    {% generation %}    {{- '
' + message.role + '\\n' + message.content + '
' + '\\n' }}    {% endgeneration %}
    {%- elif message.role == "assistant" %}
        {% generation %}{{- '
' + message.role }}
        {%- if message.content %}
            {{- '\\n' + message.content }}
        {%- endif %}
        {%- for tool_call in message.tool_calls %}
            {%- if tool_call.function is defined %}
                {%- set tool_call = tool_call.function %}
            {%- endif %}
            {{- '\\n<tool_call>\\n{\\\"name\\\": \\\"' }}
            {{- tool_call.name }}
            {{- '\\\", \\\"arguments\\\": ' }}
            {{- tool_call.arguments | tojson }}
            {{- '}\\n</tool_call>' }}
        {%- endfor %}
        {{- '
\\n' }}{% endgeneration %}
    {%- elif message.role == "tool" %}
        {%- if (loop.index0 == 0) or (messages[loop.index0 - 1].role != "tool") %}
            {{- '
user' }}
        {%- endif %}
        {{- '\\n<tool_response>\\n' }}
        {{- message.content }}
        {{- '\\n</tool_response>' }}
        {%- if loop.last or (messages[loop.index0 + 1].role != "tool") %}
            {{- '
\\n' }}
        {%- endif %}
    {%- endif %}
{%- endfor %}
{%- if add_generation_prompt %}
    {% generation %}{{- '
assistant\\n' }}{% endgeneration %}
{%- endif %}
"""
    return modified_template


```

> modified_template = """
> {%- if tools %}
>     {{- '
system\\n' }}
>     {%- if messages[0]['role'] == 'system' %}
>         {{- messages[0]['content'] }}
>     {%- else %}
>         {{- 'You are Qwen, created by Alibaba Cloud. You are a helpful assistant.' }}
>     {%- endif %}
>     {{- "\\n\\n# Tools\\n\\nYou may call one or more functions to assist with the user query.\\n\\nYou are provided with function signatures within <tools></tools> XML tags:\\n<tools>" }}
>     {%- for tool in tools %}
>         {{- "\\n" }}
>         {{- tool | tojson }}
>     {%- endfor %}
>     {{- "\\n</tools>\\n\\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\\n<tool_call>\\n{\\\"name\\\": <function-name>, \\\"arguments\\\": <args-json-object>}\\n</tool_call>
\\n" }}
> {%- else %}
>     {%- if messages[0]['role'] == 'system' %}
>         {{- '
system\\n' + messages[0]['content'] + '
\\n' }}
>     {%- else %}
>         {{- '
system\\nYou are Qwen, created by Alibaba Cloud. You are a helpful assistant.
\\n' }}
>     {%- endif %}
> {%- endif %}
> {%- for message in messages %}
>     {%- if (message.role == "user") or (message.role == "system" and not loop.first) %}
>         {{- '
' + message.role + '\\n' + message.content + '
' + '\\n' }}
>     {%- elif (message.role == "assistant" and not message.tool_calls) %}
>     {% generation %}    {{- '
' + message.role + '\\n' + message.content + '
' + '\\n' }}    {% endgeneration %}
>     {%- elif message.role == "assistant" %}
>         {% generation %}{{- '
' + message.role }}
>         {%- if message.content %}
>             {{- '\\n' + message.content }}
>         {%- endif %}
>         {%- for tool_call in message.tool_calls %}
>             {%- if tool_call.function is defined %}
>                 {%- set tool_call = tool_call.function %}
>             {%- endif %}
>             {{- '\\n<tool_call>\\n{\\\"name\\\": \\\"' }}
>             {{- tool_call.name }}
>             {{- '\\\", \\\"arguments\\\": ' }}
>             {{- tool_call.arguments | tojson }}
>             {{- '}\\n</tool_call>' }}
>         {%- endfor %}
>         {{- '
\\n' }}{% endgeneration %}
>     {%- elif message.role == "tool" %}
>         {%- if (loop.index0 == 0) or (messages[loop.index0 - 1].role != "tool") %}
>             {{- '
user' }}
>         {%- endif %}
>         {{- '\\n<tool_response>\\n' }}
>         {{- message.content }}
>         {{- '\\n</tool_response>' }}
>         {%- if loop.last or (messages[loop.index0 + 1].role != "tool") %}
>             {{- '
\\n' }}
>         {%- endif %}
>     {%- endif %}
> {%- endfor %}
> {%- if add_generation_prompt %}
>     {% generation %}{{- '
assistant\\n' }}{% endgeneration %}
> {%- endif %}
> """

it works for my code, thanks!