Chat Template Upgrades
Author: Rocketknight1
Pull request: no

Hey all, I was thinking about the chat templates API and I realized something important. When you're generating responses from the model, you want the prompt to include the message history, but **also the tokens that indicate the start of a bot response**. That makes sure that the model actually replies to you, instead of continuing the user response or something like that.

However, there are other cases when we don't want the template to do that. For example, when formatting messages for training you don't want to add any extra generation prompt at the end.

A big goal of chat templates was to be useful for both generation and training, so it's important we have some way to support both use-cases!

## Example

Consider the standard ChatML template:

```

user
Hi, this is a user message.

bot
Hi, this is a bot reply.

user
Hi, this is the next user message.
```

If the user wants to generate a bot reply, however, the actual input they prompt with should end with `
bot`:

```

user
Hi, this is a user message.

bot
Hi, this is a bot reply.

user
Hi, this is the next user message.

bot
```

The reason for this is you want the bot to **write a bot response** and not continue a user message, or write some other special tokens, or any other weird thing like that. However, when using `apply_chat_template` to format chat data for training, you don't want to add `
bot to the end, because you don't want to generate further text.

Because the prompt that indicates the start of a bot message varies between models, it has to be part of a template, and we can't just hardcode it in `ConversationPipeline` or something like that. Therefore, we need some way for our templates to flexibly include this prompt or not!

## Possible solutions

1) We add a kwarg to `apply_chat_template` to indicate whether the bot message start tokens should be appended at the end:

```python
>>> tokenizer.apply_chat_template(messages, tokenize=False)
"""
user
Hi, this is a user message.

bot
Hi, this is a bot reply.

user
Hi, this is the next user message."""

>>> tokenizer.apply_chat_template(messages, add_generation_prompt=True, tokenize=False)
"""
user
Hi, this is a user message.

bot
Hi, this is a bot reply.

user
Hi, this is the next user message.

bot"""
```

2: We assume that chat histories that end in a **user** message are for generation, and chat histories that end in a **bot** message are for training, since this is usually how it works. In this option, we could automatically add the bot message start tokens at the end of any input when the final message has `role == 'user'`.

```python
>>> chat_history = [
    {"role": "user", "content": "Hi, this is a user message."},
    {"role": "bot", "content": "Hi, this is a bot reply."}
]
>>> tokenizer.apply_chat_template(chat_history)
"""
user
Hi, this is a user message.

bot
Hi, this is a bot reply."""

>>> chat_history.append(
    {"role": "user", "content": "Hi, this is the next user message."}
)  # Now the conversation ends with a user message, so a generation prompt is added
>>> tokenizer.apply_chat_template(chat_history)
"""
user
Hi, this is a user message.

bot
Hi, this is a bot reply.

user
Hi, this is the next user message.

bot"""
```

Alternatively, we could combine those solutions - add the kwarg from option 1, but with a default value of `None`, and if the kwarg is set to `None` then we follow the rules in option 2.


Comments:
can u assign it to me

I'll probably be taking this PR myself, @draksham, but thanks for the offer!

Here's my current best plan:

1) Add a `add_generation_prompt` kwarg to `apply_chat_template`. 
2) Update our default chat templates to use this kwarg. When the kwarg is `True`, we add a prompt for a bot message (like `
bot`) at the end of the formatted string
3) Update our existing `default_chat_template` settings to use the kwarg - we can check in Jinja is the variable is defined, and default to `False` if it isn't to retain backward compatibility.
4) Update the documentation to explain the kwarg and encourage users to include support for it in their templates.

Hi. Thanks for adding this. 

When I seem to use `add_generation_prompt` for a tokenizer, it seems to have no effect. Example below:

```
>>> from transformers import AutoTokenizer
>>> chat_history = [
    {"role": "user", "content": "Hi, this is a user message."},
    {"role": "assistant", "content": "Hi, this is a bot reply."},
    {"role": "user", "content": "Oh I see!"}
]
>>> tokenizer = AutoTokenizer.from_pretrained("HuggingFaceH4/zephyr-7b-alpha")
>>> tokenizer.apply_chat_template(chat_history, tokenize=False, add_generation_prompt=True)
"""

Hi, this is a user message.



Hi, this is a bot reply.



Oh I see!

"""
```

When I checked the chat_template of the tokenizer, I got:
```
"{% for message in messages %}\n{% if message['role'] == 'user' %}\n{{ '
\n' + message['content'] + eos_token }}\n{% elif message['role'] == 'system' %}\n{{ '
\n' + message['content'] + eos_token }}\n{% elif message['role'] == 'assistant' %}\n{{ '
\n'  + message['content'] + eos_token }}\n{% endif %}\n{% if loop.last and add_generation_prompt %}\n{{ '
' }}\n{% endif %}\n{% endfor %}"
```

So, the logic for `add_generation_prompt` seems to be there. But somehow it is not getting used.

Hi @AdirthaBorgohain, this feature has only been added very recently and is only working on `main` right now. We're working on a patch release ASAP, but in the meantime, you can install from `main` with `pip install --upgrade git+https://github.com/huggingface/transformers.git`