apply_chat_template fails if the model's jinja template doesn't support a system prompt
Author: unoriginalscreenname
Pull request: no

### System Info

All the latest versions

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Some of these models reject the system prompt in their chat template. It's very inconsistent and there doesn't seem to be a way to tell if a model accepts a system prompt or not. You end up getting this error if you pass in a system prompt and format it using the chat template function:

jinja2.exceptions.TemplateError: Conversation roles must alternate user/assistant/user/assistant/

I'm really surprised I haven't seen anybody raise this. Am I missing something? How do you tell if a model takes a system prompt or not?

here is an example from the PIVOT-10.7B-Mistral-V0.2 model:

  "chat_template": "{{ bos_token }}{% for message in messages %}{% if (message['role'] == 'user') != (loop.index0 % 2 == 0) %}{{ raise_exception('Conversation roles must alternate user/assistant/user/assistant/...') }}{% endif %}{% if message['role'] == 'user' %}{{ '
 ' + message['content'] + ' 
' }}{% elif message['role'] == 'assistant' %}{{ message['content'] + eos_token}}{% else %}{{ raise_exception('Only user and assistant roles are supported!') }}{% endif %}{% endfor %}",


### Expected behavior

I would expect that apply_chat_template woudl be able to read or parse either the template or config file and determine if a system prompt was expected.  if it wasn't, then i would expect it would either ignore it or merge it with the first instruction in the chat.

Comments:
Second this issue, just ran into this on Mistral as well.

cc @Rocketknight1 

Hi @unoriginalscreenname and @lithafnium, this was discussed in #27922. I'll paste the relevant part of my reply here:

> Unfortunately, this is an unavoidable consequence of how these models were trained. Some models were trained with system prompts as part of the training data, and other models were not. When a model was not trained with a system prompt, it will not have any tokens that it can use to represent a system prompt, and trying to insert a prompt will confuse the model and probably significantly reduce the output quality.

> In the cases when a model was trained without a system prompt, the model's chat template can be configured to raise an error if a `system` message is included in the input, and this is indeed what happens with some models (e.g. LLaMA/Mistral/Mixtral). This is correct and intended behaviour, and there isn't really any way to "fix" it without retraining the models!

> The only solution I can suggest is that there is usually a different fine-tune of most models that supports a system prompt. For example, instead of Mistral-instruct you can use [Zephyr-beta](https://huggingface.co/HuggingFaceH4/zephyr-7b-beta), and instead of Mixtral-instruct you can use [Dolphin](https://huggingface.co/cognitivecomputations/dolphin-2.7-mixtral-8x7b). Both of these models were trained with system prompts, and will understand them correctly (and apply them in their chat template).

It is possible to write a chat template that ignores system messages or merges them into the first user message like you suggested, but this is not what the authors of the LLaMA/Mistral/Mixtral prompts chose to do. Although it seems like a clean solution, in practice I think there is a significant risk that just blending the system message into the user message would create a confusing message that would reduce model performance.

Also, please note that **chat templates are set in individual model repos, not in the code of Transformers itself**. As a result, we at Hugging Face don't actually have the power to overrule model authors and force their models to accept system prompts if their chosen template doesn't support them! See our [blog post](https://huggingface.co/blog/chat-templates) about chat templates for more information.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Could we have an attribute like `.support_system_message` in the tokenizer?