apply_chat_template() function, in particular with the chat_template = "rag"
Author: willxxy
Pull request: no

I am trying to figure out what's the best way to format the chat template with rag. I am following the tutorial [here](https://huggingface.co/docs/transformers/main/en/chat_extras). The only difference is that the model card for the tutorial is using `CohereForAI/c4ai-command-r-v01-4bit`.  When I follow the tutorial using a different model card, in this example `meta-llama/Llama-3.2-1B-Instruct`, the printed output of `print(tokenizer.decode(input_ids))` is "rag". Is this an expected behavior? I am guessing this may be because RAG is not supported for llama 3.2 1b instruct per [the documentation in this code](https://github.com/huggingface/transformers/blob/main/src/transformers/tokenization_utils_base.py#L1530). Although I read this and I tried looking at the [linked website](https://huggingface.co/docs/transformers/main/en/chat_templating#automated-function-conversion-for-tool-use) in the documentation, however, I am still left confused. Right now I am constructing this myself, but I wanted to see what the best practice of injecting the RAG content (e.g., in the system prompt? before the user query? etc.)

```
documents = [
    {
        "title": "The Moon: Our Age-Old Foe", 
        "text": "Man has always dreamed of destroying the moon. In this essay, I shall..."
    },
    {
        "title": "The Sun: Our Age-Old Friend",
        "text": "Although often underappreciated, the sun provides several notable benefits..."
    }
]

from transformers import AutoTokenizer, AutoModelForCausalLM

# Load the model and tokenizer
model_card = "meta-llama/Llama-3.2-1B-Instruct"
tokenizer = AutoTokenizer.from_pretrained(model_card)
model = AutoModelForCausalLM.from_pretrained(model_card, device_map="auto")
device = model.device # Get the device the model is loaded on

# Define conversation input
conversation = [
    {"role": "user", "content": "What has Man always dreamed of?"}
]

input_ids = tokenizer.apply_chat_template(
    conversation=conversation,
    documents=documents,
    chat_template="rag",
    tokenize=True,
    add_generation_prompt=True,
    return_tensors="pt").to(device)

print(tokenizer.decode(input_ids))
# Generate a response 
generated_tokens = model.generate(
    input_ids,
    max_new_tokens=100,
    do_sample=True,
    temperature=0.3,
    )

# Decode and print the generated text along with generation prompt
generated_text = tokenizer.decode(generated_tokens[0])
print(generated_text)
```

Comments:
Hi @willxxy, this is a quirk of `apply_chat_template()` - it can accept either **the name** of a template or **an entire template**. When you use it with Command-R, `tokenizer.chat_template` has a template named `rag`, and so passing `chat_template="rag"` selects that template. If you try that with another model, there is no template named `rag`, and so the model treats the string "rag" as a Jinja template, which just renders "rag" as you saw.

The fundamental issue here is that only some models were trained for RAG and include RAG templates, and Llama wasn't one of them!

I see, that makes sense. Thank you so much for the prompt and detailed reply!