Repository: BerriAI/litellm
Pull request: fix Llama models messages to prompt conversion for AWS Bedrock
Fix: #3297

----
@krrishdholakia I saw that you added support for Llama-3 recently in commit [df7db2b](https://github.com/BerriAI/litellm/commit/df7db2b870d2e1201888bb625c446e4473759ffb) but that didn't cover Llama Bedrock models.

Please let me know if this looks good otherwise.

Problem statement:
[Bug]: Gibberish output of Llama-3 models on AWS Bedrock
### What happened?

Llama-3 models are not working properly on AWS Bedrock right now and output gibberish. Probably because the instruct models now follow a different template.

```python
response = client.chat.completions.create(
    model="llama-3-8b", 
    messages = [
        {
            "role": "user",
            "content": "hi"
        }
    ],
    max_tokens=16
)

print(response.choices[0].message.content)
```

### Relevant log output

```shell
K, et al. (2014). "A systematic review of the effectiveness
```


### Twitter / LinkedIn details

_No response_

Hints from issue discussion:
I noticed this as well. It seems like the same for Llama 2 on Bedrock, not just Llama 3?

Commit message: fix Llama models message to prompt conversion in for AWS Bedrock provider

Patch to review:
diff --git a/litellm/llms/bedrock.py b/litellm/llms/bedrock.py
index ef6dbfb1b8ff..149b68472452 100644
--- a/litellm/llms/bedrock.py
+++ b/litellm/llms/bedrock.py
@@ -653,6 +653,10 @@ def convert_messages_to_prompt(model, messages, provider, custom_prompt_dict):
         prompt = prompt_factory(
             model=model, messages=messages, custom_llm_provider="bedrock"
         )
+    elif provider == "meta":
+        prompt = prompt_factory(
+            model=model, messages=messages, custom_llm_provider="bedrock"
+        )
     else:
         prompt = ""
         for message in messages:
diff --git a/litellm/llms/prompt_templates/factory.py b/litellm/llms/prompt_templates/factory.py
index a6d1d64386f5..00c6229c9ddd 100644
--- a/litellm/llms/prompt_templates/factory.py
+++ b/litellm/llms/prompt_templates/factory.py
@@ -1346,6 +1346,16 @@ def prompt_factory(
                 return anthropic_pt(messages=messages)
         elif "mistral." in model:
             return mistral_instruct_pt(messages=messages)
+        elif "llama2" in model:
+            return llama_2_chat_pt(messages=messages)
+        elif "llama3" in model:
+            return hf_chat_template(
+                model=model,
+                messages=messages,
+                chat_template=known_tokenizer_config[  # type: ignore
+                    "meta-llama/Meta-Llama-3-8B-Instruct"
+                ]["tokenizer"]["chat_template"],
+            )
     elif custom_llm_provider == "perplexity":
         for message in messages:
             message.pop("name", None)


Code review comments:
diff_hunk: @@ -1346,6 +1346,16 @@ def prompt_factory(
                 return anthropic_pt(messages=messages)
         elif "mistral." in model:
             return mistral_instruct_pt(messages=messages)
+        elif "llama2" in model:
+            return llama_2_chat_pt(messages=messages)
+        elif "llama3" in model:
path: litellm/llms/prompt_templates/factory.py
text: @user1:
can you check if it's an 'instruct' model before applying the chat template (same for llama2). 



@author:
Bedrock only supports Llama-2-Chat and Llama-3-Instruct models [1].

Nonetheless, I think it still makes sense to add a check just to be future proof. Will update.

 [1] https://docs.aws.amazon.com/bedrock/latest/userguide/model-ids.html#model-ids-arns

@user1:
i see non-chat llama2 models on bedrock

<img width="1134" alt="Screenshot 2024-04-25 at 11 20 10 AM" src="https://github.com/BerriAI/litellm/assets/17561003/1a138a2e-97ff-4bac-84bc-855b527fe437">