File size: 7,237 Bytes
220079e afbecb4 220079e afbecb4 220079e afbecb4 220079e f867b9d 220079e f867b9d 220079e f867b9d 666809c 220079e f867b9d 220079e afbecb4 220079e afbecb4 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 |
---
language:
- en
license: llama3
tags:
- fireplace
- valiant
- valiant-labs
- llama
- llama-3
- llama-3-instruct
- llama-3-instruct-70b
- 70b
- function-calling
- conversational
- chat
- instruct
pipeline_tag: text-generation
model_type: llama
model-index:
- name: Llama3-70B-Fireplace
results:
- task:
type: text-generation
name: Text Generation
dataset:
name: IFEval (0-Shot)
type: HuggingFaceH4/ifeval
args:
num_few_shot: 0
metrics:
- type: inst_level_strict_acc and prompt_level_strict_acc
value: 77.74
name: strict accuracy
source:
url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=ValiantLabs/Llama3-70B-Fireplace
name: Open LLM Leaderboard
- task:
type: text-generation
name: Text Generation
dataset:
name: BBH (3-Shot)
type: BBH
args:
num_few_shot: 3
metrics:
- type: acc_norm
value: 49.56
name: normalized accuracy
source:
url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=ValiantLabs/Llama3-70B-Fireplace
name: Open LLM Leaderboard
- task:
type: text-generation
name: Text Generation
dataset:
name: MATH Lvl 5 (4-Shot)
type: hendrycks/competition_math
args:
num_few_shot: 4
metrics:
- type: exact_match
value: 19.64
name: exact match
source:
url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=ValiantLabs/Llama3-70B-Fireplace
name: Open LLM Leaderboard
- task:
type: text-generation
name: Text Generation
dataset:
name: GPQA (0-shot)
type: Idavidrein/gpqa
args:
num_few_shot: 0
metrics:
- type: acc_norm
value: 13.98
name: acc_norm
source:
url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=ValiantLabs/Llama3-70B-Fireplace
name: Open LLM Leaderboard
- task:
type: text-generation
name: Text Generation
dataset:
name: MuSR (0-shot)
type: TAUR-Lab/MuSR
args:
num_few_shot: 0
metrics:
- type: acc_norm
value: 16.77
name: acc_norm
source:
url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=ValiantLabs/Llama3-70B-Fireplace
name: Open LLM Leaderboard
- task:
type: text-generation
name: Text Generation
dataset:
name: MMLU-PRO (5-shot)
type: TIGER-Lab/MMLU-Pro
config: main
split: test
args:
num_few_shot: 5
metrics:
- type: acc
value: 43.25
name: accuracy
source:
url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=ValiantLabs/Llama3-70B-Fireplace
name: Open LLM Leaderboard
---
Fireplace is a function-calling model for Llama 3 70b Instruct.
- combines function-calling abilities with a high-performance, versatile chat model
- system message function-calling utilizing the Llama 3 Instruct format
This version of Fireplace focuses on combining chat-instruct and system-message function-calling only.
**We've just released Fireplace 2 for Llama 3.1 8b**, which includes inline function calls as one of several technical skills (including JSON, SQL, and more!)
[Try it today!](https://huggingface.co/ValiantLabs/Llama3.1-8B-Fireplace2)
## Version
This is the **2024-05-09** release of Fireplace for Llama 3 70b.
We've also released Fireplace 2 for Llama 3.1 8b and we're working on more Fireplace releases to come :)
## Prompting Guide
Fireplace uses the [Llama 3 Instruct](https://huggingface.co/meta-llama/Meta-Llama-3-70B-Instruct) prompt format:
<|begin_of_text|><|start_header_id|>system<|end_header_id|>{{ system_prompt }}<|eot_id|><|start_header_id|>user<|end_header_id|>{{ user_msg_1 }}<|eot_id|><|start_header_id|>assistant<|end_header_id|>{{ model_answer_1 }}<|eot_id|>
Example input for function calling:
<|begin_of_text|><|start_header_id|>system<|end_header_id|>\n\n You are Fireplace, an expert code assistant with access to the following functions. Use them if required - { "name": "calculate_tip", "description": "Calculate the tip amount for a bill", "parameters": { "type": "object", "properties": { "bill_amount": { "type": "number", "description": "The total amount of the bill" }, "tip_percentage": { "type": "number", "description": "The percentage of tip to be given" } }, "required": [ "bill_amount", "tip_percentage" ] } } { "name": "check_website_availability", "description": "Check the availability of a website", "parameters": { "type": "object", "properties": { "url": { "type": "string", "description": "The URL of the website" } }, "required": [ "url" ] } } <|eot_id|><|start_header_id|>user<|end_header_id|>\n\nHi, I need help with calculating a tip. My bill is $100 and I want to leave a 30% tip. <|eot_id|><|start_header_id|>assistant<|end_header_id|>
For assistant handling of function responses, deliver them in a new user message:
<|start_header_id|>user<|end_header_id|>\n\n FUNCTION RESPONSE: {"status": "success", "message": "Email has been sent successfully"} <|eot_id|>
## WARNING: text-generation-webui
When using Llama 3 Instruct models (including Fireplace) with [text-generation-webui](https://github.com/oobabooga/text-generation-webui/tree/main) note that a current bug in webui can result in incorrect reading of the model's ending tokens, causing unfinished outputs and incorrect structure.
For a [temporary workaround](https://github.com/oobabooga/text-generation-webui/issues/5885) if you encounter this issue, edit Fireplace's tokenizer_config file as indicated:
from "eos_token": "<|end_of_text|>",
to "eos_token": "<|eot_id|>",
## The Model
Fireplace is built on top of Llama 3 70b Instruct, the highest performance open-source model currently available.
This version of Fireplace uses the [glaiveai/glaive-function-calling-v2](https://huggingface.co/datasets/glaiveai/glaive-function-calling-v2) dataset converted to Llama 3 Instruct format.
# [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard)
Detailed results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/details_ValiantLabs__Llama3-70B-Fireplace)
| Metric |Value|
|-------------------|----:|
|Avg. |36.82|
|IFEval (0-Shot) |77.74|
|BBH (3-Shot) |49.56|
|MATH Lvl 5 (4-Shot)|19.64|
|GPQA (0-shot) |13.98|
|MuSR (0-shot) |16.77|
|MMLU-PRO (5-shot) |43.25|
Fireplace is created by [Valiant Labs.](http://valiantlabs.ca/)
[Check out our HuggingFace page for Shining Valiant 2 and our other models!](https://huggingface.co/ValiantLabs)
[Follow us on X for updates on our models!](https://twitter.com/valiant_labs)
We care about open source.
For everyone to use.
We encourage others to finetune further from our models.
|