|
--- |
|
library_name: transformers |
|
license: llama2 |
|
datasets: |
|
- aqua_rat |
|
- microsoft/orca-math-word-problems-200k |
|
- m-a-p/CodeFeedback-Filtered-Instruction |
|
--- |
|
|
|
# Llama-3-Smaug-8B |
|
|
|
### Built with Meta Llama 3 |
|
|
|
This model was built using a new Smaug recipe for improving performance on real world multi-turn conversations applied to |
|
[meta-llama/Meta-Llama-3-70B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3-70B-Instruct). |
|
|
|
The model outperforms Llama-3-70B-Instruct substantially, and is on par with GPT-4-Turbo, on MT-Bench (see below). |
|
|
|
### Model Description |
|
|
|
- **Developed by:** [Abacus.AI](https://abacus.ai) |
|
- **License:** https://llama.meta.com/llama3/license/ |
|
- **Finetuned from model:** [meta-llama/Meta-Llama-3-70B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3-70B-Instruct). |
|
|
|
|
|
## Evaluation |
|
|
|
### MT-Bench |
|
|
|
``` |
|
########## First turn ########## |
|
score |
|
model turn |
|
Smaug-Llama-3-70B-Instruct 1 9.40000 |
|
GPT-4-Turbo 1 9.37500 |
|
Meta-Llama-3-70B-Instruct 1 9.21250 |
|
########## Second turn ########## |
|
score |
|
model turn |
|
Smaug-Llama-3-70B-Instruct 2 9.0125 |
|
GPT-4-Turbo 2 9.0000 |
|
Meta-Llama-3-70B-Instruct 2 8.8000 |
|
########## Average ########## |
|
score |
|
model |
|
Smaug-Llama-3-70B-Instruct 9.206250 |
|
GPT-4-Turbo 9.187500 |
|
Meta-Llama-3-70B-Instruct 9.006250 |
|
``` |
|
|
|
| Model | First turn | Second Turn | Average | |
|
| :---- | ---------: | ----------: | ------: | |
|
| Smaug-Llama-3-70B-Instruct | 9.40 | 9.01 | 9.21 | |
|
| GPT-4-Turbo | 9.38 | 9.00 | 9.19 | |
|
| Meta-Llama-3-70B-Instruct | 9.21 | 8.80 | 9.01 | |
|
|
|
This version of Smaug uses new techniques and new data compared to [Smaug-72B](https://huggingface.co/abacusai/Smaug-72B-v0.1), and more information will be released later on. For now, see the previous Smaug paper: https://arxiv.org/abs/2402.13228. |
