metadata

library_name: transformers
license: llama2

Llama-3-Smaug-8B

Built with Meta Llama 3

This model was built using the Smaug recipe for improving performance on real world multi-turn conversations applied to meta-llama/Meta-Llama-3-8B.

Model Description

Developed by: Abacus.AI
License: https://llama.meta.com/llama3/license/
Finetuned from model: meta-llama/Meta-Llama-3-8B.

Evaluation

MT-Bench

########## First turn ##########
                   score
model             turn
Llama-3-Smaug-8B 1   8.79375
Meta-Llama-3-8B-Instruct 1   8.53125
########## Second turn ##########
                   score
model             turn
Llama-3-Smaug-8B 2   7.7250
Meta-Llama-3-8B-Instruct 2   7.5500
########## Average ##########
                 score
model
Llama-3-Smaug-8B  8.259375
Meta-Llama-3-8B-Instruct 8.040625

Model	First turn	Second Turn	Average
Llama-3-Smaug-8B	8.79	7.73	8.26
Llama-3-8B-Instruct	8.53	7.55	8.04

This version of Smaug uses new techniques and new data compared to Smaug-72B, and more information will be released later on. For now, see the previous Smaug paper: https://arxiv.org/abs/2402.13228.