jacobmorrison
commited on
Commit
•
6dc0b1c
1
Parent(s):
1548cdc
Create README.md
Browse files
README.md
ADDED
@@ -0,0 +1,166 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
+
---
|
2 |
+
license: llama3.1
|
3 |
+
language:
|
4 |
+
- en
|
5 |
+
pipeline_tag: text-classification
|
6 |
+
datasets:
|
7 |
+
- allenai/llama-3.1-tulu-3-8b-preference-mixture
|
8 |
+
base_model:
|
9 |
+
- allenai/Llama-3.1-Tulu-3-8B-SFT
|
10 |
+
---
|
11 |
+
|
12 |
+
<img src="https://huggingface.co/datasets/allenai/blog-images/resolve/main/tulu3/Tulu3-logo.png" alt="Tulu 3 banner" width="800" style="margin-left:'auto' margin-right:'auto' display:'block'"/>
|
13 |
+
|
14 |
+
# Llama-3.1-Tulu-3-8B-RM
|
15 |
+
|
16 |
+
Tülu3 is a leading instruction following model family, offering fully open-source data, code, and recipes designed to serve as a comprehensive guide for modern post-training techniques.
|
17 |
+
Tülu3 is designed for state-of-the-art performance on a diversity of tasks in addition to chat, such as MATH, GSM8K, and IFEval.
|
18 |
+
|
19 |
+
## Model description
|
20 |
+
|
21 |
+
- **Model type:** A model trained on a mix of publicly available, synthetic and human-created datasets.
|
22 |
+
- **Language(s) (NLP):** Primarily English
|
23 |
+
- **License:** Llama 3.1 Community License Agreement
|
24 |
+
- **Finetuned from model:** allenai/Llama-3.1-Tulu-3-8B-SFT
|
25 |
+
|
26 |
+
### Model Sources
|
27 |
+
|
28 |
+
- **Training Repository:** https://github.com/allenai/open-instruct
|
29 |
+
- **Eval Repository:** https://github.com/allenai/olmes
|
30 |
+
- **Paper:** https://allenai.org/papers/tulu-3-report.pdf (arXiv soon)
|
31 |
+
- **Demo:** https://playground.allenai.org/
|
32 |
+
|
33 |
+
### Model Family
|
34 |
+
|
35 |
+
| **Stage** | **Llama 3.1 8B** | **Llama 3.1 70B** |
|
36 |
+
|----------------------|----------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------|
|
37 |
+
| **Base Model** | [meta-llama/Llama-3.1-8B](https://huggingface.co/meta-llama/Llama-3.1-8B) | [meta-llama/Llama-3.1-70B](https://huggingface.co/meta-llama/Llama-3.1-70B) |
|
38 |
+
| **SFT** | [allenai/Llama-3.1-Tulu-3-8B-SFT](https://huggingface.co/allenai/Llama-3.1-Tulu-3-8B-SFT) | [allenai/Llama-3.1-Tulu-3-70B-SFT](https://huggingface.co/allenai/Llama-3.1-Tulu-3-70B-SFT) |
|
39 |
+
| **DPO** | [allenai/Llama-3.1-Tulu-3-8B-DPO](https://huggingface.co/allenai/Llama-3.1-Tulu-3-8B-DPO) | [allenai/Llama-3.1-Tulu-3-70B-DPO](https://huggingface.co/allenai/Llama-3.1-Tulu-3-70B-DPO) |
|
40 |
+
| **Final Models (RLVR)** | [allenai/Llama-3.1-Tulu-3-8B](https://huggingface.co/allenai/Llama-3.1-Tulu-3-8B) | [allenai/Llama-3.1-Tulu-3-70B](https://huggingface.co/allenai/Llama-3.1-Tulu-3-70B) |
|
41 |
+
| **Reward Model (RM)**| [allenai/Llama-3.1-Tulu-3-8B-RM](https://huggingface.co/allenai/Llama-3.1-Tulu-3-8B-RM) | (Same as 8B) |
|
42 |
+
|
43 |
+
## Using the model
|
44 |
+
|
45 |
+
### Loading with HuggingFace
|
46 |
+
|
47 |
+
To load the model with HuggingFace, use the following snippet:
|
48 |
+
```
|
49 |
+
from transformers import AutoModelForSequenceClassification
|
50 |
+
|
51 |
+
tulu_model = AutoModelForSequenceClassification.from_pretrained("allenai/Llama-3.1-Tulu-3-8B-RM")
|
52 |
+
```
|
53 |
+
|
54 |
+
### Chat template
|
55 |
+
|
56 |
+
The chat template for our models is formatted as:
|
57 |
+
```
|
58 |
+
<|user|>\nHow are you doing?\n<|assistant|>\nI'm just a computer program, so I don't have feelings, but I'm functioning as expected. How can I assist you today?<|endoftext|>
|
59 |
+
```
|
60 |
+
Or with new lines expanded:
|
61 |
+
```
|
62 |
+
<|user|>
|
63 |
+
How are you doing?
|
64 |
+
<|assistant|>
|
65 |
+
I'm just a computer program, so I don't have feelings, but I'm functioning as expected. How can I assist you today?<|endoftext|>
|
66 |
+
```
|
67 |
+
It is embedded within the tokenizer as well, for `tokenizer.apply_chat_template`.
|
68 |
+
|
69 |
+
### Bias, Risks, and Limitations
|
70 |
+
|
71 |
+
The Tülu3 models have limited safety training, but are not deployed automatically with in-the-loop filtering of responses like ChatGPT, so the model can produce problematic outputs (especially when prompted to do so).
|
72 |
+
It is also unknown what the size and composition of the corpus was used to train the base Llama 3.1 models, however it is likely to have included a mix of Web data and technical sources like books and code.
|
73 |
+
See the Falcon 180B model card for an example of this.
|
74 |
+
|
75 |
+
|
76 |
+
## Performance
|
77 |
+
|
78 |
+
| Benchmark (eval) | Tülu 3 SFT 8B | Tülu 3 DPO 8B | Tülu 3 8B | Llama 3.1 8B Instruct | Qwen 2.5 7B Instruct | Magpie 8B | Gemma 2 9B Instruct | Ministral 8B Instruct |
|
79 |
+
|---------------------------------|----------------|----------------|------------|------------------------|----------------------|-----------|---------------------|-----------------------|
|
80 |
+
| **Avg.** | 60.4 | 64.4 | **64.8** | 62.2 | 57.8 | 44.7 | 55.2 | 58.3 |
|
81 |
+
| **MMLU (0 shot, CoT)** | 65.9 | 68.7 | 68.2 | 71.2 | **76.6** | 62.0 | 74.6 | 68.5 |
|
82 |
+
| **PopQA (15 shot)** | **29.3** | 29.3 | 29.1 | 20.2 | 18.1 | 22.5 | 28.3 | 20.2 |
|
83 |
+
| **TruthfulQA (6 shot)** | 46.8 | 56.1 | 55.0 | 55.1 | **63.1** | 57.0 | 61.4 | 55.5 |
|
84 |
+
| **BigBenchHard (3 shot, CoT)** | **67.9** | 65.8 | 66.0 | 62.8 | 21.7 | 0.9 | 2.5 | 56.2 |
|
85 |
+
| **DROP (3 shot)** | 61.3 | 62.5 | **62.6** | 61.5 | 54.4 | 49.4 | 58.8 | 56.2 |
|
86 |
+
| **MATH (4 shot CoT, Flex)** | 31.5 | 42.0 | **43.7** | 42.5 | 14.8 | 5.1 | 29.8 | 40.0 |
|
87 |
+
| **GSM8K (8 shot, CoT)** | 76.2 | 84.3 | **87.6** | 83.4 | 83.8 | 61.2 | 79.7 | 80.0 |
|
88 |
+
| **HumanEval (pass@10)** | 86.2 | 83.9 | 83.9 | 86.3 | **93.1** | 75.4 | 71.7 | 91.0 |
|
89 |
+
| **HumanEval+ (pass@10)** | 81.4 | 78.6 | 79.2 | 82.9 | **89.7** | 69.1 | 67.0 | 88.5 |
|
90 |
+
| **IFEval (prompt loose)** | 72.8 | 81.1 | **82.4** | 80.6 | 74.7 | 38.8 | 69.9 | 56.4 |
|
91 |
+
| **AlpacaEval 2 (LC % win)** | 12.4 | 33.5 | 34.5 | 24.2 | 29.0 | **49.0** | 43.7 | 31.4 |
|
92 |
+
| **Safety (6 task avg.)** | **93.1** | 87.2 | 85.5 | 75.2 | 75.0 | 46.4 | 75.5 | 56.2 |
|
93 |
+
|
94 |
+
| Benchmark (eval) | Tülu 3 70B SFT | Tülu 3 DPO 70B | Tülu 3 70B | Llama 3.1 70B Instruct | Qwen 2.5 72B Instruct | Hermes 3 Llama 3.1 70B | Nemotron Llama 3.1 70B |
|
95 |
+
|---------------------------------|-----------------|-----------------|-------------|-------------------------|-----------------------|------------------------|-------------------------|
|
96 |
+
| **Avg.** | 72.6 | 75.9 | **76.0** | 73.4 | 71.5 | 68.3 | 65.5 |
|
97 |
+
| **MMLU (0 shot, CoT)** | 78.9 | 83.3 | 83.1 | 85.3 | **85.5** | 80.4 | 83.8 |
|
98 |
+
| **PopQA (15 shot)** | **48.6** | 46.3 | 46.5 | 46.4 | 30.6 | 48.1 | 36.4 |
|
99 |
+
| **TruthfulQA (6 shot)** | 55.7 | 67.9 | 67.6 | 66.8 | **69.9** | 66.5 | 62.6 |
|
100 |
+
| **BigBenchHard (3 shot, CoT)** | **82.7** | 81.8 | 82.0 | 73.8 | 67.2 | 82.1 | 0.7 |
|
101 |
+
| **DROP (3 shot)** | **77.2** | 74.1 | 74.3 | 77.0 | 34.2 | 73.2 | 68.8 |
|
102 |
+
| **MATH (4 shot CoT, Flex)** | 53.7 | 62.3 | 63.0 | 56.4 | **74.3** | 41.9 | 55.0 |
|
103 |
+
| **GSM8K (8 shot, CoT)** | 91.1 | 93.5 | 93.5 | **93.7** | 89.5 | 90.0 | 84.7 |
|
104 |
+
| **HumanEval (pass@10)** | 92.9 | 92.4 | 92.4 | 93.6 | 94.0 | 89.6 | **94.1** |
|
105 |
+
| **HumanEval+ (pass@10)** | 87.3 | 88.4 | 88.0 | 89.5 | **90.8** | 85.9 | 85.5 |
|
106 |
+
| **IFEval (prompt loose)** | 82.1 | 82.6 | 83.2 | **88.0** | 87.6 | 76.0 | 79.9 |
|
107 |
+
| **AlpacaEval 2 (LC % win)** | 26.3 | 49.6 | 49.8 | 33.4 | 47.7 | 28.4 | **66.1** |
|
108 |
+
| **Safety (6 task avg.)** | **94.4** | 89.0 | 88.3 | 76.5 | 87.0 | 57.9 | 69.0 |
|
109 |
+
|
110 |
+
|
111 |
+
## Hyperparamters
|
112 |
+
|
113 |
+
Reward Modeling:
|
114 |
+
- **Learning Rate**: 3E-6
|
115 |
+
- **Effective Batch Size:** 256
|
116 |
+
- **Max. Sequence Length:** 2048
|
117 |
+
- **Learning Rate Schedule:** Linear
|
118 |
+
- **Num. Epochs:** 1
|
119 |
+
- **Grad. Norm Threshold:** 1.0
|
120 |
+
|
121 |
+
## License and use
|
122 |
+
|
123 |
+
All Llama 3.1 Tülu3 models are released under Meta's [Llama 3.1 Community License Agreement](https://www.llama.com/llama3_1/license/).
|
124 |
+
Llama 3.1 is licensed under the Llama 3.1 Community License, Copyright © Meta Platforms, Inc.
|
125 |
+
Tülu3 is intended for research and educational use.
|
126 |
+
For more information, please see our [Responsible Use Guidelines](https://allenai.org/responsible-use).
|
127 |
+
|
128 |
+
The models have been fine-tuned using a dataset mix with outputs generated from third party models and are subject to additional terms:
|
129 |
+
[Gemma Terms of Use](https://ai.google.dev/gemma/terms) and [Qwen License Agreement](https://huggingface.co/Qwen/Qwen2.5-72B-Instruct/blob/main/LICENSE) (models were improved using Qwen 2.5).
|
130 |
+
|
131 |
+
|
132 |
+
## Citation
|
133 |
+
|
134 |
+
If Tülu3 or any of the related materials were helpful to your work, please cite:
|
135 |
+
```
|
136 |
+
@article{lambert2024tulu3,
|
137 |
+
title = {Tülu 3: Pushing Frontiers in Open Language Model Post-Training},
|
138 |
+
author = {
|
139 |
+
Nathan Lambert and
|
140 |
+
Jacob Morrison and
|
141 |
+
Valentina Pyatkin and
|
142 |
+
Shengyi Huang and
|
143 |
+
Hamish Ivison and
|
144 |
+
Faeze Brahman and
|
145 |
+
Lester James V. Miranda and
|
146 |
+
Alisa Liu and
|
147 |
+
Nouha Dziri and
|
148 |
+
Shane Lyu and
|
149 |
+
Yuling Gu and
|
150 |
+
Saumya Malik and
|
151 |
+
Victoria Graf and
|
152 |
+
Jena D. Hwang and
|
153 |
+
Jiangjiang Yang and
|
154 |
+
Ronan Le Bras and
|
155 |
+
Oyvind Tafjord and
|
156 |
+
Chris Wilhelm and
|
157 |
+
Luca Soldaini and
|
158 |
+
Noah A. Smith and
|
159 |
+
Yizhong Wang and
|
160 |
+
Pradeep Dasigi and
|
161 |
+
Hannaneh Hajishirzi
|
162 |
+
},
|
163 |
+
year = {2024},
|
164 |
+
email = {tulu@allenai.org}
|
165 |
+
}
|
166 |
+
```
|