File size: 6,823 Bytes
f94e614 d2c8b43 b3afce6 d2c8b43 f94e614 b3afce6 01247f4 b3afce6 9480d31 b3afce6 9480d31 b3afce6 9480d31 b3afce6 9480d31 b3afce6 9480d31 b3afce6 01247f4 d2c8b43 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 |
---
license: llama3
library_name: transformers
tags:
- mergekit
- merge
base_model:
- Undi95/Meta-Llama-3-8B-Instruct-hf
model-index:
- name: Llama-3-8B-Ultra-Instruct
results:
- task:
type: text-generation
name: Text Generation
dataset:
name: AI2 Reasoning Challenge (25-Shot)
type: ai2_arc
config: ARC-Challenge
split: test
args:
num_few_shot: 25
metrics:
- type: acc_norm
value: 64.59
name: normalized accuracy
source:
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=elinas/Llama-3-8B-Ultra-Instruct
name: Open LLM Leaderboard
- task:
type: text-generation
name: Text Generation
dataset:
name: HellaSwag (10-Shot)
type: hellaswag
split: validation
args:
num_few_shot: 10
metrics:
- type: acc_norm
value: 81.63
name: normalized accuracy
source:
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=elinas/Llama-3-8B-Ultra-Instruct
name: Open LLM Leaderboard
- task:
type: text-generation
name: Text Generation
dataset:
name: MMLU (5-Shot)
type: cais/mmlu
config: all
split: test
args:
num_few_shot: 5
metrics:
- type: acc
value: 68.32
name: accuracy
source:
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=elinas/Llama-3-8B-Ultra-Instruct
name: Open LLM Leaderboard
- task:
type: text-generation
name: Text Generation
dataset:
name: TruthfulQA (0-shot)
type: truthful_qa
config: multiple_choice
split: validation
args:
num_few_shot: 0
metrics:
- type: mc2
value: 52.8
source:
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=elinas/Llama-3-8B-Ultra-Instruct
name: Open LLM Leaderboard
- task:
type: text-generation
name: Text Generation
dataset:
name: Winogrande (5-shot)
type: winogrande
config: winogrande_xl
split: validation
args:
num_few_shot: 5
metrics:
- type: acc
value: 76.95
name: accuracy
source:
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=elinas/Llama-3-8B-Ultra-Instruct
name: Open LLM Leaderboard
- task:
type: text-generation
name: Text Generation
dataset:
name: GSM8k (5-shot)
type: gsm8k
config: main
split: test
args:
num_few_shot: 5
metrics:
- type: acc
value: 70.36
name: accuracy
source:
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=elinas/Llama-3-8B-Ultra-Instruct
name: Open LLM Leaderboard
---
# Llama-3-8B-Ultra-Instruct
This is a merge of pre-trained language models created using [mergekit](https://github.com/cg123/mergekit).
Hello everyone this is Dampf, creator of the Destroyer series!
*looks around* Oh, now I'm on Elinas' HF account. As you can see, I'm quite the traveler!
This time, I'm introducing you to 8B-Ultra-Instruct. It is a small general purpose model that combines the most powerful instruct models with enticing roleplaying models. It will introduce better RAG capabilities in the form of Bagel to Llama 3 8B Instruct as well as German multilanguage, higher general intelligence and vision support. A model focused on Biology adds knowledge in the medical field.
As for roleplay, it features two of the hottest models right now. Those are known to be high quality and for being uncensored. So this model might put out harmful responses. We are not responsible for what you do with this model and please take everything the model says with a huge grain of salt.
Lastly, you might notice I'm conversative with the weight values in the final merge. This is because I believe L8B Instruct is a very dense model that's already great and doesn't need a lot more data. So instead of reaching the weight value of 1 in a ties merge, I'm only using a total of 0,65. This is to preserve Llama Instruct's intelligence and knowledge, while adding a little bit of the aforementioned models as salt in the soup.
A huge thank you for all the creators of the datasets. Those include Undi95, Jon Durbin, Aaditya, VAGOsolutions, Teknium, Camel and many more. They deserve all the credit. And of course, thank you Elinas for providing the compute.
## Merge Details
### Merge Method
This model was merged using the [DARE](https://arxiv.org/abs/2311.03099) [TIES](https://arxiv.org/abs/2306.01708) merge method using [Undi95/Meta-Llama-3-8B-Instruct-hf](https://huggingface.co/Undi95/Meta-Llama-3-8B-Instruct-hf) as a base.
### Models Merged
The following models were included in the merge:
* llama-3-8B-ultra-instruct/InstructPart
* llama-3-8B-ultra-instruct/RPPart
### Configuration
The following YAML configuration was used to produce this model:
```yaml
models:
- model: ChaoticNeutrals/Poppy_Porpoise-v0.7-L3-8B
parameters:
weight: 0.4
- model: Undi95/Llama-3-LewdPlay-8B-evo
parameters:
weight: 0.5
- model: jondurbin/bagel-8b-v1.0
parameters:
weight: 0.1
merge_method: dare_ties
dtype: bfloat16
base_model: Undi95/Meta-Llama-3-8B-hf
name: RPPart
---
models:
- model: Weyaxi/Einstein-v6.1-Llama3-8B
parameters:
weight: 0.6
- model: VAGOsolutions/Llama-3-SauerkrautLM-8b-Instruct
parameters:
weight: 0.3
- model: aaditya/OpenBioLLM-Llama3-8B
parameters:
weight: 0.1
merge_method: dare_ties
base_model: Undi95/Meta-Llama-3-8B-hf
dtype: bfloat16
name: InstructPart
---
models:
- model: RPPart
parameters:
weight: 0.39
- model: InstructPart
parameters:
weight: 0.26
merge_method: dare_ties
base_model: Undi95/Meta-Llama-3-8B-Instruct-hf
dtype: bfloat16
name: Llama-3-8B-Ultra-Instruct
```
### Chat Template (Llama 3 Official)
```
<|begin_of_text|><|start_header_id|>system<|end_header_id|>
{system_prompt}<|eot_id|><|start_header_id|>user<|end_header_id|>
{input}<|eot_id|><|start_header_id|>assistant<|end_header_id|>
{output}<|eot_id|>
```
# [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard)
Detailed results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/details_elinas__Llama-3-8B-Ultra-Instruct)
| Metric |Value|
|---------------------------------|----:|
|Avg. |69.11|
|AI2 Reasoning Challenge (25-Shot)|64.59|
|HellaSwag (10-Shot) |81.63|
|MMLU (5-Shot) |68.32|
|TruthfulQA (0-shot) |52.80|
|Winogrande (5-shot) |76.95|
|GSM8k (5-shot) |70.36|
|