Description
This repo contains bf16 files of Nyxene-v1-11B. Just new version with some new things.
Model used
- Intel/neural-chat-7b-v3-3-Slerp
- AIDC-ai-business/Marcoroni-7B-v3
- rwitz/go-bruins-v2
- chargoddard/loyal-piano-m7-cdpo
Prompt template
Just use chatml.
The secret sauce
go-bruins-loyal-piano-11B :
slices:
- sources:
- model: rwitz/go-bruins-v2
layer_range: [0, 24]
- sources:
- model: chargoddard/loyal-piano-m7-cdpo
layer_range: [8, 32]
merge_method: passthrough
dtype: bfloat16
neural-marcoroni-11B :
slices:
- sources:
- model: AIDC-ai-business/Marcoroni-7B-v3
layer_range: [0, 24]
- sources:
- model: Intel/neural-chat-7b-v3-3-Slerp
layer_range: [8, 32]
merge_method: passthrough
dtype: bfloat16
Nyxene-11B :
slices:
- sources:
- model: "./go-bruins-loyal-piano-11B"
layer_range: [0, 48]
- model: "./neural-marcoroni-11B"
layer_range: [0, 48]
merge_method: slerp
base_model: "./go-bruins-loyal-piano-11B"
parameters:
t:
- filter: lm_head
value: [0.5]
- filter: embed_tokens
value: [0.75]
- filter: self_attn
value: [0.75, 0.25]
- filter: mlp
value: [0.25, 0.75]
- filter: layernorm
value: [0.5, 0.5]
- filter: modelnorm
value: [0.5]
- value: 0.5 # fallback for rest of tensors
dtype: bfloat16
I use mergekit for all the manipulation told here.
Thanks to the Undi95 for the original 11B mistral merge recipe.
Open LLM Leaderboard Evaluation Results
Detailed results can be found here
Metric | Value |
---|---|
Avg. | 70.72 |
AI2 Reasoning Challenge (25-Shot) | 69.62 |
HellaSwag (10-Shot) | 85.33 |
MMLU (5-Shot) | 64.75 |
TruthfulQA (0-shot) | 60.91 |
Winogrande (5-shot) | 80.19 |
GSM8k (5-shot) | 63.53 |
- Downloads last month
- 719
This model does not have enough activity to be deployed to Inference API (serverless) yet. Increase its social
visibility and check back later, or deploy to Inference Endpoints (dedicated)
instead.
Model tree for beberik/Nyxene-v3-11B
Evaluation results
- normalized accuracy on AI2 Reasoning Challenge (25-Shot)test set Open LLM Leaderboard69.620
- normalized accuracy on HellaSwag (10-Shot)validation set Open LLM Leaderboard85.330
- accuracy on MMLU (5-Shot)test set Open LLM Leaderboard64.750
- mc2 on TruthfulQA (0-shot)validation set Open LLM Leaderboard60.910
- accuracy on Winogrande (5-shot)validation set Open LLM Leaderboard80.190
- accuracy on GSM8k (5-shot)test set Open LLM Leaderboard63.530