|
--- |
|
license: apache-2.0 |
|
language: |
|
- en |
|
- zh |
|
base_model: |
|
- Qwen/Qwen2.5-14B |
|
- Qwen/Qwen2.5-14B-Instruct |
|
- Qwen/Qwen2.5-14B-Instruct-1M |
|
- EVA-UNIT-01/EVA-Qwen2.5-14B-v0.2 |
|
- Azure99/Blossom-V6-14B |
|
- arcee-ai/Virtuoso-Small-v2 |
|
pipeline_tag: text-generation |
|
tags: |
|
- merge |
|
model-index: |
|
- name: Qwen2.5-14B-1M-YOYO-V3 |
|
results: |
|
- task: |
|
type: text-generation |
|
name: Text Generation |
|
dataset: |
|
name: IFEval (0-Shot) |
|
type: HuggingFaceH4/ifeval |
|
args: |
|
num_few_shot: 0 |
|
metrics: |
|
- type: inst_level_strict_acc and prompt_level_strict_acc |
|
value: 83.98 |
|
name: strict accuracy |
|
source: |
|
url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=YOYO-AI/Qwen2.5-14B-1M-YOYO-V3 |
|
name: Open LLM Leaderboard |
|
- task: |
|
type: text-generation |
|
name: Text Generation |
|
dataset: |
|
name: BBH (3-Shot) |
|
type: BBH |
|
args: |
|
num_few_shot: 3 |
|
metrics: |
|
- type: acc_norm |
|
value: 49.47 |
|
name: normalized accuracy |
|
source: |
|
url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=YOYO-AI/Qwen2.5-14B-1M-YOYO-V3 |
|
name: Open LLM Leaderboard |
|
- task: |
|
type: text-generation |
|
name: Text Generation |
|
dataset: |
|
name: MATH Lvl 5 (4-Shot) |
|
type: hendrycks/competition_math |
|
args: |
|
num_few_shot: 4 |
|
metrics: |
|
- type: exact_match |
|
value: 53.55 |
|
name: exact match |
|
source: |
|
url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=YOYO-AI/Qwen2.5-14B-1M-YOYO-V3 |
|
name: Open LLM Leaderboard |
|
- task: |
|
type: text-generation |
|
name: Text Generation |
|
dataset: |
|
name: GPQA (0-shot) |
|
type: Idavidrein/gpqa |
|
args: |
|
num_few_shot: 0 |
|
metrics: |
|
- type: acc_norm |
|
value: 10.51 |
|
name: acc_norm |
|
source: |
|
url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=YOYO-AI/Qwen2.5-14B-1M-YOYO-V3 |
|
name: Open LLM Leaderboard |
|
- task: |
|
type: text-generation |
|
name: Text Generation |
|
dataset: |
|
name: MuSR (0-shot) |
|
type: TAUR-Lab/MuSR |
|
args: |
|
num_few_shot: 0 |
|
metrics: |
|
- type: acc_norm |
|
value: 11.10 |
|
name: acc_norm |
|
source: |
|
url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=YOYO-AI/Qwen2.5-14B-1M-YOYO-V3 |
|
name: Open LLM Leaderboard |
|
- task: |
|
type: text-generation |
|
name: Text Generation |
|
dataset: |
|
name: MMLU-PRO (5-shot) |
|
type: TIGER-Lab/MMLU-Pro |
|
config: main |
|
split: test |
|
args: |
|
num_few_shot: 5 |
|
metrics: |
|
- type: acc |
|
value: 46.74 |
|
name: accuracy |
|
source: |
|
url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=YOYO-AI/Qwen2.5-14B-1M-YOYO-V3 |
|
name: Open LLM Leaderboard |
|
--- |
|
|
|
 |
|
# Qwen2.5-14B-1M-YOYO-V3 |
|
This time, I not only released the model but also shared some model merging insights that might be even more valuable than the model itself. |
|
|
|
Let’s start by looking at the initial merge configuration (YAML): |
|
```yaml |
|
merge_method: model_stock |
|
base_model: Qwen/Qwen2.5-14B |
|
models: |
|
- model: Qwen/Qwen2.5-14B-instruct |
|
- model: Qwen/Qwen2.5-14B-instruct-1M |
|
dtype: bfloat16 |
|
``` |
|
Does it seem like there are no issues at all? However, merged models occasionally exhibit **uncontrollable outputs**, likely due to significant discrepancies between instruction-tuned models and base models. |
|
|
|
To address this, I first attempted to directly integrate a fine-tuned model with smaller divergence from the base model, such as **Virtuoso-Small-v2**. |
|
|
|
This gave rise to [Qwen2.5-14B-YOYO-latest-V2](https://huggingface.co/YOYO-AI/Qwen2.5-14B-YOYO-latest-V2). |
|
```yaml |
|
merge_method: model_stock |
|
base_model: Qwen/Qwen2.5-14B |
|
models: |
|
- model: Qwen/Qwen2.5-14B-instruct |
|
- model: Qwen/Qwen2.5-14B-instruct-1M |
|
- model: arcee-ai/Virtuoso-Small-v2 |
|
dtype: bfloat16 |
|
name: Qwen2.5-14B-YOYO-latest-V2 |
|
``` |
|
Although the uncontrollable output issue has been addressed, the model still lacks stability. |
|
|
|
Through practical experimentation, I found that first merging **"high-divergence"** models (significantly different from the base) into **"low-divergence"** models (closer to the base) using the [DELLA](https://arxiv.org/abs/2406.11617) method, then applying the [Model Stock](https://arxiv.org/abs/2403.19522) method, ultimately produces a model that is not only more stable but also achieves better performance. |
|
|
|
## Key models used: |
|
*1. Low-divergence, high-performance models:* |
|
|
|
- Virtuoso-Small-v2 |
|
- Blossom-V6-14B |
|
|
|
*2. High-divergence, instruction-focused models:* |
|
|
|
- Qwen2.5-14B-instruct |
|
- Qwen2.5-14B-instruct-1M |
|
|
|
## DELLA Merge Configuration: |
|
```yaml |
|
models: |
|
- model: Qwen/Qwen2.5-14B-Instruct |
|
parameters: |
|
density: 1 |
|
weight: 1 |
|
lambda: 0.9 |
|
merge_method: della |
|
base_model: arcee-ai/Virtuoso-Small-v2 |
|
parameters: |
|
density: 1 |
|
weight: 1 |
|
lambda: 0.9 |
|
normalize: true |
|
int8_mask: true |
|
dtype: bfloat16 |
|
tokenizer_source: base |
|
name: Qwen2.5-14B-YOYO-della1 |
|
``` |
|
```yaml |
|
models: |
|
- model: Qwen/Qwen2.5-14B-Instruct-1M |
|
parameters: |
|
density: 1 |
|
weight: 1 |
|
lambda: 0.9 |
|
merge_method: della |
|
base_model: arcee-ai/Virtuoso-Small-v2 |
|
parameters: |
|
density: 1 |
|
weight: 1 |
|
lambda: 0.9 |
|
normalize: true |
|
int8_mask: true |
|
dtype: bfloat16 |
|
tokenizer_source: base |
|
name: Qwen2.5-14B-YOYO-della2 |
|
``` |
|
```yaml |
|
models: |
|
- model: Qwen/Qwen2.5-14B-Instruct |
|
parameters: |
|
density: 1 |
|
weight: 1 |
|
lambda: 0.9 |
|
merge_method: della |
|
base_model: Azure99/Blossom-V6-14B |
|
parameters: |
|
density: 1 |
|
weight: 1 |
|
lambda: 0.9 |
|
normalize: true |
|
int8_mask: true |
|
dtype: bfloat16 |
|
tokenizer_source: base |
|
name: Qwen2.5-14B-YOYO-della3 |
|
``` |
|
```yaml |
|
models: |
|
- model: Qwen/Qwen2.5-14B-Instruct-1M |
|
parameters: |
|
density: 1 |
|
weight: 1 |
|
lambda: 0.9 |
|
merge_method: della |
|
base_model: Azure99/Blossom-V6-14B |
|
parameters: |
|
density: 1 |
|
weight: 1 |
|
lambda: 0.9 |
|
normalize: true |
|
int8_mask: true |
|
dtype: bfloat16 |
|
tokenizer_source: base |
|
name: Qwen2.5-14B-YOYO-della4 |
|
``` |
|
This approach yielded four variants: |
|
- `Qwen2.5-14B-YOYO-della1` |
|
- `Qwen2.5-14B-YOYO-della2` |
|
- `Qwen2.5-14B-YOYO-della3` |
|
- `Qwen2.5-14B-YOYO-della4` |
|
|
|
## Base Model: |
|
To enhance base model roleplay and creative writing capabilities, I applied the same strategy: |
|
```yaml |
|
models: |
|
- model: EVA-UNIT-01/EVA-Qwen2.5-14B-v0.2 |
|
parameters: |
|
density: 1 |
|
weight: 1 |
|
lambda: 0.9 |
|
merge_method: della |
|
base_model: Qwen/Qwen2.5-14B |
|
parameters: |
|
density: 1 |
|
weight: 1 |
|
lambda: 0.9 |
|
normalize: true |
|
int8_mask: true |
|
dtype: bfloat16 |
|
tokenizer_source: base |
|
name: EVA-Qwen2.5-14B-base |
|
``` |
|
Next, I extended the context length using the SCE method: |
|
```yaml |
|
merge_method: sce |
|
models: |
|
- model: EVA-Qwen2.5-14B-base |
|
base_model: Qwen/Qwen2.5-14B-Instruct-1M |
|
parameters: |
|
select_topk: 1 |
|
dtype: bfloat16 |
|
tokenizer_source: base |
|
normalize: true |
|
int8_mask: true |
|
name: Qwen2.5-14B-pro |
|
``` |
|
## Final Merge Step: |
|
```yaml |
|
merge_method: model_stock |
|
base_model: Qwen2.5-14B-pro |
|
models: |
|
- model: Qwen2.5-14B-YOYO-della1 |
|
- model: Qwen2.5-14B-YOYO-della2 |
|
- model: Qwen2.5-14B-YOYO-della3 |
|
- model: Qwen2.5-14B-YOYO-della4 |
|
dtype: bfloat16 |
|
tokenizer_source: base |
|
int8_mask: true |
|
normalize: true |
|
name: Qwen2.5-14B-1M-YOYO-V3 |
|
``` |
|
I hope this helps! |
|
|
|
# [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard) |
|
Detailed results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/YOYO-AI__Qwen2.5-14B-1M-YOYO-V3-details) |
|
|
|
| Metric |Value| |
|
|-------------------|----:| |
|
|Avg. |42.56| |
|
|IFEval (0-Shot) |83.98| |
|
|BBH (3-Shot) |49.47| |
|
|MATH Lvl 5 (4-Shot)|53.55| |
|
|GPQA (0-shot) |10.51| |
|
|MuSR (0-shot) |11.10| |
|
|MMLU-PRO (5-shot) |46.74| |
|
|
