|
---
|
|
license: llama3.1
|
|
datasets:
|
|
- survivi/Llama-3-SynE-Dataset
|
|
- hfl/stem_zh_instruction
|
|
- llamafactory/alpaca_zh
|
|
- llamafactory/alpaca_gpt4_zh
|
|
- hfl/ruozhiba_gpt4
|
|
- codingsteven/Llama-3-8B-chat
|
|
language:
|
|
- zh
|
|
base_model:
|
|
- meta-llama/Llama-3.1-8B
|
|
model-index:
|
|
- name: Control-LLM-Llama3.1-8B-SynE-Concat16-Dlerp
|
|
results:
|
|
- task:
|
|
type: pretraining-evaluation
|
|
dataset:
|
|
type: mixed
|
|
name: Pretraining Evaluation Dataset
|
|
metrics:
|
|
- name: exact_match,strict-match (meta_pretrain)
|
|
type: exact_match
|
|
value: 0.48514264142803215
|
|
stderr: 0.003513307445696379
|
|
verified: false
|
|
- name: exact_match,strict-match (meta_bbh_3shot_cot_pretrain)
|
|
type: exact_match
|
|
value: 0.6817693134695131
|
|
stderr: 0.0057729694388110805
|
|
verified: false
|
|
- name: acc,none (meta_mmlu_5shot_pretrain)
|
|
type: accuracy
|
|
value: 0.65596068936049
|
|
stderr: 0.0040090726054856874
|
|
verified: false
|
|
- name: exact_match,strict-match (meta_mmlu_pro_5shot_pretrain)
|
|
type: exact_match
|
|
value: 0.3787400265957447
|
|
stderr: 0.004422383756050139
|
|
verified: false
|
|
- task:
|
|
type: chinese-evaluation
|
|
dataset:
|
|
type: mixed
|
|
name: Chinese Evaluation Dataset
|
|
metrics:
|
|
- name: exact_match,strict-match (zh_pretrain_multishot)
|
|
type: exact_match
|
|
value: 0.44848391089108913
|
|
stderr: 0.004255614019851072
|
|
verified: false
|
|
- name: acc,none (ceval-valid)
|
|
type: accuracy
|
|
value: 0.5698365527488856
|
|
stderr: 0.012893833892221353
|
|
verified: false
|
|
- name: exact_match,strict-match (ceval-valid-pretrain-cot_zh)
|
|
type: exact_match
|
|
value: 0.4472511144130758
|
|
stderr: 0.013203606600472227
|
|
verified: false
|
|
- name: acc,none (cmmlu)
|
|
type: accuracy
|
|
value: 0.5602659298912105
|
|
stderr: 0.0044928840587441605
|
|
verified: false
|
|
- name: exact_match,strict-match (cmmlu_pretrain_cot_zh)
|
|
type: exact_match
|
|
value: 0.4486271801070627
|
|
stderr: 0.00449553418468653
|
|
verified: false
|
|
---
|
|
# Control-LLM-Llama3.1-8B-SynE-Concat16-Dlerp
|
|
This is a fine-tuned model of Llama-3.1-8B for muliligual-Chinese tasks on SynE dataset by Control LLM-Concat16-Dlerp.
|
|
|
|
## Evaluation Results
|
|
Here is an overview of the evaluation results and findings:
|
|
|
|
### Benchmark Results Table
|
|
|
|
The table below summarizes evaluation results across Chinese tasks and original capabilities.
|
|
|
|
| **Model** | **CEval** | **CEvalC** | **CMMLU** | **CMMLUC** | **C-Avg** | **BBH** | **MLU** | **MLUP** | **O-Avg** | **Overall** |
|
|
|--------------------|-----------|------------|-----------|------------|-----------|---------|---------|----------|-----------|-------------|
|
|
| Llama3.1-8B | 48.3 | 12.8 | 51.1 | 14.1 | 13.9 | 65.2 | 65.4 | 35.5 | 45.9 | 29.9 |
|
|
| Llama-3-SynE | 57.7 | 22.3 | 57.1 | 22.8 | 22.8 | 61.9 | 64.0 | 32.6 | 42.9 | 32.9 |
|
|
| Full Param Tune | 59.0 | 40.2 | **60.2** | 44.3 | 43.8 | 64.8 | 64.9 | 35.0 | 45.4 | 44.6 |
|
|
| Stack Expansion | 56.0 | 32.7 | 55.2 | 33.4 | 33.3 | 62.3 | 65.6 | 35.3 | 44.8 | 39.1 |
|
|
| Concat-Lerp | 57.1 | 34.8 | 57.0 | 37.4 | 37.1 | 64.4 | 64.6 | 35.8 | 45.9 | 41.5 |
|
|
| Hybrid Expansion | **58.9** | 44.7 | 57.9 | 44.3 | 44.4 | 65.1 | **65.7**| 36.9 | 46.8 | 45.6 |
|
|
| **Control LLM*** | 57.0 | **44.7** | 56.0 | **44.9** | **44.8** | **68.2**| 65.6 | **37.9** | **48.5** | **46.7** |
|
|
|
|
---
|
|
|
|
### Explanation:
|
|
- **CEval**: Chinese Evaluation
|
|
- **CEvalC**: Chinese Evaluation (CoT - Chain of Thought)
|
|
- **CMMLU**: Chinese MMLU
|
|
- **CMMLUC**: Chinese MMLU (CoT)
|
|
- **C-Avg**: Chinese - Size Weighted Average across CEval, CEvalC, CMMLU, and CMMLUC
|
|
- **BBH**: BigBench Hard
|
|
- **MLU**: MMLU (Massive Multitask Language Understanding)
|
|
- **MLUP**: MMLU Pro
|
|
- **O-Avg**: Original Capability - Size Weighted Average across BBH, MLU, and MLUP
|
|
- **Overall**: Combined average across all tasks
|
|
|
