amd
/

X-EcoMLA-3B3B-fixed-kv816-DPO

Safetensors

llama

alignment-handbook

Generated from Trainer

Model card Files Files and versions

xet

Community

Mingyuyang-1 commited on Jun 25

Commit

b2324c7

1 Parent(s): 30a8ac1

Update README.md

Browse files

Files changed (1) hide show

README.md +19 -18

README.md CHANGED Viewed

@@ -104,24 +104,25 @@ python benchmark/llm_eval/lm_harness_eval.py \
 X-EcoMLA was evaluated on the Language Model Harness benchmark for zero-shot tasks and compared against its base model and other post-training methods. The results demonstrate that Zebra-Llama provides a superior balance of performance and efficiency.
 | Tasks             | Metric   |  Llama-3.2-3B-Instruct | X-EcoMLA-3B3B-fixed-kv816-DPO | X-EcoMLA-3B3B-dynamic-0.95-DPO |
 |-------------------|----------|----------------: |----------------: |----------------:|
-| arc_challenge     | acc      | 0.3575 (±0.0140) | 0.3643 (±0.0141) | 0.3686 (±0.0141)|
-|                   | acc_norm | 0.3797 (±0.0142) | 0.3993 (±0.0143) | 0.4121 (±0.0144)|
-| arc_easy          | acc      | 0.6843 (±0.0095) | 0.6873 (±0.0095) | 0.6932 (±0.0095)|
-|                   | acc_norm | 0.6351 (±0.0099) | 0.6389 (±0.0099) | 0.6486 (±0.0098)|
-| hellaswag         | acc      | 0.4506 (±0.0050) | 0.4483 (±0.0050) | 0.4459 (±0.0050)|
-|                   | acc_norm | 0.6077 (±0.0049) | 0.6073 (±0.0049) | 0.6096 (±0.0049)|
-| mmlu              | acc      | 0.4609 (±0.0918) | 0.4239 (±0.0785) | 0.4286 (±0.0809)|
-| - humanities      | acc      | 0.4397 (±0.0763) | 0.4064 (±0.0663) | 0.4013 (±0.0733)|
-| - other           | acc      | 0.5204 (±0.0868) | 0.4583 (±0.0760) | 0.4747 (±0.0774)|
-| - social_sciences | acc      | 0.5109 (±0.0843) | 0.4686 (±0.0735) | 0.4729 (±0.0734)|
-| - stem            | acc      | 0.3850 (±0.0900) | 0.3723 (±0.0818) | 0.3806 (±0.0798)|
-| openbookqa        | acc      | 0.2440 (±0.0192) | 0.2560 (±0.0195) | 0.2660 (±0.0198)|
-|                   | acc_norm | 0.3500 (±0.0214) | 0.3780 (±0.0217) | 0.3760 (±0.0217)|
-| piqa              | acc      | 0.7405 (±0.0102) | 0.7443 (±0.0102) | 0.7301 (±0.0104)|
-|                   | acc_norm | 0.7437 (±0.0102) | 0.7492 (±0.0101) | 0.7443 (±0.0102)|
-| pubmedqa          | acc      | 0.6020 (±0.0219) | 0.5880 (±0.0220) | 0.5860 (±0.0220)|
-| race              | acc      | 0.3809 (±0.0150) | 0.4077 (±0.0152) | 0.3923 (±0.0151)|
-| winogrande        | acc      | 0.5967 (±0.0138) | 0.6054 (±0.0137) | 0.5833 (±0.0139)|
 ## Conclusion
 X-EcoMLA demonstrates an efficient technique to upcycle pre-trained Transformers into MLA modules to compress KV cache. This work highlights the viability of post-training hybridization as a cost-effective and environmentally sustainable alternative to full retraining, paving the way for the deployment of powerful LLMs in resource-constrained environments.

 X-EcoMLA was evaluated on the Language Model Harness benchmark for zero-shot tasks and compared against its base model and other post-training methods. The results demonstrate that Zebra-Llama provides a superior balance of performance and efficiency.
 | Tasks             | Metric   |  Llama-3.2-3B-Instruct | X-EcoMLA-3B3B-fixed-kv816-DPO | X-EcoMLA-3B3B-dynamic-0.95-DPO |
 |-------------------|----------|----------------: |----------------: |----------------:|
+| arc_challenge     | acc      |    0.4369±0.0145 |    0.4753±0.0146 |   0.4710±0.0146 |
+|                   | acc_norm |    0.4590±0.0146 |    0.4821±0.0146 |   0.4846±0.0146 |
+| arc_easy          | acc      |    0.7428±0.0090 |    0.7660±0.0087 |   0.7580±0.0088 |
+|                   | acc_norm |    0.6776±0.0096 |    0.7045±0.0094 |   0.6999±0.0094 |
+| hellaswag         | acc      |    0.5222±0.0050 |    0.5288±0.0050 |   0.5320±0.0050 |
+|                   | acc_norm |    0.7036±0.0046 |    0.7224±0.0045 |   0.7226±0.0045 |
+| mmlu              | acc      |    0.6046±0.1057 |    0.5742±0.1014 |   0.5773±0.1028 |
+| - humanities      | acc      |    0.5926±0.0826 |    0.5507±0.0843 |   0.5518±0.0851 |
+| - other           | acc      |    0.6598±0.1118 |    0.6312±0.1011 |   0.6344±0.1070 |
+| - social_sciences | acc      |    0.6701±0.0712 |    0.6383±0.0741 |   0.6422±0.0765 |
+| - stem            | acc      |    0.5043±0.1122 |    0.4906±0.1089 |   0.4960±0.1071 |
+| openbookqa        | acc      |    0.2740±0.0200 |    0.2920±0.0204 |   0.3000±0.0205 |
+|                   | acc_norm |    0.3620±0.0215 |    0.3840±0.0218 |   0.3940±0.0219 |
+| piqa              | acc      |    0.7606±0.0100 |    0.7573±0.0100 |   0.7579±0.0100 |
+|                   | acc_norm |    0.7557±0.0100 |    0.7655±0.0099 |   0.7579±0.0100 |
+| pubmedqa          | acc      |    0.6960±0.0206 |    0.6680±0.0211 |   0.6840±0.0208 |
+| race              | acc      |    0.4077±0.0152 |    0.4622±0.0154 |   0.4632±0.0154 |
+| winogrande        | acc      |    0.6717±0.0132 |    0.6859±0.0130 |   0.6590±0.0133 |
 ## Conclusion
 X-EcoMLA demonstrates an efficient technique to upcycle pre-trained Transformers into MLA modules to compress KV cache. This work highlights the viability of post-training hybridization as a cost-effective and environmentally sustainable alternative to full retraining, paving the way for the deployment of powerful LLMs in resource-constrained environments.