RichardErkhov/survivi_-_Llama-3-SynE-gguf

Quantization made by Richard Erkhov.

Github

Discord

Request more models

Llama-3-SynE - GGUF

• Model creator: https://huggingface.co/survivi/
• Original model: https://huggingface.co/survivi/Llama-3-SynE/

Name	Quant method	Size
Llama-3-SynE.Q2_K.gguf	Q2_K	2.96GB
Llama-3-SynE.IQ3_XS.gguf	IQ3_XS	3.28GB
Llama-3-SynE.IQ3_S.gguf	IQ3_S	3.43GB
Llama-3-SynE.Q3_K_S.gguf	Q3_K_S	3.41GB
Llama-3-SynE.IQ3_M.gguf	IQ3_M	3.52GB
Llama-3-SynE.Q3_K.gguf	Q3_K	3.74GB
Llama-3-SynE.Q3_K_M.gguf	Q3_K_M	3.74GB
Llama-3-SynE.Q3_K_L.gguf	Q3_K_L	4.03GB
Llama-3-SynE.IQ4_XS.gguf	IQ4_XS	4.18GB
Llama-3-SynE.Q4_0.gguf	Q4_0	4.34GB
Llama-3-SynE.IQ4_NL.gguf	IQ4_NL	4.38GB
Llama-3-SynE.Q4_K_S.gguf	Q4_K_S	4.37GB
Llama-3-SynE.Q4_K.gguf	Q4_K	4.58GB
Llama-3-SynE.Q4_K_M.gguf	Q4_K_M	4.58GB
Llama-3-SynE.Q4_1.gguf	Q4_1	4.78GB
Llama-3-SynE.Q5_0.gguf	Q5_0	5.21GB
Llama-3-SynE.Q5_K_S.gguf	Q5_K_S	5.21GB
Llama-3-SynE.Q5_K.gguf	Q5_K	5.34GB
Llama-3-SynE.Q5_K_M.gguf	Q5_K_M	5.34GB
Llama-3-SynE.Q5_1.gguf	Q5_1	5.65GB
Llama-3-SynE.Q6_K.gguf	Q6_K	6.14GB
Llama-3-SynE.Q8_0.gguf	Q8_0	7.95GB

Original model description:

language: - en - zh datasets: - survivi/Llama-3-SynE-Dataset library_name: transformers pipeline_tag: text-generation

📄 Report   |   💻 GitHub Repo

🔍 English  |  简体中文

Here is the Llama-3-SynE model. The continual pre-training dataset is also available here.

---

News

• 🌟🌟 2024/12/17: We released the code used for continual pre-training and data preparation. The code contains detailed documentation comments.
• ✨✨ 2024/08/12: We released the continual pre-training dataset.
• ✨✨ 2024/08/10: We released the Llama-3-SynE model.
• ✨ 2024/07/26: We released the technical report, welcome to check it out!

Model Introduction

Llama-3-SynE (Synthetic data Enhanced Llama-3) is a significantly enhanced version of Llama-3 (8B), achieved through continual pre-training (CPT) to improve its Chinese language ability and scientific reasoning capability. By employing a meticulously designed data mixture and curriculum strategy, Llama-3-SynE successfully enhances new abilities while maintaining the original model’s performance. This enhancement process involves utilizing existing datasets and synthesizing high-quality datasets specifically designed for targeted tasks.

Key features of Llama-3-SynE include:

• Enhanced Chinese Language Capabilities: Achieved through topic-based data mixture and perplexity-based data curriculum.
• Improved Scientific Reasoning: Utilizing synthetic datasets to enhance multi-disciplinary scientific knowledge.
• Efficient CPT: Only consuming around 100 billion tokens, making it a cost-effective solution.

Model List

Model	Type	Seq Length	Download
Llama-3-SynE	Base	8K	🤗 Huggingface

BenchMark

We divide all evaluation benchmarks into two groups. The first group is major benchmarks, which aim to evaluate the comprehensive capacities of LLMs. Note that we include commonly used math and code benchmarks in this group because it is standard practice to use these benchmarks for evaluating various general-purpose LLMs.

The second group is scientific benchmarks, which have a broader coverage of multidisciplinary scientific knowledge.

We report the eight-shot performance on GSM8K, ASDiv, and MAWPS, five-shot for C-Eval, CMMLU, MMLU, MATH, GaoKao, SciQ, SciEval, SAT-Math, and AQUA-RAT, three-shot for MBPP. For HumanEval and ARC, we report the zero-shot evaluation performance. The best and second best are in bold and underlined, respectively.

Major Benchmarks

Models	MMLU	C-Eval	CMMLU	MATH	GSM8K	ASDiv	MAWPS	SAT-Math	HumanEval	MBPP
Llama-3-8B	66.60	49.43	51.03	16.20	54.40	72.10	89.30	38.64	36.59	47.00
DCLM-7B	64.01	41.24	40.89	14.10	39.20	67.10	83.40	41.36	21.95	32.60
Mistral-7B-v0.3	63.54	42.74	43.72	12.30	40.50	67.50	87.50	40.45	25.61	36.00
Llama-3-Chinese-8B	64.10	50.14	51.20	3.60	0.80	1.90	0.60	36.82	9.76	14.80
MAmmoTH2-8B	64.89	46.56	45.90	34.10	61.70	82.80	91.50	41.36	17.68	38.80
Galactica-6.7B	37.13	26.72	25.53	5.30	9.60	40.90	51.70	23.18	7.31	2.00
Llama-3-SynE (ours)	65.19	58.24	57.34	28.20	60.80	81.00	94.10	43.64	42.07	45.60

On Chinese evaluation benchmarks (such as C-Eval and CMMLU), Llama-3-SynE significantly outperforms the base model Llama-3 (8B), indicating that our method is very effective in improving Chinese language capabilities.

On English evaluation benchmarks (such as MMLU, MATH, and code evaluation benchmarks), Llama-3-SynE demonstrates comparable or better performance than the base model, indicating that our method effectively addresses the issue of catastrophic forgetting during the CPT process.

Scientific Benchmarks

"PHY", "CHE", and "BIO" denote the physics, chemistry, and biology sub-tasks of the corresponding benchmarks.

Models	SciEval PHY	SciEval CHE	SciEval BIO	SciEval Avg.	SciQ	GaoKao MathQA	GaoKao CHE	GaoKao BIO	ARC Easy	ARC Challenge	ARC Avg.	AQUA-RAT
Llama-3-8B	46.95	63.45	74.53	65.47	90.90	27.92	32.85	43.81	91.37	77.73	84.51	27.95
DCLM-7B	56.71	64.39	72.03	66.25	92.50	29.06	31.40	37.14	89.52	76.37	82.94	20.08
Mistral-7B-v0.3	48.17	59.41	68.89	61.51	89.40	30.48	30.92	41.43	87.33	74.74	81.04	23.23
Llama-3-Chinese-8B	48.17	67.34	73.90	67.34	89.20	27.64	30.43	38.57	88.22	70.48	79.35	27.56
MAmmoTH2-8B	49.39	69.36	76.83	69.60	90.20	32.19	36.23	49.05	92.85	84.30	88.57	27.17
Galactica-6.7B	34.76	43.39	54.07	46.27	71.50	23.65	27.05	24.76	65.91	46.76	56.33	20.87
Llama-3-SynE (ours)	53.66	67.81	77.45	69.60	91.20	31.05	51.21	69.52	91.58	80.97	86.28	28.74

On scientific evaluation benchmarks (such as SciEval, GaoKao, and ARC), Llama-3-SynE significantly outperforms the base model, particularly showing remarkable improvement in Chinese scientific benchmarks (for example, a 25.71% improvement in the GaoKao biology subtest).

Quick Start

Use the transformers backend for inference:

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

model_path = "survivi/Llama-3-SynE"
tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_path, torch_dtype=torch.bfloat16, trust_remote_code=True
)
model.to("cuda:0")
model.eval()
prompt = "Hello world!"
inputs = tokenizer(prompt, return_tensors="pt")
inputs = inputs.to("cuda")
pred = model.generate(
    **inputs,
    max_new_tokens=2048,
    repetition_penalty=1.05,
    temperature=0.5,
    top_k=5,
    top_p=0.85,
    do_sample=True
)
pred = pred[0][len(inputs.input_ids[0]) :]
output = tokenizer.decode(pred, skip_special_tokens=True)
print(output)

Use the vLLM backend for inference:

from transformers import AutoTokenizer
from vllm import LLM, SamplingParams

model_path = "survivi/Llama-3-SynE"
tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)
sampling_params = SamplingParams(
    max_tokens=2048,
    repetition_penalty=1.05,
    temperature=0.5,
    top_k=5,
    top_p=0.85,
)
llm = LLM(
    model=model_path,
    tensor_parallel_size=1,
    trust_remote_code=True,
)
prompt = "Hello world!"
output = llm.generate(prompt, sampling_params)
output = output[0].outputs[0].text
print(output)

License

This project is built upon Meta's Llama-3 model. The use of Llama-3-SynE model weights must follow the Llama-3 license agreement. The code in this open-source repository follows the Apache 2.0 license.

Citation

If you find our work helpful, please consider citing the following paper:

@article{jie2024llama3syne,
  title={Towards Effective and Efficient Continual Pre-training of Large Language Models},
  author={Chen, Jie and Chen, Zhipeng and Wang, Jiapeng and Zhou, Kun and Zhu, Yutao and Jiang, Jinhao and Min, Yingqian and Zhao, Wayne Xin and Dou, Zhicheng and Mao, Jiaxin and others},
  journal={arXiv preprint arXiv:2407.18743},
  year={2024}
}