license: wtfpl
datasets:
- JosephusCheung/GuanacoDataset
- Open-Orca/OpenOrca
- stingning/ultrachat
- meta-math/MetaMathQA
- liuhaotian/LLaVA-Instruct-150K
- jondurbin/airoboros-3.1
- WizardLM/WizardLM_evol_instruct_V2_196k
- RyokoAI/ShareGPT52K
- RyokoAI/Fandom23K
- milashkaarshif/MoeGirlPedia_wikitext_raw_archive
- wikipedia
- wiki_lingua
- fnlp/moss-003-sft-data
- garage-bAInd/Open-Platypus
- LDJnr/Puffin
- openbmb/llava_zh
- BAAI/COIG
- TigerResearch/tigerbot-zhihu-zh-10k
- liwu/MNBVC
- teknium/openhermes
language:
- en
- zh
pipeline_tag: text-generation
tags:
- llama
- llama2
- qwen
- causallm
Original model https://huggingface.co/CausalLM/7B
Image drawn by GPT-4 DALL·E 3 TL;DR: Perhaps this 7B model, better than all existing models <= 33B, in most quantitative evaluations...
CausalLM 7B - Fully Compatible with Meta LLaMA 2
Use the transformers library that does not require remote/external code to load the model, AutoModelForCausalLM and AutoTokenizer (or manually specify LlamaForCausalLM to load LM, GPT2Tokenizer to load Tokenizer), and model quantization is fully compatible with GGUF (llama.cpp), GPTQ, and AWQ.
Recent Updates: DPO-α Version outperforms Zephyr-β in MT-Bench
llama.cpp GGUF models GPT2Tokenizer fixed by Kerfuffle on https://github.com/ggerganov/llama.cpp/pull/3743, new models are reuploaded.
Thanks TheBloke for GGUF quants: https://huggingface.co/TheBloke/CausalLM-7B-GGUF
Caution: Unofficial GPTQ and AWQ models may have issues as they use Wikitext for calibration, while this model has undergone considerable training on a synthesized Wikipedia conversation dataset.
It is not recommended to use any form of quantization, but rather to use smaller-sized models, as the 7B and 14B versions have high consistency. However, if you do use model quantization, please use GGUF.
Read Me:
Also see 14B Version
This model was trained based on the model weights of Qwen (and LLaMA2 was used, yes, for calculating some initial weights), you may also need to comply with the commercial use restrictions of these two models depending on the situation. The training process utilized a model architecture that was identical to LLaMA2, using the same attention calculation method as the original MHA LLaMA2 models, and no additional scaling applied to the Rotary Positional Encoding (RoPE).
We manually curated a SFT dataset of 1.3B tokens for training, utilizing open source datasets from Hugging Face. For most of these sentences, we performed manual or synthetic rewrites and generated alternate language versions using larger language models. Additionally, we conducted augmented text training using carefully selected entries from Wikipedia, as well as featured entries from Fandom and filtered entries from Moegirlpedia. In order to strike a balance between efficiency and quality, 100% of the data used for training was synthetic data, no direct use of text from the internet or original texts from publicly available datasets was employed for fine-tuning.
The 7B version of the model is a distilled version of the 14B model, specifically designed for speculative sampling. Therefore, it is important to exercise caution when directly using the model, as it may produce hallucinations or unreliable outputs.
Please note that the model was trained on unfiltered internet data. Since we do not have the capacity to vet all of it, there may be a substantial amount of objectionable content, pornography, violence, and offensive language present that we are unable to remove. Therefore, you will still need to complete your own checks on the model's safety and filter keywords in the output. Due to computational resource constraints, we are presently unable to implement RLHF for the model's ethics and safety, nor training on SFT samples that refuse to answer certain questions for restrictive fine-tuning.
Bonus: The model underwent some fine-tuning on the prompt format introduced in LLaVA1.5 that is unrelated to image attention calculation. Therefore, aligning the ViT Projection module with frozen LM under visual instructions would enable rapid implementation of effective multimodal capabilities.
PROMPT FORMAT:
System Prompt must not be empty!
MMLU:
stem ACC: 56.83
Humanities ACC: 58.79
other ACC: 70.04
social ACC: 72.41
AVERAGE ACC:63.82 (Outperforms / Equal to the best Mistral-7B Chat-style fine-tunes, ChatGLM3-6B and ALL other models under 33B.)
CEval (Val):
STEM acc: 61.67
Social Science acc: 81.94
Humanities acc: 77.19
Other acc: 68.35
Hard acc:48.03
AVERAGE acc:70.27 (Outperforms ALL 7B models currently, including ChatGLM3-6B.)
GSM8K
Zero-shot ACC 0.5921152388172858 (Outperforms WizardMath-7B and Qwen-7B)
MT-Behch on DPO Version
| Model | MT-Bench |
|---|---|
| GPT-4 | 8.99 |
| GPT-3.5-Turbo | 7.94 |
| Zephyr-7b-β (Overfitting) | 7.34 |
| Zephyr-7b-α | 6.88 |
| CausalLM/14B-DPO-α | 7.618868 |
| CausalLM/7B-DPO-α | 7.038125 |