Edit model card

CausalLM 14B-EXL2


4bpw h6
3.5bpw h6

Experimental exl2 quantization for CausalLM-14B for Exllamav2.
I had some issues during quantization process, so I suspect it might have quality issues.
3.5bpw version barely fits my 12GB VRAM but has unusually high perplexity for wikitext dataset.
I couldn't measure perplexity for 4bpw version and to compare it with TheBloke's GPTQ, so I have no idea if my quantization has issues or it supposed to be like this.

You could try this exl2 version but I'd recommend to use TheBloke's GPTQ version instead.

How to run

This quantization method uses GPU and requires Exllamav2 loader which can be found in following applications:

Text Generation Webui



Original model card:


Image drawn by GPT-4 DALL·E 3 TL;DR: Perhaps better than all existing models < 70B, in most quantitative evaluations...

CausalLM 14B - Fully Compatible with Meta LLaMA 2

Use the transformers library that does not require remote/external code to load the model, AutoModelForCausalLM and AutoTokenizer (or manually specify LlamaForCausalLM to load LM, GPT2Tokenizer to load Tokenizer), and model quantization is fully compatible with GGUF (llama.cpp), GPTQ, and AWQ.

News: DPO ver. Rank #1 ~13B - SOTA model of its size on 🤗 Open LLM Leaderboard

Recent Updates: DPO-α Version outperforms Zephyr-β in MT-Bench

Friendly reminder: If your VRAM is insufficient, you should use the 7B model instead of the quantized version.

Compared to the quantized versions, the 7B version and the 14B version demonstrate a high level of consistency.

llama.cpp GGUF models GPT2Tokenizer fixed by Kerfuffle on https://github.com/ggerganov/llama.cpp/pull/3743, new models are now reuploaded.

Thanks TheBloke for GGUF quants: https://huggingface.co/TheBloke/CausalLM-14B-GGUF

Caution: Unofficial GPTQ and AWQ models may have issues as they use Wikitext for calibration, while this model has undergone considerable training on a synthesized Wikipedia conversation dataset.

It is not recommended to use any form of quantization, but rather to use smaller-sized models, as the 7B and 14B versions have high consistency. However, if you do use model quantization, please use GGUF.

Read Me:

Also see 7B Version

This model was trained based on the model weights of Qwen (and LLaMA2 was used, yes, for calculating some initial weights), you may also need to comply with the commercial use restrictions of these two models depending on the situation. The training process utilized a model architecture that was identical to LLaMA2, using the same attention calculation method as the original MHA LLaMA2 models, and no additional scaling applied to the Rotary Positional Encoding (RoPE).

We manually curated a SFT dataset of 1.3B tokens for training, utilizing open source datasets from Hugging Face. For most of these sentences, we performed manual or synthetic rewrites and generated alternate language versions using larger language models. Additionally, we conducted augmented text training using carefully selected entries from Wikipedia, as well as featured entries from Fandom and filtered entries from Moegirlpedia. In order to strike a balance between efficiency and quality, 100% of the data used for training was synthetic data, no direct use of text from the internet or original texts from publicly available datasets was employed for fine-tuning.

The 7B version of the model is a distilled version of the 14B model, specifically designed for speculative sampling. Therefore, it is important to exercise caution when directly using the model, as it may produce hallucinations or unreliable outputs.

Please note that the model was trained on unfiltered internet data. Since we do not have the capacity to vet all of it, there may be a substantial amount of objectionable content, pornography, violence, and offensive language present that we are unable to remove. Therefore, you will still need to complete your own checks on the model's safety and filter keywords in the output. Due to computational resource constraints, we are presently unable to implement RLHF for the model's ethics and safety, nor training on SFT samples that refuse to answer certain questions for restrictive fine-tuning.

Bonus: The model underwent some fine-tuning on the prompt format introduced in LLaVA1.5 that is unrelated to image attention calculation. Therefore, aligning the ViT Projection module with frozen LM under visual instructions would enable rapid implementation of effective multimodal capabilities.



System Prompt must not be empty!


stem ACC: 64.19

Humanities ACC: 61.40

other ACC: 71.64

social ACC: 75.37

AVERAGE ACC:67.36 (Outperforms ALL models under 70B, very close to those best 70B fine-tunes)

CEval (Val):

STEM ACC: 66.71

Social Science ACC: 85.10

Humanities ACC: 76.68

Other ACC: 70.23

Hard ACC:54.71

AVERAGE ACC:73.10 (Outperforms Qwen-14B, and GPT-4)


Zero-shot ACC 0.7012888551933283 (Outperforms MetaMath-13B, Qwen-14B)

AlpacaEval Leaderboard

win_rate standard_error n_wins n_wins_base n_draws n_total mode avg_length
causallm-14b 88.26087 1.116333 705 89 11 805 community 1391

Win rate 88.26% on AlpacaEval Leaderboard view raw

MT-Behch on DPO Version

Model MT-Bench
GPT-4 8.99
GPT-3.5-Turbo 7.94
Zephyr-7b-β (Overfitting) 7.34
Zephyr-7b-α 6.88
CausalLM/14B-DPO-α 7.618868
CausalLM/7B-DPO-α 7.038125

Other languages

We are currently unable to produce accurate benchmark templates for non-QA tasks (languages other than English and Chinese). However, we will be working on other language versions of the QA-Task challenge in the near future.

Japanese Benchmark

Task Version Metric Value Stderr
jcommonsenseqa-1.1-0.6 1.1 acc 0.8213 ± 0.0115

JCommonsenseQA benchmark result is very, very close to Japanese Stable LM Gamma 7B (83.47), current SOTA Japanese LM. However, our model was not trained on a particularly large amount of text in Japanese. This seems to reflect the cross-language transferability of metalinguistics.

🤗 Open LLM Leaderboard

SOTA chat model of its size on 🤗 Open LLM Leaderboard.

Dec 3, 2023 DPO Version Rank #1 non-base model, of its size on 🤗 Open LLM Leaderboard, outperforms ALL ~13B chat models.


因果语言模型 14B - 与 Meta LLaMA 2 完全兼容

使用无需远程/外部代码的transformers库加载模型,AutoModelForCausalLM和AutoTokenizer(或者手动指定LlamaForCausalLM加载LM, GPT2Tokenizer加载Tokenizer),并且模型量化与GGUF(llama.cpp)、GPTQ、AWQ完全兼容。

新消息:DPO 版本在~13B排名第1 🤗 Open LLM 排行榜上同尺寸的所有模型中评分最高

最近更新: DPO-α Version 在 MT-Bench 超过 Zephyr-β


与量化版本相比,7B 版本和 14B 版本具有高度的一致性。

llama.cpp GGUF models GPT2Tokenizer 支持由 Kerfuffle 修复于 https://github.com/ggerganov/llama.cpp/pull/3743,新模型稍后上传。

感谢 TheBloke 制作 GGUF 版本量化模型: https://huggingface.co/TheBloke/CausalLM-14B-GGUF

注意: 非官方 GPTQ 和 AWQ 模型可能存在问题,因为它们使用 Wikitext 进行校准,而该模型已经在合成的 Wikipedia 对话数据集上经过了大量的训练。

不建议使用任何形式的量化,而是使用较小尺寸的模型,因为7B和14B版本具有较高的一致性。 但是,如果您确实使用模型量化,请使用 GGUF。



该模型是基于Qwen的权重(并使用了LLaMA2权重,是的,用于计算一些权重初始化),您根据情况可能还需要遵守这两个模型的商业使用限制。训练过程中使用了与LLaMA2相同的模型结构,使用原始MHA LLaMA2模型的相同注意力计算方法,对旋转位置编码(RoPE)没有进行额外的缩放。

我们手动筛选了一个包含13亿个标记的SFT数据集进行训练,利用了Hugging Face的开源数据集。对于大多数句子,我们进行了手动或合成改写,并使用更大的语言模型生成了其他语言版本。此外,我们还使用了精心挑选的来自维基百科的条目、来自Fandom的精选条目以及来自萌娘百科的过滤条目进行增强文本训练。为了在效率和质量之间取得平衡,训练所使用的100%数据都是合成数据,没有直接使用来自互联网或公开可用数据集的原始文本进行微调。






















AlpacaEval Leaderboard

win_rate standard_error n_wins n_wins_base n_draws n_total mode avg_length
causallm-14b 88.26087 1.116333 705 89 11 805 community 1391

AlpacaEval Leaderboard 胜率 88.26% view raw

DPO 版本的 MT-Behch

Model MT-Bench
GPT-4 8.99
GPT-3.5-Turbo 7.94
Zephyr-7b-β (Overfitting) 7.34
Zephyr-7b-α 6.88
CausalLM/14B-DPO-α 7.618868
CausalLM/7B-DPO-α 7.038125


我们目前无法为非 QA 任务(英语和中文以外的语言)生成准确的基准模板。 不过,我们将在不久的将来开发其他语言版本的 QA-Task 挑战。


Task Version Metric Value Stderr
jcommonsenseqa-1.1-0.6 1.1 acc 0.8213 ± 0.0115

JCommonsenseQA 基准测试结果非常非常接近 Japanese Stable LM Gamma 7B (83.47),当前 SOTA 日文 LM 。然而,我们的模型并未在日文上进行特别的大量文本训练。这似乎能体现元语言的跨语言迁移能力。

🤗 Open LLM 排行榜

Dec 3, 2023 DPO版本在🤗 Open LLM 排行榜上~13B的所有聊天模型中排名第1 image/png

Downloads last month
Inference Examples
Inference API has been turned off for this model.

Finetuned from

Datasets used to train cgus/CausalLM-14B-exl2