Hugging Face's logo

ChenMnZ
/
Llama-3-70b-instruct-EfficientQAT-w2g128

Text Generation
Transformers
Safetensors
llama
conversational
text-generation-inference
Inference Endpoints
Model card Files Community
ChenMnZ commited on
Commit
5b870a1
•
1 Parent(s): 2976027

Upload folder using huggingface_hub

Browse files
Files changed (10) hide show
  1. README.md +89 -0
  2. config.json +3 -2
  3. generation_config.json +1 -1
  4. model-00001-of-00005.safetensors +1 -1
  5. model-00002-of-00005.safetensors +1 -1
  6. model-00003-of-00005.safetensors +1 -1
  7. model-00004-of-00005.safetensors +1 -1
  8. model-00005-of-00005.safetensors +1 -1
  9. model.safetensors +3 -0
  10. tokenizer.json +0 -1
README.md ADDED
@@ -0,0 +1,89 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # EfficientQAT
2
+
3
+ [EfficientQAT](https://arxiv.org/abs/2407.11062) is a novel quantization technical, which pushes the limitation of uniform (INT) quantization in an efficient manner. Due to the leverage of standard INT quantization, the quantized model of EfficientQAT can also be transferred into other formats, such as GPTQ, BitBLAS, etc.
4
+
5
+
6
+ In this repo, we provide three type checkpoints, one is EQAT, indicats the original checkpoints of EfficientQAT. The other two are GPTQ and BitBLAS respectively.
7
+
8
+
9
+ ## Model Zoo
10
+
11
+ We provide a number of prequantized EfficientQAT models as follows:
12
+
13
+ - WikiText2 PPL is measured in 2048 context length.
14
+ - Avg. Accuracy indicate the average accuracy in 5 zero-shot reasoning tasks (WinoGrande,PIQA,HellaSwag,Arc-Easy, Arc-Challenge) with [lm-eval v0.4.2](https://github.com/EleutherAI/lm-evaluation-harness).
15
+ - 1GB = $10^9$ Bit
16
+ - Hub Link: EQAT indicates the original checkpoints. We also transfer the checkpoints into GPTQ and BitBLAS formats, which can be loaded directly through [GPTQModel](https://github.com/ModelCloud/GPTQModel). (PS: [GPTQModel](https://github.com/ModelCloud/GPTQModel) is a official bug-fixed repo of AutoGPTQ, which would be merged into [AutoGPTQ](https://github.com/AutoGPTQ/AutoGPTQ) in future.)
17
+
18
+ | Model | Quantization | WikiText2 PPL | Avg. Accuracy | Model Size (GB) | Hub link|
19
+ |-------|--------------|---------------|---------------|-----------------|----------|
20
+ Llama-2-7B|fp16|5.47|64.86|13.2|-|
21
+ Llama-2-7B|w4g128|5.53|64.27|3.7|[EQAT](https://huggingface.co/ChenMnZ/Llama-2-7b-EfficientQAT-w4g128)\|[GPTQ](https://huggingface.co/ChenMnZ/Llama-2-7b-EfficientQAT-w4g128-GPTQ)\|[BitBLAS](Llama-2-7b-EfficientQAT-w4g128-BitBLAS)|
22
+ Llama-2-7B|w3g128|5.81|64.02|3.1|[EQAT](https://huggingface.co/ChenMnZ/Llama-2-7b-EfficientQAT-w3g128)|
23
+ Llama-2-7B|w2g64|6.86|60.14|2.3|[EQAT](https://huggingface.co/ChenMnZ/Llama-2-7b-EfficientQAT-w2g64)\|[GPTQ](https://huggingface.co/ChenMnZ/Llama-2-7b-EfficientQAT-w2g64-GPTQ)\|[BitBLAS](Llama-2-7b-EfficientQAT-w2g64-BitBLAS)|
24
+ Llama-2-7B|w2g128|7.17|59.50|2.2|[EQAT](https://huggingface.co/ChenMnZ/Llama-2-7b-EfficientQAT-w2g128)\|[GPTQ](https://huggingface.co/ChenMnZ/Llama-2-7b-EfficientQAT-w2g128-GPTQ)\|[BitBLAS](Llama-2-7b-EfficientQAT-w2g128-BitBLAS)|
25
+ Llama-2-13B|fp16|4.88|67.81|25.4|-|
26
+ Llama-2-13B|w4g128|4.93|67.52|6.8|[EQAT](https://huggingface.co/ChenMnZ/Llama-2-13b-EfficientQAT-w4g128)\|[GPTQ](https://huggingface.co/ChenMnZ/Llama-2-7b-EfficientQAT-w4g128-GPTQ)\|[BitBLAS](Llama-2-7b-EfficientQAT-w4g128-BitBLAS)|
27
+ Llama-2-13B|w3g128|5.12|67.28|5.6|[EQAT](https://huggingface.co/ChenMnZ/Llama-2-13b-EfficientQAT-w3g128)|
28
+ Llama-2-13B|w2g64|5.96|64.88|4.0|[EQAT](https://huggingface.co/ChenMnZ/Llama-2-13b-EfficientQAT-w2g64)\|[GPTQ](https://huggingface.co/ChenMnZ/Llama-2-13b-EfficientQAT-w2g64-GPTQ)\|[BitBLAS](Llama-2-13b-EfficientQAT-w2g64-BitBLAS)|
29
+ Llama-2-13B|w2g128|6.08|63.88|3.8|[EQAT](https://huggingface.co/ChenMnZ/Llama-2-13b-EfficientQAT-w2g128)\|[GPTQ](https://huggingface.co/ChenMnZ/Llama-2-13b-EfficientQAT-w2g128-GPTQ)\|[BitBLAS](Llama-2-13b-EfficientQAT-w2g128-BitBLAS)|
30
+ Llama-2-70B|fp16|3.32|72.41|131.6|-|
31
+ Llama-2-70B|w4g128|3.39|72.62|35.8|[EQAT](https://huggingface.co/ChenMnZ/Llama-2-70b-EfficientQAT-w4g128)\|[GPTQ](https://huggingface.co/ChenMnZ/Llama-2-70b-EfficientQAT-w4g128-GPTQ)\|[BitBLAS](Llama-2-70b-EfficientQAT-w4g128-BitBLAS)|
32
+ Llama-2-70B|w3g128|3.61|71.76|29.1|[EQAT](https://huggingface.co/ChenMnZ/Llama-2-70b-EfficientQAT-w3g128)|
33
+ Llama-2-70B|w2g64|4.52|69.48|20.1|[EQAT](https://huggingface.co/ChenMnZ/Llama-2-70b-EfficientQAT-w2g64)\|[GPTQ](https://huggingface.co/ChenMnZ/Llama-2-70b-EfficientQAT-w2g64-GPTQ)\|[BitBLAS](Llama-2-70b-EfficientQAT-w2g64-BitBLAS)|
34
+ Llama-2-70B|w2g128|4.61|68.93|18.9|[EQAT](https://huggingface.co/ChenMnZ/Llama-2-70b-EfficientQAT-w2g128)\|[GPTQ](https://huggingface.co/ChenMnZ/Llama-2-70b-EfficientQAT-w2g128-GPTQ)\|[BitBLAS](Llama-2-70b-EfficientQAT-w2g128-BitBLAS)|
35
+ Llama-3-8B|fp16|6.14|68.58|13.0|-|
36
+ Llama-3-8B|w4g128|6.47|68.43|5.4|[EQAT](https://huggingface.co/ChenMnZ/Llama-3-8b-EfficientQAT-w4g128)\|[GPTQ](https://huggingface.co/ChenMnZ/Llama-3-8b-EfficientQAT-w4g128-GPTQ)\|[BitBLAS](Llama-3-8b-EfficientQAT-w4g128-BitBLAS)|
37
+ Llama-3-8B|w3g128|7.09|67.35|4.7|[EQAT](https://huggingface.co/ChenMnZ/Llama-3-8b-EfficientQAT-w3g128)|
38
+ Llama-3-8B|w2g64|9.41|60.76|3.9|[EQAT](https://huggingface.co/ChenMnZ/Llama-3-8b-EfficientQAT-w2g64)\|[GPTQ](https://huggingface.co/ChenMnZ/Llama-3-8b-EfficientQAT-w4g128-GPTQ)\|[BitBLAS](Llama-3-8b-EfficientQAT-w2g64-BitBLAS)|
39
+ Llama-3-8B|w2g128|9.80|59.36|3.8|[EQAT](https://huggingface.co/ChenMnZ/Llama-3-8b-EfficientQAT-w2g128)\|[GPTQ](https://huggingface.co/ChenMnZ/Llama-3-8b-EfficientQAT-w2g128-GPTQ)\|[BitBLAS](Llama-3-8b-EfficientQAT-w2g128-BitBLAS)|
40
+ Llama-3-70B|fp16|2.85|75.33|137.8|-|
41
+ Llama-3-70B|w4g128|3.17|74.57|38.9|[EQAT](https://huggingface.co/ChenMnZ/Llama-3-70b-EfficientQAT-w4g128)\|[GPTQ](https://huggingface.co/ChenMnZ/Llama-3-70b-EfficientQAT-w4g128-GPTQ)\|[BitBLAS](Llama-3-70b-EfficientQAT-w4g128-BitBLAS)|
42
+ Llama-3-70B|w3g128|4.19|72.42|32.2|[EQAT](https://huggingface.co/ChenMnZ/Llama-3-70b-EfficientQAT-w3g128)|
43
+ Llama-3-70B|w2g64|6.08|67.89|23.2|[EQAT](https://huggingface.co/ChenMnZ/Llama-3-70b-EfficientQAT-w2g64)\|[GPTQ](https://huggingface.co/ChenMnZ/Llama-3-70b-EfficientQAT-w2g64-GPTQ)|
44
+ Llama-3-70B|w2g128|6.38|67.57|22.0|[EQAT](https://huggingface.co/ChenMnZ/Llama-3-70b-EfficientQAT-w2g128)\|[GPTQ](https://huggingface.co/ChenMnZ/Llama-3-70b-EfficientQAT-w2g128-GPTQ)\|[BitBLAS](Llama-3-70b-EfficientQAT-w2g128-BitBLAS)|
45
+ Llama-3-8B-Instruct|fp16|8.29|68.43|13.0|-|
46
+ Llama-3-8B-Instruct|w4g128|7.93|68.39|5.4|[EQAT](https://huggingface.co/ChenMnZ/Llama-3-8b-instruct-EfficientQAT-w4g128)\|[GPTQ](https://huggingface.co/ChenMnZ/Llama-3-8b-instruct-EfficientQAT-w4g128-GPTQ)\|[BitBLAS](Llama-3-8b-instruct-EfficientQAT-w4g128-BitBLAS)|
47
+ Llama-3-8B-Instruct|w3g128|8.55|67.24|4.7|[EQAT](https://huggingface.co/ChenMnZ/Llama-3-8b-instruct-EfficientQAT-w3g128)|
48
+ Llama-3-8B-Instruct|w2g64|11.19|60.66|3.9|[EQAT](https://huggingface.co/ChenMnZ/Llama-3-8b-instruct-EfficientQAT-w2g64)\|[GPTQ](https://huggingface.co/ChenMnZ/Llama-3-8b-instruct-EfficientQAT-w2g64-GPTQ)\|[BitBLAS](Llama-3-8b-instruct-EfficientQAT-w2g64-BitBLAS)|
49
+ Llama-3-8B-Instruct|w2g128|11.73|60.16|3.8|[EQAT](https://huggingface.co/ChenMnZ/Llama-3-8b-instruct-EfficientQAT-w2g128)\|[GPTQ](https://huggingface.co/ChenMnZ/Llama-3-8b-instruct-EfficientQAT-w2g128-GPTQ)\|[BitBLAS](Llama-3-8b-instruct-EfficientQAT-w2g128-BitBLAS)|
50
+ Llama-3-70B-Instruct|fp16|5.33|73.78|137.8|-|
51
+ Llama-3-70B-Instruct|w4g128|5.35|73.47|38.9|[EQAT](https://huggingface.co/ChenMnZ/Llama-3-70b-instruct-EfficientQAT-w4g128)\|[GPTQ](https://huggingface.co/ChenMnZ/Llama-3-70b-instruct-EfficientQAT-w4g128-GPTQ)\|[BitBLAS](Llama-3-70b-instruct-EfficientQAT-w4g128-BitBLAS)|
52
+ Llama-3-70B-Instruct|w3g128|5.65|72.87|32.2|[EQAT](https://huggingface.co/ChenMnZ/Llama-3-70b-instruct-EfficientQAT-w3g128)|
53
+ Llama-3-70B-Instruct|w2g64|7.86|67.64|23.2|[EQAT](https://huggingface.co/ChenMnZ/Llama-3-70b-instruct-EfficientQAT-w2g64)\|[GPTQ](https://huggingface.co/ChenMnZ/Llama-3-70b-instruct-EfficientQAT-w2g64-GPTQ)\|[BitBLAS](Llama-3-70b-instruct-EfficientQAT-w2g64-BitBLAS)|
54
+ Llama-3-70B-Instruct|w2g128|8.14|67.54|22.0|[EQAT](https://huggingface.co/ChenMnZ/Llama-3-70b-instruct-EfficientQAT-w2g128)\|[GPTQ](https://huggingface.co/ChenMnZ/Llama-3-70b-instruct-EfficientQAT-w2g128-GPTQ)\|[BitBLAS](Llama-3-70b-instruct-EfficientQAT-w2g128-BitBLAS)|
55
+
56
+ ## Usage of EQAT models
57
+ Please refer [https://github.com/OpenGVLab/EfficientQAT](https://github.com/OpenGVLab/EfficientQAT?tab=readme-ov-file#inference) for details.
58
+
59
+ ## Usage of GPTQ and BitBLAS models
60
+ Below is an example to inference with GPTQ or BitBLAS quantized formats.
61
+ ```Python
62
+ from transformers import AutoTokenizer
63
+ from gptqmodel import GPTQModel
64
+
65
+ quant_dir = "ChenMnZ/Llama-2-7b-EfficientQAT-w2g128-GPTQ"
66
+ # quant_dir = "ChenMnZ/Llama-2-7b-EfficientQAT-w2g128-BitBLAS"
67
+ # or local path
68
+
69
+ tokenizer = AutoTokenizer.from_pretrained(quant_dir, use_fast=True)
70
+
71
+
72
+ # load quantized model to the first GPU
73
+ model = GPTQModel.from_quantized(quant_dir)
74
+
75
+ # inference with model.generate
76
+ print(tokenizer.decode(model.generate(**tokenizer("Model quantization is", return_tensors="pt").to(model.device))[0]))
77
+ ```
78
+
79
+
80
+ ## Citation
81
+ If you found this work useful, please consider citing:
82
+ ```
83
+ @article{efficientqat,
84
+ title={EfficientQAT: Efficient Quantization-Aware Training for Large Language Models},
85
+ author={Chen, Mengzhao and Shao, Wenqi and Xu, Peng and Wang, Jiahao and Gao, Peng and Zhang, Kaipeng and Qiao, Yu and Luo, Ping},
86
+ journal={arXiv preprint arXiv:2407.11062},
87
+ year={2024}
88
+ }
89
+ ```
config.json CHANGED
@@ -1,5 +1,5 @@
1
  {
2
- "_name_or_path": "/cpfs01/user/chenmengzhao/efficientqat_repo/e2e-qp-output/llama3-70b-instruct-w2g128-mix-8192-lr2e-5/checkpoint-256",
3
  "architectures": [
4
  "LlamaForCausalLM"
5
  ],
@@ -12,6 +12,7 @@
12
  "initializer_range": 0.02,
13
  "intermediate_size": 28672,
14
  "max_position_embeddings": 8192,
 
15
  "model_type": "llama",
16
  "num_attention_heads": 64,
17
  "num_hidden_layers": 80,
@@ -22,7 +23,7 @@
22
  "rope_theta": 500000.0,
23
  "tie_word_embeddings": false,
24
  "torch_dtype": "float16",
25
- "transformers_version": "4.40.1",
26
  "use_cache": false,
27
  "vocab_size": 128257
28
  }
 
1
  {
2
+ "_name_or_path": "/cpfs01/user/chenmengzhao/efficientqat_repo/EfficientQAT/output/e2e-qp-output/Llama-3-70b-instruct-w2g128-deita-8192/checkpoint-306",
3
  "architectures": [
4
  "LlamaForCausalLM"
5
  ],
 
12
  "initializer_range": 0.02,
13
  "intermediate_size": 28672,
14
  "max_position_embeddings": 8192,
15
+ "mlp_bias": false,
16
  "model_type": "llama",
17
  "num_attention_heads": 64,
18
  "num_hidden_layers": 80,
 
23
  "rope_theta": 500000.0,
24
  "tie_word_embeddings": false,
25
  "torch_dtype": "float16",
26
+ "transformers_version": "4.42.4",
27
  "use_cache": false,
28
  "vocab_size": 128257
29
  }
generation_config.json CHANGED
@@ -2,6 +2,6 @@
2
  "_from_model_config": true,
3
  "bos_token_id": 128000,
4
  "eos_token_id": 128001,
5
- "transformers_version": "4.40.1",
6
  "use_cache": false
7
  }
 
2
  "_from_model_config": true,
3
  "bos_token_id": 128000,
4
  "eos_token_id": 128001,
5
+ "transformers_version": "4.42.4",
6
  "use_cache": false
7
  }
model-00001-of-00005.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:79eb7c27eb40f09c7a23c0ab2949314e1aa3e02ff3e32b0746fc8e19d3de73d5
3
  size 4960020216
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:f8d6498ee588649242aabd76f558d73376f887945d70470a811ad594df81a510
3
  size 4960020216
model-00002-of-00005.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:135377a5c6c22ce04f01494bc1d89fdf115ddb8e534c48f72cd3450b0e490cf9
3
  size 4981662032
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:15e2d6b6d2741c49d35b14758405e97cc4c9d0ad0adcc0c9640666eaef1db1cb
3
  size 4981662032
model-00003-of-00005.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:6ebc3bc9177ac20bbd06aad2a8fa82ec720b12f396fd8b6615437808a872d88a
3
  size 4999731664
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:46cd117c00f0c03146810dbb516602bd79e6adb24ec6271ea9c91a6767f02f09
3
  size 4999731664
model-00004-of-00005.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:7d3bdf11ceb6027bfb27485b08714561da74cd0306393ff749134eb3a08a42c0
3
  size 4985823880
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:7a34e01c112048f65ceef610bc39f6e02f1ad266d336f52b16e5549c3092e71f
3
  size 4985823880
model-00005-of-00005.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:d026fbcb2c5ce6dd46e0165e6b0f62cf62b051337462490b5b49a04cd4e59667
3
  size 2619301168
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:e3e4a4a5e65114c81513a4fd4950c48215b33f9b9766fa416ffba7734be6d6f6
3
  size 2619301168
model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:95da378ae5d6ca532ab024de02dd956ed716c531da030a21543b377eda8e0e04
3
+ size 22546542376
tokenizer.json CHANGED
@@ -2357,7 +2357,6 @@
2357
  "end_of_word_suffix": null,
2358
  "fuse_unk": false,
2359
  "byte_fallback": false,
2360
- "ignore_merges": false,
2361
  "vocab": {
2362
  "!": 0,
2363
  "\"": 1,
 
2357
  "end_of_word_suffix": null,
2358
  "fuse_unk": false,
2359
  "byte_fallback": false,
 
2360
  "vocab": {
2361
  "!": 0,
2362
  "\"": 1,