test

MODEL_LICENSE

@@ -9,17 +9,14 @@ The ChatGLM3-6B License
 2. 许可授予
 
 根据本许可的条款和条件，许可方特此授予您非排他性、全球性、不可转让、不可再许可、可撤销、免版税的版权许可。
-本许可允许您免费使用本仓库中的所有开源模型进行学术研究，对于希望将模型用于商业目的的用户，需在[这里](https://open.bigmodel.cn/mla/form)完成登记。
-经过登记的用户可以免费使用本模型进行商业活动，但必须遵守本许可的所有条款和条件。
+
 上述版权声明和本许可声明应包含在本软件的所有副本或重要部分中。
 
 3.限制
 
 您不得出于任何军事或非法目的使用、复制、修改、合并、发布、分发、复制或创建本软件的全部或部分衍生作品。
 
-您不得利用本软件从事任何危害国家安全和国家统一，危害社会公共利益及公序良俗，侵犯他人商业秘密、知识产权、名誉权、肖像权、财产权等权益的行为。
-
-您在使用中应遵循使用地所适用的法律法规政策、道德规范等要求。
+您不得利用本软件从事任何危害国家安全和国家统一、危害社会公共利益、侵犯人身权益的行为。
 
 4.免责声明
 
@@ -43,15 +40,15 @@ The ChatGLM3-6B License
 
 2. License Grant
 
-Under the terms and conditions of this license, the Licensor hereby grants you a non-exclusive, worldwide, non-transferable, non-sublicensable, revocable, royalty-free copyright license.
-This license permits you to use all open-source models in this repository for academic research free. Users who wish to use the models for commercial purposes must register [here](https://open.bigmodel.cn/mla/form).
-Registered users may use the models for commercial activities free of charge, but must comply with all terms and conditions of this license.
-The license notice shall be included in all copies or substantial portions of the Software.
+Subject to the terms and conditions of this License, the Licensor hereby grants to you a non-exclusive, worldwide, non-transferable, non-sublicensable, revocable, royalty-free copyright license to use the Software.
+
+The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
+
+3. Restriction
+
+You will not use, copy, modify, merge, publish, distribute, reproduce, or create derivative works of the Software, in whole or in part, for any military, or illegal purposes.
 
-3. Restrictions
-You are not allowed to use, copy, modify, merge, publish, distribute, copy or create all or part of the derivative works of this software for any military or illegal purposes.
-You are not allowed to use this software to engage in any behavior that endangers national security and unity, endangers social public interests and public order, infringes on the rights and interests of others such as trade secrets, intellectual property rights, reputation rights, portrait rights, and property rights.
-You should comply with the applicable laws, regulations, policies, ethical standards, and other requirements in the place of use during use.
+You will not use the Software for any act that may undermine China's national security and national unity, harm the public interest of society, or infringe upon the rights and interests of human beings.
 
 4. Disclaimer
 

README.md

@@ -19,11 +19,6 @@ tags:
 📍Experience the larger-scale ChatGLM model at <a href="https://www.chatglm.cn">chatglm.cn</a>
 </p>
 
-## GLM-4 开源模型
-
-我们已经发布最新的 **GLM-4** 模型，该模型在多个指标上有了新的突破，您可以在以下两个渠道体验我们的最新模型。
-+ [GLM-4 开源模型](https://huggingface.co/THUDM/glm-4-9b-chat) 我们已经开源了 GLM-4-9B 系列模型，在各项指标的测试上有明显提升，欢迎尝试。
-
 ## 介绍 (Introduction)
 ChatGLM3-6B 是 ChatGLM 系列最新一代的开源模型，在保留了前两代模型对话流畅、部署门槛低等众多优秀特性的基础上，ChatGLM3-6B 引入了如下特性：
 
@@ -86,15 +81,22 @@ The code in this repository is open-sourced under the [Apache-2.0 license](LICEN
 
 如果你觉得我们的工作有帮助的话，请考虑引用下列论文。
 
-If you find our work helpful, please consider citing the following paper.
+If you find our work helpful, please consider citing the following papers.
 
 ```
-@misc{glm2024chatglm,
-      title={ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools}, 
-      author={Team GLM and Aohan Zeng and Bin Xu and Bowen Wang and Chenhui Zhang and Da Yin and Diego Rojas and Guanyu Feng and Hanlin Zhao and Hanyu Lai and Hao Yu and Hongning Wang and Jiadai Sun and Jiajie Zhang and Jiale Cheng and Jiayi Gui and Jie Tang and Jing Zhang and Juanzi Li and Lei Zhao and Lindong Wu and Lucen Zhong and Mingdao Liu and Minlie Huang and Peng Zhang and Qinkai Zheng and Rui Lu and Shuaiqi Duan and Shudan Zhang and Shulin Cao and Shuxun Yang and Weng Lam Tam and Wenyi Zhao and Xiao Liu and Xiao Xia and Xiaohan Zhang and Xiaotao Gu and Xin Lv and Xinghan Liu and Xinyi Liu and Xinyue Yang and Xixuan Song and Xunkai Zhang and Yifan An and Yifan Xu and Yilin Niu and Yuantao Yang and Yueyan Li and Yushi Bai and Yuxiao Dong and Zehan Qi and Zhaoyu Wang and Zhen Yang and Zhengxiao Du and Zhenyu Hou and Zihan Wang},
-      year={2024},
-      eprint={2406.12793},
-      archivePrefix={arXiv},
-      primaryClass={id='cs.CL' full_name='Computation and Language' is_active=True alt_name='cmp-lg' in_archive='cs' is_general=False description='Covers natural language processing. Roughly includes material in ACM Subject Class I.2.7. Note that work on artificial languages (programming languages, logics, formal systems) that does not explicitly address natural-language issues broadly construed (natural-language processing, computational linguistics, speech, text retrieval, etc.) is not appropriate for this area.'}
+@article{zeng2022glm,
+  title={Glm-130b: An open bilingual pre-trained model},
+  author={Zeng, Aohan and Liu, Xiao and Du, Zhengxiao and Wang, Zihan and Lai, Hanyu and Ding, Ming and Yang, Zhuoyi and Xu, Yifan and Zheng, Wendi and Xia, Xiao and others},
+  journal={arXiv preprint arXiv:2210.02414},
+  year={2022}
 }
 ```
+```
+@inproceedings{du2022glm,
+  title={GLM: General Language Model Pretraining with Autoregressive Blank Infilling},
+  author={Du, Zhengxiao and Qian, Yujie and Liu, Xiao and Ding, Ming and Qiu, Jiezhong and Yang, Zhilin and Tang, Jie},
+  booktitle={Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
+  pages={320--335},
+  year={2022}
+}
+```

model-00001-of-00007.safetensors

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:b1052386eac358a18add3d0f92521c85ab338979da8eeb08a6499555b857f80d
-size 1827774160

model-00002-of-00007.safetensors

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:f88d7f69a3940711b87308eb20dd0544df22aff81134fa0d5160d3c859f5d321
-size 1968291888

model-00003-of-00007.safetensors

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:447d41b7c5e7b2558905c98733469aa9e132540c91e13c4cdd7bfc58b60cc650
-size 1927406936

model-00004-of-00007.safetensors

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:386305288e2bcc4a26fcffef433c94592d155bf36e365f054f63d3e6b55a1429
-size 1815217640

model-00005-of-00007.safetensors

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:6c304e2821774d649997d1f5ae5fa12afd4d1dc5fea2bf6e8b829405b81fa7f2
-size 1968291920

model-00006-of-00007.safetensors

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:ef45bbd33d6d540d3d3523bb169b52b91b9e265176ab7efc6838233bc3b7e95d
-size 1927406960

model-00007-of-00007.safetensors

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:6eb545c6aa8a839c56a78b80331fcb4fa403341925f79fb6a761e2e847bf1e40
-size 1052805400

model.safetensors.index.json

@@ -1,207 +0,0 @@
-{
-    "metadata": {
-        "total_size": 12487168064
-    },
-    "weight_map": {
-        "transformer.embedding.word_embeddings.weight": "model-00001-of-00007.safetensors",
-        "transformer.encoder.final_layernorm.weight": "model-00007-of-00007.safetensors",
-        "transformer.encoder.layers.0.input_layernorm.weight": "model-00001-of-00007.safetensors",
-        "transformer.encoder.layers.0.mlp.dense_4h_to_h.weight": "model-00001-of-00007.safetensors",
-        "transformer.encoder.layers.0.mlp.dense_h_to_4h.weight": "model-00001-of-00007.safetensors",
-        "transformer.encoder.layers.0.post_attention_layernorm.weight": "model-00001-of-00007.safetensors",
-        "transformer.encoder.layers.0.self_attention.dense.weight": "model-00001-of-00007.safetensors",
-        "transformer.encoder.layers.0.self_attention.query_key_value.bias": "model-00001-of-00007.safetensors",
-        "transformer.encoder.layers.0.self_attention.query_key_value.weight": "model-00001-of-00007.safetensors",
-        "transformer.encoder.layers.1.input_layernorm.weight": "model-00001-of-00007.safetensors",
-        "transformer.encoder.layers.1.mlp.dense_4h_to_h.weight": "model-00001-of-00007.safetensors",
-        "transformer.encoder.layers.1.mlp.dense_h_to_4h.weight": "model-00001-of-00007.safetensors",
-        "transformer.encoder.layers.1.post_attention_layernorm.weight": "model-00001-of-00007.safetensors",
-        "transformer.encoder.layers.1.self_attention.dense.weight": "model-00001-of-00007.safetensors",
-        "transformer.encoder.layers.1.self_attention.query_key_value.bias": "model-00001-of-00007.safetensors",
-        "transformer.encoder.layers.1.self_attention.query_key_value.weight": "model-00001-of-00007.safetensors",
-        "transformer.encoder.layers.10.input_layernorm.weight": "model-00003-of-00007.safetensors",
-        "transformer.encoder.layers.10.mlp.dense_4h_to_h.weight": "model-00003-of-00007.safetensors",
-        "transformer.encoder.layers.10.mlp.dense_h_to_4h.weight": "model-00003-of-00007.safetensors",
-        "transformer.encoder.layers.10.post_attention_layernorm.weight": "model-00003-of-00007.safetensors",
-        "transformer.encoder.layers.10.self_attention.dense.weight": "model-00003-of-00007.safetensors",
-        "transformer.encoder.layers.10.self_attention.query_key_value.bias": "model-00003-of-00007.safetensors",
-        "transformer.encoder.layers.10.self_attention.query_key_value.weight": "model-00003-of-00007.safetensors",
-        "transformer.encoder.layers.11.input_layernorm.weight": "model-00003-of-00007.safetensors",
-        "transformer.encoder.layers.11.mlp.dense_4h_to_h.weight": "model-00003-of-00007.safetensors",
-        "transformer.encoder.layers.11.mlp.dense_h_to_4h.weight": "model-00003-of-00007.safetensors",
-        "transformer.encoder.layers.11.post_attention_layernorm.weight": "model-00003-of-00007.safetensors",
-        "transformer.encoder.layers.11.self_attention.dense.weight": "model-00003-of-00007.safetensors",
-        "transformer.encoder.layers.11.self_attention.query_key_value.bias": "model-00003-of-00007.safetensors",
-        "transformer.encoder.layers.11.self_attention.query_key_value.weight": "model-00003-of-00007.safetensors",
-        "transformer.encoder.layers.12.input_layernorm.weight": "model-00003-of-00007.safetensors",
-        "transformer.encoder.layers.12.mlp.dense_4h_to_h.weight": "model-00004-of-00007.safetensors",
-        "transformer.encoder.layers.12.mlp.dense_h_to_4h.weight": "model-00003-of-00007.safetensors",
-        "transformer.encoder.layers.12.post_attention_layernorm.weight": "model-00003-of-00007.safetensors",
-        "transformer.encoder.layers.12.self_attention.dense.weight": "model-00003-of-00007.safetensors",
-        "transformer.encoder.layers.12.self_attention.query_key_value.bias": "model-00003-of-00007.safetensors",
-        "transformer.encoder.layers.12.self_attention.query_key_value.weight": "model-00003-of-00007.safetensors",
-        "transformer.encoder.layers.13.input_layernorm.weight": "model-00004-of-00007.safetensors",
-        "transformer.encoder.layers.13.mlp.dense_4h_to_h.weight": "model-00004-of-00007.safetensors",
-        "transformer.encoder.layers.13.mlp.dense_h_to_4h.weight": "model-00004-of-00007.safetensors",
-        "transformer.encoder.layers.13.post_attention_layernorm.weight": "model-00004-of-00007.safetensors",
-        "transformer.encoder.layers.13.self_attention.dense.weight": "model-00004-of-00007.safetensors",
-        "transformer.encoder.layers.13.self_attention.query_key_value.bias": "model-00004-of-00007.safetensors",
-        "transformer.encoder.layers.13.self_attention.query_key_value.weight": "model-00004-of-00007.safetensors",
-        "transformer.encoder.layers.14.input_layernorm.weight": "model-00004-of-00007.safetensors",
-        "transformer.encoder.layers.14.mlp.dense_4h_to_h.weight": "model-00004-of-00007.safetensors",
-        "transformer.encoder.layers.14.mlp.dense_h_to_4h.weight": "model-00004-of-00007.safetensors",
-        "transformer.encoder.layers.14.post_attention_layernorm.weight": "model-00004-of-00007.safetensors",
-        "transformer.encoder.layers.14.self_attention.dense.weight": "model-00004-of-00007.safetensors",
-        "transformer.encoder.layers.14.self_attention.query_key_value.bias": "model-00004-of-00007.safetensors",
-        "transformer.encoder.layers.14.self_attention.query_key_value.weight": "model-00004-of-00007.safetensors",
-        "transformer.encoder.layers.15.input_layernorm.weight": "model-00004-of-00007.safetensors",
-        "transformer.encoder.layers.15.mlp.dense_4h_to_h.weight": "model-00004-of-00007.safetensors",
-        "transformer.encoder.layers.15.mlp.dense_h_to_4h.weight": "model-00004-of-00007.safetensors",
-        "transformer.encoder.layers.15.post_attention_layernorm.weight": "model-00004-of-00007.safetensors",
-        "transformer.encoder.layers.15.self_attention.dense.weight": "model-00004-of-00007.safetensors",
-        "transformer.encoder.layers.15.self_attention.query_key_value.bias": "model-00004-of-00007.safetensors",
-        "transformer.encoder.layers.15.self_attention.query_key_value.weight": "model-00004-of-00007.safetensors",
-        "transformer.encoder.layers.16.input_layernorm.weight": "model-00004-of-00007.safetensors",
-        "transformer.encoder.layers.16.mlp.dense_4h_to_h.weight": "model-00004-of-00007.safetensors",
-        "transformer.encoder.layers.16.mlp.dense_h_to_4h.weight": "model-00004-of-00007.safetensors",
-        "transformer.encoder.layers.16.post_attention_layernorm.weight": "model-00004-of-00007.safetensors",
-        "transformer.encoder.layers.16.self_attention.dense.weight": "model-00004-of-00007.safetensors",
-        "transformer.encoder.layers.16.self_attention.query_key_value.bias": "model-00004-of-00007.safetensors",
-        "transformer.encoder.layers.16.self_attention.query_key_value.weight": "model-00004-of-00007.safetensors",
-        "transformer.encoder.layers.17.input_layernorm.weight": "model-00004-of-00007.safetensors",
-        "transformer.encoder.layers.17.mlp.dense_4h_to_h.weight": "model-00005-of-00007.safetensors",
-        "transformer.encoder.layers.17.mlp.dense_h_to_4h.weight": "model-00005-of-00007.safetensors",
-        "transformer.encoder.layers.17.post_attention_layernorm.weight": "model-00004-of-00007.safetensors",
-        "transformer.encoder.layers.17.self_attention.dense.weight": "model-00004-of-00007.safetensors",
-        "transformer.encoder.layers.17.self_attention.query_key_value.bias": "model-00004-of-00007.safetensors",
-        "transformer.encoder.layers.17.self_attention.query_key_value.weight": "model-00004-of-00007.safetensors",
-        "transformer.encoder.layers.18.input_layernorm.weight": "model-00005-of-00007.safetensors",
-        "transformer.encoder.layers.18.mlp.dense_4h_to_h.weight": "model-00005-of-00007.safetensors",
-        "transformer.encoder.layers.18.mlp.dense_h_to_4h.weight": "model-00005-of-00007.safetensors",
-        "transformer.encoder.layers.18.post_attention_layernorm.weight": "model-00005-of-00007.safetensors",
-        "transformer.encoder.layers.18.self_attention.dense.weight": "model-00005-of-00007.safetensors",
-        "transformer.encoder.layers.18.self_attention.query_key_value.bias": "model-00005-of-00007.safetensors",
-        "transformer.encoder.layers.18.self_attention.query_key_value.weight": "model-00005-of-00007.safetensors",
-        "transformer.encoder.layers.19.input_layernorm.weight": "model-00005-of-00007.safetensors",
-        "transformer.encoder.layers.19.mlp.dense_4h_to_h.weight": "model-00005-of-00007.safetensors",
-        "transformer.encoder.layers.19.mlp.dense_h_to_4h.weight": "model-00005-of-00007.safetensors",
-        "transformer.encoder.layers.19.post_attention_layernorm.weight": "model-00005-of-00007.safetensors",
-        "transformer.encoder.layers.19.self_attention.dense.weight": "model-00005-of-00007.safetensors",
-        "transformer.encoder.layers.19.self_attention.query_key_value.bias": "model-00005-of-00007.safetensors",
-        "transformer.encoder.layers.19.self_attention.query_key_value.weight": "model-00005-of-00007.safetensors",
-        "transformer.encoder.layers.2.input_layernorm.weight": "model-00001-of-00007.safetensors",
-        "transformer.encoder.layers.2.mlp.dense_4h_to_h.weight": "model-00001-of-00007.safetensors",
-        "transformer.encoder.layers.2.mlp.dense_h_to_4h.weight": "model-00001-of-00007.safetensors",
-        "transformer.encoder.layers.2.post_attention_layernorm.weight": "model-00001-of-00007.safetensors",
-        "transformer.encoder.layers.2.self_attention.dense.weight": "model-00001-of-00007.safetensors",
-        "transformer.encoder.layers.2.self_attention.query_key_value.bias": "model-00001-of-00007.safetensors",
-        "transformer.encoder.layers.2.self_attention.query_key_value.weight": "model-00001-of-00007.safetensors",
-        "transformer.encoder.layers.20.input_layernorm.weight": "model-00005-of-00007.safetensors",
-        "transformer.encoder.layers.20.mlp.dense_4h_to_h.weight": "model-00005-of-00007.safetensors",
-        "transformer.encoder.layers.20.mlp.dense_h_to_4h.weight": "model-00005-of-00007.safetensors",
-        "transformer.encoder.layers.20.post_attention_layernorm.weight": "model-00005-of-00007.safetensors",
-        "transformer.encoder.layers.20.self_attention.dense.weight": "model-00005-of-00007.safetensors",
-        "transformer.encoder.layers.20.self_attention.query_key_value.bias": "model-00005-of-00007.safetensors",
-        "transformer.encoder.layers.20.self_attention.query_key_value.weight": "model-00005-of-00007.safetensors",
-        "transformer.encoder.layers.21.input_layernorm.weight": "model-00005-of-00007.safetensors",
-        "transformer.encoder.layers.21.mlp.dense_4h_to_h.weight": "model-00005-of-00007.safetensors",
-        "transformer.encoder.layers.21.mlp.dense_h_to_4h.weight": "model-00005-of-00007.safetensors",
-        "transformer.encoder.layers.21.post_attention_layernorm.weight": "model-00005-of-00007.safetensors",
-        "transformer.encoder.layers.21.self_attention.dense.weight": "model-00005-of-00007.safetensors",
-        "transformer.encoder.layers.21.self_attention.query_key_value.bias": "model-00005-of-00007.safetensors",
-        "transformer.encoder.layers.21.self_attention.query_key_value.weight": "model-00005-of-00007.safetensors",
-        "transformer.encoder.layers.22.input_layernorm.weight": "model-00005-of-00007.safetensors",
-        "transformer.encoder.layers.22.mlp.dense_4h_to_h.weight": "model-00006-of-00007.safetensors",
-        "transformer.encoder.layers.22.mlp.dense_h_to_4h.weight": "model-00006-of-00007.safetensors",
-        "transformer.encoder.layers.22.post_attention_layernorm.weight": "model-00006-of-00007.safetensors",
-        "transformer.encoder.layers.22.self_attention.dense.weight": "model-00006-of-00007.safetensors",
-        "transformer.encoder.layers.22.self_attention.query_key_value.bias": "model-00006-of-00007.safetensors",
-        "transformer.encoder.layers.22.self_attention.query_key_value.weight": "model-00006-of-00007.safetensors",
-        "transformer.encoder.layers.23.input_layernorm.weight": "model-00006-of-00007.safetensors",
-        "transformer.encoder.layers.23.mlp.dense_4h_to_h.weight": "model-00006-of-00007.safetensors",
-        "transformer.encoder.layers.23.mlp.dense_h_to_4h.weight": "model-00006-of-00007.safetensors",
-        "transformer.encoder.layers.23.post_attention_layernorm.weight": "model-00006-of-00007.safetensors",
-        "transformer.encoder.layers.23.self_attention.dense.weight": "model-00006-of-00007.safetensors",
-        "transformer.encoder.layers.23.self_attention.query_key_value.bias": "model-00006-of-00007.safetensors",
-        "transformer.encoder.layers.23.self_attention.query_key_value.weight": "model-00006-of-00007.safetensors",
-        "transformer.encoder.layers.24.input_layernorm.weight": "model-00006-of-00007.safetensors",
-        "transformer.encoder.layers.24.mlp.dense_4h_to_h.weight": "model-00006-of-00007.safetensors",
-        "transformer.encoder.layers.24.mlp.dense_h_to_4h.weight": "model-00006-of-00007.safetensors",
-        "transformer.encoder.layers.24.post_attention_layernorm.weight": "model-00006-of-00007.safetensors",
-        "transformer.encoder.layers.24.self_attention.dense.weight": "model-00006-of-00007.safetensors",
-        "transformer.encoder.layers.24.self_attention.query_key_value.bias": "model-00006-of-00007.safetensors",
-        "transformer.encoder.layers.24.self_attention.query_key_value.weight": "model-00006-of-00007.safetensors",
-        "transformer.encoder.layers.25.input_layernorm.weight": "model-00006-of-00007.safetensors",
-        "transformer.encoder.layers.25.mlp.dense_4h_to_h.weight": "model-00006-of-00007.safetensors",
-        "transformer.encoder.layers.25.mlp.dense_h_to_4h.weight": "model-00006-of-00007.safetensors",
-        "transformer.encoder.layers.25.post_attention_layernorm.weight": "model-00006-of-00007.safetensors",
-        "transformer.encoder.layers.25.self_attention.dense.weight": "model-00006-of-00007.safetensors",
-        "transformer.encoder.layers.25.self_attention.query_key_value.bias": "model-00006-of-00007.safetensors",
-        "transformer.encoder.layers.25.self_attention.query_key_value.weight": "model-00006-of-00007.safetensors",
-        "transformer.encoder.layers.26.input_layernorm.weight": "model-00006-of-00007.safetensors",
-        "transformer.encoder.layers.26.mlp.dense_4h_to_h.weight": "model-00007-of-00007.safetensors",
-        "transformer.encoder.layers.26.mlp.dense_h_to_4h.weight": "model-00006-of-00007.safetensors",
-        "transformer.encoder.layers.26.post_attention_layernorm.weight": "model-00006-of-00007.safetensors",
-        "transformer.encoder.layers.26.self_attention.dense.weight": "model-00006-of-00007.safetensors",
-        "transformer.encoder.layers.26.self_attention.query_key_value.bias": "model-00006-of-00007.safetensors",
-        "transformer.encoder.layers.26.self_attention.query_key_value.weight": "model-00006-of-00007.safetensors",
-        "transformer.encoder.layers.27.input_layernorm.weight": "model-00007-of-00007.safetensors",
-        "transformer.encoder.layers.27.mlp.dense_4h_to_h.weight": "model-00007-of-00007.safetensors",
-        "transformer.encoder.layers.27.mlp.dense_h_to_4h.weight": "model-00007-of-00007.safetensors",
-        "transformer.encoder.layers.27.post_attention_layernorm.weight": "model-00007-of-00007.safetensors",
-        "transformer.encoder.layers.27.self_attention.dense.weight": "model-00007-of-00007.safetensors",
-        "transformer.encoder.layers.27.self_attention.query_key_value.bias": "model-00007-of-00007.safetensors",
-        "transformer.encoder.layers.27.self_attention.query_key_value.weight": "model-00007-of-00007.safetensors",
-        "transformer.encoder.layers.3.input_layernorm.weight": "model-00001-of-00007.safetensors",
-        "transformer.encoder.layers.3.mlp.dense_4h_to_h.weight": "model-00002-of-00007.safetensors",
-        "transformer.encoder.layers.3.mlp.dense_h_to_4h.weight": "model-00002-of-00007.safetensors",
-        "transformer.encoder.layers.3.post_attention_layernorm.weight": "model-00001-of-00007.safetensors",
-        "transformer.encoder.layers.3.self_attention.dense.weight": "model-00001-of-00007.safetensors",
-        "transformer.encoder.layers.3.self_attention.query_key_value.bias": "model-00001-of-00007.safetensors",
-        "transformer.encoder.layers.3.self_attention.query_key_value.weight": "model-00001-of-00007.safetensors",
-        "transformer.encoder.layers.4.input_layernorm.weight": "model-00002-of-00007.safetensors",
-        "transformer.encoder.layers.4.mlp.dense_4h_to_h.weight": "model-00002-of-00007.safetensors",
-        "transformer.encoder.layers.4.mlp.dense_h_to_4h.weight": "model-00002-of-00007.safetensors",
-        "transformer.encoder.layers.4.post_attention_layernorm.weight": "model-00002-of-00007.safetensors",
-        "transformer.encoder.layers.4.self_attention.dense.weight": "model-00002-of-00007.safetensors",
-        "transformer.encoder.layers.4.self_attention.query_key_value.bias": "model-00002-of-00007.safetensors",
-        "transformer.encoder.layers.4.self_attention.query_key_value.weight": "model-00002-of-00007.safetensors",
-        "transformer.encoder.layers.5.input_layernorm.weight": "model-00002-of-00007.safetensors",
-        "transformer.encoder.layers.5.mlp.dense_4h_to_h.weight": "model-00002-of-00007.safetensors",
-        "transformer.encoder.layers.5.mlp.dense_h_to_4h.weight": "model-00002-of-00007.safetensors",
-        "transformer.encoder.layers.5.post_attention_layernorm.weight": "model-00002-of-00007.safetensors",
-        "transformer.encoder.layers.5.self_attention.dense.weight": "model-00002-of-00007.safetensors",
-        "transformer.encoder.layers.5.self_attention.query_key_value.bias": "model-00002-of-00007.safetensors",
-        "transformer.encoder.layers.5.self_attention.query_key_value.weight": "model-00002-of-00007.safetensors",
-        "transformer.encoder.layers.6.input_layernorm.weight": "model-00002-of-00007.safetensors",
-        "transformer.encoder.layers.6.mlp.dense_4h_to_h.weight": "model-00002-of-00007.safetensors",
-        "transformer.encoder.layers.6.mlp.dense_h_to_4h.weight": "model-00002-of-00007.safetensors",
-        "transformer.encoder.layers.6.post_attention_layernorm.weight": "model-00002-of-00007.safetensors",
-        "transformer.encoder.layers.6.self_attention.dense.weight": "model-00002-of-00007.safetensors",
-        "transformer.encoder.layers.6.self_attention.query_key_value.bias": "model-00002-of-00007.safetensors",
-        "transformer.encoder.layers.6.self_attention.query_key_value.weight": "model-00002-of-00007.safetensors",
-        "transformer.encoder.layers.7.input_layernorm.weight": "model-00002-of-00007.safetensors",
-        "transformer.encoder.layers.7.mlp.dense_4h_to_h.weight": "model-00002-of-00007.safetensors",
-        "transformer.encoder.layers.7.mlp.dense_h_to_4h.weight": "model-00002-of-00007.safetensors",
-        "transformer.encoder.layers.7.post_attention_layernorm.weight": "model-00002-of-00007.safetensors",
-        "transformer.encoder.layers.7.self_attention.dense.weight": "model-00002-of-00007.safetensors",
-        "transformer.encoder.layers.7.self_attention.query_key_value.bias": "model-00002-of-00007.safetensors",
-        "transformer.encoder.layers.7.self_attention.query_key_value.weight": "model-00002-of-00007.safetensors",
-        "transformer.encoder.layers.8.input_layernorm.weight": "model-00002-of-00007.safetensors",
-        "transformer.encoder.layers.8.mlp.dense_4h_to_h.weight": "model-00003-of-00007.safetensors",
-        "transformer.encoder.layers.8.mlp.dense_h_to_4h.weight": "model-00003-of-00007.safetensors",
-        "transformer.encoder.layers.8.post_attention_layernorm.weight": "model-00003-of-00007.safetensors",
-        "transformer.encoder.layers.8.self_attention.dense.weight": "model-00003-of-00007.safetensors",
-        "transformer.encoder.layers.8.self_attention.query_key_value.bias": "model-00003-of-00007.safetensors",
-        "transformer.encoder.layers.8.self_attention.query_key_value.weight": "model-00003-of-00007.safetensors",
-        "transformer.encoder.layers.9.input_layernorm.weight": "model-00003-of-00007.safetensors",
-        "transformer.encoder.layers.9.mlp.dense_4h_to_h.weight": "model-00003-of-00007.safetensors",
-        "transformer.encoder.layers.9.mlp.dense_h_to_4h.weight": "model-00003-of-00007.safetensors",
-        "transformer.encoder.layers.9.post_attention_layernorm.weight": "model-00003-of-00007.safetensors",
-        "transformer.encoder.layers.9.self_attention.dense.weight": "model-00003-of-00007.safetensors",
-        "transformer.encoder.layers.9.self_attention.query_key_value.bias": "model-00003-of-00007.safetensors",
-        "transformer.encoder.layers.9.self_attention.query_key_value.weight": "model-00003-of-00007.safetensors",
-        "transformer.output_layer.weight": "model-00007-of-00007.safetensors",
-        "transformer.rotary_pos_emb.inv_freq": "model-00001-of-00007.safetensors"
-    }
-}

modeling_chatglm.py

@@ -14,7 +14,6 @@ from torch.nn import CrossEntropyLoss, LayerNorm, MSELoss, BCEWithLogitsLoss
 from torch.nn.utils import skip_init
 from typing import Optional, Tuple, Union, List, Callable, Dict, Any
 from copy import deepcopy
-import transformers
 
 from transformers.modeling_outputs import (
     BaseModelOutputWithPast,
@@ -46,9 +45,6 @@ CHATGLM_6B_PRETRAINED_MODEL_ARCHIVE_LIST = [
     # See all ChatGLM models at https://huggingface.co/models?filter=chatglm
 ]
 
-is_transformers_4_42_or_higher = int(transformers.__version__.split(".")[1]) >= 42
-is_transformers_4_44_or_higher = int(transformers.__version__.split(".")[1]) >= 44
-
 
 def default_init(cls, *args, **kwargs):
     return cls(*args, **kwargs)
@@ -185,7 +181,7 @@ def apply_rotary_pos_emb(x: torch.Tensor, rope_cache: torch.Tensor) -> torch.Ten
 class RMSNorm(torch.nn.Module):
     def __init__(self, normalized_shape, eps=1e-5, device=None, dtype=None, **kwargs):
         super().__init__()
-        self.weight = torch.nn.Parameter(torch.ones(normalized_shape, device=device, dtype=dtype))
+        self.weight = torch.nn.Parameter(torch.empty(normalized_shape, device=device, dtype=dtype))
         self.eps = eps
 
     def forward(self, hidden_states: torch.Tensor):
@@ -638,8 +634,7 @@ class GLMTransformer(torch.nn.Module):
                     attention_mask,
                     rotary_pos_emb,
                     kv_caches[index],
-                    use_cache,
-                    use_reentrant=False
+                    use_cache
                 )
             else:
                 layer_ret = layer(
@@ -702,9 +697,9 @@ class ChatGLMPreTrainedModel(PreTrainedModel):
         position_ids = torch.arange(seq_length, dtype=torch.long, device=device).unsqueeze(0).repeat(batch_size, 1)
         return position_ids
 
-    def gradient_checkpointing_enable(self, gradient_checkpointing_kwargs=None):
-        if not self.supports_gradient_checkpointing:
-            raise ValueError(f"{self.__class__.__name__} does not support gradient checkpointing.")
+    def _set_gradient_checkpointing(self, module, value=False):
+        if isinstance(module, GLMTransformer):
+            module.gradient_checkpointing = value
 
 
 class Embedding(torch.nn.Module):
@@ -773,9 +768,6 @@ class ChatGLMModel(ChatGLMPreTrainedModel):
     def get_input_embeddings(self):
         return self.embedding.word_embeddings
 
-    def set_input_embeddings(self, value):
-        self.embedding.word_embeddings = value
-
     def get_prompt(self, batch_size, device, dtype=torch.half):
         prefix_tokens = self.prefix_tokens.unsqueeze(0).expand(batch_size, -1).to(device)
         past_key_values = self.prefix_encoder(prefix_tokens).type(dtype)
@@ -876,19 +868,9 @@ class ChatGLMForConditionalGeneration(ChatGLMPreTrainedModel):
             standardize_cache_format: bool = False,
     ) -> Dict[str, Any]:
         # update past_key_values
-        if is_transformers_4_44_or_higher:
-            model_kwargs["past_key_values"] = self._extract_past_from_model_output(
-                outputs
-            )[1]
-        elif is_transformers_4_42_or_higher:
-            # update past_key_values
-            model_kwargs["past_key_values"] = self._extract_past_from_model_output(
-                outputs, standardize_cache_format=standardize_cache_format
-            )[1]
-        else:
-            model_kwargs["past_key_values"] = self._extract_past_from_model_output(
-                outputs, standardize_cache_format=standardize_cache_format
-            )
+        model_kwargs["past_key_values"] = self._extract_past_from_model_output(
+            outputs, standardize_cache_format=standardize_cache_format
+        )
 
         # update attention mask
         if "attention_mask" in model_kwargs:
@@ -1018,10 +1000,7 @@ class ChatGLMForConditionalGeneration(ChatGLMPreTrainedModel):
         content = ""
         history = deepcopy(history)
         for response in output.split("<|assistant|>"):
-            if "\n" in response:
-                metadata, content = response.split("\n", maxsplit=1)
-            else:
-                metadata, content = "", response
+            metadata, content = response.split("\n", maxsplit=1)
             if not metadata.strip():
                 content = content.strip()
                 history.append({"role": "assistant", "metadata": metadata, "content": content})

quantization.py

@@ -1,17 +1,20 @@
+from torch.nn import Linear
 from torch.nn.parameter import Parameter
+
 import bz2
 import torch
 import base64
 import ctypes
 from transformers.utils import logging
+
 from typing import List
+from functools import partial
 
 logger = logging.get_logger(__name__)
 
 try:
     from cpm_kernels.kernels.base import LazyKernelCModule, KernelFunction, round_up
 
-
     class Kernel:
         def __init__(self, code: bytes, function_names: List[str]):
             self.code = code
@@ -21,7 +24,6 @@ try:
             for name in self._function_names:
                 setattr(self, name, KernelFunction(self._cmodule, name))
 
-
     quantization_code = "$QlpoOTFBWSZTWU9yuJUAQHN//////////f/n/8/n///n//bt4dTidcVx8X3V9FV/92/v4B7/AD5FBQFAAAChSgKpFCFAFVSigUAAAEKhSgUUqgFBKigqVREQAABQBQIANDTTIGI00BkZBkNGE0A0BkBkGQGRkaNAaAGQNBoGgDIAAYIGTI0DQAQAaGmmQMRpoDIyDIaMJoBoDIDIMgMjI0aA0AMgaDQNAGQAAwQMmRoGgAgA0NNMgYjTQGRkGQ0YTQDQGQGQZAZGRo0BoAZA0GgaAMgABggZMjQNABABoaaZAxGmgMjIMhowmgGgMgMgyAyMjRoDQAyBoNA0AZAADBAyZGgaAAmqU1NEgJqnptU/Sn4jRR6J6epk2pqb1Q/SgAPUGgyNNGjQ2SBpoAZAAGg0NB6mgDIAAAAA2oaApSREBNAARhGiYEaEwU8pvImlP0k2aam1GaGqbFNM1MHpTwmkepmyU9R6nqPKekHqNNPUxNGhp6n6p6QaZ6o9TG1GMqcoV9ly6nRanHlq6zPNbnGZNi6HSug+2nPiZ13XcnFYZW+45W11CumhzYhchOJ2GLLV1OBjBjGf4TptOddTSOcVxhqYZMYwZXZZY00zI1paX5X9J+b+f4e+x43RXSxXPOdquiGpduatGyXneN696M9t4HU2eR5XX/kPhP261NTx3JO1Ow7LyuDmeo9a7d351T1ZxnvnrvYnrXv/hXxPCeuYx2XsNmO003eg9J3Z6U7b23meJ4ri01OdzTk9BNO96brz+qT5nuvvH3ds/G+m/JcG/F2XYuhXlvO+jP7U3XgrzPN/lr8Sf1n6j4j7jZs+s/T0tNaNNYzTs12rxjwztHlnire3Nzc3N1wuBwOBwXBvZfoHpD7rFmR99V5vj3aXza3xdBbXMalubTg/jIv5dfAi54Pdc75j4z412n3Npj3Ld/ENm7a3b/Cod6h/ret1/5vn/C+l+gdslMvgPSLJ8d8q+U66fevYn/tW1chleEtNTGlcHCbLRlq0tHzF5tsbbZZfHjjLgZu42XCuC3NrdjTasZGNzgxPIrGqp7r3p7L2p5XjnpPSmTd5XtzqnB6U87zzg1Ol0zd0zsLszxR6lkxp35u6/teL0L0W922cR7Lu1lpL9CsHirzuM2T+BgsyViT6LHcm0/Vr6U/7LGGyJeqTEjt0PHWhF5mCT7R9mtlDwriYv0Tyr/OxYt6qp5r0mPVT0608TqnqMZaarU2nFwrTzzlrs1ed7z1ux60wyr4ydCaTi3enW8x68x0zU7tXSlcmPSW1mGpWJMg4zmPC2lK96tp0OE80y4MfEvnZj8zGluR6b22ki1Ou9V2nCd9xovcPvcYMZYy0lvN60ScZ45vN6yeCeeXFb1lVjnnCar5fwXwE2bzJ4HI1XVPXfXZMm44GUsMpYsmLB65TuVdm0cl0b+i/wGNN66XjeV7zuPpHcnK/juhhjdfId5jMdE5nN0dGmmm2zZs2cexD5n9p/dY352XsvXHaZNWWsmmS1atjR452nYudzvqv2HMRyvNNnlMcDl3R2+yx2uVrBubTW9icHDVtbNXlZm7jma1rM4VurZZd2y6nUau7ZXZ7bVU+mnoOVxZGMrVmvX60605JwmzGZhhhjTWtaaaMaaGTGmNMZasY0iX8VMUl8eepaIrzGSpemWOQyZORk2bNpjUybMmxqYmknCGCFynutfksaZpjTNMaaatM0xsxcGR0sociNqxNSmhhR1ZJPbsn8qyF0t2qH6iYBclclalbtTTcHTDsPaX6rlnElph2Jyumumtynv2Kk8GI7rsvXbIcJgHJOSaSXnnGaI3m87RtVXJOZ/YtgdTE6Wpha6ZlE8ayXkef1fh602r2WwvfMXtMdLlkfnLFdYYwYso+bWqm7yJqHXZGw2nrS5ZanSYnWlxBxMF1V940K2wdrI7R6OYf7DGGamMmTSbRhlS45xmVOumF1EyPCmHrrN8wwZOOrdNtLeMtzFzDlWnfTBxMk2NaXIZHBYxYLD4w8yju0ao65Vz1OIXoS9dLanwCe1PWrYuWMqf1if1z2k2yYfKJ741PDgno1ZQ8DRqvUny3mNoWTzGO6m1DkrJI8JiR5cSd+vZdGOO8nrMoc5+NDUFsMSXaZJeNlMmGLtJsovOsUp7I9S5VojKxF6bTVEelXqlfJobQr3LozSh2Jk7VcrVMfhXqszGWMzNqGhqZY0OadxkyyMssKugZR0KNFXBHlqwmJgTE/BNVMk6ItJXZMR0H47GpXv/DMOvNkmVuaV1PRfEdxuqc7Hcd+ZV/zTLaRxWk0nl9CdCeM6mn5rstHIBcpiuwmUZXeq81DacHI2rmrZ5SuE5mOZd6LQrZg9mx32TprA8BMo5jKN6yLTCi3WzQaZSuhzTtM1fUTGVpG8Tw+KXI0tjEpiWxtLYynOlktSbVlaI5kxP8TDH8kx50xoxi5KcA4pcja8KWLRlO/Ks6q06ergnvm1ca3Tq8Uw7LTUsmWyctXPWmpitl/uvGcWTGXGuAXDfhqazGmjkxcJW5hMMMMpYsXl2TZYtVOddG3XCarUt6Ptq9CZXSNzyuRzqRZOjsxdBbFVz6OA5HI43r1jityVlVpVkxmOsyaYWE1NTGq1sOVh36mHMcxtSvcy70edG0ZGR3I1Go1GRlV7mWWo1G0ZGRqlvH40l7o4m5xMWLLLYyNjnqc8556mdPqLJ31n/1nWOncxzG1tizrHs/Z+d2vP/B/l8wdJ6rHUn2nbbDq4p6htFtYzMMMTaZis1K5GKzGNmxhmUx2DDlZ/qNnIx41xnaMfCZWYaZWtNLTNW8ND4Fw1MyZOCdM428suKG1ehW8TesOydg7J+YYcD4cYR+8dFK6M4E3HM9ZfRNNL+Sn6rsl4DsrDl2HpPCnfxjGXtbZtYys1ttlyJ4T+BvexjGWRjMszK4Jpc77D3GyuVD7q0+G8m9G+2+rGm7cOR2y7FdtY2XUYx/oNlfRYxhMYyYZkyyg55enna9Kt/FFi6GMMwYwdwxWgxGMLKYmUyGExTKMZkMFhkymKuh0NOBNnBu+23LdwDoZYYzGGMxtORaTU1pjTGWTTGGtMrNWUsyyTTLLG1qy2ZjbK2DBllWqxMtBMaYZQmcE7zvvRcTkclUwdkxTaSdyySt/7fpL+T1v516Ji97fwr5JbLu305zMn5+GMTTZ9F+y7ExwmGVfG44yxn3dLv6l5i+Wth1jCrDq21nW9LqvvDzz3Vf3LLH/O/32TJ/erx3bXftO4eF+G956D952K/An4NfvOpjFjExjevP/UmE0fIoZXx6/w6lX/no3D0bLt+ixjieBM6ksRd0yB4Lt2SwYNE+gd1detlZWUnpiZfGfFaK+4PyCa/v18V8X75pe9fLXzp7l3VjF76vWZmHwGz1IZNWT7b8yddJ4q5kyrVdfru6atWc7bVYztL9Jf4GXvT+Y8m9/YsXP6H018a8D4XVOqvfzqeR+6yZOD8dPv0+U7/q5Pl+2dNb0MjzGVH5p6MNQ7cOWvw62U9aHE8DprDek+McLyvDz+te+9Zhq5+YTruufMcWMabqysTmZVWjKPfnK0wyVcrsuhjZRdLkHNvD72b9abriOSGIxiLixMOoalNPXzy+wT/tf+U6HHONfsz+xe8ufHBdQWWGWLA9if0rsnmrxK5LvRZQeWsTCsrmOYy8VteVfuRfcVTtDLItLIsMYxZLdU/DbtSemxF6Z6Zo5WBXE4tFdCyVMMXMTEMZXVlS6Xec2T4e0tHsRcEuWshcJ2YsNF5rUx1E8ifCq6Z+ZP7qdCeu/aTwFd53l16/o0NOw6O3dLavP4Hbi4RdmuDk6DoYaninC0+o4uZjbJ7Rxeu0/FbuFg+q7DVS6fQe0rZ6NDGUNNU6DEqOaLTicKnYZMnBWruljQxoaS3dZhocDge0bSTyOvdAbG5hxe2xji7E/L55xX13wWNDi6HCekcFxfCPGxY0MXC+s7afWaMdDyjyr+o8Rudm/NabOZvdl274zH4f5XK9z6On1Pe/K5TdPAslg77BjuO6Y3eO7GqvOPG/stknp1leyvLL0Z7bl9I4noMvLkzytLhWYzrOZzLXCORe028rORzOg4N/L0HlMOQ3Pgmnbb6KczlabORpu980q37TBqRu0/p3PO6234Bl03Ynuz+9W7gnsEcmvYaYY3aMYY0wx3pYd+ujsXauWdaY5Xkbtl23fPzFHiDB/QMo0yFjBllYxTQYYyxkrwn7JufwJ/PfgJ+C83X69ni6zvXcnyXabv0ncbLwsceS+RNlyN2mnneJtX0ngYO0+e+0+UnA+Wch3ji8hj5an4h+i6XBySU4n+R0roVcbw5yvHrmr4Yw8Y7x6c+9POPYHI5HI5HI5HI5HGXGww4nE4nrVyOR8XeqPEO7PLOiukYa3Novk5hV4cdtYZLI93e+uxff2jRo0aNGjRo0aNG1bVtW1dy3m83m8+tQ5ZzHw3nObwOu8La9Rc1dtkdS8A3eTk823tnktXWlxN6Oixe06zrN70Isd9jiOgZFq9yfkPqP/SLhN2Myl8jDM43bl1nbcb4cO57jlh8Jow6pzXZdL4dyODTuuhu77FyO27DdwdRxmvO+O+3N2+BdqyTwLHVczDVY4UPE4O66/ZO2cx1LFzVdSXtF7G4HMbrauOHRw6c8FdZ5m9fHZHYZXfTlZquyynSyTTKke6vcffSD9pzPA/G7n7jxPmuhc1DHMynPMrGL6AdewYmwu5ko+UUyTwrMv27rPH1v1nGqd87+p6N6LU8k3NEng53xXyHS97+44OSg/sy/hn+Se6yfYNjW0/uTgP+PvWYzLMmjhcLB/gGpri6H83/84eUXWT6T9Hsv7785z/7z4icpW+zfXypuR7rx/gMdZb1/wC678pcs8/2a3mDitGHxl9mfPlll5MafWWqxk/eYuTDgcNMzDGWLWvsuglNxs53GtN6uWpktlW1tZZYcuinMMWmnNnJydze3b2Y1McBxrBkXw799izLMZZYyy0TkbsGM4p03S2uVu5s/XXUdSdec6smVxZYYGpVmT8A+8ajuEyV5FatkvVru2x6uxGXXbH4A+jvgP4GMYy3iPLXzq/6z65+E005ey+cwMZD3fZcqc6xpjTFjQ0P3U+e++cPYmTIwj0nrK5NPTfl3WvpfLtXDcb2HQMudYOxFXQBor4L4T6vrOauFctYXJQ++NUWmJe5bmx1jDiZS1dTqWxo4GR8jm3fttpmPHppk9PEyv4/y8/sO07XacOmcqc0x2Vi9BvNJvN5oW8x4mOsydpidRxMYJPx06m1bqPzq9KtK8sxXNXFodD/+MYYaJTLwOhc9brCsV18oOR1i4tXChyTkq4lf4y1Ke+9axjDHqs1mfBbMXuP4Hzi+X7t8vzv7bHerrUPgPCxhjre4fXdfLNtNM+Jd+Zdh8xd8wP87uNPoPgv4W7/5P2BuxfsMabNnMnza+54Pdi5U671GPZY8CehX8Voeoo7FHpkeEc6715FwHZrIrUrHaviPUbPZHND+IhczrP6FcYvhOZ0Di/ETt0OI+YwNWR9r7tpf6WDeZKZDB1+z2IthOl1mPyb5FluvEx9h9d0NnM0Y1XPFkWIsk1WotJ0PBMmkvjvQTd0e71tfeV+8r8lQ/tpzpsmxJ+InrI/dj2UajUajVTUajatRqNRtGo1Go1Go4wjeMpZFMVV9CHbofPraLsJ3JpWV2XOoanCuFky4y3PPNxucK2uKC1Lbdb1eo+m5XomN6HfeZsabHLHRX/K+offtNGGmHWctcVcG44MdSqsOLY9VzX+Zxfxn2HPdWTpzWvkrtJ8M5zorrKcquRytJ5N5DZmcaW02l76nWO+BqPXm1A2Ry/0q71dH/mqrqeFjkYxjEXtsX8qubTk67rGycyqsdm4tZx5D6D5hhi0waaWmiaMP81Yjii5qxPlPuU/GfTL1Y5E6Jyfiq63qTa39A4J0sOGDgO9WF9bOXl0XfPRbsY2bPNKPy1YrFYrFYmRhhlTIyMjJWJYZHXuCXI8OoXsvfljGLFicNifpp2XunoPiG1wtx3p1Tah+/DD66OnVtVXP9rKbVxOnL0tR/rHtqB5UDErUVcl11D4qqvjpOcxX7armUNJB3LpW6bxVvD08e8h3odKKvyCFZBdSh2FVcST9xV3n3T8t1j7Kr9qgrqXg+13Pt5U7JCvFXVIV1YG5lRhkVYZJYYDDD4KOIMoHCp26WS8GB7uBh2zIdgq/PKyInjV2STShuoapUdCpX1yTwqq/z1VvET7Kh5nVPkO8YyxjLt2MaaMmWTLQvx3qnzltnXW0p2jxgbEtSny/Osv8Y9pLMXYoHVPAhkVdWVeODhR6q9/Sxe2liwwZWMVvFXfRkeIDxAePUPIrdJ4ey6yquzH+PD/bUOWAu05qVHtFd8rrKHSoeNIOUqrYr3FXyToqfYJgwmJdKpXXOwYYegNNGMzfZPp/t3t/DVs4zjNTN61rRqaWaa4NYbRjTa0tWwy2Y2tGN8ZO8ofNKq4j9SL7I+cSm4/6ovLV5HNXLI0jJidwrtk6ynCaP6Z++GjRlWS3tLeW129Mi9evxU9mtz6s5J3Z7M2ngTgnKvmpomxpaLCzPfmx0JWE+m3NLDDGOX47RctdYYNK5jakdqLkRlI39n590T5zctGSwwZZDJj6kW8XSi6ot2MmWWJ0DUT3nuvebBudScjZ79g8cWJ8av0k+/bE5WKd5MdbFpbDVMxu1DVMmtNZGJvq1mtRbn6M+g/kP0FwDwr7quZs7xosNGpbscyxhhd9TyJyFwbLcxlTasg75vW7TsV5K7ji44XPMMrdoj+Y3rT0Hie62nlYV/pwczzOmdLqLhYkzGMzCZWGMQzGMSsZYY6Di1t4nlJ+Em63mJxrVLxPbYxNEdgc1dU2iOKyoYYWjNrEeHTYybVk0atSa7ehuwsWMWTqn1TrnS6hYsi71d1+s+k+ic70e20fzE/VaTdxT9ZtU4GIXdeNx3X77guYYfpHeTQjaMX6brOu4OY4K7Y2d9mbHarI5ox3p4GpJ2Vd/Tst60f7j999pppjR+Q/Qf8J/VaORs3cji7FfFuN61+ui9s8hix1OCh5KGVV23BPXvZfz3CLyHpix+exi8z/KnCnosY2eunor+cxyPO/xJ0vKey9OvE9VjqaYu0x3Z3jd6o2b1T12D+F8l232lwaaacD5LE8LBxu7WTlbWraWpew8Xexjel3E+wWD4APITdNqR8F3R3T0lunCQ4GaE9R37DxeCYfcHi4xci5ovKfxVs55y2hf+65E/Xdp6jR5nrebTmi5incpkyOjs50JvrZwstbbW6kfuuQw+2mykf/EXNFzxfKTrxew929TR6bWnGL//F3JFOFCQT3K4lQ"
 
     kernels = Kernel(
@@ -120,12 +122,11 @@ def extract_weight_to_half(weight: torch.Tensor, scale_list: torch.Tensor, sourc
 
 
 class QuantizedLinear(torch.nn.Module):
-    def __init__(self, weight_bit_width: int, weight, bias=None, device="cuda", dtype=None, empty_init=False):
+    def __init__(self, weight_bit_width: int, weight, bias=None, device="cpu", dtype=None, empty_init=False, *args,
+                 **kwargs):
         super().__init__()
-        weight = weight.to(device)  # ensure the weight is on the cuda device
-        assert str(weight.device).startswith(
-            'cuda'), 'The weights that need to be quantified should be on the CUDA device'
         self.weight_bit_width = weight_bit_width
+
         shape = weight.shape
 
         if weight is None or empty_init:
@@ -153,7 +154,7 @@ def quantize(model, weight_bit_width, empty_init=False, device=None):
     for layer in model.layers:
         layer.self_attention.query_key_value = QuantizedLinear(
             weight_bit_width=weight_bit_width,
-            weight=layer.self_attention.query_key_value.weight,
+            weight=layer.self_attention.query_key_value.weight.to(torch.cuda.current_device()),
             bias=layer.self_attention.query_key_value.bias,
             dtype=layer.self_attention.query_key_value.weight.dtype,
             device=layer.self_attention.query_key_value.weight.device if device is None else device,
@@ -161,7 +162,7 @@ def quantize(model, weight_bit_width, empty_init=False, device=None):
         )
         layer.self_attention.dense = QuantizedLinear(
             weight_bit_width=weight_bit_width,
-            weight=layer.self_attention.dense.weight,
+            weight=layer.self_attention.dense.weight.to(torch.cuda.current_device()),
             bias=layer.self_attention.dense.bias,
             dtype=layer.self_attention.dense.weight.dtype,
             device=layer.self_attention.dense.weight.device if device is None else device,
@@ -169,7 +170,7 @@ def quantize(model, weight_bit_width, empty_init=False, device=None):
         )
         layer.mlp.dense_h_to_4h = QuantizedLinear(
             weight_bit_width=weight_bit_width,
-            weight=layer.mlp.dense_h_to_4h.weight,
+            weight=layer.mlp.dense_h_to_4h.weight.to(torch.cuda.current_device()),
             bias=layer.mlp.dense_h_to_4h.bias,
             dtype=layer.mlp.dense_h_to_4h.weight.dtype,
             device=layer.mlp.dense_h_to_4h.weight.device if device is None else device,
@@ -177,7 +178,7 @@ def quantize(model, weight_bit_width, empty_init=False, device=None):
         )
         layer.mlp.dense_4h_to_h = QuantizedLinear(
             weight_bit_width=weight_bit_width,
-            weight=layer.mlp.dense_4h_to_h.weight,
+            weight=layer.mlp.dense_4h_to_h.weight.to(torch.cuda.current_device()),
             bias=layer.mlp.dense_4h_to_h.bias,
             dtype=layer.mlp.dense_4h_to_h.weight.dtype,
             device=layer.mlp.dense_4h_to_h.weight.device if device is None else device,

special_tokens_map.json

@@ -1 +0,0 @@
-{}

tokenization_chatglm.py

@@ -1,6 +1,6 @@
 import json
 import os
-import re
+import torch
 from typing import List, Optional, Union, Dict
 from sentencepiece import SentencePieceProcessor
 from transformers import PreTrainedTokenizer
@@ -8,9 +8,6 @@ from transformers.utils import logging, PaddingStrategy
 from transformers.tokenization_utils_base import EncodedInput, BatchEncoding
 
 
-logger = logging.get_logger(__name__)
-
-
 class SPTokenizer:
     def __init__(self, model_path: str):
         # reload tokenizer
@@ -24,30 +21,17 @@ class SPTokenizer:
         self.pad_id: int = self.sp_model.unk_id()
         assert self.sp_model.vocab_size() == self.sp_model.get_piece_size()
 
-        role_special_tokens = ["<|system|>", "<|user|>", "<|assistant|>", "<|observation|>"]
-        special_tokens = ["[MASK]", "[gMASK]", "[sMASK]", "sop", "eop"] + role_special_tokens
+        special_tokens = ["[MASK]", "[gMASK]", "[sMASK]", "sop", "eop", "<|system|>", "<|user|>", "<|assistant|>",
+                          "<|observation|>"]
         self.special_tokens = {}
         self.index_special_tokens = {}
         for token in special_tokens:
             self.special_tokens[token] = self.n_words
             self.index_special_tokens[self.n_words] = token
             self.n_words += 1
-        self.role_special_token_expression = "|".join([re.escape(token) for token in special_tokens]) # for apply_chat_template
-
-    def tokenize(self, s: str, encode_special_tokens=False):
-        if encode_special_tokens:
-            last_index = 0
-            t = []
-            for match in re.finditer(self.role_special_token_expression, s):
-                if last_index < match.start():
-                    t.extend(self.sp_model.EncodeAsPieces(s[last_index:match.start()]))
-                t.append(s[match.start():match.end()])
-                last_index = match.end()
-            if last_index < len(s):
-                t.extend(self.sp_model.EncodeAsPieces(s[last_index:]))
-            return t
-        else:
-            return self.sp_model.EncodeAsPieces(s)
+
+    def tokenize(self, s: str):
+        return self.sp_model.EncodeAsPieces(s)
 
     def encode(self, s: str, bos: bool = False, eos: bool = False) -> List[int]:
         assert type(s) is str
@@ -86,40 +70,27 @@ class SPTokenizer:
         """Converts an index (integer) in a token (str) using the vocab."""
         if index in self.index_special_tokens:
             return self.index_special_tokens[index]
-        if index in [self.eos_id, self.bos_id, self.pad_id] or index < 0 or index > self.sp_model.vocab_size():
+        if index in [self.eos_id, self.bos_id, self.pad_id] or index < 0:
             return ""
         return self.sp_model.IdToPiece(index)
 
 
 class ChatGLMTokenizer(PreTrainedTokenizer):
-
     vocab_files_names = {"vocab_file": "tokenizer.model"}
+
     model_input_names = ["input_ids", "attention_mask", "position_ids"]
 
-    def __init__(
-        self,
-        vocab_file,
-        padding_side="left",
-        clean_up_tokenization_spaces=False,
-        encode_special_tokens=False,
-        **kwargs
-    ):
+    def __init__(self, vocab_file, padding_side="left", clean_up_tokenization_spaces=False, **kwargs):
         self.name = "GLMTokenizer"
+
         self.vocab_file = vocab_file
         self.tokenizer = SPTokenizer(vocab_file)
         self.special_tokens = {
             "<bos>": self.tokenizer.bos_id,
             "<eos>": self.tokenizer.eos_id,
-            "<unk>": self.tokenizer.pad_id,
             "<pad>": self.tokenizer.pad_id
         }
-        self.encode_special_tokens = encode_special_tokens
-
-        super().__init__(
-            padding_side=padding_side,
-            clean_up_tokenization_spaces=clean_up_tokenization_spaces,
-            **kwargs
-        )
+        super().__init__(padding_side=padding_side, clean_up_tokenization_spaces=clean_up_tokenization_spaces, **kwargs)
 
     def get_command(self, token):
         if token in self.special_tokens:
@@ -129,40 +100,24 @@ class ChatGLMTokenizer(PreTrainedTokenizer):
 
     @property
     def unk_token(self) -> str:
-        return self.tokenizer.sp_model.IdToPiece(self.get_command("<unk>"))
+        return "<unk>"
 
     @property
     def pad_token(self) -> str:
-        return self.tokenizer.sp_model.IdToPiece(self.get_command("<pad>"))
+        return "<unk>"
 
     @property
-    def eos_token(self) -> str:
-        return self.tokenizer.sp_model.IdToPiece(self.get_command("<eos>"))
-
-    @property
-    def unk_token_id(self) -> int:
-        return self.get_command("<unk>")
+    def pad_token_id(self):
+        return self.get_command("<pad>")
 
     @property
-    def pad_token_id(self) -> int:
-        return self.get_command("<pad>")
+    def eos_token(self) -> str:
+        return "</s>"
 
     @property
     def eos_token_id(self):
         return self.get_command("<eos>")
 
-    @unk_token.setter
-    def unk_token(self, value):
-        logger.warning("Setting unk_token is not supported, use the default one.")
-
-    @pad_token.setter
-    def pad_token(self, value):
-        logger.warning("Setting pad_token is not supported, use the default one.")
-
-    @eos_token.setter
-    def eos_token(self, value):
-        logger.warning("Setting eos_token is not supported, use the default one.")
-
     @property
     def vocab_size(self):
         return self.tokenizer.n_words
@@ -174,7 +129,7 @@ class ChatGLMTokenizer(PreTrainedTokenizer):
         return vocab
 
     def _tokenize(self, text, **kwargs):
-        return self.tokenizer.tokenize(text, encode_special_tokens=self.encode_special_tokens)
+        return self.tokenizer.tokenize(text)
 
     def _convert_token_to_id(self, token):
         """ Converts a token (str) in an id using the vocab. """
@@ -240,7 +195,7 @@ class ChatGLMTokenizer(PreTrainedTokenizer):
         return self.batch_encode_plus([input_ids], return_tensors="pt", is_split_into_words=True)
 
     def build_inputs_with_special_tokens(
-        self, token_ids_0: List[int], token_ids_1: Optional[List[int]] = None
+            self, token_ids_0: List[int], token_ids_1: Optional[List[int]] = None
     ) -> List[int]:
         """
         Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and
@@ -265,14 +220,12 @@ class ChatGLMTokenizer(PreTrainedTokenizer):
         return token_ids_0
 
     def _pad(
-        self,
-        encoded_inputs: Union[Dict[str, EncodedInput], BatchEncoding],
-        max_length: Optional[int] = None,
-        padding_strategy: PaddingStrategy = PaddingStrategy.DO_NOT_PAD,
-        pad_to_multiple_of: Optional[int] = None,
-        return_attention_mask: Optional[bool] = None,
-        padding_side: Optional[bool] = None,
-        **kwargs
+            self,
+            encoded_inputs: Union[Dict[str, EncodedInput], BatchEncoding],
+            max_length: Optional[int] = None,
+            padding_strategy: PaddingStrategy = PaddingStrategy.DO_NOT_PAD,
+            pad_to_multiple_of: Optional[int] = None,
+            return_attention_mask: Optional[bool] = None,
     ) -> dict:
         """
         Pad encoded inputs (on left/right and up to predefined length or max length in the batch)
@@ -299,7 +252,6 @@ class ChatGLMTokenizer(PreTrainedTokenizer):
         """
         # Load from model defaults
         assert self.padding_side == "left"
-        assert padding_side is None or padding_side == "left"
 
         required_input = encoded_inputs[self.model_input_names[0]]
         seq_length = len(required_input)

tokenizer_config.json

@@ -1,52 +1,12 @@
 {
-  "added_tokens_decoder": {
-    "64790": {
-      "content": "[gMASK]",
-      "lstrip": false,
-      "normalized": true,
-      "rstrip": false,
-      "single_word": false,
-      "special": false
-    },
-    "64792": {
-      "content": "sop",
-      "lstrip": false,
-      "normalized": true,
-      "rstrip": false,
-      "single_word": false,
-      "special": false
-    },
-    "64795": {
-      "content": "<|user|>",
-      "lstrip": false,
-      "normalized": true,
-      "rstrip": false,
-      "single_word": false,
-      "special": false
-    },
-    "64796": {
-      "content": "<|assistant|>",
-      "lstrip": false,
-      "normalized": true,
-      "rstrip": false,
-      "single_word": false,
-      "special": false
-    }
-  },
+  "name_or_path": "THUDM/chatglm3-6b",
+  "remove_space": false,
+  "do_lower_case": false,
+  "tokenizer_class": "ChatGLMTokenizer",
   "auto_map": {
     "AutoTokenizer": [
       "tokenization_chatglm.ChatGLMTokenizer",
       null
-    ]
-  },
-  "chat_template": "{% for message in messages %}{% if loop.first %}[gMASK]sop<|{{ message['role'] }}|>\n {{ message['content'] }}{% else %}<|{{ message['role'] }}|>\n {{ message['content'] }}{% endif %}{% endfor %}{% if add_generation_prompt %}<|assistant|>{% endif %}",
-  "clean_up_tokenization_spaces": false,
-  "do_lower_case": false,
-  "eos_token": "</s>",
-  "model_max_length": 1000000000000000019884624838656,
-  "pad_token": "<unk>",
-  "padding_side": "left",
-  "remove_space": false,
-  "tokenizer_class": "ChatGLMTokenizer",
-  "unk_token": "<unk>"
+      ]
+  }
 }