Stanislas commited on
Commit
711c7bf
1 Parent(s): 23975c1

Initial commit

Browse files
LICENSE ADDED
@@ -0,0 +1,201 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ Apache License
2
+ Version 2.0, January 2004
3
+ http://www.apache.org/licenses/
4
+
5
+ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
6
+
7
+ 1. Definitions.
8
+
9
+ "License" shall mean the terms and conditions for use, reproduction,
10
+ and distribution as defined by Sections 1 through 9 of this document.
11
+
12
+ "Licensor" shall mean the copyright owner or entity authorized by
13
+ the copyright owner that is granting the License.
14
+
15
+ "Legal Entity" shall mean the union of the acting entity and all
16
+ other entities that control, are controlled by, or are under common
17
+ control with that entity. For the purposes of this definition,
18
+ "control" means (i) the power, direct or indirect, to cause the
19
+ direction or management of such entity, whether by contract or
20
+ otherwise, or (ii) ownership of fifty percent (50%) or more of the
21
+ outstanding shares, or (iii) beneficial ownership of such entity.
22
+
23
+ "You" (or "Your") shall mean an individual or Legal Entity
24
+ exercising permissions granted by this License.
25
+
26
+ "Source" form shall mean the preferred form for making modifications,
27
+ including but not limited to software source code, documentation
28
+ source, and configuration files.
29
+
30
+ "Object" form shall mean any form resulting from mechanical
31
+ transformation or translation of a Source form, including but
32
+ not limited to compiled object code, generated documentation,
33
+ and conversions to other media types.
34
+
35
+ "Work" shall mean the work of authorship, whether in Source or
36
+ Object form, made available under the License, as indicated by a
37
+ copyright notice that is included in or attached to the work
38
+ (an example is provided in the Appendix below).
39
+
40
+ "Derivative Works" shall mean any work, whether in Source or Object
41
+ form, that is based on (or derived from) the Work and for which the
42
+ editorial revisions, annotations, elaborations, or other modifications
43
+ represent, as a whole, an original work of authorship. For the purposes
44
+ of this License, Derivative Works shall not include works that remain
45
+ separable from, or merely link (or bind by name) to the interfaces of,
46
+ the Work and Derivative Works thereof.
47
+
48
+ "Contribution" shall mean any work of authorship, including
49
+ the original version of the Work and any modifications or additions
50
+ to that Work or Derivative Works thereof, that is intentionally
51
+ submitted to Licensor for inclusion in the Work by the copyright owner
52
+ or by an individual or Legal Entity authorized to submit on behalf of
53
+ the copyright owner. For the purposes of this definition, "submitted"
54
+ means any form of electronic, verbal, or written communication sent
55
+ to the Licensor or its representatives, including but not limited to
56
+ communication on electronic mailing lists, source code control systems,
57
+ and issue tracking systems that are managed by, or on behalf of, the
58
+ Licensor for the purpose of discussing and improving the Work, but
59
+ excluding communication that is conspicuously marked or otherwise
60
+ designated in writing by the copyright owner as "Not a Contribution."
61
+
62
+ "Contributor" shall mean Licensor and any individual or Legal Entity
63
+ on behalf of whom a Contribution has been received by Licensor and
64
+ subsequently incorporated within the Work.
65
+
66
+ 2. Grant of Copyright License. Subject to the terms and conditions of
67
+ this License, each Contributor hereby grants to You a perpetual,
68
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
69
+ copyright license to reproduce, prepare Derivative Works of,
70
+ publicly display, publicly perform, sublicense, and distribute the
71
+ Work and such Derivative Works in Source or Object form.
72
+
73
+ 3. Grant of Patent License. Subject to the terms and conditions of
74
+ this License, each Contributor hereby grants to You a perpetual,
75
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
76
+ (except as stated in this section) patent license to make, have made,
77
+ use, offer to sell, sell, import, and otherwise transfer the Work,
78
+ where such license applies only to those patent claims licensable
79
+ by such Contributor that are necessarily infringed by their
80
+ Contribution(s) alone or by combination of their Contribution(s)
81
+ with the Work to which such Contribution(s) was submitted. If You
82
+ institute patent litigation against any entity (including a
83
+ cross-claim or counterclaim in a lawsuit) alleging that the Work
84
+ or a Contribution incorporated within the Work constitutes direct
85
+ or contributory patent infringement, then any patent licenses
86
+ granted to You under this License for that Work shall terminate
87
+ as of the date such litigation is filed.
88
+
89
+ 4. Redistribution. You may reproduce and distribute copies of the
90
+ Work or Derivative Works thereof in any medium, with or without
91
+ modifications, and in Source or Object form, provided that You
92
+ meet the following conditions:
93
+
94
+ (a) You must give any other recipients of the Work or
95
+ Derivative Works a copy of this License; and
96
+
97
+ (b) You must cause any modified files to carry prominent notices
98
+ stating that You changed the files; and
99
+
100
+ (c) You must retain, in the Source form of any Derivative Works
101
+ that You distribute, all copyright, patent, trademark, and
102
+ attribution notices from the Source form of the Work,
103
+ excluding those notices that do not pertain to any part of
104
+ the Derivative Works; and
105
+
106
+ (d) If the Work includes a "NOTICE" text file as part of its
107
+ distribution, then any Derivative Works that You distribute must
108
+ include a readable copy of the attribution notices contained
109
+ within such NOTICE file, excluding those notices that do not
110
+ pertain to any part of the Derivative Works, in at least one
111
+ of the following places: within a NOTICE text file distributed
112
+ as part of the Derivative Works; within the Source form or
113
+ documentation, if provided along with the Derivative Works; or,
114
+ within a display generated by the Derivative Works, if and
115
+ wherever such third-party notices normally appear. The contents
116
+ of the NOTICE file are for informational purposes only and
117
+ do not modify the License. You may add Your own attribution
118
+ notices within Derivative Works that You distribute, alongside
119
+ or as an addendum to the NOTICE text from the Work, provided
120
+ that such additional attribution notices cannot be construed
121
+ as modifying the License.
122
+
123
+ You may add Your own copyright statement to Your modifications and
124
+ may provide additional or different license terms and conditions
125
+ for use, reproduction, or distribution of Your modifications, or
126
+ for any such Derivative Works as a whole, provided Your use,
127
+ reproduction, and distribution of the Work otherwise complies with
128
+ the conditions stated in this License.
129
+
130
+ 5. Submission of Contributions. Unless You explicitly state otherwise,
131
+ any Contribution intentionally submitted for inclusion in the Work
132
+ by You to the Licensor shall be under the terms and conditions of
133
+ this License, without any additional terms or conditions.
134
+ Notwithstanding the above, nothing herein shall supersede or modify
135
+ the terms of any separate license agreement you may have executed
136
+ with Licensor regarding such Contributions.
137
+
138
+ 6. Trademarks. This License does not grant permission to use the trade
139
+ names, trademarks, service marks, or product names of the Licensor,
140
+ except as required for reasonable and customary use in describing the
141
+ origin of the Work and reproducing the content of the NOTICE file.
142
+
143
+ 7. Disclaimer of Warranty. Unless required by applicable law or
144
+ agreed to in writing, Licensor provides the Work (and each
145
+ Contributor provides its Contributions) on an "AS IS" BASIS,
146
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
147
+ implied, including, without limitation, any warranties or conditions
148
+ of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
149
+ PARTICULAR PURPOSE. You are solely responsible for determining the
150
+ appropriateness of using or redistributing the Work and assume any
151
+ risks associated with Your exercise of permissions under this License.
152
+
153
+ 8. Limitation of Liability. In no event and under no legal theory,
154
+ whether in tort (including negligence), contract, or otherwise,
155
+ unless required by applicable law (such as deliberate and grossly
156
+ negligent acts) or agreed to in writing, shall any Contributor be
157
+ liable to You for damages, including any direct, indirect, special,
158
+ incidental, or consequential damages of any character arising as a
159
+ result of this License or out of the use or inability to use the
160
+ Work (including but not limited to damages for loss of goodwill,
161
+ work stoppage, computer failure or malfunction, or any and all
162
+ other commercial damages or losses), even if such Contributor
163
+ has been advised of the possibility of such damages.
164
+
165
+ 9. Accepting Warranty or Additional Liability. While redistributing
166
+ the Work or Derivative Works thereof, You may choose to offer,
167
+ and charge a fee for, acceptance of support, warranty, indemnity,
168
+ or other liability obligations and/or rights consistent with this
169
+ License. However, in accepting such obligations, You may act only
170
+ on Your own behalf and on Your sole responsibility, not on behalf
171
+ of any other Contributor, and only if You agree to indemnify,
172
+ defend, and hold each Contributor harmless for any liability
173
+ incurred by, or claims asserted against, such Contributor by reason
174
+ of your accepting any such warranty or additional liability.
175
+
176
+ END OF TERMS AND CONDITIONS
177
+
178
+ APPENDIX: How to apply the Apache License to your work.
179
+
180
+ To apply the Apache License to your work, attach the following
181
+ boilerplate notice, with the fields enclosed by brackets "[]"
182
+ replaced with your own identifying information. (Don't include
183
+ the brackets!) The text should be enclosed in the appropriate
184
+ comment syntax for the file format. We also recommend that a
185
+ file or class name and description of purpose be included on the
186
+ same "printed page" as the copyright notice for easier
187
+ identification within third-party archives.
188
+
189
+ Copyright Zhengxiao Du
190
+
191
+ Licensed under the Apache License, Version 2.0 (the "License");
192
+ you may not use this file except in compliance with the License.
193
+ You may obtain a copy of the License at
194
+
195
+ http://www.apache.org/licenses/LICENSE-2.0
196
+
197
+ Unless required by applicable law or agreed to in writing, software
198
+ distributed under the License is distributed on an "AS IS" BASIS,
199
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
200
+ See the License for the specific language governing permissions and
201
+ limitations under the License.
MODEL_LICENSE ADDED
@@ -0,0 +1,65 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ The CodeGeeX2-6B License
2
+
3
+ 1. 定义
4
+
5
+ “许可方”是指分发其软件的 CodeGeeX2-6B 模型团队。
6
+
7
+ “软件”是指根据本许可提供的 CodeGeeX2-6B 模型参数。
8
+
9
+ 2. 许可授予
10
+
11
+ 根据本许可的条款和条件,许可方特此授予您非排他性、全球性、不可转让、不可再许可、可撤销、免版税的版权许可。
12
+
13
+ 上述版权声明和本许可声明应包含在本软件的所有副本或重要部分中。
14
+
15
+ 3. 限制
16
+
17
+ 您不得出于任何军事或非法目的使用、复制、修改、合并、发布、分发、复制或创建本软件的全部或部分衍生作品。
18
+
19
+ 您不得利用本软件从事任何危害国家安全和国家统一、危害社会公共利益、侵犯人身权益的行为。
20
+
21
+ 4. 免责声明
22
+
23
+ 本软件“按原样”提供,不提供任何明示或暗示的保证,包括但不限于对适销性、特定用途的适用性和非侵权性的保证。 在任何情况下,作者或版权持有人均不对任何索赔、损害或其他责任负责,无论是在合同诉讼、侵权行为还是其他方面,由软件或软件的使用或其他交易引起、由软件引起或与之相关 软件。
24
+
25
+ 5. 责任限制
26
+
27
+ 除适用法律禁止的范围外,在任何情况下且根据任何法律理论,无论是基于侵权行为、疏忽、合同、责任或其他原因,任何许可方均不对您承担任何直接、间接、特殊、偶然、示范性、 或间接损害,或任何其他商业损失,即使许可人已被告知此类损害的可能性。
28
+
29
+ 6.争议解决
30
+
31
+ 本许可受中华人民共和国法律管辖并按其解释。 因本许可引起的或与本许可有关的任何争议应提交北京市海淀区人民法院。
32
+
33
+ 请注意,许可证可能会更新到更全面的版本。 有关许可和版权的任何问题,请通过 license@zhipuai.cn 与我们联系。
34
+
35
+ 1. Definitions
36
+
37
+ “Licensor” means the CodeGeeX2-6B Model Team that distributes its Software.
38
+
39
+ “Software” means the CodeGeeX2-6B model parameters made available under this license.
40
+
41
+ 2. License Grant
42
+
43
+ Subject to the terms and conditions of this License, the Licensor hereby grants to you a non-exclusive, worldwide, non-transferable, non-sublicensable, revocable, royalty-free copyright license to use the Software.
44
+
45
+ The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
46
+
47
+ 3. Restriction
48
+
49
+ You will not use, copy, modify, merge, publish, distribute, reproduce, or create derivative works of the Software, in whole or in part, for any military, or illegal purposes.
50
+
51
+ You will not use the Software for any act that may undermine China's national security and national unity, harm the public interest of society, or infringe upon the rights and interests of human beings.
52
+
53
+ 4. Disclaimer
54
+
55
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
56
+
57
+ 5. Limitation of Liability
58
+
59
+ EXCEPT TO THE EXTENT PROHIBITED BY APPLICABLE LAW, IN NO EVENT AND UNDER NO LEGAL THEORY, WHETHER BASED IN TORT, NEGLIGENCE, CONTRACT, LIABILITY, OR OTHERWISE WILL ANY LICENSOR BE LIABLE TO YOU FOR ANY DIRECT, INDIRECT, SPECIAL, INCIDENTAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES, OR ANY OTHER COMMERCIAL LOSSES, EVEN IF THE LICENSOR HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.
60
+
61
+ 6. Dispute Resolution
62
+
63
+ This license shall be governed and construed in accordance with the laws of People’s Republic of China. Any dispute arising from or in connection with this License shall be submitted to Haidian District People's Court in Beijing.
64
+
65
+ Note that the license is subject to update to a more comprehensive version. For any questions related to the license and copyright, please contact us at license@zhipuai.cn.
config.json ADDED
@@ -0,0 +1,41 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "_name_or_path": "THUDM/codegeex2-6b",
3
+ "model_type": "chatglm",
4
+ "architectures": [
5
+ "ChatGLMModel"
6
+ ],
7
+ "auto_map": {
8
+ "AutoConfig": "configuration_chatglm.ChatGLMConfig",
9
+ "AutoModel": "modeling_chatglm.ChatGLMForConditionalGeneration",
10
+ "AutoModelForSeq2SeqLM": "modeling_chatglm.ChatGLMForConditionalGeneration"
11
+ },
12
+ "add_bias_linear": false,
13
+ "add_qkv_bias": true,
14
+ "apply_query_key_layer_scaling": true,
15
+ "apply_residual_connection_post_layernorm": false,
16
+ "attention_dropout": 0.0,
17
+ "attention_softmax_in_fp32": true,
18
+ "bias_dropout_fusion": true,
19
+ "ffn_hidden_size": 13696,
20
+ "fp32_residual_connection": false,
21
+ "hidden_dropout": 0.0,
22
+ "hidden_size": 4096,
23
+ "interleaved_qkv": false,
24
+ "kv_channels": 128,
25
+ "layernorm_epsilon": 1e-05,
26
+ "multi_query_attention": true,
27
+ "multi_query_group_num": 2,
28
+ "num_attention_heads": 32,
29
+ "num_layers": 28,
30
+ "original_rope": true,
31
+ "padded_vocab_size": 65024,
32
+ "post_layer_norm": true,
33
+ "rmsnorm": true,
34
+ "rotary_percent": 0.5,
35
+ "seq_length": 8192,
36
+ "use_cache": true,
37
+ "torch_dtype": "bfloat16",
38
+ "transformers_version": "4.27.1",
39
+ "tie_word_embeddings": false,
40
+ "eos_token_id": 2
41
+ }
configuration_chatglm.py ADDED
@@ -0,0 +1,58 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ from transformers import PretrainedConfig
2
+
3
+
4
+ class ChatGLMConfig(PretrainedConfig):
5
+ model_type = "chatglm"
6
+ def __init__(
7
+ self,
8
+ num_layers=28,
9
+ padded_vocab_size=65024,
10
+ hidden_size=4096,
11
+ ffn_hidden_size=13696,
12
+ kv_channels=128,
13
+ num_attention_heads=32,
14
+ seq_length=2048,
15
+ hidden_dropout=0.0,
16
+ attention_dropout=0.0,
17
+ layernorm_epsilon=1e-5,
18
+ rmsnorm=True,
19
+ apply_residual_connection_post_layernorm=False,
20
+ post_layer_norm=True,
21
+ add_bias_linear=False,
22
+ add_qkv_bias=False,
23
+ interleaved_qkv=False,
24
+ bias_dropout_fusion=True,
25
+ rotary_percent=1.0,
26
+ multi_query_attention=False,
27
+ multi_query_group_num=1,
28
+ apply_query_key_layer_scaling=True,
29
+ attention_softmax_in_fp32=True,
30
+ fp32_residual_connection=False,
31
+ quantization_bit=0,
32
+ **kwargs
33
+ ):
34
+ self.num_layers = num_layers
35
+ self.padded_vocab_size = padded_vocab_size
36
+ self.hidden_size = hidden_size
37
+ self.ffn_hidden_size = ffn_hidden_size
38
+ self.kv_channels = kv_channels
39
+ self.num_attention_heads = num_attention_heads
40
+ self.seq_length = seq_length
41
+ self.hidden_dropout = hidden_dropout
42
+ self.attention_dropout = attention_dropout
43
+ self.layernorm_epsilon = layernorm_epsilon
44
+ self.rmsnorm = rmsnorm
45
+ self.apply_residual_connection_post_layernorm = apply_residual_connection_post_layernorm
46
+ self.post_layer_norm = post_layer_norm
47
+ self.add_bias_linear = add_bias_linear
48
+ self.add_qkv_bias = add_qkv_bias
49
+ self.interleaved_qkv = interleaved_qkv
50
+ self.bias_dropout_fusion = bias_dropout_fusion
51
+ self.rotary_percent = rotary_percent
52
+ self.multi_query_attention = multi_query_attention
53
+ self.multi_query_group_num = multi_query_group_num
54
+ self.apply_query_key_layer_scaling = apply_query_key_layer_scaling
55
+ self.attention_softmax_in_fp32 = attention_softmax_in_fp32
56
+ self.fp32_residual_connection = fp32_residual_connection
57
+ self.quantization_bit = quantization_bit
58
+ super().__init__(**kwargs)
modeling_chatglm.py ADDED
@@ -0,0 +1,1092 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """ PyTorch ChatGLM model. """
2
+
3
+ import math
4
+ import copy
5
+ import warnings
6
+ import re
7
+ import sys
8
+
9
+ import torch
10
+ import torch.utils.checkpoint
11
+ import torch.nn.functional as F
12
+ from torch import nn
13
+ from torch.nn import CrossEntropyLoss, LayerNorm
14
+ from torch.nn.utils import skip_init
15
+ from typing import Optional, Tuple, Union, List, Callable, Dict, Any
16
+
17
+ from transformers.modeling_outputs import (
18
+ BaseModelOutputWithPast,
19
+ CausalLMOutputWithPast,
20
+ )
21
+ from transformers.modeling_utils import PreTrainedModel
22
+ from transformers.utils import logging
23
+ from transformers.generation.logits_process import LogitsProcessor
24
+ from transformers.generation.utils import LogitsProcessorList, StoppingCriteriaList, GenerationConfig, ModelOutput
25
+
26
+ from .configuration_chatglm import ChatGLMConfig
27
+
28
+ # flags required to enable jit fusion kernels
29
+
30
+ if sys.platform != 'darwin':
31
+ torch._C._jit_set_profiling_mode(False)
32
+ torch._C._jit_set_profiling_executor(False)
33
+ torch._C._jit_override_can_fuse_on_cpu(True)
34
+ torch._C._jit_override_can_fuse_on_gpu(True)
35
+
36
+ logger = logging.get_logger(__name__)
37
+
38
+ _CHECKPOINT_FOR_DOC = "THUDM/ChatGLM-6B"
39
+ _CONFIG_FOR_DOC = "ChatGLM6BConfig"
40
+
41
+ CHATGLM_6B_PRETRAINED_MODEL_ARCHIVE_LIST = [
42
+ "THUDM/chatglm-6b",
43
+ # See all ChatGLM-6B models at https://huggingface.co/models?filter=chatglm
44
+ ]
45
+
46
+
47
+ def default_init(cls, *args, **kwargs):
48
+ return cls(*args, **kwargs)
49
+
50
+
51
+ class InvalidScoreLogitsProcessor(LogitsProcessor):
52
+ def __call__(self, input_ids: torch.LongTensor, scores: torch.FloatTensor) -> torch.FloatTensor:
53
+ if torch.isnan(scores).any() or torch.isinf(scores).any():
54
+ scores.zero_()
55
+ scores[..., 5] = 5e4
56
+ return scores
57
+
58
+
59
+ def split_tensor_along_last_dim(
60
+ tensor: torch.Tensor,
61
+ num_partitions: int,
62
+ contiguous_split_chunks: bool = False,
63
+ ) -> List[torch.Tensor]:
64
+ """Split a tensor along its last dimension.
65
+
66
+ Arguments:
67
+ tensor: input tensor.
68
+ num_partitions: number of partitions to split the tensor
69
+ contiguous_split_chunks: If True, make each chunk contiguous
70
+ in memory.
71
+
72
+ Returns:
73
+ A list of Tensors
74
+ """
75
+ # Get the size and dimension.
76
+ last_dim = tensor.dim() - 1
77
+ last_dim_size = tensor.size()[last_dim] // num_partitions
78
+ # Split.
79
+ tensor_list = torch.split(tensor, last_dim_size, dim=last_dim)
80
+ # Note: torch.split does not create contiguous tensors by default.
81
+ if contiguous_split_chunks:
82
+ return tuple(chunk.contiguous() for chunk in tensor_list)
83
+
84
+ return tensor_list
85
+
86
+
87
+ class RotaryEmbedding(nn.Module):
88
+ def __init__(self, dim, original_impl=False, device=None, dtype=None):
89
+ super().__init__()
90
+ inv_freq = 1.0 / (10000 ** (torch.arange(0, dim, 2, device=device, dtype=dtype) / dim))
91
+ self.register_buffer("inv_freq", inv_freq)
92
+ self.dim = dim
93
+ self.original_impl = original_impl
94
+
95
+ def forward_original_impl(
96
+ self, seq_len: int, n_elem: int, dtype: torch.dtype, device: torch.device, base: int = 10000
97
+ ):
98
+ """Enhanced Transformer with Rotary Position Embedding.
99
+
100
+ Derived from: https://github.com/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/
101
+ transformers/rope/__init__.py. MIT License:
102
+ https://github.com/labmlai/annotated_deep_learning_paper_implementations/blob/master/license.
103
+ """
104
+ # $\Theta = {\theta_i = 10000^{\frac{2(i-1)}{d}}, i \in [1, 2, ..., \frac{d}{2}]}$
105
+ theta = 1.0 / (base ** (torch.arange(0, n_elem, 2, dtype=dtype, device=device) / n_elem))
106
+
107
+ # Create position indexes `[0, 1, ..., seq_len - 1]`
108
+ seq_idx = torch.arange(seq_len, dtype=dtype, device=device)
109
+
110
+ # Calculate the product of position index and $\theta_i$
111
+ idx_theta = torch.outer(seq_idx, theta).float()
112
+
113
+ cache = torch.stack([torch.cos(idx_theta), torch.sin(idx_theta)], dim=-1)
114
+
115
+ # this is to mimic the behaviour of complex32, else we will get different results
116
+ if dtype in (torch.float16, torch.bfloat16, torch.int8):
117
+ cache = cache.bfloat16() if dtype == torch.bfloat16 else cache.half()
118
+ return cache
119
+
120
+ def forward(self, max_seq_len, offset=0):
121
+ if self.original_impl:
122
+ return self.forward_original_impl(
123
+ max_seq_len, self.dim, dtype=self.inv_freq.dtype, device=self.inv_freq.device
124
+ )
125
+
126
+
127
+ @torch.jit.script
128
+ def apply_rotary_pos_emb_original(x: torch.Tensor, rope_cache: torch.Tensor) -> torch.Tensor:
129
+ # x: [sq, b, np, hn]
130
+ sq, b, np, hn = x.size(0), x.size(1), x.size(2), x.size(3)
131
+ rot_dim = rope_cache.shape[-2] * 2
132
+ x, x_pass = x[..., :rot_dim], x[..., rot_dim:]
133
+ # truncate to support variable sizes
134
+ rope_cache = rope_cache[:sq]
135
+ xshaped = x.reshape(sq, -1, np, rot_dim // 2, 2)
136
+ rope_cache = rope_cache.view(sq, -1, 1, xshaped.size(3), 2)
137
+ x_out2 = torch.stack(
138
+ [
139
+ xshaped[..., 0] * rope_cache[..., 0] - xshaped[..., 1] * rope_cache[..., 1],
140
+ xshaped[..., 1] * rope_cache[..., 0] + xshaped[..., 0] * rope_cache[..., 1],
141
+ ],
142
+ -1,
143
+ )
144
+ x_out2 = x_out2.flatten(3)
145
+ return torch.cat((x_out2, x_pass), dim=-1)
146
+
147
+
148
+ class RMSNorm(torch.nn.Module):
149
+ def __init__(self, normalized_shape, eps=1e-5, device=None, dtype=None, **kwargs):
150
+ super().__init__()
151
+ self.weight = torch.nn.Parameter(torch.empty(normalized_shape, device=device, dtype=dtype))
152
+ self.eps = eps
153
+
154
+ def forward(self, input: torch.Tensor):
155
+ norm_x = torch.mean(input * input, dim=-1, keepdim=True)
156
+ x_normed = input * torch.rsqrt(norm_x + self.eps)
157
+ return self.weight * x_normed
158
+
159
+
160
+ class CoreAttention(torch.nn.Module):
161
+ def __init__(self, config: ChatGLMConfig, layer_number):
162
+ super(CoreAttention, self).__init__()
163
+
164
+ self.apply_query_key_layer_scaling = config.apply_query_key_layer_scaling
165
+ self.attention_softmax_in_fp32 = config.attention_softmax_in_fp32
166
+ if self.apply_query_key_layer_scaling:
167
+ self.attention_softmax_in_fp32 = True
168
+ self.layer_number = max(1, layer_number)
169
+
170
+ projection_size = config.kv_channels * config.num_attention_heads
171
+
172
+ # Per attention head and per partition values.
173
+ self.hidden_size_per_partition = projection_size
174
+ self.hidden_size_per_attention_head = projection_size // config.num_attention_heads
175
+ self.num_attention_heads_per_partition = config.num_attention_heads
176
+
177
+ coeff = None
178
+ self.norm_factor = math.sqrt(self.hidden_size_per_attention_head)
179
+ if self.apply_query_key_layer_scaling:
180
+ coeff = self.layer_number
181
+ self.norm_factor *= coeff
182
+ self.coeff = coeff
183
+
184
+ self.attention_dropout = torch.nn.Dropout(config.attention_dropout)
185
+
186
+ def forward(self, query_layer, key_layer, value_layer, attention_mask):
187
+ pytorch_major_version = int(torch.__version__.split('.')[0])
188
+ if pytorch_major_version >= 2:
189
+ query_layer, key_layer, value_layer = [k.permute(1, 2, 0, 3) for k in [query_layer, key_layer, value_layer]]
190
+ if attention_mask is None and query_layer.shape[2] == key_layer.shape[2]:
191
+ context_layer = torch.nn.functional.scaled_dot_product_attention(query_layer, key_layer, value_layer,
192
+ is_causal=True)
193
+ else:
194
+ if attention_mask is not None:
195
+ attention_mask = ~attention_mask
196
+ context_layer = torch.nn.functional.scaled_dot_product_attention(query_layer, key_layer, value_layer,
197
+ attention_mask)
198
+ context_layer = context_layer.permute(2, 0, 1, 3)
199
+ new_context_layer_shape = context_layer.size()[:-2] + (self.hidden_size_per_partition,)
200
+ context_layer = context_layer.reshape(*new_context_layer_shape)
201
+ else:
202
+ # Raw attention scores
203
+
204
+ # [b, np, sq, sk]
205
+ output_size = (query_layer.size(1), query_layer.size(2), query_layer.size(0), key_layer.size(0))
206
+
207
+ # [sq, b, np, hn] -> [sq, b * np, hn]
208
+ query_layer = query_layer.view(output_size[2], output_size[0] * output_size[1], -1)
209
+ # [sk, b, np, hn] -> [sk, b * np, hn]
210
+ key_layer = key_layer.view(output_size[3], output_size[0] * output_size[1], -1)
211
+
212
+ # preallocting input tensor: [b * np, sq, sk]
213
+ matmul_input_buffer = torch.empty(
214
+ output_size[0] * output_size[1], output_size[2], output_size[3], dtype=query_layer.dtype,
215
+ device=query_layer.device
216
+ )
217
+
218
+ # Raw attention scores. [b * np, sq, sk]
219
+ matmul_result = torch.baddbmm(
220
+ matmul_input_buffer,
221
+ query_layer.transpose(0, 1), # [b * np, sq, hn]
222
+ key_layer.transpose(0, 1).transpose(1, 2), # [b * np, hn, sk]
223
+ beta=0.0,
224
+ alpha=(1.0 / self.norm_factor),
225
+ )
226
+
227
+ # change view to [b, np, sq, sk]
228
+ attention_scores = matmul_result.view(*output_size)
229
+
230
+ # ===========================
231
+ # Attention probs and dropout
232
+ # ===========================
233
+
234
+ # attention scores and attention mask [b, np, sq, sk]
235
+ if self.attention_softmax_in_fp32:
236
+ attention_scores = attention_scores.float()
237
+ if self.coeff is not None:
238
+ attention_scores = attention_scores * self.coeff
239
+ if attention_mask is None and attention_scores.shape[2] == attention_scores.shape[3]:
240
+ attention_mask = torch.ones(output_size[0], 1, output_size[2], output_size[3],
241
+ device=attention_scores.device, dtype=torch.bool)
242
+ attention_mask.tril_()
243
+ attention_mask = ~attention_mask
244
+ if attention_mask is not None:
245
+ attention_scores = attention_scores.masked_fill(attention_mask, float("-inf"))
246
+ attention_probs = F.softmax(attention_scores, dim=-1)
247
+ attention_probs = attention_probs.type_as(value_layer)
248
+
249
+ # This is actually dropping out entire tokens to attend to, which might
250
+ # seem a bit unusual, but is taken from the original Transformer paper.
251
+ attention_probs = self.attention_dropout(attention_probs)
252
+ # =========================
253
+ # Context layer. [sq, b, hp]
254
+ # =========================
255
+
256
+ # value_layer -> context layer.
257
+ # [sk, b, np, hn] --> [b, np, sq, hn]
258
+
259
+ # context layer shape: [b, np, sq, hn]
260
+ output_size = (value_layer.size(1), value_layer.size(2), query_layer.size(0), value_layer.size(3))
261
+ # change view [sk, b * np, hn]
262
+ value_layer = value_layer.view(value_layer.size(0), output_size[0] * output_size[1], -1)
263
+ # change view [b * np, sq, sk]
264
+ attention_probs = attention_probs.view(output_size[0] * output_size[1], output_size[2], -1)
265
+ # matmul: [b * np, sq, hn]
266
+ context_layer = torch.bmm(attention_probs, value_layer.transpose(0, 1))
267
+ # change view [b, np, sq, hn]
268
+ context_layer = context_layer.view(*output_size)
269
+ # [b, np, sq, hn] --> [sq, b, np, hn]
270
+ context_layer = context_layer.permute(2, 0, 1, 3).contiguous()
271
+ # [sq, b, np, hn] --> [sq, b, hp]
272
+ new_context_layer_shape = context_layer.size()[:-2] + (self.hidden_size_per_partition,)
273
+ context_layer = context_layer.view(*new_context_layer_shape)
274
+
275
+ return context_layer
276
+
277
+
278
+ class SelfAttention(torch.nn.Module):
279
+ """Parallel self-attention layer abstract class.
280
+
281
+ Self-attention layer takes input with size [s, b, h]
282
+ and returns output of the same size.
283
+ """
284
+
285
+ def __init__(self, config: ChatGLMConfig, layer_number, device=None):
286
+ super(SelfAttention, self).__init__()
287
+ self.layer_number = max(1, layer_number)
288
+
289
+ self.projection_size = config.kv_channels * config.num_attention_heads
290
+
291
+ # Per attention head and per partition values.
292
+ self.hidden_size_per_attention_head = self.projection_size // config.num_attention_heads
293
+ self.num_attention_heads_per_partition = config.num_attention_heads
294
+
295
+ self.multi_query_attention = config.multi_query_attention
296
+ self.qkv_hidden_size = 3 * self.projection_size
297
+ if self.multi_query_attention:
298
+ self.num_multi_query_groups_per_partition = config.multi_query_group_num
299
+ self.qkv_hidden_size = (
300
+ self.projection_size + 2 * self.hidden_size_per_attention_head * config.multi_query_group_num
301
+ )
302
+ self.query_key_value = nn.Linear(config.hidden_size, self.qkv_hidden_size,
303
+ bias=config.add_bias_linear or config.add_qkv_bias,
304
+ device=device, **_config_to_kwargs(config)
305
+ )
306
+
307
+ self.core_attention = CoreAttention(config, self.layer_number)
308
+
309
+ # Output.
310
+ self.dense = nn.Linear(self.projection_size, config.hidden_size, bias=config.add_bias_linear,
311
+ device=device, **_config_to_kwargs(config)
312
+ )
313
+
314
+ self.interleaved_qkv = config.interleaved_qkv
315
+
316
+ def _allocate_memory(self, inference_max_sequence_len, batch_size, device=None, dtype=None):
317
+ if self.multi_query_attention:
318
+ num_attention_heads = self.num_multi_query_groups_per_partition
319
+ else:
320
+ num_attention_heads = self.num_attention_heads_per_partition
321
+ return torch.empty(
322
+ inference_max_sequence_len,
323
+ batch_size,
324
+ num_attention_heads,
325
+ self.hidden_size_per_attention_head,
326
+ dtype=dtype,
327
+ device=device,
328
+ )
329
+
330
+ def forward(
331
+ self, hidden_states, attention_mask, rotary_pos_emb, kv_cache=None, use_cache=True
332
+ ):
333
+ # hidden_states: [sq, b, h]
334
+
335
+ # =================================================
336
+ # Pre-allocate memory for key-values for inference.
337
+ # =================================================
338
+ # =====================
339
+ # Query, Key, and Value
340
+ # =====================
341
+
342
+ # Attention heads [sq, b, h] --> [sq, b, (np * 3 * hn)]
343
+ mixed_x_layer = self.query_key_value(hidden_states)
344
+
345
+ if self.multi_query_attention:
346
+ (query_layer, key_layer, value_layer) = mixed_x_layer.split(
347
+ [
348
+ self.num_attention_heads_per_partition * self.hidden_size_per_attention_head,
349
+ self.num_multi_query_groups_per_partition * self.hidden_size_per_attention_head,
350
+ self.num_multi_query_groups_per_partition * self.hidden_size_per_attention_head,
351
+ ],
352
+ dim=-1,
353
+ )
354
+ query_layer = query_layer.view(
355
+ query_layer.size()[:-1] + (self.num_attention_heads_per_partition, self.hidden_size_per_attention_head)
356
+ )
357
+ key_layer = key_layer.view(
358
+ key_layer.size()[:-1] + (self.num_multi_query_groups_per_partition, self.hidden_size_per_attention_head)
359
+ )
360
+ value_layer = value_layer.view(
361
+ value_layer.size()[:-1]
362
+ + (self.num_multi_query_groups_per_partition, self.hidden_size_per_attention_head)
363
+ )
364
+ else:
365
+ if self.interleaved_qkv:
366
+ new_tensor_shape = mixed_x_layer.size()[:-1] + \
367
+ (self.num_attention_heads_per_partition,
368
+ 3 * self.hidden_size_per_attention_head)
369
+ mixed_x_layer = mixed_x_layer.view(*new_tensor_shape)
370
+
371
+ # [sq, b, np, 3 * hn] --> 3 [sq, b, np, hn]
372
+ (query_layer, key_layer, value_layer) = split_tensor_along_last_dim(mixed_x_layer, 3)
373
+
374
+ if not self.interleaved_qkv:
375
+ query_layer = query_layer.view(
376
+ query_layer.size()[:-1] + (
377
+ self.num_attention_heads_per_partition, self.hidden_size_per_attention_head)
378
+ ).contiguous()
379
+ key_layer = key_layer.view(
380
+ key_layer.size()[:-1] + (
381
+ self.num_attention_heads_per_partition, self.hidden_size_per_attention_head)
382
+ ).contiguous()
383
+ value_layer = value_layer.view(
384
+ value_layer.size()[:-1] + (
385
+ self.num_attention_heads_per_partition, self.hidden_size_per_attention_head)
386
+ ).contiguous()
387
+
388
+ # apply relative positional encoding (rotary embedding)
389
+ if rotary_pos_emb is not None:
390
+ query_layer = apply_rotary_pos_emb_original(query_layer, rotary_pos_emb)
391
+ key_layer = apply_rotary_pos_emb_original(key_layer, rotary_pos_emb)
392
+
393
+ # adjust key and value for inference
394
+ if use_cache:
395
+ if kv_cache is not None:
396
+ cache_k, cache_v = kv_cache
397
+ key_layer = torch.cat((cache_k, key_layer), dim=0)
398
+ value_layer = torch.cat((cache_v, value_layer), dim=0)
399
+ kv_cache = (key_layer, value_layer)
400
+ else:
401
+ kv_cache = None
402
+
403
+ if self.multi_query_attention:
404
+ key_layer = key_layer.unsqueeze(-2)
405
+ key_layer = key_layer.expand(
406
+ -1, -1, -1, self.num_attention_heads_per_partition // self.num_multi_query_groups_per_partition, -1
407
+ )
408
+ key_layer = key_layer.contiguous().view(
409
+ key_layer.size()[:2] + (self.num_attention_heads_per_partition, self.hidden_size_per_attention_head)
410
+ )
411
+ value_layer = value_layer.unsqueeze(-2)
412
+ value_layer = value_layer.expand(
413
+ -1, -1, -1, self.num_attention_heads_per_partition // self.num_multi_query_groups_per_partition, -1
414
+ )
415
+ value_layer = value_layer.contiguous().view(
416
+ value_layer.size()[:2] + (self.num_attention_heads_per_partition, self.hidden_size_per_attention_head)
417
+ )
418
+
419
+ # ==================================
420
+ # core attention computation
421
+ # ==================================
422
+
423
+ context_layer = self.core_attention(query_layer, key_layer, value_layer, attention_mask)
424
+
425
+ # =================
426
+ # Output. [sq, b, h]
427
+ # =================
428
+
429
+ output = self.dense(context_layer)
430
+
431
+ return output, kv_cache
432
+
433
+
434
+ def _config_to_kwargs(args):
435
+ common_kwargs = {
436
+ "dtype": args.torch_dtype,
437
+ }
438
+ return common_kwargs
439
+
440
+
441
+ class MLP(torch.nn.Module):
442
+ """MLP.
443
+
444
+ MLP will take the input with h hidden state, project it to 4*h
445
+ hidden dimension, perform nonlinear transformation, and project the
446
+ state back into h hidden dimension.
447
+ """
448
+
449
+ def __init__(self, config: ChatGLMConfig, device=None):
450
+ super(MLP, self).__init__()
451
+
452
+ self.add_bias = config.add_bias_linear
453
+
454
+ # Project to 4h. If using swiglu double the output width, see https://arxiv.org/pdf/2002.05202.pdf
455
+ self.dense_h_to_4h = nn.Linear(
456
+ config.hidden_size,
457
+ config.ffn_hidden_size * 2,
458
+ bias=self.add_bias,
459
+ device=device,
460
+ **_config_to_kwargs(config)
461
+ )
462
+
463
+ def swiglu(x):
464
+ x = torch.chunk(x, 2, dim=-1)
465
+ return F.silu(x[0]) * x[1]
466
+
467
+ self.activation_func = swiglu
468
+
469
+ # Project back to h.
470
+ self.dense_4h_to_h = nn.Linear(
471
+ config.ffn_hidden_size,
472
+ config.hidden_size,
473
+ bias=self.add_bias,
474
+ device=device,
475
+ **_config_to_kwargs(config)
476
+ )
477
+
478
+ def forward(self, hidden_states):
479
+ # [s, b, 4hp]
480
+ intermediate_parallel = self.dense_h_to_4h(hidden_states)
481
+ intermediate_parallel = self.activation_func(intermediate_parallel)
482
+ # [s, b, h]
483
+ output = self.dense_4h_to_h(intermediate_parallel)
484
+ return output
485
+
486
+
487
+ class GLMBlock(torch.nn.Module):
488
+ """A single transformer layer.
489
+
490
+ Transformer layer takes input with size [s, b, h] and returns an
491
+ output of the same size.
492
+ """
493
+
494
+ def __init__(self, config: ChatGLMConfig, layer_number, device=None):
495
+ super(GLMBlock, self).__init__()
496
+ self.layer_number = layer_number
497
+
498
+ self.apply_residual_connection_post_layernorm = config.apply_residual_connection_post_layernorm
499
+
500
+ self.fp32_residual_connection = config.fp32_residual_connection
501
+
502
+ LayerNormFunc = RMSNorm if config.rmsnorm else LayerNorm
503
+ # Layernorm on the input data.
504
+ self.input_layernorm = LayerNormFunc(config.hidden_size, eps=config.layernorm_epsilon, device=device,
505
+ dtype=config.torch_dtype)
506
+
507
+ # Self attention.
508
+ self.self_attention = SelfAttention(config, layer_number, device=device)
509
+ self.hidden_dropout = config.hidden_dropout
510
+
511
+ # Layernorm on the attention output
512
+ self.post_attention_layernorm = LayerNormFunc(config.hidden_size, eps=config.layernorm_epsilon, device=device,
513
+ dtype=config.torch_dtype)
514
+
515
+ # MLP
516
+ self.mlp = MLP(config, device=device)
517
+
518
+ def forward(
519
+ self, hidden_states, attention_mask, rotary_pos_emb, kv_cache=None, use_cache=True,
520
+ ):
521
+ # hidden_states: [s, b, h]
522
+
523
+ # Layer norm at the beginning of the transformer layer.
524
+ layernorm_output = self.input_layernorm(hidden_states)
525
+ # Self attention.
526
+ attention_output, kv_cache = self.self_attention(
527
+ layernorm_output,
528
+ attention_mask,
529
+ rotary_pos_emb,
530
+ kv_cache=kv_cache,
531
+ use_cache=use_cache
532
+ )
533
+
534
+ # Residual connection.
535
+ if self.apply_residual_connection_post_layernorm:
536
+ residual = layernorm_output
537
+ else:
538
+ residual = hidden_states
539
+
540
+ layernorm_input = torch.nn.functional.dropout(attention_output, p=self.hidden_dropout, training=self.training)
541
+ layernorm_input = residual + layernorm_input
542
+
543
+ # Layer norm post the self attention.
544
+ layernorm_output = self.post_attention_layernorm(layernorm_input)
545
+
546
+ # MLP.
547
+ mlp_output = self.mlp(layernorm_output)
548
+
549
+ # Second residual connection.
550
+ if self.apply_residual_connection_post_layernorm:
551
+ residual = layernorm_output
552
+ else:
553
+ residual = layernorm_input
554
+
555
+ output = torch.nn.functional.dropout(mlp_output, p=self.hidden_dropout, training=self.training)
556
+ output = residual + output
557
+
558
+ return output, kv_cache
559
+
560
+
561
+ class GLMTransformer(torch.nn.Module):
562
+ """Transformer class."""
563
+
564
+ def __init__(self, config: ChatGLMConfig, device=None):
565
+ super(GLMTransformer, self).__init__()
566
+
567
+ self.fp32_residual_connection = config.fp32_residual_connection
568
+ self.post_layer_norm = config.post_layer_norm
569
+
570
+ # Number of layers.
571
+ self.num_layers = config.num_layers
572
+
573
+ # Transformer layers.
574
+ def build_layer(layer_number):
575
+ return GLMBlock(config, layer_number, device=device)
576
+
577
+ self.layers = torch.nn.ModuleList([build_layer(i + 1) for i in range(self.num_layers)])
578
+
579
+ if self.post_layer_norm:
580
+ LayerNormFunc = RMSNorm if config.rmsnorm else LayerNorm
581
+ # Final layer norm before output.
582
+ self.final_layernorm = LayerNormFunc(config.hidden_size, eps=config.layernorm_epsilon, device=device,
583
+ dtype=config.torch_dtype)
584
+
585
+ def _get_layer(self, layer_number):
586
+ return self.layers[layer_number]
587
+
588
+ def forward(
589
+ self, hidden_states, attention_mask, rotary_pos_emb, kv_caches=None,
590
+ use_cache: Optional[bool] = True,
591
+ output_hidden_states: Optional[bool] = False,
592
+ ):
593
+ if not kv_caches:
594
+ kv_caches = [None for _ in range(self.num_layers)]
595
+ presents = () if use_cache else None
596
+ all_self_attentions = None
597
+ all_hidden_states = () if output_hidden_states else None
598
+ for index in range(self.num_layers):
599
+ if output_hidden_states:
600
+ all_hidden_states = all_hidden_states + (hidden_states,)
601
+
602
+ layer = self._get_layer(index)
603
+
604
+ hidden_states, kv_cache = layer(
605
+ hidden_states,
606
+ attention_mask,
607
+ rotary_pos_emb,
608
+ kv_cache=kv_caches[index],
609
+ use_cache=use_cache
610
+ )
611
+ if use_cache:
612
+ presents = presents + (kv_cache,)
613
+
614
+ if output_hidden_states:
615
+ all_hidden_states = all_hidden_states + (hidden_states,)
616
+
617
+ # Final layer norm.
618
+ if self.post_layer_norm:
619
+ hidden_states = self.final_layernorm(hidden_states)
620
+
621
+ return hidden_states, presents, all_hidden_states, all_self_attentions
622
+
623
+
624
+ class ChatGLMPreTrainedModel(PreTrainedModel):
625
+ """
626
+ An abstract class to handle weights initialization and
627
+ a simple interface for downloading and loading pretrained models.
628
+ """
629
+
630
+ is_parallelizable = False
631
+ supports_gradient_checkpointing = True
632
+ config_class = ChatGLMConfig
633
+ base_model_prefix = "transformer"
634
+ _no_split_modules = ["GLMBlock"]
635
+
636
+ def _init_weights(self, module: nn.Module):
637
+ """Initialize the weights."""
638
+ return
639
+
640
+ def get_masks(self, input_ids, past_key_values, padding_mask=None):
641
+ batch_size, seq_length = input_ids.shape
642
+ full_attention_mask = torch.ones(batch_size, seq_length, seq_length, device=input_ids.device)
643
+ full_attention_mask.tril_()
644
+ past_length = 0
645
+ if past_key_values:
646
+ past_length = past_key_values[0][0].shape[0]
647
+ if past_length:
648
+ full_attention_mask = torch.cat((torch.ones(batch_size, seq_length, past_length,
649
+ device=input_ids.device), full_attention_mask), dim=-1)
650
+ if padding_mask is not None:
651
+ full_attention_mask = full_attention_mask * padding_mask.unsqueeze(1)
652
+ if not past_length and padding_mask is not None:
653
+ full_attention_mask -= padding_mask.unsqueeze(-1) - 1
654
+ full_attention_mask = (full_attention_mask < 0.5).bool()
655
+ full_attention_mask.unsqueeze_(1)
656
+ return full_attention_mask
657
+
658
+ def get_position_ids(self, input_ids, device):
659
+ batch_size, seq_length = input_ids.shape
660
+ position_ids = torch.arange(seq_length, dtype=torch.long, device=device).unsqueeze(0).repeat(batch_size, 1)
661
+ return position_ids
662
+
663
+ def _set_gradient_checkpointing(self, module, value=False):
664
+ if isinstance(module, ChatGLMModel):
665
+ module.gradient_checkpointing = value
666
+
667
+
668
+ class Embedding(torch.nn.Module):
669
+ """Language model embeddings."""
670
+
671
+ def __init__(self, config: ChatGLMConfig, device=None):
672
+ super(Embedding, self).__init__()
673
+
674
+ self.hidden_size = config.hidden_size
675
+ # Word embeddings (parallel).
676
+ self.word_embeddings = nn.Embedding(
677
+ config.padded_vocab_size,
678
+ self.hidden_size,
679
+ dtype=config.torch_dtype,
680
+ device=device
681
+ )
682
+ self.fp32_residual_connection = config.fp32_residual_connection
683
+
684
+ def forward(self, input_ids):
685
+ # Embeddings.
686
+ words_embeddings = self.word_embeddings(input_ids)
687
+ embeddings = words_embeddings
688
+ # Data format change to avoid explicit tranposes : [b s h] --> [s b h].
689
+ embeddings = embeddings.transpose(0, 1).contiguous()
690
+ # If the input flag for fp32 residual connection is set, convert for float.
691
+ if self.fp32_residual_connection:
692
+ embeddings = embeddings.float()
693
+ return embeddings
694
+
695
+
696
+ class ChatGLMModel(ChatGLMPreTrainedModel):
697
+ def __init__(self, config: ChatGLMConfig, device=None, empty_init=True):
698
+ super().__init__(config)
699
+ if empty_init:
700
+ init_method = skip_init
701
+ else:
702
+ init_method = default_init
703
+ init_kwargs = {}
704
+ if device is not None:
705
+ init_kwargs["device"] = device
706
+ self.embedding = init_method(Embedding, config, **init_kwargs)
707
+
708
+ # Rotary positional embeddings
709
+ self.seq_length = config.seq_length
710
+ rotary_dim = (
711
+ config.hidden_size // config.num_attention_heads if config.kv_channels is None else config.kv_channels
712
+ )
713
+
714
+ if config.rotary_percent < 1.0:
715
+ rotary_dim = int(rotary_dim * config.rotary_percent)
716
+
717
+ # partial rotary embeddings, which is better than full rotary
718
+ # Wang and Komatsuzaki et al
719
+ # https://github.com/kingoflolz/mesh-transformer-jax/
720
+ self.rotary_pos_emb = RotaryEmbedding(rotary_dim, original_impl=config.original_rope, device=device,
721
+ dtype=config.torch_dtype)
722
+ self.encoder = init_method(GLMTransformer, config, **init_kwargs)
723
+ self.output_layer = init_method(nn.Linear, config.hidden_size, config.padded_vocab_size, bias=False,
724
+ dtype=config.torch_dtype, **init_kwargs)
725
+ self.gradient_checkpointing = False
726
+
727
+ def forward(
728
+ self,
729
+ input_ids,
730
+ position_ids: Optional[torch.Tensor] = None,
731
+ attention_mask: Optional[torch.BoolTensor] = None,
732
+ full_attention_mask: Optional[torch.BoolTensor] = None,
733
+ past_key_values: Optional[Tuple[Tuple[torch.Tensor, torch.Tensor], ...]] = None,
734
+ inputs_embeds: Optional[torch.Tensor] = None,
735
+ use_cache: Optional[bool] = None,
736
+ output_hidden_states: Optional[bool] = None,
737
+ return_dict: Optional[bool] = None,
738
+ ):
739
+ output_hidden_states = (
740
+ output_hidden_states if output_hidden_states is not None else self.config.output_hidden_states
741
+ )
742
+ use_cache = use_cache if use_cache is not None else self.config.use_cache
743
+ return_dict = return_dict if return_dict is not None else self.config.use_return_dict
744
+
745
+ batch_size, seq_length = input_ids.shape
746
+
747
+ if inputs_embeds is None:
748
+ inputs_embeds = self.embedding(input_ids)
749
+
750
+ if full_attention_mask is None and attention_mask is not None and not attention_mask.all():
751
+ full_attention_mask = self.get_masks(input_ids, past_key_values, padding_mask=attention_mask)
752
+
753
+ # Rotary positional embeddings
754
+ rotary_pos_emb = self.rotary_pos_emb(self.seq_length)
755
+ if position_ids is not None:
756
+ rotary_pos_emb = rotary_pos_emb[position_ids]
757
+ else:
758
+ rotary_pos_emb = rotary_pos_emb[None, :seq_length]
759
+ rotary_pos_emb = rotary_pos_emb.transpose(0, 1).contiguous()
760
+
761
+ # Run encoder.
762
+ hidden_states, presents, all_hidden_states, all_self_attentions = self.encoder(
763
+ inputs_embeds, full_attention_mask, rotary_pos_emb=rotary_pos_emb,
764
+ kv_caches=past_key_values, use_cache=use_cache, output_hidden_states=output_hidden_states
765
+ )
766
+
767
+ if not return_dict:
768
+ return tuple(v for v in [hidden_states, presents, all_hidden_states, all_self_attentions] if v is not None)
769
+
770
+ return BaseModelOutputWithPast(
771
+ last_hidden_state=hidden_states,
772
+ past_key_values=presents,
773
+ hidden_states=all_hidden_states,
774
+ attentions=all_self_attentions,
775
+ )
776
+
777
+ def quantize(self, weight_bit_width: int):
778
+ from .quantization import quantize
779
+ quantize(self.encoder, weight_bit_width)
780
+ return self
781
+
782
+
783
+ class ChatGLMForConditionalGeneration(ChatGLMPreTrainedModel):
784
+ def __init__(self, config: ChatGLMConfig, empty_init=True, device=None):
785
+ super().__init__(config)
786
+
787
+ self.max_sequence_length = config.max_length
788
+ self.transformer = ChatGLMModel(config, empty_init=empty_init, device=device)
789
+ self.config = config
790
+ self.quantized = False
791
+
792
+ if self.config.quantization_bit:
793
+ self.quantize(self.config.quantization_bit, empty_init=True)
794
+
795
+ def _update_model_kwargs_for_generation(
796
+ self,
797
+ outputs: ModelOutput,
798
+ model_kwargs: Dict[str, Any],
799
+ is_encoder_decoder: bool = False,
800
+ standardize_cache_format: bool = False,
801
+ ) -> Dict[str, Any]:
802
+ # update past_key_values
803
+ model_kwargs["past_key_values"] = self._extract_past_from_model_output(
804
+ outputs, standardize_cache_format=standardize_cache_format
805
+ )
806
+
807
+ # update attention mask
808
+ if "attention_mask" in model_kwargs:
809
+ attention_mask = model_kwargs["attention_mask"]
810
+ model_kwargs["attention_mask"] = torch.cat(
811
+ [attention_mask, attention_mask.new_ones((attention_mask.shape[0], 1))], dim=-1
812
+ )
813
+
814
+ # update position ids
815
+ if "position_ids" in model_kwargs:
816
+ position_ids = model_kwargs["position_ids"]
817
+ new_position_id = position_ids[..., -1:].clone()
818
+ new_position_id += 1
819
+ model_kwargs["position_ids"] = torch.cat(
820
+ [position_ids, new_position_id], dim=-1
821
+ )
822
+
823
+ return model_kwargs
824
+
825
+ def prepare_inputs_for_generation(
826
+ self,
827
+ input_ids: torch.LongTensor,
828
+ past_key_values: Optional[torch.Tensor] = None,
829
+ attention_mask: Optional[torch.Tensor] = None,
830
+ position_ids: Optional[torch.Tensor] = None,
831
+ input_pos: int = None,
832
+ **kwargs
833
+ ) -> dict:
834
+ # only last token for input_ids if past is not None
835
+ if past_key_values is not None:
836
+ if position_ids is None:
837
+ position_ids = self.get_position_ids(input_ids, device=input_ids.device)
838
+ position_ids = position_ids[..., -1:]
839
+ input_ids = input_ids[:, -1:]
840
+ return {
841
+ "input_ids": input_ids,
842
+ "past_key_values": past_key_values,
843
+ "position_ids": position_ids,
844
+ "attention_mask": attention_mask
845
+ }
846
+
847
+ def forward(
848
+ self,
849
+ input_ids: Optional[torch.Tensor] = None,
850
+ position_ids: Optional[torch.Tensor] = None,
851
+ attention_mask: Optional[torch.Tensor] = None,
852
+ past_key_values: Optional[Tuple[torch.FloatTensor]] = None,
853
+ inputs_embeds: Optional[torch.Tensor] = None,
854
+ labels: Optional[torch.Tensor] = None,
855
+ use_cache: Optional[bool] = None,
856
+ output_attentions: Optional[bool] = None,
857
+ output_hidden_states: Optional[bool] = None,
858
+ return_dict: Optional[bool] = None,
859
+ ):
860
+ use_cache = use_cache if use_cache is not None else self.config.use_cache
861
+ return_dict = return_dict if return_dict is not None else self.config.use_return_dict
862
+
863
+ transformer_outputs = self.transformer(
864
+ input_ids=input_ids,
865
+ position_ids=position_ids,
866
+ attention_mask=attention_mask,
867
+ past_key_values=past_key_values,
868
+ inputs_embeds=inputs_embeds,
869
+ use_cache=use_cache,
870
+ output_hidden_states=output_hidden_states,
871
+ return_dict=return_dict,
872
+ )
873
+
874
+ hidden_states = transformer_outputs[0]
875
+
876
+ lm_logits = self.transformer.output_layer(hidden_states)
877
+ lm_logits = lm_logits.transpose(0, 1).contiguous()
878
+
879
+ loss = None
880
+ if labels is not None:
881
+ lm_logits = lm_logits.to(torch.float32)
882
+
883
+ # Shift so that tokens < n predict n
884
+ shift_logits = lm_logits[..., :-1, :].contiguous()
885
+ shift_labels = labels[..., 1:].contiguous()
886
+ # Flatten the tokens
887
+ loss_fct = CrossEntropyLoss(ignore_index=-100)
888
+ loss = loss_fct(shift_logits.view(-1, shift_logits.size(-1)), shift_labels.view(-1))
889
+
890
+ lm_logits = lm_logits.to(hidden_states.dtype)
891
+ loss = loss.to(hidden_states.dtype)
892
+
893
+ if not return_dict:
894
+ output = (lm_logits,) + transformer_outputs[1:]
895
+ return ((loss,) + output) if loss is not None else output
896
+
897
+ return CausalLMOutputWithPast(
898
+ loss=loss,
899
+ logits=lm_logits,
900
+ past_key_values=transformer_outputs.past_key_values,
901
+ hidden_states=transformer_outputs.hidden_states,
902
+ attentions=transformer_outputs.attentions,
903
+ )
904
+
905
+ @staticmethod
906
+ def _reorder_cache(
907
+ past: Tuple[Tuple[torch.Tensor, torch.Tensor], ...], beam_idx: torch.LongTensor
908
+ ) -> Tuple[Tuple[torch.Tensor, torch.Tensor], ...]:
909
+ """
910
+ This function is used to re-order the `past_key_values` cache if [`~PreTrainedModel.beam_search`] or
911
+ [`~PreTrainedModel.beam_sample`] is called. This is required to match `past_key_values` with the correct
912
+ beam_idx at every generation step.
913
+
914
+ Output shares the same memory storage as `past`.
915
+ """
916
+ return tuple(
917
+ (
918
+ layer_past[0].index_select(1, beam_idx.to(layer_past[0].device)),
919
+ layer_past[1].index_select(1, beam_idx.to(layer_past[1].device)),
920
+ )
921
+ for layer_past in past
922
+ )
923
+
924
+ def process_response(self, response):
925
+ response = response.strip()
926
+ response = response.replace("[[训练时间]]", "2023年")
927
+ return response
928
+
929
+ def build_inputs(self, tokenizer, query: str, history: List[Tuple[str, str]] = None):
930
+ prompt = ""
931
+ for i, (old_query, response) in enumerate(history):
932
+ prompt += "[Round {}]\n\n问:{}\n\n答:{}\n\n".format(i + 1, old_query, response)
933
+ prompt += "[Round {}]\n\n问:{}\n\n答:".format(len(history) + 1, query)
934
+ inputs = tokenizer([prompt], return_tensors="pt")
935
+ inputs = inputs.to(self.device)
936
+ return inputs
937
+
938
+ @torch.no_grad()
939
+ def chat(self, tokenizer, query: str, history: List[Tuple[str, str]] = None, max_length: int = 2048, num_beams=1,
940
+ do_sample=True, top_p=0.8, temperature=0.8, logits_processor=None, **kwargs):
941
+ if history is None:
942
+ history = []
943
+ if logits_processor is None:
944
+ logits_processor = LogitsProcessorList()
945
+ logits_processor.append(InvalidScoreLogitsProcessor())
946
+ gen_kwargs = {"max_length": max_length, "num_beams": num_beams, "do_sample": do_sample, "top_p": top_p,
947
+ "temperature": temperature, "logits_processor": logits_processor, **kwargs}
948
+ inputs = self.build_inputs(tokenizer, query, history=history)
949
+ outputs = self.generate(**inputs, **gen_kwargs)
950
+ outputs = outputs.tolist()[0][len(inputs["input_ids"][0]):]
951
+ response = tokenizer.decode(outputs)
952
+ response = self.process_response(response)
953
+ history = history + [(query, response)]
954
+ return response, history
955
+
956
+ @torch.no_grad()
957
+ def stream_chat(self, tokenizer, query: str, history: List[Tuple[str, str]] = None, max_length: int = 2048,
958
+ do_sample=True, top_p=0.7, temperature=0.95, logits_processor=None, **kwargs):
959
+ if history is None:
960
+ history = []
961
+ if logits_processor is None:
962
+ logits_processor = LogitsProcessorList()
963
+ logits_processor.append(InvalidScoreLogitsProcessor())
964
+ gen_kwargs = {"max_length": max_length, "do_sample": do_sample, "top_p": top_p,
965
+ "temperature": temperature, "logits_processor": logits_processor, **kwargs}
966
+ inputs = self.build_inputs(tokenizer, query, history=history)
967
+ for outputs in self.stream_generate(**inputs, **gen_kwargs):
968
+ outputs = outputs.tolist()[0][len(inputs["input_ids"][0]):]
969
+ response = tokenizer.decode(outputs)
970
+ response = self.process_response(response)
971
+ new_history = history + [(query, response)]
972
+ yield response, new_history
973
+
974
+ @torch.no_grad()
975
+ def stream_generate(
976
+ self,
977
+ input_ids,
978
+ generation_config: Optional[GenerationConfig] = None,
979
+ logits_processor: Optional[LogitsProcessorList] = None,
980
+ stopping_criteria: Optional[StoppingCriteriaList] = None,
981
+ prefix_allowed_tokens_fn: Optional[Callable[[int, torch.Tensor], List[int]]] = None,
982
+ **kwargs,
983
+ ):
984
+ batch_size, input_ids_seq_length = input_ids.shape[0], input_ids.shape[-1]
985
+
986
+ if generation_config is None:
987
+ generation_config = self.generation_config
988
+ generation_config = copy.deepcopy(generation_config)
989
+ model_kwargs = generation_config.update(**kwargs)
990
+ bos_token_id, eos_token_id = generation_config.bos_token_id, generation_config.eos_token_id
991
+
992
+ if isinstance(eos_token_id, int):
993
+ eos_token_id = [eos_token_id]
994
+
995
+ has_default_max_length = kwargs.get("max_length") is None and generation_config.max_length is not None
996
+ if has_default_max_length and generation_config.max_new_tokens is None:
997
+ warnings.warn(
998
+ f"Using `max_length`'s default ({generation_config.max_length}) to control the generation length. "
999
+ "This behaviour is deprecated and will be removed from the config in v5 of Transformers -- we"
1000
+ " recommend using `max_new_tokens` to control the maximum length of the generation.",
1001
+ UserWarning,
1002
+ )
1003
+ elif generation_config.max_new_tokens is not None:
1004
+ generation_config.max_length = generation_config.max_new_tokens + input_ids_seq_length
1005
+ if not has_default_max_length:
1006
+ logger.warn(
1007
+ f"Both `max_new_tokens` (={generation_config.max_new_tokens}) and `max_length`(="
1008
+ f"{generation_config.max_length}) seem to have been set. `max_new_tokens` will take precedence. "
1009
+ "Please refer to the documentation for more information. "
1010
+ "(https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)",
1011
+ UserWarning,
1012
+ )
1013
+
1014
+ if input_ids_seq_length >= generation_config.max_length:
1015
+ input_ids_string = "decoder_input_ids" if self.config.is_encoder_decoder else "input_ids"
1016
+ logger.warning(
1017
+ f"Input length of {input_ids_string} is {input_ids_seq_length}, but `max_length` is set to"
1018
+ f" {generation_config.max_length}. This can lead to unexpected behavior. You should consider"
1019
+ " increasing `max_new_tokens`."
1020
+ )
1021
+
1022
+ # 2. Set generation parameters if not already defined
1023
+ logits_processor = logits_processor if logits_processor is not None else LogitsProcessorList()
1024
+ stopping_criteria = stopping_criteria if stopping_criteria is not None else StoppingCriteriaList()
1025
+
1026
+ logits_processor = self._get_logits_processor(
1027
+ generation_config=generation_config,
1028
+ input_ids_seq_length=input_ids_seq_length,
1029
+ encoder_input_ids=input_ids,
1030
+ prefix_allowed_tokens_fn=prefix_allowed_tokens_fn,
1031
+ logits_processor=logits_processor,
1032
+ )
1033
+
1034
+ stopping_criteria = self._get_stopping_criteria(
1035
+ generation_config=generation_config, stopping_criteria=stopping_criteria
1036
+ )
1037
+ logits_warper = self._get_logits_warper(generation_config)
1038
+
1039
+ unfinished_sequences = input_ids.new(input_ids.shape[0]).fill_(1)
1040
+ scores = None
1041
+ while True:
1042
+ model_inputs = self.prepare_inputs_for_generation(input_ids, **model_kwargs)
1043
+ # forward pass to get next token
1044
+ outputs = self(
1045
+ **model_inputs,
1046
+ return_dict=True,
1047
+ output_attentions=False,
1048
+ output_hidden_states=False,
1049
+ )
1050
+
1051
+ next_token_logits = outputs.logits[:, -1, :]
1052
+
1053
+ # pre-process distribution
1054
+ next_token_scores = logits_processor(input_ids, next_token_logits)
1055
+ next_token_scores = logits_warper(input_ids, next_token_scores)
1056
+
1057
+ # sample
1058
+ probs = nn.functional.softmax(next_token_scores, dim=-1)
1059
+ if generation_config.do_sample:
1060
+ next_tokens = torch.multinomial(probs, num_samples=1).squeeze(1)
1061
+ else:
1062
+ next_tokens = torch.argmax(probs, dim=-1)
1063
+
1064
+ # update generated ids, model inputs, and length for next step
1065
+ input_ids = torch.cat([input_ids, next_tokens[:, None]], dim=-1)
1066
+ model_kwargs = self._update_model_kwargs_for_generation(
1067
+ outputs, model_kwargs, is_encoder_decoder=self.config.is_encoder_decoder
1068
+ )
1069
+ unfinished_sequences = unfinished_sequences.mul((sum(next_tokens != i for i in eos_token_id)).long())
1070
+
1071
+ # stop when each sentence is finished, or if we exceed the maximum length
1072
+ if unfinished_sequences.max() == 0 or stopping_criteria(input_ids, scores):
1073
+ break
1074
+ yield input_ids
1075
+
1076
+ def quantize(self, bits: int, empty_init=False, device=None, **kwargs):
1077
+ if bits == 0:
1078
+ return
1079
+
1080
+ from .quantization import quantize
1081
+
1082
+ if self.quantized:
1083
+ logger.info("Already quantized.")
1084
+ return self
1085
+
1086
+ self.quantized = True
1087
+
1088
+ self.config.quantization_bit = bits
1089
+
1090
+ self.transformer.encoder = quantize(self.transformer.encoder, bits, empty_init=empty_init, device=device,
1091
+ **kwargs)
1092
+ return self
pytorch_model-00001-of-00007.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:39d9ea931a3cb444ad53fd33853e003d22953a5fcd34204e6df7fb61d746b748
3
+ size 1827781090
pytorch_model-00002-of-00007.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:7706f7420577fbe254f2f80c430084cec9c6879de5094a94a627f9d2517a5c8d
3
+ size 1968299480
pytorch_model-00003-of-00007.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:7bb6903f7cc06c3bd1be76a2880a56f92ae9e8c6a734380d9ce48d98f5bba958
3
+ size 1927415036
pytorch_model-00004-of-00007.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:1f3412c7584797d1576dc8ff55ad0d863f6c3b52d1da720a2d3a45266ebed568
3
+ size 1815225998
pytorch_model-00005-of-00007.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:bdfddc4596fc323ea16be00059d0e7cf255e180c1818c92138137f3c260b4c06
3
+ size 1968299544
pytorch_model-00006-of-00007.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:b19d2d2f1bccbca9a7f09fcd8975191e435acd5be0a557a51140dc6d8a1fffa5
3
+ size 1927415036
pytorch_model-00007-of-00007.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:7bde960e5c66595c46022163579938d0e0c3e37f6b006b2d1249ba85407b5e1c
3
+ size 1052808542
pytorch_model.bin.index.json ADDED
@@ -0,0 +1,207 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "metadata": {
3
+ "total_size": 12487168064
4
+ },
5
+ "weight_map": {
6
+ "transformer.embedding.word_embeddings.weight": "pytorch_model-00001-of-00007.bin",
7
+ "transformer.encoder.final_layernorm.weight": "pytorch_model-00007-of-00007.bin",
8
+ "transformer.encoder.layers.0.input_layernorm.weight": "pytorch_model-00001-of-00007.bin",
9
+ "transformer.encoder.layers.0.mlp.dense_4h_to_h.weight": "pytorch_model-00001-of-00007.bin",
10
+ "transformer.encoder.layers.0.mlp.dense_h_to_4h.weight": "pytorch_model-00001-of-00007.bin",
11
+ "transformer.encoder.layers.0.post_attention_layernorm.weight": "pytorch_model-00001-of-00007.bin",
12
+ "transformer.encoder.layers.0.self_attention.dense.weight": "pytorch_model-00001-of-00007.bin",
13
+ "transformer.encoder.layers.0.self_attention.query_key_value.bias": "pytorch_model-00001-of-00007.bin",
14
+ "transformer.encoder.layers.0.self_attention.query_key_value.weight": "pytorch_model-00001-of-00007.bin",
15
+ "transformer.encoder.layers.1.input_layernorm.weight": "pytorch_model-00001-of-00007.bin",
16
+ "transformer.encoder.layers.1.mlp.dense_4h_to_h.weight": "pytorch_model-00001-of-00007.bin",
17
+ "transformer.encoder.layers.1.mlp.dense_h_to_4h.weight": "pytorch_model-00001-of-00007.bin",
18
+ "transformer.encoder.layers.1.post_attention_layernorm.weight": "pytorch_model-00001-of-00007.bin",
19
+ "transformer.encoder.layers.1.self_attention.dense.weight": "pytorch_model-00001-of-00007.bin",
20
+ "transformer.encoder.layers.1.self_attention.query_key_value.bias": "pytorch_model-00001-of-00007.bin",
21
+ "transformer.encoder.layers.1.self_attention.query_key_value.weight": "pytorch_model-00001-of-00007.bin",
22
+ "transformer.encoder.layers.10.input_layernorm.weight": "pytorch_model-00003-of-00007.bin",
23
+ "transformer.encoder.layers.10.mlp.dense_4h_to_h.weight": "pytorch_model-00003-of-00007.bin",
24
+ "transformer.encoder.layers.10.mlp.dense_h_to_4h.weight": "pytorch_model-00003-of-00007.bin",
25
+ "transformer.encoder.layers.10.post_attention_layernorm.weight": "pytorch_model-00003-of-00007.bin",
26
+ "transformer.encoder.layers.10.self_attention.dense.weight": "pytorch_model-00003-of-00007.bin",
27
+ "transformer.encoder.layers.10.self_attention.query_key_value.bias": "pytorch_model-00003-of-00007.bin",
28
+ "transformer.encoder.layers.10.self_attention.query_key_value.weight": "pytorch_model-00003-of-00007.bin",
29
+ "transformer.encoder.layers.11.input_layernorm.weight": "pytorch_model-00003-of-00007.bin",
30
+ "transformer.encoder.layers.11.mlp.dense_4h_to_h.weight": "pytorch_model-00003-of-00007.bin",
31
+ "transformer.encoder.layers.11.mlp.dense_h_to_4h.weight": "pytorch_model-00003-of-00007.bin",
32
+ "transformer.encoder.layers.11.post_attention_layernorm.weight": "pytorch_model-00003-of-00007.bin",
33
+ "transformer.encoder.layers.11.self_attention.dense.weight": "pytorch_model-00003-of-00007.bin",
34
+ "transformer.encoder.layers.11.self_attention.query_key_value.bias": "pytorch_model-00003-of-00007.bin",
35
+ "transformer.encoder.layers.11.self_attention.query_key_value.weight": "pytorch_model-00003-of-00007.bin",
36
+ "transformer.encoder.layers.12.input_layernorm.weight": "pytorch_model-00003-of-00007.bin",
37
+ "transformer.encoder.layers.12.mlp.dense_4h_to_h.weight": "pytorch_model-00004-of-00007.bin",
38
+ "transformer.encoder.layers.12.mlp.dense_h_to_4h.weight": "pytorch_model-00003-of-00007.bin",
39
+ "transformer.encoder.layers.12.post_attention_layernorm.weight": "pytorch_model-00003-of-00007.bin",
40
+ "transformer.encoder.layers.12.self_attention.dense.weight": "pytorch_model-00003-of-00007.bin",
41
+ "transformer.encoder.layers.12.self_attention.query_key_value.bias": "pytorch_model-00003-of-00007.bin",
42
+ "transformer.encoder.layers.12.self_attention.query_key_value.weight": "pytorch_model-00003-of-00007.bin",
43
+ "transformer.encoder.layers.13.input_layernorm.weight": "pytorch_model-00004-of-00007.bin",
44
+ "transformer.encoder.layers.13.mlp.dense_4h_to_h.weight": "pytorch_model-00004-of-00007.bin",
45
+ "transformer.encoder.layers.13.mlp.dense_h_to_4h.weight": "pytorch_model-00004-of-00007.bin",
46
+ "transformer.encoder.layers.13.post_attention_layernorm.weight": "pytorch_model-00004-of-00007.bin",
47
+ "transformer.encoder.layers.13.self_attention.dense.weight": "pytorch_model-00004-of-00007.bin",
48
+ "transformer.encoder.layers.13.self_attention.query_key_value.bias": "pytorch_model-00004-of-00007.bin",
49
+ "transformer.encoder.layers.13.self_attention.query_key_value.weight": "pytorch_model-00004-of-00007.bin",
50
+ "transformer.encoder.layers.14.input_layernorm.weight": "pytorch_model-00004-of-00007.bin",
51
+ "transformer.encoder.layers.14.mlp.dense_4h_to_h.weight": "pytorch_model-00004-of-00007.bin",
52
+ "transformer.encoder.layers.14.mlp.dense_h_to_4h.weight": "pytorch_model-00004-of-00007.bin",
53
+ "transformer.encoder.layers.14.post_attention_layernorm.weight": "pytorch_model-00004-of-00007.bin",
54
+ "transformer.encoder.layers.14.self_attention.dense.weight": "pytorch_model-00004-of-00007.bin",
55
+ "transformer.encoder.layers.14.self_attention.query_key_value.bias": "pytorch_model-00004-of-00007.bin",
56
+ "transformer.encoder.layers.14.self_attention.query_key_value.weight": "pytorch_model-00004-of-00007.bin",
57
+ "transformer.encoder.layers.15.input_layernorm.weight": "pytorch_model-00004-of-00007.bin",
58
+ "transformer.encoder.layers.15.mlp.dense_4h_to_h.weight": "pytorch_model-00004-of-00007.bin",
59
+ "transformer.encoder.layers.15.mlp.dense_h_to_4h.weight": "pytorch_model-00004-of-00007.bin",
60
+ "transformer.encoder.layers.15.post_attention_layernorm.weight": "pytorch_model-00004-of-00007.bin",
61
+ "transformer.encoder.layers.15.self_attention.dense.weight": "pytorch_model-00004-of-00007.bin",
62
+ "transformer.encoder.layers.15.self_attention.query_key_value.bias": "pytorch_model-00004-of-00007.bin",
63
+ "transformer.encoder.layers.15.self_attention.query_key_value.weight": "pytorch_model-00004-of-00007.bin",
64
+ "transformer.encoder.layers.16.input_layernorm.weight": "pytorch_model-00004-of-00007.bin",
65
+ "transformer.encoder.layers.16.mlp.dense_4h_to_h.weight": "pytorch_model-00004-of-00007.bin",
66
+ "transformer.encoder.layers.16.mlp.dense_h_to_4h.weight": "pytorch_model-00004-of-00007.bin",
67
+ "transformer.encoder.layers.16.post_attention_layernorm.weight": "pytorch_model-00004-of-00007.bin",
68
+ "transformer.encoder.layers.16.self_attention.dense.weight": "pytorch_model-00004-of-00007.bin",
69
+ "transformer.encoder.layers.16.self_attention.query_key_value.bias": "pytorch_model-00004-of-00007.bin",
70
+ "transformer.encoder.layers.16.self_attention.query_key_value.weight": "pytorch_model-00004-of-00007.bin",
71
+ "transformer.encoder.layers.17.input_layernorm.weight": "pytorch_model-00004-of-00007.bin",
72
+ "transformer.encoder.layers.17.mlp.dense_4h_to_h.weight": "pytorch_model-00005-of-00007.bin",
73
+ "transformer.encoder.layers.17.mlp.dense_h_to_4h.weight": "pytorch_model-00005-of-00007.bin",
74
+ "transformer.encoder.layers.17.post_attention_layernorm.weight": "pytorch_model-00004-of-00007.bin",
75
+ "transformer.encoder.layers.17.self_attention.dense.weight": "pytorch_model-00004-of-00007.bin",
76
+ "transformer.encoder.layers.17.self_attention.query_key_value.bias": "pytorch_model-00004-of-00007.bin",
77
+ "transformer.encoder.layers.17.self_attention.query_key_value.weight": "pytorch_model-00004-of-00007.bin",
78
+ "transformer.encoder.layers.18.input_layernorm.weight": "pytorch_model-00005-of-00007.bin",
79
+ "transformer.encoder.layers.18.mlp.dense_4h_to_h.weight": "pytorch_model-00005-of-00007.bin",
80
+ "transformer.encoder.layers.18.mlp.dense_h_to_4h.weight": "pytorch_model-00005-of-00007.bin",
81
+ "transformer.encoder.layers.18.post_attention_layernorm.weight": "pytorch_model-00005-of-00007.bin",
82
+ "transformer.encoder.layers.18.self_attention.dense.weight": "pytorch_model-00005-of-00007.bin",
83
+ "transformer.encoder.layers.18.self_attention.query_key_value.bias": "pytorch_model-00005-of-00007.bin",
84
+ "transformer.encoder.layers.18.self_attention.query_key_value.weight": "pytorch_model-00005-of-00007.bin",
85
+ "transformer.encoder.layers.19.input_layernorm.weight": "pytorch_model-00005-of-00007.bin",
86
+ "transformer.encoder.layers.19.mlp.dense_4h_to_h.weight": "pytorch_model-00005-of-00007.bin",
87
+ "transformer.encoder.layers.19.mlp.dense_h_to_4h.weight": "pytorch_model-00005-of-00007.bin",
88
+ "transformer.encoder.layers.19.post_attention_layernorm.weight": "pytorch_model-00005-of-00007.bin",
89
+ "transformer.encoder.layers.19.self_attention.dense.weight": "pytorch_model-00005-of-00007.bin",
90
+ "transformer.encoder.layers.19.self_attention.query_key_value.bias": "pytorch_model-00005-of-00007.bin",
91
+ "transformer.encoder.layers.19.self_attention.query_key_value.weight": "pytorch_model-00005-of-00007.bin",
92
+ "transformer.encoder.layers.2.input_layernorm.weight": "pytorch_model-00001-of-00007.bin",
93
+ "transformer.encoder.layers.2.mlp.dense_4h_to_h.weight": "pytorch_model-00001-of-00007.bin",
94
+ "transformer.encoder.layers.2.mlp.dense_h_to_4h.weight": "pytorch_model-00001-of-00007.bin",
95
+ "transformer.encoder.layers.2.post_attention_layernorm.weight": "pytorch_model-00001-of-00007.bin",
96
+ "transformer.encoder.layers.2.self_attention.dense.weight": "pytorch_model-00001-of-00007.bin",
97
+ "transformer.encoder.layers.2.self_attention.query_key_value.bias": "pytorch_model-00001-of-00007.bin",
98
+ "transformer.encoder.layers.2.self_attention.query_key_value.weight": "pytorch_model-00001-of-00007.bin",
99
+ "transformer.encoder.layers.20.input_layernorm.weight": "pytorch_model-00005-of-00007.bin",
100
+ "transformer.encoder.layers.20.mlp.dense_4h_to_h.weight": "pytorch_model-00005-of-00007.bin",
101
+ "transformer.encoder.layers.20.mlp.dense_h_to_4h.weight": "pytorch_model-00005-of-00007.bin",
102
+ "transformer.encoder.layers.20.post_attention_layernorm.weight": "pytorch_model-00005-of-00007.bin",
103
+ "transformer.encoder.layers.20.self_attention.dense.weight": "pytorch_model-00005-of-00007.bin",
104
+ "transformer.encoder.layers.20.self_attention.query_key_value.bias": "pytorch_model-00005-of-00007.bin",
105
+ "transformer.encoder.layers.20.self_attention.query_key_value.weight": "pytorch_model-00005-of-00007.bin",
106
+ "transformer.encoder.layers.21.input_layernorm.weight": "pytorch_model-00005-of-00007.bin",
107
+ "transformer.encoder.layers.21.mlp.dense_4h_to_h.weight": "pytorch_model-00005-of-00007.bin",
108
+ "transformer.encoder.layers.21.mlp.dense_h_to_4h.weight": "pytorch_model-00005-of-00007.bin",
109
+ "transformer.encoder.layers.21.post_attention_layernorm.weight": "pytorch_model-00005-of-00007.bin",
110
+ "transformer.encoder.layers.21.self_attention.dense.weight": "pytorch_model-00005-of-00007.bin",
111
+ "transformer.encoder.layers.21.self_attention.query_key_value.bias": "pytorch_model-00005-of-00007.bin",
112
+ "transformer.encoder.layers.21.self_attention.query_key_value.weight": "pytorch_model-00005-of-00007.bin",
113
+ "transformer.encoder.layers.22.input_layernorm.weight": "pytorch_model-00005-of-00007.bin",
114
+ "transformer.encoder.layers.22.mlp.dense_4h_to_h.weight": "pytorch_model-00006-of-00007.bin",
115
+ "transformer.encoder.layers.22.mlp.dense_h_to_4h.weight": "pytorch_model-00006-of-00007.bin",
116
+ "transformer.encoder.layers.22.post_attention_layernorm.weight": "pytorch_model-00006-of-00007.bin",
117
+ "transformer.encoder.layers.22.self_attention.dense.weight": "pytorch_model-00006-of-00007.bin",
118
+ "transformer.encoder.layers.22.self_attention.query_key_value.bias": "pytorch_model-00006-of-00007.bin",
119
+ "transformer.encoder.layers.22.self_attention.query_key_value.weight": "pytorch_model-00006-of-00007.bin",
120
+ "transformer.encoder.layers.23.input_layernorm.weight": "pytorch_model-00006-of-00007.bin",
121
+ "transformer.encoder.layers.23.mlp.dense_4h_to_h.weight": "pytorch_model-00006-of-00007.bin",
122
+ "transformer.encoder.layers.23.mlp.dense_h_to_4h.weight": "pytorch_model-00006-of-00007.bin",
123
+ "transformer.encoder.layers.23.post_attention_layernorm.weight": "pytorch_model-00006-of-00007.bin",
124
+ "transformer.encoder.layers.23.self_attention.dense.weight": "pytorch_model-00006-of-00007.bin",
125
+ "transformer.encoder.layers.23.self_attention.query_key_value.bias": "pytorch_model-00006-of-00007.bin",
126
+ "transformer.encoder.layers.23.self_attention.query_key_value.weight": "pytorch_model-00006-of-00007.bin",
127
+ "transformer.encoder.layers.24.input_layernorm.weight": "pytorch_model-00006-of-00007.bin",
128
+ "transformer.encoder.layers.24.mlp.dense_4h_to_h.weight": "pytorch_model-00006-of-00007.bin",
129
+ "transformer.encoder.layers.24.mlp.dense_h_to_4h.weight": "pytorch_model-00006-of-00007.bin",
130
+ "transformer.encoder.layers.24.post_attention_layernorm.weight": "pytorch_model-00006-of-00007.bin",
131
+ "transformer.encoder.layers.24.self_attention.dense.weight": "pytorch_model-00006-of-00007.bin",
132
+ "transformer.encoder.layers.24.self_attention.query_key_value.bias": "pytorch_model-00006-of-00007.bin",
133
+ "transformer.encoder.layers.24.self_attention.query_key_value.weight": "pytorch_model-00006-of-00007.bin",
134
+ "transformer.encoder.layers.25.input_layernorm.weight": "pytorch_model-00006-of-00007.bin",
135
+ "transformer.encoder.layers.25.mlp.dense_4h_to_h.weight": "pytorch_model-00006-of-00007.bin",
136
+ "transformer.encoder.layers.25.mlp.dense_h_to_4h.weight": "pytorch_model-00006-of-00007.bin",
137
+ "transformer.encoder.layers.25.post_attention_layernorm.weight": "pytorch_model-00006-of-00007.bin",
138
+ "transformer.encoder.layers.25.self_attention.dense.weight": "pytorch_model-00006-of-00007.bin",
139
+ "transformer.encoder.layers.25.self_attention.query_key_value.bias": "pytorch_model-00006-of-00007.bin",
140
+ "transformer.encoder.layers.25.self_attention.query_key_value.weight": "pytorch_model-00006-of-00007.bin",
141
+ "transformer.encoder.layers.26.input_layernorm.weight": "pytorch_model-00006-of-00007.bin",
142
+ "transformer.encoder.layers.26.mlp.dense_4h_to_h.weight": "pytorch_model-00007-of-00007.bin",
143
+ "transformer.encoder.layers.26.mlp.dense_h_to_4h.weight": "pytorch_model-00006-of-00007.bin",
144
+ "transformer.encoder.layers.26.post_attention_layernorm.weight": "pytorch_model-00006-of-00007.bin",
145
+ "transformer.encoder.layers.26.self_attention.dense.weight": "pytorch_model-00006-of-00007.bin",
146
+ "transformer.encoder.layers.26.self_attention.query_key_value.bias": "pytorch_model-00006-of-00007.bin",
147
+ "transformer.encoder.layers.26.self_attention.query_key_value.weight": "pytorch_model-00006-of-00007.bin",
148
+ "transformer.encoder.layers.27.input_layernorm.weight": "pytorch_model-00007-of-00007.bin",
149
+ "transformer.encoder.layers.27.mlp.dense_4h_to_h.weight": "pytorch_model-00007-of-00007.bin",
150
+ "transformer.encoder.layers.27.mlp.dense_h_to_4h.weight": "pytorch_model-00007-of-00007.bin",
151
+ "transformer.encoder.layers.27.post_attention_layernorm.weight": "pytorch_model-00007-of-00007.bin",
152
+ "transformer.encoder.layers.27.self_attention.dense.weight": "pytorch_model-00007-of-00007.bin",
153
+ "transformer.encoder.layers.27.self_attention.query_key_value.bias": "pytorch_model-00007-of-00007.bin",
154
+ "transformer.encoder.layers.27.self_attention.query_key_value.weight": "pytorch_model-00007-of-00007.bin",
155
+ "transformer.encoder.layers.3.input_layernorm.weight": "pytorch_model-00001-of-00007.bin",
156
+ "transformer.encoder.layers.3.mlp.dense_4h_to_h.weight": "pytorch_model-00002-of-00007.bin",
157
+ "transformer.encoder.layers.3.mlp.dense_h_to_4h.weight": "pytorch_model-00002-of-00007.bin",
158
+ "transformer.encoder.layers.3.post_attention_layernorm.weight": "pytorch_model-00001-of-00007.bin",
159
+ "transformer.encoder.layers.3.self_attention.dense.weight": "pytorch_model-00001-of-00007.bin",
160
+ "transformer.encoder.layers.3.self_attention.query_key_value.bias": "pytorch_model-00001-of-00007.bin",
161
+ "transformer.encoder.layers.3.self_attention.query_key_value.weight": "pytorch_model-00001-of-00007.bin",
162
+ "transformer.encoder.layers.4.input_layernorm.weight": "pytorch_model-00002-of-00007.bin",
163
+ "transformer.encoder.layers.4.mlp.dense_4h_to_h.weight": "pytorch_model-00002-of-00007.bin",
164
+ "transformer.encoder.layers.4.mlp.dense_h_to_4h.weight": "pytorch_model-00002-of-00007.bin",
165
+ "transformer.encoder.layers.4.post_attention_layernorm.weight": "pytorch_model-00002-of-00007.bin",
166
+ "transformer.encoder.layers.4.self_attention.dense.weight": "pytorch_model-00002-of-00007.bin",
167
+ "transformer.encoder.layers.4.self_attention.query_key_value.bias": "pytorch_model-00002-of-00007.bin",
168
+ "transformer.encoder.layers.4.self_attention.query_key_value.weight": "pytorch_model-00002-of-00007.bin",
169
+ "transformer.encoder.layers.5.input_layernorm.weight": "pytorch_model-00002-of-00007.bin",
170
+ "transformer.encoder.layers.5.mlp.dense_4h_to_h.weight": "pytorch_model-00002-of-00007.bin",
171
+ "transformer.encoder.layers.5.mlp.dense_h_to_4h.weight": "pytorch_model-00002-of-00007.bin",
172
+ "transformer.encoder.layers.5.post_attention_layernorm.weight": "pytorch_model-00002-of-00007.bin",
173
+ "transformer.encoder.layers.5.self_attention.dense.weight": "pytorch_model-00002-of-00007.bin",
174
+ "transformer.encoder.layers.5.self_attention.query_key_value.bias": "pytorch_model-00002-of-00007.bin",
175
+ "transformer.encoder.layers.5.self_attention.query_key_value.weight": "pytorch_model-00002-of-00007.bin",
176
+ "transformer.encoder.layers.6.input_layernorm.weight": "pytorch_model-00002-of-00007.bin",
177
+ "transformer.encoder.layers.6.mlp.dense_4h_to_h.weight": "pytorch_model-00002-of-00007.bin",
178
+ "transformer.encoder.layers.6.mlp.dense_h_to_4h.weight": "pytorch_model-00002-of-00007.bin",
179
+ "transformer.encoder.layers.6.post_attention_layernorm.weight": "pytorch_model-00002-of-00007.bin",
180
+ "transformer.encoder.layers.6.self_attention.dense.weight": "pytorch_model-00002-of-00007.bin",
181
+ "transformer.encoder.layers.6.self_attention.query_key_value.bias": "pytorch_model-00002-of-00007.bin",
182
+ "transformer.encoder.layers.6.self_attention.query_key_value.weight": "pytorch_model-00002-of-00007.bin",
183
+ "transformer.encoder.layers.7.input_layernorm.weight": "pytorch_model-00002-of-00007.bin",
184
+ "transformer.encoder.layers.7.mlp.dense_4h_to_h.weight": "pytorch_model-00002-of-00007.bin",
185
+ "transformer.encoder.layers.7.mlp.dense_h_to_4h.weight": "pytorch_model-00002-of-00007.bin",
186
+ "transformer.encoder.layers.7.post_attention_layernorm.weight": "pytorch_model-00002-of-00007.bin",
187
+ "transformer.encoder.layers.7.self_attention.dense.weight": "pytorch_model-00002-of-00007.bin",
188
+ "transformer.encoder.layers.7.self_attention.query_key_value.bias": "pytorch_model-00002-of-00007.bin",
189
+ "transformer.encoder.layers.7.self_attention.query_key_value.weight": "pytorch_model-00002-of-00007.bin",
190
+ "transformer.encoder.layers.8.input_layernorm.weight": "pytorch_model-00002-of-00007.bin",
191
+ "transformer.encoder.layers.8.mlp.dense_4h_to_h.weight": "pytorch_model-00003-of-00007.bin",
192
+ "transformer.encoder.layers.8.mlp.dense_h_to_4h.weight": "pytorch_model-00003-of-00007.bin",
193
+ "transformer.encoder.layers.8.post_attention_layernorm.weight": "pytorch_model-00003-of-00007.bin",
194
+ "transformer.encoder.layers.8.self_attention.dense.weight": "pytorch_model-00003-of-00007.bin",
195
+ "transformer.encoder.layers.8.self_attention.query_key_value.bias": "pytorch_model-00003-of-00007.bin",
196
+ "transformer.encoder.layers.8.self_attention.query_key_value.weight": "pytorch_model-00003-of-00007.bin",
197
+ "transformer.encoder.layers.9.input_layernorm.weight": "pytorch_model-00003-of-00007.bin",
198
+ "transformer.encoder.layers.9.mlp.dense_4h_to_h.weight": "pytorch_model-00003-of-00007.bin",
199
+ "transformer.encoder.layers.9.mlp.dense_h_to_4h.weight": "pytorch_model-00003-of-00007.bin",
200
+ "transformer.encoder.layers.9.post_attention_layernorm.weight": "pytorch_model-00003-of-00007.bin",
201
+ "transformer.encoder.layers.9.self_attention.dense.weight": "pytorch_model-00003-of-00007.bin",
202
+ "transformer.encoder.layers.9.self_attention.query_key_value.bias": "pytorch_model-00003-of-00007.bin",
203
+ "transformer.encoder.layers.9.self_attention.query_key_value.weight": "pytorch_model-00003-of-00007.bin",
204
+ "transformer.output_layer.weight": "pytorch_model-00007-of-00007.bin",
205
+ "transformer.rotary_pos_emb.inv_freq": "pytorch_model-00001-of-00007.bin"
206
+ }
207
+ }
quantization.py ADDED
@@ -0,0 +1,190 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ from torch.nn import Linear
2
+ from torch.nn.parameter import Parameter
3
+
4
+ import bz2
5
+ import torch
6
+ import base64
7
+ import ctypes
8
+ from transformers.utils import logging
9
+
10
+ from typing import List
11
+ from functools import partial
12
+
13
+ logger = logging.get_logger(__name__)
14
+
15
+ try:
16
+ from cpm_kernels.kernels.base import LazyKernelCModule, KernelFunction, round_up
17
+
18
+ class Kernel:
19
+ def __init__(self, code: bytes, function_names: List[str]):
20
+ self.code = code
21
+ self._function_names = function_names
22
+ self._cmodule = LazyKernelCModule(self.code)
23
+
24
+ for name in self._function_names:
25
+ setattr(self, name, KernelFunction(self._cmodule, name))
26
+
27
+ quantization_code = "QlpoOTFBWSZTWVAm3YoAX6P//////////f///8///v////T988fldcV++XXV9tX/92/f4Cq/AfQJKoEgJABEBVFKUoAABJUEEgAAFAoAoUAAAACUgAKAUCgKBQAEQBQqQCQFUKKAAAAAAAg0ZBpoDRoaDI0DQaAABpo0aDQ00ANAAZNNAMI0NAAANAYgANAAA0AGQDQEGjINNAaNDQZGgaDQAANNGjQaGmgBoADJpoBhGhoAABoDEABoAAGgAyAaAg0ZBpoDRoaDI0DQaAABpo0aDQ00ANAAZNNAMI0NAAANAYgANAAA0AGQDQEGjINNAaNDQZGgaDQAANNGjQaGmgBoADJpoBhGhoAABoDEABoAAGgAyAaAJqkiFBPVPAIk/JTxop+kj2qD1HlGgGgBkeo0Mj0g9Ro0AA0GQAAaA00aDIAAAaAGgAAAAKUkQgQAJpoAmACaaNAUzTaFGEeibRqBtT0phtSbUyabSbaU8RPUwDQTTyaTPVD0TTTQHqY0E2oaekyGnpmk9TapzBX4PjD9Rh0N1y56Owx5s4cFjbDVlaZWpk0jxF1h9irs9plzmx27Vta3tmzll2zSxN7a4s2scLLiudzHZFcIXawofiI6arpuhz13WLLed1zO0zGmmtWMaMZprTrmzrFlLa6hWq0mVZWTJjHDTGVplppjrWG04o23tmYzE32bLY2fTv3jZ3fQ9DWXmPOeCu30uxd6bq7Sw6VdKTmHOWDeObVu2ODV7pd28Behumzy51nWfEee8T3zvPReB4HpPiO89O5Pcn+B+zmmNT1becXlTz272rwPadm7L052rtPZnkXM82e09tp8F3rwXwL3b5L7S8F6LvvhHmHlttmzbZ8I3bvQYxu4G7i8l5DttOhPNcmcnOrxO+/Elff/RY+a/qWfNY+CzTGXvO27F8P3FeZd55d9m+7n9qcV8OvPfKfLfaT8d+G/F0/Ifyf5601pprS01o01pmNMd2vMnozxTxT3c8o8Y8q5LkcjkcjkcjlcVxOJxOJxONycrL8Y7FPrrI+yXE0vMlnp1pY9m2Y2nKsTyVD6NecPl0Oihpdh8p8h2p1z5Fwbvj298i4MbOLjcbdwcHwYXvQem9P6n5zzu91vAr6+64KsXXq658SdxejvUPgLqvLyyZMPhPWn+OtN66GVhfObNNpxGqw4re2NmGYZXCaN1h7VZb3ksnCqy2nFjjNm7Fu2m6tmMmK1NzdwcG84FsNlwmmKyi8yvJnCbPIem8l9vPeeu5nCe+r2J589l6zT5jU2np16c4uvPadU3dd15vPLXeeq8stTGrxTdz3zXlfasc7dwL2XfrLQH49Yj67yq9h2ZuN6yhjJVgfGea1GMVc83X9GvSf2TJlsoj1QyE6yTtmqoebhUr6dZKq3PNNeyPdT+82KJ494p3V5F4xi1b2y9AbG7Z6E2b1pwrjWpxbTJp585qu1J571HrPeNHqztm8nQrZWlbzU4uD1PLes4sm70Zs3nBk08EnWUObFqvPrx21DYVgcKdSbtq6rELGJI+bW88ydTvTi4jZq90v2OMfxMOCW9hPMeq1VBzz2nQ77cdWSXtvLxZTGWMWNRc/lNErTmmOabz9pP4tXEHanqtT7ZXuF4i8/b3VW5ue4H75wjdbmqmlo1SaWjVMNLDRhpYLUxLKeXc+wx9/Q2ftZ5LVdyfav7x/AMY0fZOFdTx3qPQem9R5rxp0P17yHXYsOtTu91jrz0/TdVnZvtN2PLtODg4ODg8CrrT81/dYsdPUzGLy0d47/gY1ZpqaaNVqyrGKy7k7b92de7HBri7DLO3wmcHA4Thc7meZO+afkP6hzqr0ZwbvJPfHE2Y55zT2idIzDrt2slmLY2mjyjK7DLLHDKdxmacdrTWWNt03S4MHkWLL5E1dyy7/gY0g2cnS/eq53OnBgsZateS6ZpycnVujDnajGRjhNHM9BpuzbbbDGzZs2abMbN2222mmMabmmmm02rY002Gpq2thXjXYcfA1pzsuhcYR6KMgq97WRJd2rKBwXXNXNc1pZaWHNYuFpYcLFwjgcKcFbnkzgF5EMgHqrPnrLVmrGWWmaY0zTGmmrTNMaYrkZHFkLmXNYuexOFYYw2Y3qrYsVTvYjnOrQqdTdqF8NXBqijlXKtC++YnF9Q/JjrTdzqnOLV673FyHTN6how5V21dNWxT3z0GqFdVcmqegztMo+E9++C2qPdmIOqdU1J6zKjrZVXKaaUcJwnfG1OM5pxbRO3Op/zWyTqnfy8Cyw65u66ykXgywTwl2qY0nXeYo9UxesxwcWOs5SPRsmUx9TOxajp1VksplifDHCcdyYdKNK4Fg7TKbm9egTqJqdRks9GriJzmVZYD4VYWrKg5duODVq59zIymXA/AZkedamLJot1iyC3rA8iVl7FhXnqPKVod6WUysXnWy7arce4w1wW8YXNYumnStPhqw3zGOLFiy52VjCcBtRpblzGlXIZGMWHjHdkvfJLaS6ip6Uq4oPq1YTyVXqz2KtlOUsB5V/HS/4zaV9oV7BoW7ErzJMheFo1X053JpUxdysurGxVcTBR38yE87Cru1zvUZMcd5PZZQ6F9NL4BtFG4yUO0yRjFTxWBiysnl1aiPoawkfNGTjWIrJlDw7JI0yh5Veik+0hbUh+wujPU7bdRduZO5XtrpNqxjB1Mfub3xlhu2bG1mxsZC0tjLDLGqnUnu5jKxhMYZC7FYk7CTaFvS+LJhWKwK+i52bKxtBlLur9ndihuod44Wh8cYrmYYsUeW7vdXdtUybisndXEw2Dttl3DnY3pHSx3zsV7WqasmlbUWldDSug8U6E2YHknWND+I+m0XCWOdZZS0ojExXZuiGqMnQPBVhebC+IWhwrmOh0NONdVee2eco5JzljrHgoaJ1mKnVVguR1NC3m60p46rKNcDVYZVbsk2ZVswqHhZTeZIc3Wv3rU7rNG1hYYWpaaNJG5DzOVaIMZXKuhWq2rFHOrEw+gaH/ytMYxj3ysaYMYuNDdRxrvq1OZ3YWzwHy8XY2Vxd1tXGtqOx45+E00YdxpoWMZtpvPxjUYy1OqrrVpNTI2rdl6JyGGMY04Sca41qsA472Mrz3Fq1poc+NMHAOMnKsrRGSaMNU0ZSaS0YaqYlo1TRkbKmktjY0dC5q2i1W03NjeNmmyT+K/IY+qMaY7Fc1c9VDjXKXQ2smVmBmYwxh4Dw2hHCuFZWVhh30llPejFOec84zpuhdNY2XGK/1sitFbGRYqZODNGWNa0zKwZWMWxqGxstNVlW0ufYd+VYc855qD10vRR9ZH0cbRqmqrUaq8ON41G8bxqNRqq9dLLeN43jaNRqNRvS4x91JfgHI4rkssuRoxaWWWliyy2NGjY6JV0TonPvgJh33YY8ixqatNeW79cDmrmrRrJnfO8cjwV+D534P337r+B/A6pPsK62ldk6qG1lmLFZZNpmAzVR2mGGVljLFsbGLMmHZWTHBi8lb17LsmiafK2bFs00PVZxVYPiOQ0WTOLZNaeGaq4IaHavBN0V6CnKtKvCPCPCORo9eaOXEylgp+0cyp/ZZXTq4RwLI9yLqeuMfRMND6KvYd0P1h3TE6KlsMe+eJHisyyy9LbM2tMeSYtbbmY5keUud9dyvYsmLLLEwywwxxVo7BdLi+wcnffBchdKh8wafDbQeM3231MabG4vAXTHS6g1XfGNl2WLFhqfAGU88yxkmDDDBlhbjcxpVdliXVDvZd3Z59Y69DE6mMWYWMLkc7XcMkb1YZTgaDVqzMYwwyZMssaVlleQjTUsDEzFYViZGQyLGBirCsm2Maq1VlWZYxHB0pqN5XFuwxuxjk3RscC6jJhjJjLGctN5otK1NamrMxi0ymhosZjGZBjGZYRssYw0jDmMLBjFmsxkxu0Wqa0aRYZWCxkbrFYtoljVyaakVnF3HMrwPGdDnMdDnQYjuJirUDwIxKfAZJX5vuV0z438Zk/tz26+C/dV+e/VaxjDK02dV85j8pwTGTGTDJjJycbLHz6OwlfCny2pO/WQsrvvhexP29zeo3t7k4+PvseLmtmcLmY21bGLFft1/dY9WffGy9gfpnyrsse2MtWVq7d6a+S8p/KdyvuZ8+v0n9JYx6hYaYrGHCti+/qP36M2XrQsr2C/xalPr/Wb/WY1fqWx/sauNesM/VqyE+LWELx737CYWFor7s8A7FbEysF+oxR6hLD4z474zaJ8Z+6VaPNfnT/2e/ezfAYzT7Jlttr5lpd2fHfCnx54Z474cxZYsezXssww916yrgaYxljF+E+rZZdihzl4hixTR5D0L22XPu1YzO6w0MvzJ899W+UxzP7j7E/wNXU/bPdan6E++n9586bn76vs30S7df97+bP9c3feq/AV9w93yl6PxzpreFkemZE/FHnKtKvNO28gXRX7uuAPgquY7rr1g3lXzT0HtnUYe29s24oxzaRdl2Tg5GMMaboyso7LCTQyr5CyczGGSOyyVc5o7Ru03Ac1cpnQ3S4H0J1ThXKdLnFyNitGRNMYdDGnamSr+G919sy1e/uJ92x7l7iW93jgx3K8udKuabQZWGGGGWUevfFX9Vq/YXw9vfeWPZXrV8DfxL/W5uu+2fSf4b71j1zjPJoZJetWFYkyh3jvHqjZf4qHy0l4lDVDai+o/AODYj2WFXsnrmnOsKjicXptiulZVViyn9kuoTRGPTP4K7e9I7Lz0+FtmxW118zPMh3se438i9hbX7i+iuhDqGKjqdNDE1MyXwLUf1Z9+4XhfWnxnw5/he+nw39+dT8Wf37x2Pzb9Bj0EvLLPqScafyaf2ldx4lfLPkq9lJ6711xl9SFo3R6SPLMvSvJdZvXyfZNPOodP0eeh253pc3Wcze0+c7I7zgfy7jPf1s7DZ/KbL+e7TVdl9scx7TT8R6S/iPAc1buy7Tobl2djSR5DJv4XWL4E6g3ezC4shbNScXU5FsqVxMXCdTShZQxu3NcknhrQavIR/TLTYsbFqTTKu24mpOJudRtEugdDC0bGXvy+EWjk+4aNNOd1mHXZds8I1NXa3OYcjnvt7yjV+G9CcX+Q+5YaL+87k/DfQuzOvY9ecw53z1jF6zRjTRpO2d5jH5bFfltGOw39McHQd2veese85hwPFafkuBt6Lnd91PUOLm7zxPJvN3cn2rc/G+lwfZ/tNOx538zmN68oYTyWixhY8TyFjGG52Ly3oHjHPwK7acGOFsK9Wsr3ab995BNzzJeycF4jxOBjDp5rSa81pcSzGdee0zpWdDds6U3rkroXJtXovNcx0ug7PG6j3XB2HYbnuvOcHQOYfrjvzlW4nvmUOybOTZp0GNmzHge/bSnjOA1Raczbpqy7PPcVc92Hf78/Orc4E7Sq6XtOlYcDExhjRix4SY6WOmulpzzmmOTsOy4MbLtuc4jZ23hP5xR4lYX0l9/bRtUwxVZMLFisZS2JmMRpaZaMNNVLxH/aN0+ur8JX4btup2DsGzZo2NXk1h5TsO0eRX2bTrFj7Ji5MPHPA0PE8Z5bY9Vp1p2XacHXri0x2GjrNOw4Tz3FxTTxORjTmri4Ow8+1HacFo/EPIYnjPCfhn7l+seQOT3lY91zLHQ06Vojdc73VWzi6DqdDTpbHO4kR1P4JxOhdL/O+aaTn3h10K7Vl27s14M89LhefNGj3k+bPfjzj1ZyORyORyORyORxJxOJxOJxOJ3l6FfCXuV0TprK4TfHA2K7J6r5ibSr2x5NWGKso9mvaL2K9KtjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2rZs2bOJflwslzHwWnptnjGz2nbey3PJsrgsvE+Geu7jZpwcnFpXerG7rtODpdLGHlHNabtXhTsuiuZR7GOs7Bdc2cMR7Z0rwri8mhq89zPHYxjmWHQx3Gmj752VcKG7K4MWfBM/Bn+OhvNmMxJvhmFwvcq1Oe6N5ceLuvSF7ZzC+CcetNzY1HkNznNnwneOZtPpS4nkXlHByeUcmGzhNjnstrsu08KcBuNNm7cugydmOQ0du5hq9VpRyuefDYxsi7rjcJ6S/XDuOLBjsHUf13S3rsQvPacOTZq705runFq3nVecds4XS3PZrammRlxlk71sdZbWydutjeXDA7Sdo7L3xvVyeBqO289qu0mOhkd1u1XI77VFXmK43nNldJ6RBdZeZaVN7hcK0Wiw7PZvMRzGSZDvO92HVc2GVToYUXl0Ox4Ww8ZkOLrNY980mx3tTqupqdM3NVK7n4z9Nsq+lV7l4L66fon9Y+I1f5y/yP+XMbN5xX505HRGMZysxllYZajgcWMpymph+e/+n+jN/decq7NeZPyXuPvFb15MwHLJjPH81j+Eq6b+guU8z9OrecxieNV+kjZPaNv05+mWmJgwsJiZMWLGmj9QbFjG1+rav0LDh9ZrTJwzLJjLLRoatYWHQzLaaYx0Gq5gVYtnC0WtOpxjocpzbLBhhjJpVmmOC1XXWMMyXPFk0LnXEcM0f0s2tmq1MaN7h0GUOm/2GRmFgxjLLVT0TgxMWcydV1rbMsYx0G7nw/tHZTsK7U6CZP2s7y2jmZZGE2m6DdZQ2Yd99w8dbV5JX0H9hsX1Syr2TzJ5Dw/uH7d+zrkdFYViZZPLuk9mi+Rv/J7VjGTGOUo9KYjK+NWp/fH6Bh9BpzfZV4Svs8kz7ztugYw6FiYxjRaGmLH8NXbR2/KY7mr7c+ZctpixZkxQxwrc2v7D/a8j6v2j4b6rkp01hPQQuqi0J7rneo5/seq/ouRXJiXNC+c1IaZSe4/0PD/uP6zsOJX3VZSe69b7d5e4rqeRTz9NLbGY9Py2zbMyV6+LMTK+3fPftGPe+g4la2bNqOgsH3t1rqbJdww74yTj5rLbuWrV47rXbuVRycnI04PQea77TlJ01iumsjvMDvuc53Q6WjGNjZ4XXbN27naNnBjgcyvAbuRpgVfvw5ORsc7xn4XlOhc9ditR1q5NQPxa9utIdmvaOd/QMMYYw3qoew9Z4ni61cFdau0uxWyxJhasm+FpiWMEfavHb1srjXgr1XYd53uy5q5o5mLsWlzKndundPkcjiYdNLJL/CPzrYLYury2Xddxs3ruTirz3vO4c/cem+LfGyYwwx8RlrM+W2Yny32b71PvV877q+68k7cL0J8o9Jgw63yj/I7GLNtGj5Kdh7id4YTtsPrNzT5r+29JcWz3jg/0r4r0VczZ/CcjZ0ODndBwYxps/VafMt3Js6XJjd0OZwcXXV1HO3z7kfdlh910OLhI2cQz1qeofFfCeUr9F+ix8iPkJPMSe/qPPrpdL4j4rmLiLzTwNKXAYxUwfBfFnQq+YUOm9t4n+/XGc0jnOssX7xjFcbuwsfsWnEvefrmqp/s/WaT3mU4zFJiWTDK7VDvtRXfN3WfJNl3JwXUw3fpHJps52zDGNV33ZoaoYob6rpeS0rjXChoW87rpt26jmeJW0+fX0r6yv95JfmJzJtLaqPVR6iP44+EjZfoWLLK3jeN42jaNo2qbRtGq2jZb2K3jaNzew3jeN42jaNo2jaOUlLirCYhYlHS9l65hWXanehfOrjQ7St5pXerLvLX7jTroW6cTgwYftkbHnPGsdKuSg3XXfauivHronEcauN01zPKXAA2Vw/pqDhJ/deqw0xhh9NpdDzDHJtC85jY7q1W9P0n1B22mjDuGzR12jmrirg3buLZxWNNJpjuHFp2YFhycRpyD59fuGXRNXSOY53FnWycc13m2xfEPksd1XxXv67NVxVyc7StTxz85xK8hGPXY411nBHer/zULF1AcI9SalVNDvvpjnXNzoHtIo8g7D354HQxjG5pQ81i/Arq07Dpxj3zWxXedZVWmri1nXn+V5C/uujZpjGzS/8GxsY/dnxX7AYYNLE0aGMaaLTC/TrAdKh16xXmr2HzL2rFfgFpOScfSUdXqHeWNo6PDVW/jStU2rrnC1GGpzI/l78/PzIw2V4j0FwNNmP88o+kMrKyMjIyMjKrDDIyMjIyMjIyMMMqZGRkZKwmVivLb0OR5rrNpXWmVHunxgxiwYsVytqX5rL4B+3YbK0eQ2PKLdwR9wsKn+UH6LVfUV2S0J2U7I2d9Q8edpPYFf5lf8SBe6qrJGyS412UL5JA95J1qMV/lVbOy6EluEXgcUcULdV7Cv/hV2q+Wq66F+XArILsIXaVV0FzBXWVeB8E3PrP1VX+GBXanhcJ4xYX/IqryKqjtXZVUdmRXAyhhgMMBhlFhiqsMqMMOpHEGQF3pzFk/elyFsVieMqr+wQMkePVX8grJRyYgXkK7QVopcElqBXUkT4YV4SB4z/1qrhVFeApdvuTszWp8Uxpo1MmMGmNMZYxk0ywn1UemjRdl+A/ZDVl47ey/ZI95GyNIxQ7FehVXsNhXFQ+BeR/0H7Gw7Vvt91PuWTGzVxcTY1Ytre2bMHiSqnbO1DJU7cq7wNyvhPrysHpX3paliatDLFak4qp14yPHB44O9RO7G8l17mjjYFd1T/Vh+OhcyFuoMiK7bxnRAr2ivEV0JLqQvQCLoQLYr3FXyA6yH+JVkZSxJ1kidkd4wYdctGjJpX377Z5gj3VbOC4LnLbu2Vqno+deGt98NtNtq3Njc3bNG29WGmjTYYxWlbLVs1Y0yxuZbm5ubZbzebjlwaTZtOLk8KF+vQLhP6lD7Wqh/1UPNk8x0VxFaZWCxYOzR2EdMOC1X0ledFqtTc0VpjBY3q71D+Cj1z7h+QWle3WI3RxPvJzT+fNTyJp7ezxTwo5V499tWlWDFYrLLsKG4n7ZyLhcJbjDFjHQwM5OsrDZTK1Eu9Q1HGbkfkv9rmVsjFcYljk28wcpoOoxTDGMrDKmDLtUPvaHUodlq+PQ4WYxhjFhZciwq+afNWlfZOctHBYXfR22k9+xPNoYjafx+avsry76kw+8clXixHVYhhidnBttNMshu0q0jMMNalu2LNW4NqGV3L+bV+S85+RxF4HFzVOy2rK3PGNww0ahs6E0ww6WK7jDwqu9Vyc857uTpdLk5nAWlarwsPLmxqLmrFG68djdz15lcIcZ0OhbFH8dpWmVu+Q6Hy3Cuu4TUT6+U2ea6ppp2J1Ouk42SZjGBjFgwMYVjCxhhlZimMlmJjBGYrGOpjTLLynlxPpK3U3nBoNSO+/mP7jYxjZmlZKO2d5h0upvJGOe0qLr1u1JswvGP9TTRbFrRZbsRpdxxnoMrcHDEcGIwtNGlxrjO5P5w4ujgdl2wYZJfQeF+U/ePyn3x2Hl+Bo8ZX4TuNquDi5g/qt6XJMRHjPC5bn55wOksXYPGVqA7zCXbYTgjF/RddO45Ok4PC8TROjg89wreOiMJXjPA7hi2aV1jZ4+78p9N/wrGljR++bPyn8hj+Mxow4McHB/wOs+R62MdMxfUOsjCryUYvfrzVg8ZpckLTzYWqU9JwV697LruFD8xeMeKekDTzn47F5zSaPzHnq55PNcXB23qPSYXgH+lh0upcmPz1dp6x7D1F1mOw077wtkxxbvAcmzZYbMYbNmzx/Lc71GpzoMnte82R7rvvlPA/WTc0Y89o02Njimzicw2Vz18A3tlcDmc6sYxNGNNmz/3bq3V0zdjLhJ8A7Xcsd3uvSu8cHFX4k+tC81V0K3iXgnO5R6Z6BwRxqq7iZoluu9OpkYrr9h7p6WnKhzV1HTJecm5fZnlnjLlOZ000q4q91/mfmPbeM0dHeeFjOhjpmulbm47tw7Ryu957Zvb457muazHYR36H9cvp9YMjLBhHjllMrGWWMGTHNz3IpXSn5xpH2FYiXOXHpcs6rmbN+ppr7FDqq8qyHaofJL9G/7rwUPmXtvWx5TR9vjnof/4u5IpwoSCgTbsU"
28
+
29
+ kernels = Kernel(
30
+ bz2.decompress(base64.b64decode(quantization_code)),
31
+ [
32
+ "int4WeightCompression",
33
+ "int4WeightExtractionFloat",
34
+ "int4WeightExtractionHalf",
35
+ "int4WeightExtractionBFloat16",
36
+ "int8WeightExtractionFloat",
37
+ "int8WeightExtractionHalf",
38
+ "int8WeightExtractionBFloat16",
39
+ ],
40
+ )
41
+ except Exception as exception:
42
+ kernels = None
43
+ logger.warning("Failed to load cpm_kernels:" + str(exception))
44
+
45
+
46
+ class W8A16Linear(torch.autograd.Function):
47
+ @staticmethod
48
+ def forward(ctx, inp: torch.Tensor, quant_w: torch.Tensor, scale_w: torch.Tensor, weight_bit_width):
49
+ ctx.inp_shape = inp.size()
50
+ ctx.weight_bit_width = weight_bit_width
51
+ out_features = quant_w.size(0)
52
+ inp = inp.contiguous().view(-1, inp.size(-1))
53
+ weight = extract_weight_to_half(quant_w, scale_w, weight_bit_width)
54
+ ctx.weight_shape = weight.size()
55
+ output = inp.mm(weight.t())
56
+ ctx.save_for_backward(inp, quant_w, scale_w)
57
+ return output.view(*(ctx.inp_shape[:-1] + (out_features,)))
58
+
59
+ @staticmethod
60
+ def backward(ctx, grad_output: torch.Tensor):
61
+ inp, quant_w, scale_w = ctx.saved_tensors
62
+ weight = extract_weight_to_half(quant_w, scale_w, ctx.weight_bit_width)
63
+ grad_output = grad_output.contiguous().view(-1, weight.size(0))
64
+ grad_input = grad_output.mm(weight)
65
+ grad_weight = grad_output.t().mm(inp)
66
+ return grad_input.view(ctx.inp_shape), grad_weight.view(ctx.weight_shape), None, None
67
+
68
+
69
+ def compress_int4_weight(weight: torch.Tensor): # (n, m)
70
+ with torch.cuda.device(weight.device):
71
+ n, m = weight.size(0), weight.size(1)
72
+ assert m % 2 == 0
73
+ m = m // 2
74
+ out = torch.empty(n, m, dtype=torch.int8, device="cuda")
75
+ stream = torch.cuda.current_stream()
76
+
77
+ gridDim = (n, 1, 1)
78
+ blockDim = (min(round_up(m, 32), 1024), 1, 1)
79
+
80
+ kernels.int4WeightCompression(
81
+ gridDim,
82
+ blockDim,
83
+ 0,
84
+ stream,
85
+ [ctypes.c_void_p(weight.data_ptr()), ctypes.c_void_p(out.data_ptr()), ctypes.c_int32(n), ctypes.c_int32(m)],
86
+ )
87
+ return out
88
+
89
+
90
+ def extract_weight_to_half(weight: torch.Tensor, scale_list: torch.Tensor, source_bit_width: int):
91
+ assert scale_list.dtype in [torch.half, torch.bfloat16]
92
+ assert weight.dtype in [torch.int8]
93
+ if source_bit_width == 8:
94
+ return weight.to(scale_list.dtype) * scale_list[:, None]
95
+ elif source_bit_width == 4:
96
+ func = (
97
+ kernels.int4WeightExtractionHalf if scale_list.dtype == torch.half else kernels.int4WeightExtractionBFloat16
98
+ )
99
+ else:
100
+ assert False, "Unsupported bit-width"
101
+
102
+ with torch.cuda.device(weight.device):
103
+ n, m = weight.size(0), weight.size(1)
104
+ out = torch.empty(n, m * (8 // source_bit_width), dtype=scale_list.dtype, device="cuda")
105
+ stream = torch.cuda.current_stream()
106
+
107
+ gridDim = (n, 1, 1)
108
+ blockDim = (min(round_up(m, 32), 1024), 1, 1)
109
+
110
+ func(
111
+ gridDim,
112
+ blockDim,
113
+ 0,
114
+ stream,
115
+ [
116
+ ctypes.c_void_p(weight.data_ptr()),
117
+ ctypes.c_void_p(scale_list.data_ptr()),
118
+ ctypes.c_void_p(out.data_ptr()),
119
+ ctypes.c_int32(n),
120
+ ctypes.c_int32(m),
121
+ ],
122
+ )
123
+ return out
124
+
125
+
126
+ class QuantizedLinear(torch.nn.Module):
127
+ def __init__(self, weight_bit_width: int, weight, bias=None, device="cpu", dtype=None, empty_init=False, *args,
128
+ **kwargs):
129
+ super().__init__()
130
+ self.weight_bit_width = weight_bit_width
131
+
132
+ shape = weight.shape
133
+
134
+ if weight is None or empty_init:
135
+ self.weight = torch.empty(shape[0], shape[1] * weight_bit_width // 8, dtype=torch.int8, device=device)
136
+ self.weight_scale = torch.empty(shape[0], dtype=dtype, device=device)
137
+ else:
138
+ self.weight_scale = weight.abs().max(dim=-1).values / ((2 ** (weight_bit_width - 1)) - 1)
139
+ self.weight = torch.round(weight / self.weight_scale[:, None]).to(torch.int8)
140
+ if weight_bit_width == 4:
141
+ self.weight = compress_int4_weight(self.weight)
142
+
143
+ self.weight = Parameter(self.weight.to(device), requires_grad=False)
144
+ self.weight_scale = Parameter(self.weight_scale.to(device), requires_grad=False)
145
+ self.bias = Parameter(bias.to(device), requires_grad=False) if bias is not None else None
146
+
147
+ def forward(self, input):
148
+ output = W8A16Linear.apply(input, self.weight, self.weight_scale, self.weight_bit_width)
149
+ if self.bias is not None:
150
+ output = output + self.bias
151
+ return output
152
+
153
+
154
+ def quantize(model, weight_bit_width, empty_init=False, device=None):
155
+ """Replace fp16 linear with quantized linear"""
156
+ for layer in model.layers:
157
+ layer.self_attention.query_key_value = QuantizedLinear(
158
+ weight_bit_width=weight_bit_width,
159
+ weight=layer.self_attention.query_key_value.weight.to(torch.cuda.current_device()),
160
+ bias=layer.self_attention.query_key_value.bias,
161
+ dtype=layer.self_attention.query_key_value.weight.dtype,
162
+ device=layer.self_attention.query_key_value.weight.device if device is None else device,
163
+ empty_init=empty_init
164
+ )
165
+ layer.self_attention.dense = QuantizedLinear(
166
+ weight_bit_width=weight_bit_width,
167
+ weight=layer.self_attention.dense.weight.to(torch.cuda.current_device()),
168
+ bias=layer.self_attention.dense.bias,
169
+ dtype=layer.self_attention.dense.weight.dtype,
170
+ device=layer.self_attention.dense.weight.device if device is None else device,
171
+ empty_init=empty_init
172
+ )
173
+ layer.mlp.dense_h_to_4h = QuantizedLinear(
174
+ weight_bit_width=weight_bit_width,
175
+ weight=layer.mlp.dense_h_to_4h.weight.to(torch.cuda.current_device()),
176
+ bias=layer.mlp.dense_h_to_4h.bias,
177
+ dtype=layer.mlp.dense_h_to_4h.weight.dtype,
178
+ device=layer.mlp.dense_h_to_4h.weight.device if device is None else device,
179
+ empty_init=empty_init
180
+ )
181
+ layer.mlp.dense_4h_to_h = QuantizedLinear(
182
+ weight_bit_width=weight_bit_width,
183
+ weight=layer.mlp.dense_4h_to_h.weight.to(torch.cuda.current_device()),
184
+ bias=layer.mlp.dense_4h_to_h.bias,
185
+ dtype=layer.mlp.dense_4h_to_h.weight.dtype,
186
+ device=layer.mlp.dense_4h_to_h.weight.device if device is None else device,
187
+ empty_init=empty_init
188
+ )
189
+
190
+ return model
tokenization_chatglm.py ADDED
@@ -0,0 +1,236 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ import os
2
+ import torch
3
+ from typing import List, Optional, Union, Dict
4
+ from sentencepiece import SentencePieceProcessor
5
+ from transformers import PreTrainedTokenizer
6
+ from transformers.utils import logging, PaddingStrategy
7
+ from transformers.tokenization_utils_base import EncodedInput, BatchEncoding
8
+
9
+
10
+ class SPTokenizer:
11
+ def __init__(self, model_path: str):
12
+ # reload tokenizer
13
+ assert os.path.isfile(model_path), model_path
14
+ self.sp_model = SentencePieceProcessor(model_file=model_path)
15
+
16
+ # BOS / EOS token IDs
17
+ self.n_words: int = self.sp_model.vocab_size()
18
+ self.bos_id: int = self.sp_model.bos_id()
19
+ self.eos_id: int = self.sp_model.eos_id()
20
+ self.pad_id: int = self.sp_model.eos_id()
21
+ assert self.sp_model.vocab_size() == self.sp_model.get_piece_size()
22
+
23
+ special_tokens = ["[MASK]", "[gMASK]", "[sMASK]", "sop", "eop"]
24
+ self.special_tokens = {}
25
+ self.index_special_tokens = {}
26
+ for token in special_tokens:
27
+ self.special_tokens[token] = self.n_words
28
+ self.index_special_tokens[self.n_words] = token
29
+ self.n_words += 1
30
+
31
+ def tokenize(self, s: str):
32
+ return self.sp_model.EncodeAsPieces(s)
33
+
34
+ def encode(self, s: str, bos: bool = False, eos: bool = False) -> List[int]:
35
+ assert type(s) is str
36
+ t = self.sp_model.encode(s)
37
+ if bos:
38
+ t = [self.bos_id] + t
39
+ if eos:
40
+ t = t + [self.eos_id]
41
+ return t
42
+
43
+ def decode(self, t: List[int]) -> str:
44
+ return self.sp_model.decode(t)
45
+
46
+ def decode_tokens(self, tokens: List[str]) -> str:
47
+ text = self.sp_model.DecodePieces(tokens)
48
+ return text
49
+
50
+ def convert_token_to_id(self, token):
51
+ """ Converts a token (str) in an id using the vocab. """
52
+ if token in self.special_tokens:
53
+ return self.special_tokens[token]
54
+ return self.sp_model.PieceToId(token)
55
+
56
+ def convert_id_to_token(self, index):
57
+ """Converts an index (integer) in a token (str) using the vocab."""
58
+ if index in self.index_special_tokens:
59
+ return ""
60
+ return self.sp_model.IdToPiece(index)
61
+
62
+
63
+ class ChatGLMTokenizer(PreTrainedTokenizer):
64
+ vocab_files_names = {"vocab_file": "tokenizer.model"}
65
+
66
+ model_input_names = ["input_ids", "attention_mask", "position_ids"]
67
+
68
+ def __init__(self, vocab_file, padding_side="left", **kwargs):
69
+ super().__init__(padding_side=padding_side, clean_up_tokenization_spaces=False, **kwargs)
70
+ self.name = "GLMTokenizer"
71
+
72
+ self.tokenizer = SPTokenizer(vocab_file)
73
+ self.special_tokens = {
74
+ "<bos>": self.tokenizer.bos_id,
75
+ "<eos>": self.tokenizer.eos_id,
76
+ "<pad>": self.tokenizer.pad_id
77
+ }
78
+
79
+ def get_command(self, token):
80
+ if token in self.special_tokens:
81
+ return self.special_tokens[token]
82
+ assert token in self.tokenizer.special_tokens, f"{token} is not a special token for {self.name}"
83
+ return self.tokenizer.special_tokens[token]
84
+
85
+ @property
86
+ def pad_token(self) -> str:
87
+ return "</s>"
88
+
89
+ @property
90
+ def pad_token_id(self):
91
+ return self.get_command("<pad>")
92
+
93
+ @property
94
+ def vocab_size(self):
95
+ return self.tokenizer.n_words
96
+
97
+ def get_vocab(self):
98
+ """ Returns vocab as a dict """
99
+ vocab = {self._convert_id_to_token(i): i for i in range(self.vocab_size)}
100
+ vocab.update(self.added_tokens_encoder)
101
+ return vocab
102
+
103
+ def _tokenize(self, text, **kwargs):
104
+ return self.tokenizer.tokenize(text)
105
+
106
+ def _convert_token_to_id(self, token):
107
+ """ Converts a token (str) in an id using the vocab. """
108
+ return self.tokenizer.convert_token_to_id(token)
109
+
110
+ def _convert_id_to_token(self, index):
111
+ """Converts an index (integer) in a token (str) using the vocab."""
112
+ return self.tokenizer.convert_id_to_token(index)
113
+
114
+ def convert_tokens_to_string(self, tokens: List[str]) -> str:
115
+ return self.tokenizer.decode_tokens(tokens)
116
+
117
+ def save_vocabulary(self, save_directory, filename_prefix=None):
118
+ """
119
+ Save the vocabulary and special tokens file to a directory.
120
+
121
+ Args:
122
+ save_directory (`str`):
123
+ The directory in which to save the vocabulary.
124
+ filename_prefix (`str`, *optional*):
125
+ An optional prefix to add to the named of the saved files.
126
+
127
+ Returns:
128
+ `Tuple(str)`: Paths to the files saved.
129
+ """
130
+ if os.path.isdir(save_directory):
131
+ vocab_file = os.path.join(
132
+ save_directory, self.vocab_files_names["vocab_file"]
133
+ )
134
+ else:
135
+ vocab_file = save_directory
136
+
137
+ with open(self.vocab_file, 'rb') as fin:
138
+ proto_str = fin.read()
139
+
140
+ with open(vocab_file, "wb") as writer:
141
+ writer.write(proto_str)
142
+
143
+ return (vocab_file,)
144
+
145
+ def get_prefix_tokens(self):
146
+ prefix_tokens = [self.get_command("[gMASK]"), self.get_command("sop")]
147
+ return prefix_tokens
148
+
149
+ def build_inputs_with_special_tokens(
150
+ self, token_ids_0: List[int], token_ids_1: Optional[List[int]] = None
151
+ ) -> List[int]:
152
+ """
153
+ Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and
154
+ adding special tokens. A BERT sequence has the following format:
155
+
156
+ - single sequence: `[CLS] X [SEP]`
157
+ - pair of sequences: `[CLS] A [SEP] B [SEP]`
158
+
159
+ Args:
160
+ token_ids_0 (`List[int]`):
161
+ List of IDs to which the special tokens will be added.
162
+ token_ids_1 (`List[int]`, *optional*):
163
+ Optional second list of IDs for sequence pairs.
164
+
165
+ Returns:
166
+ `List[int]`: List of [input IDs](../glossary#input-ids) with the appropriate special tokens.
167
+ """
168
+ prefix_tokens = self.get_prefix_tokens()
169
+ token_ids_0 = prefix_tokens + token_ids_0
170
+ if token_ids_1 is not None:
171
+ token_ids_0 = token_ids_0 + token_ids_1 + [self.get_command("<eos>")]
172
+ return token_ids_0
173
+
174
+ def _pad(
175
+ self,
176
+ encoded_inputs: Union[Dict[str, EncodedInput], BatchEncoding],
177
+ max_length: Optional[int] = None,
178
+ padding_strategy: PaddingStrategy = PaddingStrategy.DO_NOT_PAD,
179
+ pad_to_multiple_of: Optional[int] = None,
180
+ return_attention_mask: Optional[bool] = None,
181
+ ) -> dict:
182
+ """
183
+ Pad encoded inputs (on left/right and up to predefined length or max length in the batch)
184
+
185
+ Args:
186
+ encoded_inputs:
187
+ Dictionary of tokenized inputs (`List[int]`) or batch of tokenized inputs (`List[List[int]]`).
188
+ max_length: maximum length of the returned list and optionally padding length (see below).
189
+ Will truncate by taking into account the special tokens.
190
+ padding_strategy: PaddingStrategy to use for padding.
191
+
192
+ - PaddingStrategy.LONGEST Pad to the longest sequence in the batch
193
+ - PaddingStrategy.MAX_LENGTH: Pad to the max length (default)
194
+ - PaddingStrategy.DO_NOT_PAD: Do not pad
195
+ The tokenizer padding sides are defined in self.padding_side:
196
+
197
+ - 'left': pads on the left of the sequences
198
+ - 'right': pads on the right of the sequences
199
+ pad_to_multiple_of: (optional) Integer if set will pad the sequence to a multiple of the provided value.
200
+ This is especially useful to enable the use of Tensor Core on NVIDIA hardware with compute capability
201
+ `>= 7.5` (Volta).
202
+ return_attention_mask:
203
+ (optional) Set to False to avoid returning attention mask (default: set to model specifics)
204
+ """
205
+ # Load from model defaults
206
+ assert self.padding_side == "left"
207
+
208
+ required_input = encoded_inputs[self.model_input_names[0]]
209
+ seq_length = len(required_input)
210
+
211
+ if padding_strategy == PaddingStrategy.LONGEST:
212
+ max_length = len(required_input)
213
+
214
+ if max_length is not None and pad_to_multiple_of is not None and (max_length % pad_to_multiple_of != 0):
215
+ max_length = ((max_length // pad_to_multiple_of) + 1) * pad_to_multiple_of
216
+
217
+ needs_to_be_padded = padding_strategy != PaddingStrategy.DO_NOT_PAD and len(required_input) != max_length
218
+
219
+ # Initialize attention mask if not present.
220
+ if max_length is not None:
221
+ if "attention_mask" not in encoded_inputs:
222
+ encoded_inputs["attention_mask"] = [1] * seq_length
223
+
224
+ if "position_ids" not in encoded_inputs:
225
+ encoded_inputs["position_ids"] = list(range(seq_length))
226
+
227
+ if needs_to_be_padded:
228
+ difference = max_length - len(required_input)
229
+
230
+ if "attention_mask" in encoded_inputs:
231
+ encoded_inputs["attention_mask"] = [0] * difference + encoded_inputs["attention_mask"]
232
+ if "position_ids" in encoded_inputs:
233
+ encoded_inputs["position_ids"] = [0] * difference + encoded_inputs["position_ids"]
234
+ encoded_inputs[self.model_input_names[0]] = [self.pad_token_id] * difference + required_input
235
+
236
+ return encoded_inputs
tokenizer.model ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:57d9fdbdfaa7cd8c0a3a38d7e8de2e6c31374b5dbc4dc4568d85585fe745812f
3
+ size 1018370
tokenizer_config.json ADDED
@@ -0,0 +1,12 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "name_or_path": "THUDM/codegeex2-6b",
3
+ "remove_space": false,
4
+ "do_lower_case": false,
5
+ "tokenizer_class": "ChatGLMTokenizer",
6
+ "auto_map": {
7
+ "AutoTokenizer": [
8
+ "tokenization_chatglm.ChatGLMTokenizer",
9
+ null
10
+ ]
11
+ }
12
+ }