allknowingroger commited on
Commit
a34b2c4
1 Parent(s): 04bf29f

Upload 10 files

Browse files
LICENSE.txt ADDED
@@ -0,0 +1,42 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ STABILITY AI NON-COMMERCIAL RESEARCH COMMUNITY LICENSE AGREEMENT
2
+ Dated: December 06, 2023
3
+
4
+ By using or distributing any portion or element of the Models, Software, Software Products or Derivative Works, you agree to be bound by this Agreement.
5
+
6
+ "Agreement" means this Stable Non-Commercial Research Community License Agreement.
7
+
8
+ “AUP” means the Stability AI Acceptable Use Policy available at https://stability.ai/use-policy, as may be updated from time to time.
9
+
10
+ "Derivative Work(s)” means (a) any derivative work of the Software Products as recognized by U.S. copyright laws and (b) any modifications to a Model, and any other model created which is based on or derived from the Model or the Model’s output. For clarity, Derivative Works do not include the output of any Model.
11
+
12
+ “Documentation” means any specifications, manuals, documentation, and other written information provided by Stability AI related to the Software.
13
+
14
+ "Licensee" or "you" means you, or your employer or any other person or entity (if you are entering into this Agreement on such person or entity's behalf), of the age required under applicable laws, rules or regulations to provide legal consent and that has legal authority to bind your employer or such other person or entity if you are entering in this Agreement on their behalf.
15
+
16
+ “Model(s)" means, collectively, Stability AI’s proprietary models and algorithms, including machine-learning models, trained model weights and other elements of the foregoing, made available under this Agreement.
17
+
18
+ “Non-Commercial Uses” means exercising any of the rights granted herein for the purpose of research or non-commercial purposes. Non-Commercial Uses does not include any production use of the Software Products or any Derivative Works.
19
+
20
+ "Stability AI" or "we" means Stability AI Ltd. and its affiliates.
21
+
22
+
23
+ "Software" means Stability AI’s proprietary software made available under this Agreement.
24
+
25
+ “Software Products” means the Models, Software and Documentation, individually or in any combination.
26
+
27
+
28
+
29
+ 1. License Rights and Redistribution.
30
+ a. Subject to your compliance with this Agreement, the AUP (which is hereby incorporated herein by reference), and the Documentation, Stability AI grants you a non-exclusive, worldwide, non-transferable, non-sublicensable, revocable, royalty free and limited license under Stability AI’s intellectual property or other rights owned or controlled by Stability AI embodied in the Software Products to use, reproduce, distribute, and create Derivative Works of, the Software Products, in each case for Non-Commercial Uses only.
31
+ b. You may not use the Software Products or Derivative Works to enable third parties to use the Software Products or Derivative Works as part of your hosted service or via your APIs, whether you are adding substantial additional functionality thereto or not. Merely distributing the Software Products or Derivative Works for download online without offering any related service (ex. by distributing the Models on HuggingFace) is not a violation of this subsection. If you wish to use the Software Products or any Derivative Works for commercial or production use or you wish to make the Software Products or any Derivative Works available to third parties via your hosted service or your APIs, contact Stability AI at https://stability.ai/contact.
32
+ c. If you distribute or make the Software Products, or any Derivative Works thereof, available to a third party, the Software Products, Derivative Works, or any portion thereof, respectively, will remain subject to this Agreement and you must (i) provide a copy of this Agreement to such third party, and (ii) retain the following attribution notice within a "Notice" text file distributed as a part of such copies: "This Stability AI Model is licensed under the Stability AI Non-Commercial Research Community License, Copyright (c) Stability AI Ltd. All Rights Reserved.” If you create a Derivative Work of a Software Product, you may add your own attribution notices to the Notice file included with the Software Product, provided that you clearly indicate which attributions apply to the Software Product and you must state in the NOTICE file that you changed the Software Product and how it was modified.
33
+ 2. Disclaimer of Warranty. UNLESS REQUIRED BY APPLICABLE LAW, THE SOFTWARE PRODUCTS AND ANY OUTPUT AND RESULTS THEREFROM ARE PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING, WITHOUT LIMITATION, ANY WARRANTIES OF TITLE, NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. YOU ARE SOLELY RESPONSIBLE FOR DETERMINING THE APPROPRIATENESS OF USING OR REDISTRIBUTING THE SOFTWARE PRODUCTS, DERIVATIVE WORKS OR ANY OUTPUT OR RESULTS AND ASSUME ANY RISKS ASSOCIATED WITH YOUR USE OF THE SOFTWARE PRODUCTS, DERIVATIVE WORKS AND ANY OUTPUT AND RESULTS.
34
+ 3. Limitation of Liability. IN NO EVENT WILL STABILITY AI OR ITS AFFILIATES BE LIABLE UNDER ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, TORT, NEGLIGENCE, PRODUCTS LIABILITY, OR OTHERWISE, ARISING OUT OF THIS AGREEMENT, FOR ANY LOST PROFITS OR ANY DIRECT, INDIRECT, SPECIAL, CONSEQUENTIAL, INCIDENTAL, EXEMPLARY OR PUNITIVE DAMAGES, EVEN IF STABILITY AI OR ITS AFFILIATES HAVE BEEN ADVISED OF THE POSSIBILITY OF ANY OF THE FOREGOING.
35
+ 4. Intellectual Property.
36
+ a. No trademark licenses are granted under this Agreement, and in connection with the Software Products or Derivative Works, neither Stability AI nor Licensee may use any name or mark owned by or associated with the other or any of its affiliates, except as required for reasonable and customary use in describing and redistributing the Software Products or Derivative Works.
37
+ b. Subject to Stability AI’s ownership of the Software Products and Derivative Works made by or for Stability AI, with respect to any Derivative Works that are made by you, as between you and Stability AI, you are and will be the owner of such Derivative Works
38
+ c. If you institute litigation or other proceedings against Stability AI (including a cross-claim or counterclaim in a lawsuit) alleging that the Software Products, Derivative Works or associated outputs or results, or any portion of any of the foregoing, constitutes infringement of intellectual property or other rights owned or licensable by you, then any licenses granted to you under this Agreement shall terminate as of the date such litigation or claim is filed or instituted. You will indemnify and hold harmless Stability AI from and against any claim by any third party arising out of or related to your use or distribution of the Software Products or Derivative Works in violation of this Agreement.
39
+ 5. Term and Termination. The term of this Agreement will commence upon your acceptance of this Agreement or access to the Software Products and will continue in full force and effect until terminated in accordance with the terms and conditions herein. Stability AI may terminate this Agreement if you are in breach of any term or condition of this Agreement. Upon termination of this Agreement, you shall delete and cease use of any Software Products or Derivative Works. Sections 2-4 shall survive the termination of this Agreement.
40
+
41
+ 6. Governing Law. This Agreement will be governed by and construed in accordance with the laws of the United States and the State of California without regard to choice of law
42
+ principles.
README.md ADDED
@@ -0,0 +1,514 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ base_model: stabilityai/stablelm-zephyr-3b
3
+ datasets:
4
+ - HuggingFaceH4/ultrachat_200k
5
+ - HuggingFaceH4/ultrafeedback_binarized
6
+ - meta-math/MetaMathQA
7
+ - WizardLM/WizardLM_evol_instruct_V2_196k
8
+ - Intel/orca_dpo_pairs
9
+ inference: false
10
+ language:
11
+ - en
12
+ license: other
13
+ model_creator: Stability AI
14
+ model_name: StableLM Zephyr 3B
15
+ model_type: stablelm
16
+ prompt_template: '<|user|>
17
+
18
+ {prompt}<|endoftext|>
19
+
20
+ <|assistant|>
21
+
22
+ '
23
+ quantized_by: TheBloke
24
+ tags:
25
+ - causal-lm
26
+ ---
27
+ <!-- markdownlint-disable MD041 -->
28
+
29
+ <!-- header start -->
30
+ <!-- 200823 -->
31
+ <div style="width: auto; margin-left: auto; margin-right: auto">
32
+ <img src="https://i.imgur.com/EBdldam.jpg" alt="TheBlokeAI" style="width: 100%; min-width: 400px; display: block; margin: auto;">
33
+ </div>
34
+ <div style="display: flex; justify-content: space-between; width: 100%;">
35
+ <div style="display: flex; flex-direction: column; align-items: flex-start;">
36
+ <p style="margin-top: 0.5em; margin-bottom: 0em;"><a href="https://discord.gg/theblokeai">Chat & support: TheBloke's Discord server</a></p>
37
+ </div>
38
+ <div style="display: flex; flex-direction: column; align-items: flex-end;">
39
+ <p style="margin-top: 0.5em; margin-bottom: 0em;"><a href="https://www.patreon.com/TheBlokeAI">Want to contribute? TheBloke's Patreon page</a></p>
40
+ </div>
41
+ </div>
42
+ <div style="text-align:center; margin-top: 0em; margin-bottom: 0em"><p style="margin-top: 0.25em; margin-bottom: 0em;">TheBloke's LLM work is generously supported by a grant from <a href="https://a16z.com">andreessen horowitz (a16z)</a></p></div>
43
+ <hr style="margin-top: 1.0em; margin-bottom: 1.0em;">
44
+ <!-- header end -->
45
+
46
+ # StableLM Zephyr 3B - GPTQ
47
+ - Model creator: [Stability AI](https://huggingface.co/stabilityai)
48
+ - Original model: [StableLM Zephyr 3B](https://huggingface.co/stabilityai/stablelm-zephyr-3b)
49
+
50
+ <!-- description start -->
51
+ # Description
52
+
53
+ This repo contains GPTQ model files for [Stability AI's StableLM Zephyr 3B](https://huggingface.co/stabilityai/stablelm-zephyr-3b).
54
+
55
+ Multiple GPTQ parameter permutations are provided; see Provided Files below for details of the options provided, their parameters, and the software used to create them.
56
+
57
+ These files were quantised using hardware kindly provided by [Massed Compute](https://massedcompute.com/).
58
+
59
+ <!-- description end -->
60
+ <!-- repositories-available start -->
61
+ ## Repositories available
62
+
63
+ * [GPTQ models for GPU inference, with multiple quantisation parameter options.](https://huggingface.co/TheBloke/stablelm-zephyr-3b-GPTQ)
64
+ * [2, 3, 4, 5, 6 and 8-bit GGUF models for CPU+GPU inference](https://huggingface.co/TheBloke/stablelm-zephyr-3b-GGUF)
65
+ * [Stability AI's original unquantised fp16 model in pytorch format, for GPU inference and for further conversions](https://huggingface.co/stabilityai/stablelm-zephyr-3b)
66
+ <!-- repositories-available end -->
67
+
68
+ <!-- prompt-template start -->
69
+ ## Prompt template: StableLM-Zephyr
70
+
71
+ ```
72
+ <|user|>
73
+ {prompt}<|endoftext|>
74
+ <|assistant|>
75
+
76
+ ```
77
+
78
+ <!-- prompt-template end -->
79
+
80
+
81
+
82
+ <!-- README_GPTQ.md-compatible clients start -->
83
+ ## Known compatible clients / servers
84
+
85
+ GPTQ models are currently supported on Linux (NVidia/AMD) and Windows (NVidia only). macOS users: please use GGUF models.
86
+
87
+ These GPTQ models are known to work in the following inference servers/webuis.
88
+
89
+ - [text-generation-webui](https://github.com/oobabooga/text-generation-webui)
90
+ - [KoboldAI United](https://github.com/henk717/koboldai)
91
+ - [LoLLMS Web UI](https://github.com/ParisNeo/lollms-webui)
92
+ - [Hugging Face Text Generation Inference (TGI)](https://github.com/huggingface/text-generation-inference)
93
+
94
+ This may not be a complete list; if you know of others, please let me know!
95
+ <!-- README_GPTQ.md-compatible clients end -->
96
+
97
+ <!-- README_GPTQ.md-provided-files start -->
98
+ ## Provided files, and GPTQ parameters
99
+
100
+ Multiple quantisation parameters are provided, to allow you to choose the best one for your hardware and requirements.
101
+
102
+ Each separate quant is in a different branch. See below for instructions on fetching from different branches.
103
+
104
+ Most GPTQ files are made with AutoGPTQ. Mistral models are currently made with Transformers.
105
+
106
+ <details>
107
+ <summary>Explanation of GPTQ parameters</summary>
108
+
109
+ - Bits: The bit size of the quantised model.
110
+ - GS: GPTQ group size. Higher numbers use less VRAM, but have lower quantisation accuracy. "None" is the lowest possible value.
111
+ - Act Order: True or False. Also known as `desc_act`. True results in better quantisation accuracy. Some GPTQ clients have had issues with models that use Act Order plus Group Size, but this is generally resolved now.
112
+ - Damp %: A GPTQ parameter that affects how samples are processed for quantisation. 0.01 is default, but 0.1 results in slightly better accuracy.
113
+ - GPTQ dataset: The calibration dataset used during quantisation. Using a dataset more appropriate to the model's training can improve quantisation accuracy. Note that the GPTQ calibration dataset is not the same as the dataset used to train the model - please refer to the original model repo for details of the training dataset(s).
114
+ - Sequence Length: The length of the dataset sequences used for quantisation. Ideally this is the same as the model sequence length. For some very long sequence models (16+K), a lower sequence length may have to be used. Note that a lower sequence length does not limit the sequence length of the quantised model. It only impacts the quantisation accuracy on longer inference sequences.
115
+ - ExLlama Compatibility: Whether this file can be loaded with ExLlama, which currently only supports Llama and Mistral models in 4-bit.
116
+
117
+ </details>
118
+
119
+ | Branch | Bits | GS | Act Order | Damp % | GPTQ Dataset | Seq Len | Size | ExLlama | Desc |
120
+ | ------ | ---- | -- | --------- | ------ | ------------ | ------- | ---- | ------- | ---- |
121
+ | [main](https://huggingface.co/TheBloke/stablelm-zephyr-3b-GPTQ/tree/main) | 4 | 128 | Yes | 0.1 | [VMware Open Instruct](https://huggingface.co/datasets/VMware/open-instruct/viewer/) | 8192 | 1.84 GB | No | 4-bit, with Act Order and group size 128g. Uses even less VRAM than 64g, but with slightly lower accuracy. |
122
+ | [gptq-4bit-32g-actorder_True](https://huggingface.co/TheBloke/stablelm-zephyr-3b-GPTQ/tree/gptq-4bit-32g-actorder_True) | 4 | 32 | Yes | 0.1 | [VMware Open Instruct](https://huggingface.co/datasets/VMware/open-instruct/viewer/) | 8192 | 1.99 GB | No | 4-bit, with Act Order and group size 32g. Gives highest possible inference quality, with maximum VRAM usage. |
123
+ | [gptq-8bit--1g-actorder_True](https://huggingface.co/TheBloke/stablelm-zephyr-3b-GPTQ/tree/gptq-8bit--1g-actorder_True) | 8 | None | Yes | 0.1 | [VMware Open Instruct](https://huggingface.co/datasets/VMware/open-instruct/viewer/) | 8192 | 3.06 GB | No | 8-bit, with Act Order. No group size, to lower VRAM requirements. |
124
+ | [gptq-8bit-128g-actorder_True](https://huggingface.co/TheBloke/stablelm-zephyr-3b-GPTQ/tree/gptq-8bit-128g-actorder_True) | 8 | 128 | Yes | 0.1 | [VMware Open Instruct](https://huggingface.co/datasets/VMware/open-instruct/viewer/) | 8192 | 3.12 GB | No | 8-bit, with group size 128g for higher inference quality and with Act Order for even higher accuracy. |
125
+ | [gptq-8bit-32g-actorder_True](https://huggingface.co/TheBloke/stablelm-zephyr-3b-GPTQ/tree/gptq-8bit-32g-actorder_True) | 8 | 32 | Yes | 0.1 | [VMware Open Instruct](https://huggingface.co/datasets/VMware/open-instruct/viewer/) | 8192 | 3.30 GB | No | 8-bit, with group size 32g and Act Order for maximum inference quality. |
126
+ | [gptq-4bit-64g-actorder_True](https://huggingface.co/TheBloke/stablelm-zephyr-3b-GPTQ/tree/gptq-4bit-64g-actorder_True) | 4 | 64 | Yes | 0.1 | [VMware Open Instruct](https://huggingface.co/datasets/VMware/open-instruct/viewer/) | 8192 | 1.89 GB | No | 4-bit, with Act Order and group size 64g. Uses less VRAM than 32g, but with slightly lower accuracy. |
127
+
128
+ <!-- README_GPTQ.md-provided-files end -->
129
+
130
+ <!-- README_GPTQ.md-download-from-branches start -->
131
+ ## How to download, including from branches
132
+
133
+ ### In text-generation-webui
134
+
135
+ To download from the `main` branch, enter `TheBloke/stablelm-zephyr-3b-GPTQ` in the "Download model" box.
136
+
137
+ To download from another branch, add `:branchname` to the end of the download name, eg `TheBloke/stablelm-zephyr-3b-GPTQ:gptq-4bit-32g-actorder_True`
138
+
139
+ ### From the command line
140
+
141
+ I recommend using the `huggingface-hub` Python library:
142
+
143
+ ```shell
144
+ pip3 install huggingface-hub
145
+ ```
146
+
147
+ To download the `main` branch to a folder called `stablelm-zephyr-3b-GPTQ`:
148
+
149
+ ```shell
150
+ mkdir stablelm-zephyr-3b-GPTQ
151
+ huggingface-cli download TheBloke/stablelm-zephyr-3b-GPTQ --local-dir stablelm-zephyr-3b-GPTQ --local-dir-use-symlinks False
152
+ ```
153
+
154
+ To download from a different branch, add the `--revision` parameter:
155
+
156
+ ```shell
157
+ mkdir stablelm-zephyr-3b-GPTQ
158
+ huggingface-cli download TheBloke/stablelm-zephyr-3b-GPTQ --revision gptq-4bit-32g-actorder_True --local-dir stablelm-zephyr-3b-GPTQ --local-dir-use-symlinks False
159
+ ```
160
+
161
+ <details>
162
+ <summary>More advanced huggingface-cli download usage</summary>
163
+
164
+ If you remove the `--local-dir-use-symlinks False` parameter, the files will instead be stored in the central Hugging Face cache directory (default location on Linux is: `~/.cache/huggingface`), and symlinks will be added to the specified `--local-dir`, pointing to their real location in the cache. This allows for interrupted downloads to be resumed, and allows you to quickly clone the repo to multiple places on disk without triggering a download again. The downside, and the reason why I don't list that as the default option, is that the files are then hidden away in a cache folder and it's harder to know where your disk space is being used, and to clear it up if/when you want to remove a download model.
165
+
166
+ The cache location can be changed with the `HF_HOME` environment variable, and/or the `--cache-dir` parameter to `huggingface-cli`.
167
+
168
+ For more documentation on downloading with `huggingface-cli`, please see: [HF -> Hub Python Library -> Download files -> Download from the CLI](https://huggingface.co/docs/huggingface_hub/guides/download#download-from-the-cli).
169
+
170
+ To accelerate downloads on fast connections (1Gbit/s or higher), install `hf_transfer`:
171
+
172
+ ```shell
173
+ pip3 install hf_transfer
174
+ ```
175
+
176
+ And set environment variable `HF_HUB_ENABLE_HF_TRANSFER` to `1`:
177
+
178
+ ```shell
179
+ mkdir stablelm-zephyr-3b-GPTQ
180
+ HF_HUB_ENABLE_HF_TRANSFER=1 huggingface-cli download TheBloke/stablelm-zephyr-3b-GPTQ --local-dir stablelm-zephyr-3b-GPTQ --local-dir-use-symlinks False
181
+ ```
182
+
183
+ Windows Command Line users: You can set the environment variable by running `set HF_HUB_ENABLE_HF_TRANSFER=1` before the download command.
184
+ </details>
185
+
186
+ ### With `git` (**not** recommended)
187
+
188
+ To clone a specific branch with `git`, use a command like this:
189
+
190
+ ```shell
191
+ git clone --single-branch --branch gptq-4bit-32g-actorder_True https://huggingface.co/TheBloke/stablelm-zephyr-3b-GPTQ
192
+ ```
193
+
194
+ Note that using Git with HF repos is strongly discouraged. It will be much slower than using `huggingface-hub`, and will use twice as much disk space as it has to store the model files twice (it stores every byte both in the intended target folder, and again in the `.git` folder as a blob.)
195
+
196
+ <!-- README_GPTQ.md-download-from-branches end -->
197
+ <!-- README_GPTQ.md-text-generation-webui start -->
198
+ ## How to easily download and use this model in [text-generation-webui](https://github.com/oobabooga/text-generation-webui)
199
+
200
+ Please make sure you're using the latest version of [text-generation-webui](https://github.com/oobabooga/text-generation-webui).
201
+
202
+ It is strongly recommended to use the text-generation-webui one-click-installers unless you're sure you know how to make a manual install.
203
+
204
+ 1. Click the **Model tab**.
205
+ 2. Under **Download custom model or LoRA**, enter `TheBloke/stablelm-zephyr-3b-GPTQ`.
206
+
207
+ - To download from a specific branch, enter for example `TheBloke/stablelm-zephyr-3b-GPTQ:gptq-4bit-32g-actorder_True`
208
+ - see Provided Files above for the list of branches for each option.
209
+
210
+ 3. Click **Download**.
211
+ 4. The model will start downloading. Once it's finished it will say "Done".
212
+ 5. In the top left, click the refresh icon next to **Model**.
213
+ 6. In the **Model** dropdown, choose the model you just downloaded: `stablelm-zephyr-3b-GPTQ`
214
+ 7. The model will automatically load, and is now ready for use!
215
+ 8. If you want any custom settings, set them and then click **Save settings for this model** followed by **Reload the Model** in the top right.
216
+
217
+ - Note that you do not need to and should not set manual GPTQ parameters any more. These are set automatically from the file `quantize_config.json`.
218
+
219
+ 9. Once you're ready, click the **Text Generation** tab and enter a prompt to get started!
220
+
221
+ <!-- README_GPTQ.md-text-generation-webui end -->
222
+
223
+ <!-- README_GPTQ.md-use-from-tgi start -->
224
+ ## Serving this model from Text Generation Inference (TGI)
225
+
226
+ It's recommended to use TGI version 1.1.0 or later. The official Docker container is: `ghcr.io/huggingface/text-generation-inference:1.1.0`
227
+
228
+ Example Docker parameters:
229
+
230
+ ```shell
231
+ --model-id TheBloke/stablelm-zephyr-3b-GPTQ --port 3000 --quantize gptq --max-input-length 3696 --max-total-tokens 4096 --max-batch-prefill-tokens 4096
232
+ ```
233
+
234
+ Example Python code for interfacing with TGI (requires huggingface-hub 0.17.0 or later):
235
+
236
+ ```shell
237
+ pip3 install huggingface-hub
238
+ ```
239
+
240
+ ```python
241
+ from huggingface_hub import InferenceClient
242
+
243
+ endpoint_url = "https://your-endpoint-url-here"
244
+
245
+ prompt = "Tell me about AI"
246
+ prompt_template=f'''<|user|>
247
+ {prompt}<|endoftext|>
248
+ <|assistant|>
249
+ '''
250
+
251
+ client = InferenceClient(endpoint_url)
252
+ response = client.text_generation(prompt,
253
+ max_new_tokens=128,
254
+ do_sample=True,
255
+ temperature=0.7,
256
+ top_p=0.95,
257
+ top_k=40,
258
+ repetition_penalty=1.1)
259
+
260
+ print(f"Model output: {response}")
261
+ ```
262
+ <!-- README_GPTQ.md-use-from-tgi end -->
263
+ <!-- README_GPTQ.md-use-from-python start -->
264
+ ## Python code example: inference from this GPTQ model
265
+
266
+ ### Install the necessary packages
267
+
268
+ Requires: Transformers 4.33.0 or later, Optimum 1.12.0 or later, and AutoGPTQ 0.4.2 or later.
269
+
270
+ ```shell
271
+ pip3 install --upgrade transformers optimum
272
+ # If using PyTorch 2.1 + CUDA 12.x:
273
+ pip3 install --upgrade auto-gptq
274
+ # or, if using PyTorch 2.1 + CUDA 11.x:
275
+ pip3 install --upgrade auto-gptq --extra-index-url https://huggingface.github.io/autogptq-index/whl/cu118/
276
+ ```
277
+
278
+ If you are using PyTorch 2.0, you will need to install AutoGPTQ from source. Likewise if you have problems with the pre-built wheels, you should try building from source:
279
+
280
+ ```shell
281
+ pip3 uninstall -y auto-gptq
282
+ git clone https://github.com/PanQiWei/AutoGPTQ
283
+ cd AutoGPTQ
284
+ git checkout v0.5.1
285
+ pip3 install .
286
+ ```
287
+
288
+ ### Example Python code
289
+
290
+ ```python
291
+ from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline
292
+
293
+ model_name_or_path = "TheBloke/stablelm-zephyr-3b-GPTQ"
294
+ # To use a different branch, change revision
295
+ # For example: revision="gptq-4bit-32g-actorder_True"
296
+ model = AutoModelForCausalLM.from_pretrained(model_name_or_path,
297
+ device_map="auto",
298
+ trust_remote_code=True,
299
+ revision="main")
300
+
301
+ tokenizer = AutoTokenizer.from_pretrained(model_name_or_path, use_fast=True)
302
+
303
+ prompt = "Tell me about AI"
304
+ prompt_template=f'''<|user|>
305
+ {prompt}<|endoftext|>
306
+ <|assistant|>
307
+ '''
308
+
309
+ print("\n\n*** Generate:")
310
+
311
+ input_ids = tokenizer(prompt_template, return_tensors='pt').input_ids.cuda()
312
+ output = model.generate(inputs=input_ids, temperature=0.7, do_sample=True, top_p=0.95, top_k=40, max_new_tokens=512)
313
+ print(tokenizer.decode(output[0]))
314
+
315
+ # Inference can also be done using transformers' pipeline
316
+
317
+ print("*** Pipeline:")
318
+ pipe = pipeline(
319
+ "text-generation",
320
+ model=model,
321
+ tokenizer=tokenizer,
322
+ max_new_tokens=512,
323
+ do_sample=True,
324
+ temperature=0.7,
325
+ top_p=0.95,
326
+ top_k=40,
327
+ repetition_penalty=1.1
328
+ )
329
+
330
+ print(pipe(prompt_template)[0]['generated_text'])
331
+ ```
332
+ <!-- README_GPTQ.md-use-from-python end -->
333
+
334
+ <!-- README_GPTQ.md-compatibility start -->
335
+ ## Compatibility
336
+
337
+ The files provided are tested to work with Transformers. For non-Mistral models, AutoGPTQ can also be used directly.
338
+
339
+ [ExLlama](https://github.com/turboderp/exllama) is compatible with Llama and Mistral models in 4-bit. Please see the Provided Files table above for per-file compatibility.
340
+
341
+ For a list of clients/servers, please see "Known compatible clients / servers", above.
342
+ <!-- README_GPTQ.md-compatibility end -->
343
+
344
+ <!-- footer start -->
345
+ <!-- 200823 -->
346
+ ## Discord
347
+
348
+ For further support, and discussions on these models and AI in general, join us at:
349
+
350
+ [TheBloke AI's Discord server](https://discord.gg/theblokeai)
351
+
352
+ ## Thanks, and how to contribute
353
+
354
+ Thanks to the [chirper.ai](https://chirper.ai) team!
355
+
356
+ Thanks to Clay from [gpus.llm-utils.org](llm-utils)!
357
+
358
+ I've had a lot of people ask if they can contribute. I enjoy providing models and helping people, and would love to be able to spend even more time doing it, as well as expanding into new projects like fine tuning/training.
359
+
360
+ If you're able and willing to contribute it will be most gratefully received and will help me to keep providing more models, and to start work on new AI projects.
361
+
362
+ Donaters will get priority support on any and all AI/LLM/model questions and requests, access to a private Discord room, plus other benefits.
363
+
364
+ * Patreon: https://patreon.com/TheBlokeAI
365
+ * Ko-Fi: https://ko-fi.com/TheBlokeAI
366
+
367
+ **Special thanks to**: Aemon Algiz.
368
+
369
+ **Patreon special mentions**: Michael Levine, 阿明, Trailburnt, Nikolai Manek, John Detwiler, Randy H, Will Dee, Sebastain Graf, NimbleBox.ai, Eugene Pentland, Emad Mostaque, Ai Maven, Jim Angel, Jeff Scroggin, Michael Davis, Manuel Alberto Morcote, Stephen Murray, Robert, Justin Joy, Luke @flexchar, Brandon Frisco, Elijah Stavena, S_X, Dan Guido, Undi ., Komninos Chatzipapas, Shadi, theTransient, Lone Striker, Raven Klaugh, jjj, Cap'n Zoog, Michel-Marie MAUDET (LINAGORA), Matthew Berman, David, Fen Risland, Omer Bin Jawed, Luke Pendergrass, Kalila, OG, Erik Bjäreholt, Rooh Singh, Joseph William Delisle, Dan Lewis, TL, John Villwock, AzureBlack, Brad, Pedro Madruga, Caitlyn Gatomon, K, jinyuan sun, Mano Prime, Alex, Jeffrey Morgan, Alicia Loh, Illia Dulskyi, Chadd, transmissions 11, fincy, Rainer Wilmers, ReadyPlayerEmma, knownsqashed, Mandus, biorpg, Deo Leter, Brandon Phillips, SuperWojo, Sean Connelly, Iucharbius, Jack West, Harry Royden McLaughlin, Nicholas, terasurfer, Vitor Caleffi, Duane Dunston, Johann-Peter Hartmann, David Ziegler, Olakabola, Ken Nordquist, Trenton Dambrowitz, Tom X Nguyen, Vadim, Ajan Kanaga, Leonard Tan, Clay Pascal, Alexandros Triantafyllidis, JM33133, Xule, vamX, ya boyyy, subjectnull, Talal Aujan, Alps Aficionado, wassieverse, Ari Malik, James Bentley, Woland, Spencer Kim, Michael Dempsey, Fred von Graf, Elle, zynix, William Richards, Stanislav Ovsiannikov, Edmond Seymore, Jonathan Leane, Martin Kemka, usrbinkat, Enrico Ros
370
+
371
+
372
+ Thank you to all my generous patrons and donaters!
373
+
374
+ And thank you again to a16z for their generous grant.
375
+
376
+ <!-- footer end -->
377
+
378
+ # Original model card: Stability AI's StableLM Zephyr 3B
379
+
380
+ # `StableLM Zephyr 3B`
381
+
382
+ ## Model Description
383
+
384
+ `StableLM Zephyr 3B` is a 3 billion parameter instruction tuned inspired by [HugginFaceH4's Zephyr 7B](https://huggingface.co/HuggingFaceH4/zephyr-7b-beta) training pipeline this model was trained on a mix of publicly available datasets, synthetic datasets using [Direct Preference Optimization (DPO)](https://arxiv.org/abs/2305.18290), evaluation for this model based on
385
+ [MT Bench](https://tatsu-lab.github.io/alpaca_eval/) and [Alpaca Benchmark](https://tatsu-lab.github.io/alpaca_eval/)
386
+
387
+ ## Usage
388
+
389
+ `StableLM Zephyr 3B` uses the following instruction format:
390
+ ```
391
+ <|user|>
392
+ List 3 synonyms for the word "tiny"<|endoftext|>
393
+ <|assistant|>
394
+ 1. Dwarf
395
+ 2. Little
396
+ 3. Petite<|endoftext|>
397
+ ```
398
+
399
+ This format is also available through the tokenizer's `apply_chat_template` method:
400
+
401
+ ```python
402
+ from transformers import AutoModelForCausalLM, AutoTokenizer
403
+
404
+ tokenizer = AutoTokenizer.from_pretrained('stabilityai/stablelm-zephyr-3b')
405
+ model = AutoModelForCausalLM.from_pretrained(
406
+ 'stabilityai/stablelm-zephyr-3b',
407
+ trust_remote_code=True,
408
+ device_map="auto"
409
+ )
410
+
411
+ prompt = [{'role': 'user', 'content': 'List 3 synonyms for the word "tiny"'}]
412
+ inputs = tokenizer.apply_chat_template(
413
+ prompt,
414
+ add_generation_prompt=True,
415
+ return_tensors='pt'
416
+ )
417
+
418
+ tokens = model.generate(
419
+ inputs.to(model.device),
420
+ max_new_tokens=1024,
421
+ temperature=0.8,
422
+ do_sample=True
423
+ )
424
+
425
+ print(tokenizer.decode(tokens[0], skip_special_tokens=False))
426
+ ```
427
+
428
+ You can also see how to run a performance optimized version of this model [here](https://github.com/eaidova/openvino_notebooks/blob/ea/stateful_chatbot/notebooks/273-stable-zephyr-3b-chatbot/273-stable-zephyr-3b-chatbot.ipynb) using [OpenVINO](https://docs.openvino.ai/2023.2/home.html) from Intel.
429
+
430
+ ## Model Details
431
+
432
+ * **Developed by**: [Stability AI](https://stability.ai/)
433
+ * **Model type**: `StableLM Zephyr 3B` model is an auto-regressive language model based on the transformer decoder architecture.
434
+ * **Language(s)**: English
435
+ * **Library**: [Alignment Handbook](https://github.com/huggingface/alignment-handbook.git)
436
+ * **Finetuned from model**: [stabilityai/stablelm-3b-4e1t](https://huggingface.co/stabilityai/stablelm-3b-4e1t)
437
+ * **License**: [StabilityAI Non-Commercial Research Community License](https://huggingface.co/stabilityai/stablelm-zephyr-3b/raw/main/LICENSE)
438
+ * **Contact**: For questions and comments about the model, please email `lm@stability.ai`
439
+
440
+ ### Training Dataset
441
+
442
+ The dataset is comprised of a mixture of open datasets large-scale datasets available on the [HuggingFace Hub](https://huggingface.co/datasets):
443
+ 1. SFT Datasets
444
+ - HuggingFaceH4/ultrachat_200k
445
+ - meta-math/MetaMathQA
446
+ - WizardLM/WizardLM_evol_instruct_V2_196k
447
+ - Open-Orca/SlimOrca
448
+ 2. Preference Datasets:
449
+ - HuggingFaceH4/ultrafeedback_binarized
450
+ - Intel/orca_dpo_pairs
451
+
452
+ ## Performance
453
+
454
+ ### MT-Bench and Alpaca Bench
455
+
456
+
457
+ <img src="https://cdn-uploads.huggingface.co/production/uploads/6310474ca119d49bc1eb0d80/8WIZS6dAlu5kSH-382pMl.png" alt="mt_bench_plot" width="600"/>
458
+
459
+ | Model | Size | Alignment | MT-Bench (score) | AlpacaEval (win rate %) |
460
+ |-------------|-----|----|---------------|--------------|
461
+ | **StableLM Zephyr 3B** 🪁 | 3B | DPO | 6.64 | 76.00 |
462
+ | StableLM Zephyr (SFT only) | 3B | SFT | 6.04 | 71.15 |
463
+ | Capybara v1.9 | 3B | dSFT | 5.94 | - |
464
+ | MPT-Chat | 7B |dSFT |5.42| -|
465
+ | Xwin-LM v0.1 | 7B| dPPO| 6.19| 87.83|
466
+ | Mistral-Instruct v0.1 | 7B| - | 6.84 |-|
467
+ | Zephyr-7b-α |7B| dDPO| 6.88| -|
468
+ | Zephyr-7b-β| 7B | dDPO | 7.34 | 90.60 |
469
+ | Falcon-Instruct | 40B |dSFT |5.17 |45.71|
470
+ | Guanaco | 65B | SFT |6.41| 71.80|
471
+ | Llama2-Chat | 70B |RLHF |6.86| 92.66|
472
+ | Vicuna v1.3 | 33B |dSFT |7.12 |88.99|
473
+ | WizardLM v1.0 | 70B |dSFT |7.71 |-|
474
+ | Xwin-LM v0.1 | 70B |dPPO |- |95.57|
475
+ | GPT-3.5-turbo | - |RLHF |7.94 |89.37|
476
+ | Claude 2 | - |RLHF |8.06| 91.36|
477
+ | GPT-4 | -| RLHF |8.99| 95.28|
478
+
479
+ ## Other benchmarks:
480
+ | Task | Value |
481
+ |-----------------------|---------------------------|
482
+ | ARC (25-shot) | 47.0 |
483
+ | HellaSwag (10-shot) | 74.2 |
484
+ | MMLU (5-shot) | 46.3 |
485
+ | TruthfulQA (0-shot) | 46.5 |
486
+ | Winogrande (5-shot) | 65.5 |
487
+ | GSM8K (5-shot) | 42.3 |
488
+ | BigBench (Avg) | 35.26 |
489
+ | AGI Benchmark (Avg) | 33.23 |
490
+
491
+ ### Training Infrastructure
492
+
493
+ * **Hardware**: `StableLM Zephyr 3B` was trained on the Stability AI cluster across 8 nodes with 8 A100 80GBs GPUs for each nodes.
494
+ * **Code Base**: We use our internal script for SFT steps and used [HuggingFace Alignment Handbook script](https://github.com/huggingface/alignment-handbook) for DPO training.
495
+
496
+ ## Commitment to Ethical AI
497
+ In line with our responsibility towards ethical AI development, `StableLM Zephyr 3B` is released with a focus on ensuring safety, reliability, and appropriateness in its applications. To this end, we have evaluated `StableLM Zephyr 3B` on 488 malicious prompts and used standard protocols to assess the harmfulness of its outputs. Compared to Zephyr-7b-β, `StableLM Zephyr 3B` reduces the number of harmful outputs as assessed by GPT-4 by 55. Additionally, we performed an internal red teaming event targeting the following abuse areas:
498
+ * **Self-Harm Methods**: (Suicide Methods, Encouragement of Self-Harm, Methods and encouragement of Eating Disorders)
499
+ * **Misinformation**: (Health, Conspiracy Theories, Social Unrest/Conflict, Political Misinformation, & Climate change)
500
+ * **Hate Speech**: (Race, Stereotypes, Immigrants, Gender, Personally Identifiable Information such as Social security numbers, Full names, ID numbers, Email addresses, and telephone numbers)
501
+
502
+ We have incorporated the findings of our malicious prompts evaluation and red teaming event into our release. Users are encouraged to fine-tune and evaluate the model to suit their specific needs, considering the potential biases and limitations found in `StableLM Zephyr 3B` and inherent in other LLM models.
503
+
504
+ ## Use and Limitations
505
+
506
+ ### Intended Use
507
+
508
+ The model is intended to be used as a foundational base model for application-specific fine-tuning. Developers must evaluate and fine-tune the model for safe performance in downstream applications.
509
+
510
+ ### Limitations and Bias
511
+
512
+ This model is not trained against adversarial inputs. We strongly recommend pairing this model with an input and output classifier to prevent harmful responses.
513
+
514
+ Through our internal red teaming, we discovered that while the model will not output harmful information if not prompted to do so, it is willing to output potentially harmful outputs or misinformation when the user requests it. Using this model will require guardrails around your inputs and outputs to ensure that any outputs returned are not misinformation or harmful. Additionally, as each use case is unique, we recommend running your own suite of tests to ensure proper performance of this model. Finally, do not use the models if they are unsuitable for your application, or for any applications that may cause deliberate or unintentional harm to others.
config (1).json ADDED
@@ -0,0 +1,57 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "_name_or_path": "/workspace/process/stabilityai_stablelm-zephyr-3b/source",
3
+ "architectures": [
4
+ "StableLMEpochForCausalLM"
5
+ ],
6
+ "auto_map": {
7
+ "AutoConfig": "configuration_stablelm_epoch.StableLMEpochConfig",
8
+ "AutoModelForCausalLM": "modeling_stablelm_epoch.StableLMEpochForCausalLM"
9
+ },
10
+ "bos_token_id": 0,
11
+ "eos_token_id": 0,
12
+ "hidden_act": "silu",
13
+ "hidden_size": 2560,
14
+ "initializer_range": 0.02,
15
+ "intermediate_size": 6912,
16
+ "max_position_embeddings": 4096,
17
+ "model_type": "stablelm_epoch",
18
+ "norm_eps": 1e-05,
19
+ "num_attention_heads": 32,
20
+ "num_heads": 32,
21
+ "num_hidden_layers": 32,
22
+ "num_key_value_heads": 32,
23
+ "pad_token_id": 0,
24
+ "pretraining_tp": 1,
25
+ "quantization_config": {
26
+ "batch_size": 1,
27
+ "bits": 4,
28
+ "block_name_to_quantize": "model.layers",
29
+ "cache_block_outputs": true,
30
+ "damp_percent": 0.1,
31
+ "desc_act": true,
32
+ "exllama_config": {
33
+ "version": 1
34
+ },
35
+ "group_size": 128,
36
+ "max_input_length": null,
37
+ "model_seqlen": 8192,
38
+ "module_name_preceding_first_block": [
39
+ "model.embed_tokens"
40
+ ],
41
+ "pad_token_id": null,
42
+ "quant_method": "gptq",
43
+ "sym": true,
44
+ "tokenizer": null,
45
+ "true_sequential": true,
46
+ "use_cuda_fp16": false,
47
+ "use_exllama": true
48
+ },
49
+ "rope_pct": 0.25,
50
+ "rope_theta": 10000,
51
+ "rotary_scaling_factor": 1.0,
52
+ "tie_word_embeddings": false,
53
+ "torch_dtype": "bfloat16",
54
+ "transformers_version": "4.35.2",
55
+ "use_cache": true,
56
+ "vocab_size": 50304
57
+ }
configuration_stablelm_epoch.py ADDED
@@ -0,0 +1,110 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # coding=utf-8
2
+ # Copyright 2023 Stability and The HuggingFace Inc. team. All rights reserved.
3
+ #
4
+ # Licensed under the Apache License, Version 2.0 (the "License");
5
+ # you may not use this file except in compliance with the License.
6
+ # You may obtain a copy of the License at
7
+ #
8
+ # http://www.apache.org/licenses/LICENSE-2.0
9
+ #
10
+ # Unless required by applicable law or agreed to in writing, software
11
+ # distributed under the License is distributed on an "AS IS" BASIS,
12
+ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13
+ # See the License for the specific language governing permissions and
14
+ # limitations under the License.
15
+ """ StableLM Epoch model configuration"""
16
+ from transformers import PretrainedConfig
17
+ from transformers.utils import logging
18
+
19
+
20
+ logger = logging.get_logger(__name__)
21
+
22
+
23
+ class StableLMEpochConfig(PretrainedConfig):
24
+ r"""
25
+ Configuration objects inherit from [`PretrainedConfig`] and can be used to control the model outputs. Read the
26
+ documentation from [`PretrainedConfig`] for more information.
27
+
28
+ Args:
29
+ vocab_size (`int`, *optional*, defaults to 50_304):
30
+ Vocabulary size of the StableLM model. Defines the number of different tokens that
31
+ can be represented by the `inputs_ids` passed when calling [`StableLMEpochModel`].
32
+ intermediate_size (`int`, *optional*, defaults to 6912):
33
+ Dimension of the MLP representations.
34
+ hidden_size (`int`, *optional*, defaults to 2560):
35
+ Dimension of the decoder layers and the pooler layer.
36
+ num_hidden_layers (`int`, *optional*, defaults to 32):
37
+ Number of hidden layers in the Transformer decoder.
38
+ num_attention_heads (`int`, *optional*, defaults to 32):
39
+ Number of attention heads for each attention layer in the Transformer encoder.
40
+ num_key_value_heads (`int`, *optional*):
41
+ This is the number of key_value heads that should be used to implement Grouped Query Attention. If
42
+ `num_key_value_heads=num_attention_heads`, the model will use Multi Head Attention (MHA), if
43
+ `num_key_value_heads=1 the model will use Multi Query Attention (MQA) otherwise GQA is used. When
44
+ converting a multi-head checkpoint to a GQA checkpoint, each group key and value head should be constructed
45
+ by meanpooling all the original heads within that group. For more details checkout [this
46
+ paper](https://arxiv.org/pdf/2305.13245.pdf). If it is not specified, will default to
47
+ `num_attention_heads`.
48
+ hidden_act (`str` or `function`, *optional*, defaults to `"silu"`):
49
+ The non-linear activation function (function or string).
50
+ rope_pct (`float`, *optional*, defaults to 1.0):
51
+ Percentage of hidden dimensions to allocate to rotary embeddings.
52
+ rope_theta (`float`, *optional*, defaults to 10000.0):
53
+ The base period of the RoPE embeddings.
54
+ max_position_embeddings (`int`, *optional*, defaults to 2048):
55
+ The maximum sequence length that this model might ever be used with.
56
+ Typically set this to something large just in case (e.g., 512 or 1024 or 2048).
57
+ initializer_range (`float`, *optional*, defaults to 1e-5):
58
+ The standard deviation of the truncated_normal_initializer for initializing
59
+ all weight matrices.
60
+ norm_eps (`float`, *optional*, defaults to 1e-8):
61
+ The epsilon used by the normalization layers.
62
+ use_cache (`bool`, *optional*, defaults to `True`):
63
+ Whether or not the model should return the last key/values attentions
64
+ (not used by all models). Only relevant if `config.is_decoder=True`.
65
+ tie_word_embeddings(`bool`, *optional*, defaults to `False`):
66
+ Whether to tie weight embeddings
67
+ """
68
+ model_type = "stablelm_epoch"
69
+ keys_to_ignore_at_inference = ["past_key_values"]
70
+
71
+ def __init__(
72
+ self,
73
+ vocab_size=50_304,
74
+ intermediate_size=6912,
75
+ hidden_size=2560,
76
+ num_hidden_layers=32,
77
+ num_attention_heads=32,
78
+ num_key_value_heads=32,
79
+ hidden_act="silu",
80
+ rope_pct=0.25,
81
+ rope_theta=10_000,
82
+ max_position_embeddings=4096,
83
+ initializer_range=0.02,
84
+ norm_eps=1.0e-5,
85
+ use_cache=True,
86
+ bos_token_id=0,
87
+ eos_token_id=2,
88
+ tie_word_embeddings=False,
89
+ **kwargs,
90
+ ):
91
+ self.vocab_size = vocab_size
92
+ self.max_position_embeddings = max_position_embeddings
93
+ self.intermediate_size = intermediate_size
94
+ self.hidden_size = hidden_size
95
+ self.num_hidden_layers = num_hidden_layers
96
+ self.num_attention_heads = num_attention_heads
97
+ self.num_key_value_heads = num_key_value_heads
98
+ self.hidden_act = hidden_act
99
+ self.rope_pct = rope_pct
100
+ self.rope_theta = rope_theta
101
+ self.initializer_range = initializer_range
102
+ self.norm_eps = norm_eps
103
+ self.use_cache = use_cache
104
+ self.tie_word_embeddings = tie_word_embeddings
105
+ super().__init__(
106
+ bos_token_id=bos_token_id,
107
+ eos_token_id=eos_token_id,
108
+ tie_word_embeddings=tie_word_embeddings,
109
+ **kwargs,
110
+ )
generation_config.json ADDED
@@ -0,0 +1,6 @@
 
 
 
 
 
 
 
1
+ {
2
+ "_from_model_config": true,
3
+ "bos_token_id": 0,
4
+ "eos_token_id": 0,
5
+ "transformers_version": "4.35.2"
6
+ }
modeling_stablelm_epoch.py ADDED
@@ -0,0 +1,687 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # coding=utf-8
2
+ # Copyright 2023 Stability AI, EleutherAI, and The HuggingFace Inc. team. All rights reserved.
3
+ #
4
+ # Licensed under the Apache License, Version 2.0 (the "License");
5
+ # you may not use this file except in compliance with the License.
6
+ # You may obtain a copy of the License at
7
+ #
8
+ # http://www.apache.org/licenses/LICENSE-2.0
9
+ #
10
+ # Unless required by applicable law or agreed to in writing, software
11
+ # distributed under the License is distributed on an "AS IS" BASIS,
12
+ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13
+ # See the License for the specific language governing permissions and
14
+ # limitations under the License.
15
+ #
16
+ # This code is based off the following work:
17
+ # https://github.com/huggingface/transformers/blob/main/src/transformers/models/llama/modeling_llama.py
18
+ # https://github.com/huggingface/transformers/blob/main/src/transformers/models/gpt_neox/modeling_gpt_neox.py
19
+ """ PyTorch StableLM Epoch model. """
20
+ from typing import Optional, Tuple, Union
21
+ import math
22
+
23
+ import torch
24
+ import torch.utils.checkpoint
25
+ from torch import nn
26
+ from torch.nn import CrossEntropyLoss
27
+ from transformers.modeling_outputs import (
28
+ BaseModelOutputWithPast,
29
+ CausalLMOutputWithPast,
30
+ )
31
+ from transformers.modeling_utils import PreTrainedModel
32
+ from transformers.utils import logging
33
+ from .configuration_stablelm_epoch import StableLMEpochConfig
34
+
35
+
36
+ logger = logging.get_logger(__name__)
37
+
38
+
39
+ # Copied from transformers.models.bart.modeling_bart._make_causal_mask
40
+ def _make_causal_mask(
41
+ input_ids_shape: torch.Size,
42
+ dtype: torch.dtype,
43
+ device: torch.device,
44
+ past_key_values_length: int = 0,
45
+ ):
46
+ """Make causal mask used for bi-directional self-attention."""
47
+ batch_size, tgt_len = input_ids_shape
48
+ mask = torch.full((tgt_len, tgt_len), torch.finfo(torch.float16).min, device=device)
49
+ mask_cond = torch.arange(mask.size(-1), device=device)
50
+ mask.masked_fill_(mask_cond < (mask_cond + 1).view(mask.size(-1), 1), 0)
51
+ mask = mask.to(dtype)
52
+ if past_key_values_length > 0:
53
+ mask = torch.cat([torch.zeros(tgt_len, past_key_values_length, dtype=dtype, device=device), mask], dim=-1)
54
+ return mask[None, None, :, :].expand(batch_size, 1, tgt_len, tgt_len + past_key_values_length)
55
+
56
+
57
+ # Copied from transformers.models.bart.modeling_bart._expand_mask
58
+ def _expand_mask(mask: torch.Tensor, dtype: torch.dtype, tgt_len: Optional[int] = None):
59
+ """Expands attention_mask from `[batch_size, seq_len]` to `[batch_size, 1, tgt_seq_len, src_seq_len]`."""
60
+ batch_size, src_len = mask.size()
61
+ tgt_len = tgt_len if tgt_len is not None else src_len
62
+
63
+ expanded_mask = mask[:, None, None, :].expand(batch_size, 1, tgt_len, src_len).to(dtype)
64
+ inverted_mask = 1.0 - expanded_mask
65
+
66
+ return inverted_mask.masked_fill(
67
+ inverted_mask.to(torch.bool), torch.finfo(dtype).min
68
+ )
69
+
70
+
71
+ class RotaryEmbedding(nn.Module):
72
+ def __init__(
73
+ self,
74
+ dim: int,
75
+ max_position_embeddings: int,
76
+ base: int = 10_000,
77
+ device: Optional[torch.device] = None,
78
+ ):
79
+ super().__init__()
80
+
81
+ self.dim = dim
82
+ self.max_position_embeddings = max_position_embeddings
83
+ self.base = base
84
+ inv_freq = 1.0 / (self.base ** (torch.arange(0, self.dim, 2, device=device, dtype=torch.float32) / self.dim))
85
+ self.register_buffer("inv_freq", inv_freq, persistent=False)
86
+
87
+ # Build here to make `torch.jit.trace` work.
88
+ self._set_cos_sin_cache(
89
+ seq_len=max_position_embeddings, device=self.inv_freq.device, dtype=torch.get_default_dtype(),
90
+ )
91
+
92
+ def _set_cos_sin_cache(self, seq_len: int, device: torch.device, dtype: torch.dtype):
93
+ self.max_seq_len_cached = seq_len
94
+ t = torch.arange(self.max_seq_len_cached, device=device, dtype=torch.float32)
95
+
96
+ # Don't do einsum, it converts fp32 to fp16 under AMP
97
+ # freqs = torch.einsum("i,j->ij", t, self.inv_freq)
98
+ freqs = torch.outer(t, self.inv_freq)
99
+ # Different from paper, but it uses a different permutation in order to obtain the same calculation
100
+ emb = torch.cat((freqs, freqs), dim=-1)
101
+ self.register_buffer("cos_cached", emb.cos()[None, None, :, :].to(dtype), persistent=False)
102
+ self.register_buffer("sin_cached", emb.sin()[None, None, :, :].to(dtype), persistent=False)
103
+
104
+ def forward(self, x: torch.Tensor, seq_len: Optional[int] = None):
105
+ # x: [batch_size, num_heads, seq_len, head_size]
106
+ if seq_len > self.max_seq_len_cached:
107
+ self._set_cos_sin_cache(seq_len=seq_len, device=x.device, dtype=torch.get_default_dtype())
108
+ return (
109
+ self.cos_cached[:, :, :seq_len, ...].to(dtype=x.dtype),
110
+ self.sin_cached[:, :, :seq_len, ...].to(dtype=x.dtype),
111
+ )
112
+
113
+
114
+ def rotate_half(x: torch.Tensor):
115
+ """Rotates half the hidden dims of the input."""
116
+ x1, x2 = torch.chunk(x, 2, dim=-1)
117
+ return torch.cat((-x2, x1), dim=-1)
118
+
119
+
120
+ def apply_rotary_pos_emb(q, k, cos, sin, position_ids):
121
+ # The first two dimensions of cos and sin are always 1, so we can `squeeze` them.
122
+ cos = cos.squeeze(1).squeeze(0) # [seq_len, dim]
123
+ sin = sin.squeeze(1).squeeze(0) # [seq_len, dim]
124
+ cos = cos[position_ids].unsqueeze(1) # [batch_size, 1, seq_len, dim]
125
+ sin = sin[position_ids].unsqueeze(1) # [batch_size, 1, seq_len, dim]
126
+ q_embed = (q * cos) + (rotate_half(q) * sin)
127
+ k_embed = (k * cos) + (rotate_half(k) * sin)
128
+ return q_embed, k_embed
129
+
130
+
131
+ class MLP(nn.Module):
132
+ def __init__(self, config: StableLMEpochConfig):
133
+ super().__init__()
134
+ self.config = config
135
+ self.hidden_size = config.hidden_size
136
+ self.intermediate_size = config.intermediate_size
137
+ self.gate_proj = nn.Linear(config.hidden_size, config.intermediate_size, bias=False)
138
+ self.up_proj = nn.Linear(config.hidden_size, config.intermediate_size, bias=False)
139
+ self.down_proj = nn.Linear(config.intermediate_size, config.hidden_size, bias=False)
140
+ self.act_fn = nn.SiLU()
141
+
142
+ def forward(self, x: torch.Tensor) -> torch.Tensor:
143
+ return self.down_proj(self.act_fn(self.gate_proj(x)) * self.up_proj(x))
144
+
145
+
146
+ def repeat_kv(hidden_states: torch.Tensor, n_rep: int) -> torch.Tensor:
147
+ """
148
+ This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch,
149
+ num_key_value_heads, seqlen, head_dim) to (batch, num_attention_heads, seqlen, head_dim)
150
+ """
151
+ batch, num_key_value_heads, slen, head_dim = hidden_states.shape
152
+ if n_rep == 1:
153
+ return hidden_states
154
+ hidden_states = hidden_states[:, :, None, :, :].expand(batch, num_key_value_heads, n_rep, slen, head_dim)
155
+ return hidden_states.reshape(batch, num_key_value_heads * n_rep, slen, head_dim)
156
+
157
+
158
+ class Attention(nn.Module):
159
+ def __init__(self, config: StableLMEpochConfig):
160
+ super().__init__()
161
+ self.config = config
162
+ self.hidden_size = config.hidden_size
163
+ self.num_heads = config.num_attention_heads
164
+ self.head_dim = self.hidden_size // self.num_heads
165
+ self.num_key_value_heads = config.num_key_value_heads
166
+ self.num_key_value_groups = self.num_heads // self.num_key_value_heads
167
+ self.max_position_embeddings = config.max_position_embeddings
168
+
169
+ if (self.head_dim * self.num_heads) != self.hidden_size:
170
+ raise ValueError(
171
+ f"hidden_size must be divisible by num_heads (got `hidden_size`: {self.hidden_size}"
172
+ f" and `num_heads`: {self.num_heads})."
173
+ )
174
+ self.q_proj = nn.Linear(self.hidden_size, self.num_heads * self.head_dim, bias=False)
175
+ self.k_proj = nn.Linear(self.hidden_size, self.num_key_value_heads * self.head_dim, bias=False)
176
+ self.v_proj = nn.Linear(self.hidden_size, self.num_key_value_heads * self.head_dim, bias=False)
177
+ self.o_proj = nn.Linear(self.hidden_size, self.hidden_size, bias=False)
178
+
179
+ self._init_rope()
180
+
181
+ def _init_rope(self):
182
+ self.rotary_ndims = int(self.head_dim * self.config.rope_pct)
183
+ self.rotary_emb = RotaryEmbedding(
184
+ self.rotary_ndims,
185
+ max_position_embeddings=self.config.max_position_embeddings,
186
+ base=self.config.rope_theta,
187
+ )
188
+
189
+ def forward(
190
+ self,
191
+ hidden_states: torch.FloatTensor,
192
+ attention_mask: torch.FloatTensor,
193
+ position_ids: torch.LongTensor,
194
+ past_key_value: Optional[Tuple[torch.Tensor]] = None,
195
+ output_attentions: Optional[bool] = False,
196
+ use_cache: Optional[bool] = False,
197
+ ) -> Tuple[torch.Tensor, Optional[torch.Tensor], Optional[Tuple[torch.Tensor]]]:
198
+ bsz, q_len, _ = hidden_states.size()
199
+
200
+ query_states = self.q_proj(hidden_states)
201
+ key_states = self.k_proj(hidden_states)
202
+ value_states = self.v_proj(hidden_states)
203
+
204
+ query_states = query_states.view(bsz, q_len, self.num_heads, self.head_dim).transpose(1, 2)
205
+ key_states = key_states.view(bsz, q_len, self.num_key_value_heads, self.head_dim).transpose(1, 2)
206
+ value_states = value_states.view(bsz, q_len, self.num_key_value_heads, self.head_dim).transpose(1, 2)
207
+
208
+ query_rot = query_states[..., : self.rotary_ndims]
209
+ query_pass = query_states[..., self.rotary_ndims :]
210
+ key_rot = key_states[..., : self.rotary_ndims]
211
+ key_pass = key_states[..., self.rotary_ndims :]
212
+
213
+ kv_seq_len = key_states.shape[-2]
214
+ if past_key_value is not None:
215
+ kv_seq_len += past_key_value[0].shape[-2]
216
+ cos, sin = self.rotary_emb(value_states, seq_len=kv_seq_len)
217
+ query_states, key_states = apply_rotary_pos_emb(query_rot, key_rot, cos, sin, position_ids)
218
+
219
+ # [batch_size, num_heads, seq_len, head_dim]
220
+ query_states = torch.cat((query_states, query_pass), dim=-1)
221
+ key_states = torch.cat((key_states, key_pass), dim=-1)
222
+
223
+ if past_key_value is not None:
224
+ # Reuse k, v, self_attention
225
+ key_states = torch.cat((past_key_value[0], key_states), dim=2)
226
+ value_states = torch.cat((past_key_value[1], value_states), dim=2)
227
+
228
+ past_key_value = (key_states, value_states) if use_cache else None
229
+
230
+ # Repeat k/v heads if n_kv_heads < n_heads
231
+ key_states = repeat_kv(key_states, self.num_key_value_groups)
232
+ value_states = repeat_kv(value_states, self.num_key_value_groups)
233
+
234
+ attn_weights = torch.matmul(query_states, key_states.transpose(2, 3)) / math.sqrt(self.head_dim)
235
+
236
+ if attn_weights.size() != (bsz, self.num_heads, q_len, kv_seq_len):
237
+ raise ValueError(
238
+ f"Attention weights should be of size {(bsz, self.num_heads, q_len, kv_seq_len)}, but is"
239
+ f" {attn_weights.size()}"
240
+ )
241
+
242
+ if attention_mask is not None:
243
+ if attention_mask.size() != (bsz, 1, q_len, kv_seq_len):
244
+ raise ValueError(
245
+ f"Attention mask should be of size {(bsz, 1, q_len, kv_seq_len)}, but is {attention_mask.size()}"
246
+ )
247
+ attn_weights = attn_weights + attention_mask
248
+
249
+ # Upcast attention to fp32
250
+ attn_weights = nn.functional.softmax(attn_weights, dim=-1, dtype=torch.float32).to(query_states.dtype)
251
+ attn_output = torch.matmul(attn_weights, value_states)
252
+
253
+ if attn_output.size() != (bsz, self.num_heads, q_len, self.head_dim):
254
+ raise ValueError(
255
+ f"`attn_output` should be of size {(bsz, self.num_heads, q_len, self.head_dim)}, but is"
256
+ f" {attn_output.size()}"
257
+ )
258
+
259
+ # Merge heads
260
+ attn_output = attn_output.transpose(1, 2).contiguous()
261
+ attn_output = attn_output.reshape(bsz, q_len, self.hidden_size)
262
+
263
+ # Final linear projection
264
+ attn_output = self.o_proj(attn_output)
265
+
266
+ if not output_attentions:
267
+ attn_weights = None
268
+
269
+ return attn_output, attn_weights, past_key_value
270
+
271
+
272
+ class DecoderLayer(nn.Module):
273
+ def __init__(self, config: StableLMEpochConfig):
274
+ super().__init__()
275
+ self.self_attn = Attention(config)
276
+ self.mlp = MLP(config)
277
+ self.input_layernorm = nn.LayerNorm(config.hidden_size, eps=config.norm_eps)
278
+ self.post_attention_layernorm = nn.LayerNorm(config.hidden_size, eps=config.norm_eps)
279
+
280
+ def forward(
281
+ self,
282
+ hidden_states: Optional[torch.FloatTensor],
283
+ attention_mask: Optional[torch.FloatTensor] = None,
284
+ position_ids: Optional[torch.LongTensor] = None,
285
+ past_key_value: Optional[Tuple[torch.Tensor]] = None,
286
+ output_attentions: Optional[bool] = False,
287
+ use_cache: Optional[bool] = False,
288
+ ) -> Union[Tuple[torch.Tensor], Optional[Tuple[torch.Tensor, Tuple[torch.FloatTensor, ...]]]]:
289
+ residual = hidden_states
290
+
291
+ hidden_states = self.input_layernorm(hidden_states)
292
+
293
+ # Self Attention
294
+ hidden_states, self_attn_weights, present_key_value = self.self_attn(
295
+ hidden_states=hidden_states,
296
+ attention_mask=attention_mask,
297
+ position_ids=position_ids,
298
+ past_key_value=past_key_value,
299
+ output_attentions=output_attentions,
300
+ use_cache=use_cache,
301
+ )
302
+ hidden_states = residual + hidden_states
303
+
304
+ # Fully Connected
305
+ residual = hidden_states
306
+ hidden_states = self.post_attention_layernorm(hidden_states)
307
+ hidden_states = self.mlp(hidden_states)
308
+ hidden_states = residual + hidden_states
309
+
310
+ outputs = (hidden_states,)
311
+
312
+ if output_attentions:
313
+ outputs += (self_attn_weights,)
314
+
315
+ if use_cache:
316
+ outputs += (present_key_value,)
317
+
318
+ return outputs
319
+
320
+
321
+ class StableLMEpochPreTrainedModel(PreTrainedModel):
322
+ """An abstract class to handle weights initialization and a simple interface
323
+ for downloading and loading pretrained models.
324
+ """
325
+
326
+ config_class = StableLMEpochConfig
327
+ base_model_prefix = "transformer"
328
+ supports_gradient_checkpointing = True
329
+ _no_split_modules = ["DecoderLayer"]
330
+ _skip_keys_device_placement = "past_key_values"
331
+
332
+ def _init_weights(self, module: nn.Module):
333
+ """Initialize the weights"""
334
+ if isinstance(module, nn.Linear):
335
+ module.weight.data.normal_(mean=0.0, std=self.config.initializer_range)
336
+ if module.bias is not None:
337
+ module.bias.data.zero_()
338
+ elif isinstance(module, nn.Embedding):
339
+ module.weight.data.normal_(mean=0.0, std=self.config.initializer_range)
340
+ if module.padding_idx is not None:
341
+ module.weight.data[module.padding_idx].zero_()
342
+ elif isinstance(module, nn.LayerNorm):
343
+ module.bias.data.zero_()
344
+ module.weight.data.fill_(1.0)
345
+
346
+ def _set_gradient_checkpointing(self, module: nn.Module, value=False):
347
+ if isinstance(module, StableLMEpochModel):
348
+ module.gradient_checkpointing = value
349
+
350
+
351
+ class StableLMEpochModel(StableLMEpochPreTrainedModel):
352
+ def __init__(self, config: StableLMEpochConfig):
353
+ super().__init__(config)
354
+ self.embed_tokens = nn.Embedding(config.vocab_size, config.hidden_size, config.pad_token_id)
355
+ self.layers = nn.ModuleList([DecoderLayer(config) for _ in range(config.num_hidden_layers)])
356
+ self.norm = nn.LayerNorm(config.hidden_size, eps=config.norm_eps)
357
+
358
+ self.gradient_checkpointing = False
359
+ # Initialize weights and apply final processing
360
+ self.post_init()
361
+
362
+ def get_input_embeddings(self):
363
+ return self.embed_tokens
364
+
365
+ def set_input_embeddings(self, value: nn.Module):
366
+ self.embed_tokens = value
367
+
368
+ # Copied from transformers.models.bart.modeling_bart.BartDecoder._prepare_decoder_attention_mask
369
+ def _prepare_decoder_attention_mask(
370
+ self,
371
+ attention_mask: torch.Tensor,
372
+ input_shape: torch.Size,
373
+ inputs_embeds: torch.Tensor,
374
+ past_key_values_length: int,
375
+ ):
376
+ # Create causal mask
377
+ # [batch_size, seq_len] -> [batch_size, 1, tgt_seq_len, src_seq_len]
378
+ combined_attention_mask = None
379
+ if input_shape[-1] > 1:
380
+ combined_attention_mask = _make_causal_mask(
381
+ input_shape,
382
+ inputs_embeds.dtype,
383
+ device=inputs_embeds.device,
384
+ past_key_values_length=past_key_values_length,
385
+ )
386
+
387
+ if attention_mask is not None:
388
+ # [batch_size, seq_len] -> [batch_size, 1, tgt_seq_len, src_seq_len]
389
+ expanded_attn_mask = _expand_mask(
390
+ attention_mask, inputs_embeds.dtype, tgt_len=input_shape[-1]
391
+ ).to(inputs_embeds.device)
392
+ combined_attention_mask = expanded_attn_mask if combined_attention_mask is None else expanded_attn_mask + combined_attention_mask
393
+
394
+ return combined_attention_mask
395
+
396
+ def forward(
397
+ self,
398
+ input_ids: Optional[torch.LongTensor] = None,
399
+ attention_mask: Optional[torch.FloatTensor] = None,
400
+ position_ids: Optional[torch.LongTensor] = None,
401
+ past_key_values: Optional[Tuple[Tuple[torch.FloatTensor]]] = None,
402
+ inputs_embeds: Optional[torch.FloatTensor] = None,
403
+ use_cache: Optional[bool] = None,
404
+ output_attentions: Optional[bool] = None,
405
+ output_hidden_states: Optional[bool] = None,
406
+ return_dict: Optional[bool] = None,
407
+ ) -> Union[Tuple, BaseModelOutputWithPast]:
408
+ output_attentions = output_attentions if output_attentions is not None else self.config.output_attentions
409
+ output_hidden_states = output_hidden_states if output_hidden_states is not None else self.config.output_hidden_states
410
+ use_cache = use_cache if use_cache is not None else self.config.use_cache
411
+
412
+ return_dict = return_dict if return_dict is not None else self.config.use_return_dict
413
+
414
+ # Retrieve input_ids and inputs_embeds
415
+ if input_ids is not None and inputs_embeds is not None:
416
+ raise ValueError(
417
+ "You cannot specify both decoder_input_ids and decoder_inputs_embeds at the same time"
418
+ )
419
+ elif input_ids is not None:
420
+ batch_size, seq_length = input_ids.shape
421
+ elif inputs_embeds is not None:
422
+ batch_size, seq_length, _ = inputs_embeds.shape
423
+ else:
424
+ raise ValueError(
425
+ "You have to specify either decoder_input_ids or decoder_inputs_embeds"
426
+ )
427
+
428
+ seq_length_with_past = seq_length
429
+ past_key_values_length = 0
430
+
431
+ if past_key_values is not None:
432
+ past_key_values_length = past_key_values[0][0].shape[2]
433
+ seq_length_with_past = seq_length_with_past + past_key_values_length
434
+
435
+ if position_ids is None:
436
+ device = input_ids.device if input_ids is not None else inputs_embeds.device
437
+ position_ids = torch.arange(
438
+ past_key_values_length,
439
+ seq_length + past_key_values_length,
440
+ dtype=torch.long,
441
+ device=device,
442
+ )
443
+ position_ids = position_ids.unsqueeze(0).view(-1, seq_length)
444
+ else:
445
+ position_ids = position_ids.view(-1, seq_length).long()
446
+
447
+ if inputs_embeds is None:
448
+ inputs_embeds = self.embed_tokens(input_ids)
449
+ # Embed positions
450
+ if attention_mask is None:
451
+ attention_mask = torch.ones(
452
+ (batch_size, seq_length_with_past),
453
+ dtype=torch.bool,
454
+ device=inputs_embeds.device,
455
+ )
456
+ attention_mask = self._prepare_decoder_attention_mask(
457
+ attention_mask,
458
+ (batch_size, seq_length),
459
+ inputs_embeds,
460
+ past_key_values_length,
461
+ )
462
+
463
+ hidden_states = inputs_embeds
464
+
465
+ if self.gradient_checkpointing and self.training:
466
+ if use_cache:
467
+ logger.warning(
468
+ "`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`..."
469
+ )
470
+ use_cache = False
471
+
472
+ # Decoder layers
473
+ all_hidden_states = () if output_hidden_states else None
474
+ all_self_attns = () if output_attentions else None
475
+ next_decoder_cache = () if use_cache else None
476
+
477
+ for idx, decoder_layer in enumerate(self.layers):
478
+ if output_hidden_states:
479
+ all_hidden_states += (hidden_states,)
480
+
481
+ past_key_value = (
482
+ past_key_values[idx] if past_key_values is not None else None
483
+ )
484
+
485
+ if self.gradient_checkpointing and self.training:
486
+
487
+ def create_custom_forward(module):
488
+ def custom_forward(*inputs):
489
+ # None for past_key_value
490
+ return module(*inputs, past_key_value, output_attentions)
491
+
492
+ return custom_forward
493
+
494
+ layer_outputs = torch.utils.checkpoint.checkpoint(
495
+ create_custom_forward(decoder_layer),
496
+ hidden_states,
497
+ attention_mask,
498
+ position_ids,
499
+ )
500
+ else:
501
+ layer_outputs = decoder_layer(
502
+ hidden_states,
503
+ attention_mask=attention_mask,
504
+ position_ids=position_ids,
505
+ past_key_value=past_key_value,
506
+ output_attentions=output_attentions,
507
+ use_cache=use_cache,
508
+ )
509
+
510
+ hidden_states = layer_outputs[0]
511
+
512
+ if use_cache:
513
+ next_decoder_cache += (layer_outputs[2 if output_attentions else 1],)
514
+
515
+ if output_attentions:
516
+ all_self_attns += (layer_outputs[1],)
517
+
518
+ hidden_states = self.norm(hidden_states)
519
+
520
+ # Add hidden states from the last decoder layer
521
+ if output_hidden_states:
522
+ all_hidden_states += (hidden_states,)
523
+
524
+ next_cache = next_decoder_cache if use_cache else None
525
+ if not return_dict:
526
+ return tuple(
527
+ v
528
+ for v in [hidden_states, next_cache, all_hidden_states, all_self_attns]
529
+ if v is not None
530
+ )
531
+ return BaseModelOutputWithPast(
532
+ last_hidden_state=hidden_states,
533
+ past_key_values=next_cache,
534
+ hidden_states=all_hidden_states,
535
+ attentions=all_self_attns,
536
+ )
537
+
538
+
539
+ class StableLMEpochForCausalLM(StableLMEpochPreTrainedModel):
540
+ _tied_weights_keys = ["lm_head.weight"]
541
+
542
+ def __init__(self, config: StableLMEpochConfig):
543
+ super().__init__(config)
544
+
545
+ self.model = StableLMEpochModel(config)
546
+ self.lm_head = nn.Linear(config.hidden_size, config.vocab_size, bias=False)
547
+
548
+ # Initialize weights and apply final processing
549
+ self.post_init()
550
+
551
+ def get_input_embeddings(self):
552
+ return self.model.embed_tokens
553
+
554
+ def set_input_embeddings(self, value):
555
+ self.model.embed_tokens = value
556
+
557
+ def get_output_embeddings(self):
558
+ return self.lm_head
559
+
560
+ def set_output_embeddings(self, new_embeddings: nn.Module):
561
+ self.lm_head = new_embeddings
562
+
563
+ def get_decoder(self):
564
+ return self.model
565
+
566
+ def set_decoder(self, decoder):
567
+ self.model = decoder
568
+
569
+ def forward(
570
+ self,
571
+ input_ids: Optional[torch.LongTensor] = None,
572
+ attention_mask: Optional[torch.FloatTensor] = None,
573
+ position_ids: Optional[torch.LongTensor] = None,
574
+ past_key_values: Optional[Tuple[Tuple[torch.FloatTensor]]] = None,
575
+ inputs_embeds: Optional[torch.FloatTensor] = None,
576
+ labels: Optional[torch.LongTensor] = None,
577
+ use_cache: Optional[bool] = None,
578
+ output_attentions: Optional[bool] = None,
579
+ output_hidden_states: Optional[bool] = None,
580
+ return_dict: Optional[bool] = None,
581
+ ) -> Union[Tuple, CausalLMOutputWithPast]:
582
+ output_attentions = (
583
+ output_attentions
584
+ if output_attentions is not None
585
+ else self.config.output_attentions
586
+ )
587
+ output_hidden_states = (
588
+ output_hidden_states
589
+ if output_hidden_states is not None
590
+ else self.config.output_hidden_states
591
+ )
592
+ return_dict = (
593
+ return_dict if return_dict is not None else self.config.use_return_dict
594
+ )
595
+
596
+ # decoder outputs consists of (dec_features, layer_state, dec_hidden, dec_attn)
597
+ outputs = self.model(
598
+ input_ids,
599
+ attention_mask=attention_mask,
600
+ position_ids=position_ids,
601
+ past_key_values=past_key_values,
602
+ inputs_embeds=inputs_embeds,
603
+ use_cache=use_cache,
604
+ output_attentions=output_attentions,
605
+ output_hidden_states=output_hidden_states,
606
+ return_dict=return_dict,
607
+ )
608
+
609
+ hidden_states = outputs[0]
610
+ logits = self.lm_head(hidden_states).float()
611
+
612
+ loss = None
613
+ if labels is not None:
614
+ # Shift so that tokens < n predict n
615
+ shift_logits = logits[..., :-1, :].contiguous()
616
+ shift_labels = labels[..., 1:].contiguous()
617
+ # Flatten the tokens
618
+ loss_fct = CrossEntropyLoss()
619
+ shift_logits = shift_logits.view(-1, self.config.vocab_size)
620
+ shift_labels = shift_labels.view(-1)
621
+ # Enable model parallelism
622
+ shift_labels = shift_labels.to(shift_logits.device)
623
+ loss = loss_fct(shift_logits, shift_labels)
624
+
625
+ if not return_dict:
626
+ output = (logits,) + outputs[1:]
627
+ return (loss,) + output if loss is not None else output
628
+
629
+ return CausalLMOutputWithPast(
630
+ loss=loss,
631
+ logits=logits,
632
+ past_key_values=outputs.past_key_values,
633
+ hidden_states=outputs.hidden_states,
634
+ attentions=outputs.attentions,
635
+ )
636
+
637
+ def prepare_inputs_for_generation(
638
+ self,
639
+ input_ids,
640
+ past_key_values: Optional[torch.Tensor] = None,
641
+ attention_mask: Optional[torch.Tensor] = None,
642
+ inputs_embeds: Optional[torch.Tensor] = None,
643
+ **kwargs,
644
+ ):
645
+ # Trim decoder_input_ids if past is used
646
+ if past_key_values and past_key_values[0] is not None:
647
+ input_ids = input_ids[:, -1:]
648
+
649
+ position_ids = kwargs.get("position_ids", None)
650
+ if attention_mask is not None and position_ids is None:
651
+ # Create position_ids on the fly for batch generation
652
+ position_ids = attention_mask.long().cumsum(-1) - 1
653
+ position_ids.masked_fill_(attention_mask == 0, 1)
654
+ if past_key_values:
655
+ position_ids = position_ids[:, -1].unsqueeze(-1)
656
+
657
+ # If `inputs_embeds` are passed, we only want to use them in the 1st generation step
658
+ if inputs_embeds is not None and past_key_values is None:
659
+ model_inputs = {"inputs_embeds": inputs_embeds}
660
+ else:
661
+ model_inputs = {"input_ids": input_ids}
662
+
663
+ model_inputs.update(
664
+ {
665
+ "attention_mask": attention_mask,
666
+ "past_key_values": past_key_values,
667
+ "use_cache": kwargs.get("use_cache"),
668
+ "position_ids": position_ids,
669
+ }
670
+ )
671
+ return model_inputs
672
+
673
+ @staticmethod
674
+ def _reorder_cache(past_key_values, beam_idx):
675
+ reordered_past = ()
676
+ for layer_past in past_key_values:
677
+ reordered_past += (
678
+ tuple(
679
+ past_state.index_select(0, beam_idx.to(past_state.device))
680
+ for past_state in layer_past
681
+ ),
682
+ )
683
+ return reordered_past
684
+
685
+
686
+ StableLMEpochConfig.register_for_auto_class()
687
+ StableLMEpochForCausalLM.register_for_auto_class("AutoModelForCausalLM")
quantize_config.json ADDED
@@ -0,0 +1,16 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "bits": [
3
+ 4
4
+ ],
5
+ "group_size": [
6
+ 128
7
+ ],
8
+ "damp_percent": [
9
+ 0.1
10
+ ],
11
+ "desc_act": [
12
+ true
13
+ ],
14
+ "sym": true,
15
+ "true_sequential": true
16
+ }
special_tokens_map.json ADDED
@@ -0,0 +1,30 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "bos_token": {
3
+ "content": "<|endoftext|>",
4
+ "lstrip": false,
5
+ "normalized": false,
6
+ "rstrip": false,
7
+ "single_word": false
8
+ },
9
+ "eos_token": {
10
+ "content": "<|endoftext|>",
11
+ "lstrip": false,
12
+ "normalized": false,
13
+ "rstrip": false,
14
+ "single_word": false
15
+ },
16
+ "pad_token": {
17
+ "content": "<|endoftext|>",
18
+ "lstrip": false,
19
+ "normalized": false,
20
+ "rstrip": false,
21
+ "single_word": false
22
+ },
23
+ "unk_token": {
24
+ "content": "<|endoftext|>",
25
+ "lstrip": false,
26
+ "normalized": false,
27
+ "rstrip": false,
28
+ "single_word": false
29
+ }
30
+ }
tokenizer.json ADDED
The diff for this file is too large to render. See raw diff
 
tokenizer_config.json ADDED
@@ -0,0 +1,213 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "add_prefix_space": false,
3
+ "added_tokens_decoder": {
4
+ "0": {
5
+ "content": "<|endoftext|>",
6
+ "lstrip": false,
7
+ "normalized": false,
8
+ "rstrip": false,
9
+ "single_word": false,
10
+ "special": true
11
+ },
12
+ "1": {
13
+ "content": "<|padding|>",
14
+ "lstrip": false,
15
+ "normalized": false,
16
+ "rstrip": false,
17
+ "single_word": false,
18
+ "special": true
19
+ },
20
+ "50254": {
21
+ "content": " ",
22
+ "lstrip": false,
23
+ "normalized": true,
24
+ "rstrip": false,
25
+ "single_word": false,
26
+ "special": false
27
+ },
28
+ "50255": {
29
+ "content": " ",
30
+ "lstrip": false,
31
+ "normalized": true,
32
+ "rstrip": false,
33
+ "single_word": false,
34
+ "special": false
35
+ },
36
+ "50256": {
37
+ "content": " ",
38
+ "lstrip": false,
39
+ "normalized": true,
40
+ "rstrip": false,
41
+ "single_word": false,
42
+ "special": false
43
+ },
44
+ "50257": {
45
+ "content": " ",
46
+ "lstrip": false,
47
+ "normalized": true,
48
+ "rstrip": false,
49
+ "single_word": false,
50
+ "special": false
51
+ },
52
+ "50258": {
53
+ "content": " ",
54
+ "lstrip": false,
55
+ "normalized": true,
56
+ "rstrip": false,
57
+ "single_word": false,
58
+ "special": false
59
+ },
60
+ "50259": {
61
+ "content": " ",
62
+ "lstrip": false,
63
+ "normalized": true,
64
+ "rstrip": false,
65
+ "single_word": false,
66
+ "special": false
67
+ },
68
+ "50260": {
69
+ "content": " ",
70
+ "lstrip": false,
71
+ "normalized": true,
72
+ "rstrip": false,
73
+ "single_word": false,
74
+ "special": false
75
+ },
76
+ "50261": {
77
+ "content": " ",
78
+ "lstrip": false,
79
+ "normalized": true,
80
+ "rstrip": false,
81
+ "single_word": false,
82
+ "special": false
83
+ },
84
+ "50262": {
85
+ "content": " ",
86
+ "lstrip": false,
87
+ "normalized": true,
88
+ "rstrip": false,
89
+ "single_word": false,
90
+ "special": false
91
+ },
92
+ "50263": {
93
+ "content": " ",
94
+ "lstrip": false,
95
+ "normalized": true,
96
+ "rstrip": false,
97
+ "single_word": false,
98
+ "special": false
99
+ },
100
+ "50264": {
101
+ "content": " ",
102
+ "lstrip": false,
103
+ "normalized": true,
104
+ "rstrip": false,
105
+ "single_word": false,
106
+ "special": false
107
+ },
108
+ "50265": {
109
+ "content": " ",
110
+ "lstrip": false,
111
+ "normalized": true,
112
+ "rstrip": false,
113
+ "single_word": false,
114
+ "special": false
115
+ },
116
+ "50266": {
117
+ "content": " ",
118
+ "lstrip": false,
119
+ "normalized": true,
120
+ "rstrip": false,
121
+ "single_word": false,
122
+ "special": false
123
+ },
124
+ "50267": {
125
+ "content": " ",
126
+ "lstrip": false,
127
+ "normalized": true,
128
+ "rstrip": false,
129
+ "single_word": false,
130
+ "special": false
131
+ },
132
+ "50268": {
133
+ "content": " ",
134
+ "lstrip": false,
135
+ "normalized": true,
136
+ "rstrip": false,
137
+ "single_word": false,
138
+ "special": false
139
+ },
140
+ "50269": {
141
+ "content": " ",
142
+ "lstrip": false,
143
+ "normalized": true,
144
+ "rstrip": false,
145
+ "single_word": false,
146
+ "special": false
147
+ },
148
+ "50270": {
149
+ "content": " ",
150
+ "lstrip": false,
151
+ "normalized": true,
152
+ "rstrip": false,
153
+ "single_word": false,
154
+ "special": false
155
+ },
156
+ "50271": {
157
+ "content": " ",
158
+ "lstrip": false,
159
+ "normalized": true,
160
+ "rstrip": false,
161
+ "single_word": false,
162
+ "special": false
163
+ },
164
+ "50272": {
165
+ "content": " ",
166
+ "lstrip": false,
167
+ "normalized": true,
168
+ "rstrip": false,
169
+ "single_word": false,
170
+ "special": false
171
+ },
172
+ "50273": {
173
+ "content": " ",
174
+ "lstrip": false,
175
+ "normalized": true,
176
+ "rstrip": false,
177
+ "single_word": false,
178
+ "special": false
179
+ },
180
+ "50274": {
181
+ "content": " ",
182
+ "lstrip": false,
183
+ "normalized": true,
184
+ "rstrip": false,
185
+ "single_word": false,
186
+ "special": false
187
+ },
188
+ "50275": {
189
+ "content": " ",
190
+ "lstrip": false,
191
+ "normalized": true,
192
+ "rstrip": false,
193
+ "single_word": false,
194
+ "special": false
195
+ },
196
+ "50276": {
197
+ "content": " ",
198
+ "lstrip": false,
199
+ "normalized": true,
200
+ "rstrip": false,
201
+ "single_word": false,
202
+ "special": false
203
+ }
204
+ },
205
+ "bos_token": "<|endoftext|>",
206
+ "chat_template": "{% for message in messages %}\n{% if message['role'] == 'user' %}\n{{ '<|user|>\n' + message['content'] + eos_token }}\n{% elif message['role'] == 'system' %}\n{{ '<|system|>\n' + message['content'] + eos_token }}\n{% elif message['role'] == 'assistant' %}\n{{ '<|assistant|>\n' + message['content'] + eos_token }}\n{% endif %}\n{% if loop.last and add_generation_prompt %}\n{{ '<|assistant|>' }}\n{% endif %}\n{% endfor %}",
207
+ "clean_up_tokenization_spaces": true,
208
+ "eos_token": "<|endoftext|>",
209
+ "model_max_length": 2048,
210
+ "pad_token": "<|endoftext|>",
211
+ "tokenizer_class": "GPTNeoXTokenizer",
212
+ "unk_token": "<|endoftext|>"
213
+ }