TheBloke commited on
Commit
93af7f0
1 Parent(s): 38e197c

Initial GPTQ model commit

Browse files
Files changed (1) hide show
  1. README.md +211 -0
README.md ADDED
@@ -0,0 +1,211 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ inference: false
3
+ license: other
4
+ ---
5
+
6
+ <!-- header start -->
7
+ <div style="width: 100%;">
8
+ <img src="https://i.imgur.com/EBdldam.jpg" alt="TheBlokeAI" style="width: 100%; min-width: 400px; display: block; margin: auto;">
9
+ </div>
10
+ <div style="display: flex; justify-content: space-between; width: 100%;">
11
+ <div style="display: flex; flex-direction: column; align-items: flex-start;">
12
+ <p><a href="https://discord.gg/theblokeai">Chat & support: my new Discord server</a></p>
13
+ </div>
14
+ <div style="display: flex; flex-direction: column; align-items: flex-end;">
15
+ <p><a href="https://www.patreon.com/TheBlokeAI">Want to contribute? TheBloke's Patreon page</a></p>
16
+ </div>
17
+ </div>
18
+ <!-- header end -->
19
+
20
+ # Eric Hartford's Samantha 33B GPTQ
21
+
22
+ These files are GPTQ 4bit model files for [Eric Hartford's Samantha 33B](https://huggingface.co/ehartford/samantha-1.1-llama-33b) merged with [Kaio Ken's SuperHOT 8K](https://huggingface.co/kaiokendev/superhot-30b-8k-no-rlhf-test).
23
+
24
+ It is the result of quantising to 4bit using [GPTQ-for-LLaMa](https://github.com/qwopqwop200/GPTQ-for-LLaMa).
25
+
26
+ **This is an experimental new GPTQ which offers up to 8K context size**
27
+
28
+ The increased context is tested to work with [ExLlama](https://github.com/turboderp/exllama), via the latest release of [text-generation-webui](https://github.com/oobabooga/text-generation-webui).
29
+
30
+ It has also been tested from Python code using AutoGPTQ, and `trust_remote_code=True`.
31
+
32
+ Code credits:
33
+ - Original concept and code for increasing context length: [kaiokendev](https://huggingface.co/kaiokendev)
34
+ - Updated Llama modelling code that includes this automatically via trust_remote_code: [emozilla](https://huggingface.co/emozilla).
35
+
36
+ Please read carefully below to see how to use it.
37
+
38
+ **NOTE**: Using the full 8K context on a 30B model will exceed 24GB VRAM.
39
+
40
+ GGML versions are not yet provided, as there is not yet support for SuperHOT in llama.cpp. This is being investigated and will hopefully come soon.
41
+
42
+ ## Repositories available
43
+
44
+ * [4-bit GPTQ models for GPU inference](https://huggingface.co/TheBloke/Samantha-33B-SuperHOT-8K-GPTQ)
45
+ * [Unquantised SuperHOT fp16 model in pytorch format, for GPU inference and for further conversions](https://huggingface.co/TheBloke/Samantha-33B-SuperHOT-8K-fp16)
46
+ * [Unquantised base fp16 model in pytorch format, for GPU inference and for further conversions](https://huggingface.co/ehartford/samantha-1.1-llama-33b)
47
+
48
+ ## How to easily download and use this model in text-generation-webui with ExLlama
49
+
50
+ Please make sure you're using the latest version of text-generation-webui
51
+
52
+ 1. Click the **Model tab**.
53
+ 2. Under **Download custom model or LoRA**, enter `TheBloke/Samantha-33B-SuperHOT-8K-GPTQ`.
54
+ 3. Click **Download**.
55
+ 4. The model will start downloading. Once it's finished it will say "Done"
56
+ 5. Untick **Autoload the model**
57
+ 6. In the top left, click the refresh icon next to **Model**.
58
+ 7. In the **Model** dropdown, choose the model you just downloaded: `Samantha-33B-SuperHOT-8K-GPTQ`
59
+ 8. To use the increased context, set the **Loader** to **ExLlama**, set **max_seq_len** to 8192 or 4096, and set **compress_pos_emb** to **4** for 8192 context, or to **2** for 4096 context.
60
+ 9. Now click **Save Settings** followed by **Reload**
61
+ 10. The model will automatically load, and is now ready for use!
62
+ 11. Once you're ready, click the **Text Generation tab** and enter a prompt to get started!
63
+
64
+ ## How to use this GPTQ model from Python code with AutoGPTQ
65
+
66
+ First make sure you have AutoGPTQ and Einops installed:
67
+
68
+ ```
69
+ pip3 install einops auto-gptq
70
+ ```
71
+
72
+ Then run the following code. Note that in order to get this to work, `config.json` has been hardcoded to a sequence length of 8192.
73
+
74
+ If you want to try 4096 instead to reduce VRAM usage, please manually edit `config.json` to set `max_position_embeddings` to the value you want.
75
+
76
+ ```python
77
+ from transformers import AutoTokenizer, pipeline, logging
78
+ from auto_gptq import AutoGPTQForCausalLM, BaseQuantizeConfig
79
+ import argparse
80
+
81
+ model_name_or_path = "TheBloke/Samantha-33B-SuperHOT-8K-GPTQ"
82
+ model_basename = "samantha-33b-superhot-8k-GPTQ-4bit-128g.no-act.order"
83
+
84
+ use_triton = False
85
+
86
+ tokenizer = AutoTokenizer.from_pretrained(model_name_or_path, use_fast=True)
87
+
88
+ model = AutoGPTQForCausalLM.from_quantized(model_name_or_path,
89
+ model_basename=model_basename,
90
+ use_safetensors=True,
91
+ trust_remote_code=True,
92
+ device_map='auto',
93
+ use_triton=use_triton,
94
+ quantize_config=None)
95
+
96
+ model.seqlen = 8192
97
+
98
+ # Note: check the prompt template is correct for this model.
99
+ prompt = "Tell me about AI"
100
+ prompt_template=f'''USER: {prompt}
101
+ ASSISTANT:'''
102
+
103
+ print("\n\n*** Generate:")
104
+
105
+ input_ids = tokenizer(prompt_template, return_tensors='pt').input_ids.cuda()
106
+ output = model.generate(inputs=input_ids, temperature=0.7, max_new_tokens=512)
107
+ print(tokenizer.decode(output[0]))
108
+
109
+ # Inference can also be done using transformers' pipeline
110
+
111
+ # Prevent printing spurious transformers error when using pipeline with AutoGPTQ
112
+ logging.set_verbosity(logging.CRITICAL)
113
+
114
+ print("*** Pipeline:")
115
+ pipe = pipeline(
116
+ "text-generation",
117
+ model=model,
118
+ tokenizer=tokenizer,
119
+ max_new_tokens=512,
120
+ temperature=0.7,
121
+ top_p=0.95,
122
+ repetition_penalty=1.15
123
+ )
124
+
125
+ print(pipe(prompt_template)[0]['generated_text'])
126
+ ```
127
+
128
+ ## Using other UIs: monkey patch
129
+
130
+ Provided in the repo is `llama_rope_scaled_monkey_patch.py`, written by @kaiokendev.
131
+
132
+ It can be theoretically be added to any Python UI or custom code to enable the same result as `trust_remote_code=True`. I have not tested this, and it should be superseded by using `trust_remote_code=True`, but I include it for completeness and for interest.
133
+
134
+ ## Provided files
135
+
136
+ **samantha-33b-superhot-8k-GPTQ-4bit-128g.no-act.order.safetensors**
137
+
138
+ This will work with AutoGPTQ, ExLlama, and CUDA versions of GPTQ-for-LLaMa. There are reports of issues with Triton mode of recent GPTQ-for-LLaMa. If you have issues, please use AutoGPTQ instead.
139
+
140
+ It was created with group_size 128 to increase inference accuracy, but without --act-order (desc_act) to increase compatibility and improve inference speed.
141
+
142
+ * `samantha-33b-superhot-8k-GPTQ-4bit-128g.no-act.order.safetensors`
143
+ * Works for use with ExLlama with increased context (4096 or 8192)
144
+ * Works with AutoGPTQ in Python code, including with increased context, if `trust_remote_code=True` is set.
145
+ * Should work with GPTQ-for-LLaMa in CUDA mode, but unknown if increased context works - TBC. May have issues with GPTQ-for-LLaMa Triton mode.
146
+ * Works with text-generation-webui, including one-click-installers.
147
+ * Parameters: Groupsize = 128. Act Order / desc_act = False.
148
+
149
+ <!-- footer start -->
150
+ ## Discord
151
+
152
+ For further support, and discussions on these models and AI in general, join us at:
153
+
154
+ [TheBloke AI's Discord server](https://discord.gg/theblokeai)
155
+
156
+ ## Thanks, and how to contribute.
157
+
158
+ Thanks to the [chirper.ai](https://chirper.ai) team!
159
+
160
+ I've had a lot of people ask if they can contribute. I enjoy providing models and helping people, and would love to be able to spend even more time doing it, as well as expanding into new projects like fine tuning/training.
161
+
162
+ If you're able and willing to contribute it will be most gratefully received and will help me to keep providing more models, and to start work on new AI projects.
163
+
164
+ Donaters will get priority support on any and all AI/LLM/model questions and requests, access to a private Discord room, plus other benefits.
165
+
166
+ * Patreon: https://patreon.com/TheBlokeAI
167
+ * Ko-Fi: https://ko-fi.com/TheBlokeAI
168
+
169
+ **Special thanks to**: Luke from CarbonQuill, Aemon Algiz, Dmitriy Samsonov.
170
+
171
+ **Patreon special mentions**: Pyrater, WelcomeToTheClub, Kalila, Mano Prime, Trenton Dambrowitz, Spiking Neurons AB, Pierre Kircher, Fen Risland, Kevin Schuppel, Luke, Rainer Wilmers, vamX, Gabriel Puliatti, Alex , Karl Bernard, Ajan Kanaga, Talal Aujan, Space Cruiser, ya boyyy, biorpg, Johann-Peter Hartmann, Asp the Wyvern, Ai Maven, Ghost , Preetika Verma, Nikolai Manek, trip7s trip, John Detwiler, Fred von Graf, Artur Olbinski, subjectnull, John Villwock, Junyu Yang, Rod A, Lone Striker, Chris McCloskey, Iucharbius , Matthew Berman, Illia Dulskyi, Khalefa Al-Ahmad, Imad Khwaja, chris gileta, Willem Michiel, Greatston Gnanesh, Derek Yates, K, Alps Aficionado, Oscar Rangel, David Flickinger, Luke Pendergrass, Deep Realms, Eugene Pentland, Cory Kujawski, terasurfer , Jonathan Leane, senxiiz, Joseph William Delisle, Sean Connelly, webtim, zynix , Nathan LeClaire.
172
+
173
+ Thank you to all my generous patrons and donaters!
174
+
175
+ <!-- footer end -->
176
+
177
+ # Original model card: Kaio Ken's SuperHOT 8K
178
+
179
+ ### SuperHOT Prototype 2 w/ 8K Context
180
+
181
+ This is a second prototype of SuperHOT, this time 30B with 8K context and no RLHF, using the same technique described in [the github blog](https://kaiokendev.github.io/til#extending-context-to-8k).
182
+ Tests have shown that the model does indeed leverage the extended context at 8K.
183
+
184
+ You will need to **use either the monkeypatch** or, if you are already using the monkeypatch, **change the scaling factor to 0.25 and the maximum sequence length to 8192**
185
+
186
+ #### Looking for Merged & Quantized Models?
187
+ - 30B 4-bit CUDA: [tmpupload/superhot-30b-8k-4bit-safetensors](https://huggingface.co/tmpupload/superhot-30b-8k-4bit-safetensors)
188
+ - 30B 4-bit CUDA 128g: [tmpupload/superhot-30b-8k-4bit-128g-safetensors](https://huggingface.co/tmpupload/superhot-30b-8k-4bit-128g-safetensors)
189
+
190
+
191
+ #### Training Details
192
+ I trained the LoRA with the following configuration:
193
+ - 1200 samples (~400 samples over 2048 sequence length)
194
+ - learning rate of 3e-4
195
+ - 3 epochs
196
+ - The exported modules are:
197
+ - q_proj
198
+ - k_proj
199
+ - v_proj
200
+ - o_proj
201
+ - no bias
202
+ - Rank = 4
203
+ - Alpha = 8
204
+ - no dropout
205
+ - weight decay of 0.1
206
+ - AdamW beta1 of 0.9 and beta2 0.99, epsilon of 1e-5
207
+ - Trained on 4-bit base model
208
+
209
+ # Original model card: Eric Hartford's Samantha 33B
210
+
211
+ No original model card was provided.