Update for Transformers GPTQ support

README.md

@@ -9,17 +9,20 @@ tags:
 ---
 
 <!-- header start -->
-<div style="width: 100%;">
-    <img src="https://i.imgur.com/EBdldam.jpg" alt="TheBlokeAI" style="width: 100%; min-width: 400px; display: block; margin: auto;">
+<!-- 200823 -->
+<div style="width: auto; margin-left: auto; margin-right: auto">
+<img src="https://i.imgur.com/EBdldam.jpg" alt="TheBlokeAI" style="width: 100%; min-width: 400px; display: block; margin: auto;">
 </div>
 <div style="display: flex; justify-content: space-between; width: 100%;">
     <div style="display: flex; flex-direction: column; align-items: flex-start;">
-        <p><a href="https://discord.gg/theblokeai">Chat & support: my new Discord server</a></p>
+        <p style="margin-top: 0.5em; margin-bottom: 0em;"><a href="https://discord.gg/theblokeai">Chat & support: TheBloke's Discord server</a></p>
     </div>
     <div style="display: flex; flex-direction: column; align-items: flex-end;">
-        <p><a href="https://www.patreon.com/TheBlokeAI">Want to contribute? TheBloke's Patreon page</a></p>
+        <p style="margin-top: 0.5em; margin-bottom: 0em;"><a href="https://www.patreon.com/TheBlokeAI">Want to contribute? TheBloke's Patreon page</a></p>
     </div>
 </div>
+<div style="text-align:center; margin-top: 0em; margin-bottom: 0em"><p style="margin-top: 0.25em; margin-bottom: 0em;">TheBloke's LLM work is generously supported by a grant from <a href="https://a16z.com">andreessen horowitz (a16z)</a></p></div>
+<hr style="margin-top: 1.0em; margin-bottom: 1.0em;">
 <!-- header end -->
 
 # Elinas' Chronos 33B merged with Kaio Ken's SuperHOT 8K GPTQ
@@ -154,6 +157,7 @@ It was created without group_size to lower VRAM requirements, and with --act-ord
   * Parameters: Groupsize = -1. Act Order / desc_act = True.
 
 <!-- footer start -->
+<!-- 200823 -->
 ## Discord
 
 For further support, and discussions on these models and AI in general, join us at:
@@ -173,12 +177,15 @@ Donaters will get priority support on any and all AI/LLM/model questions and req
 * Patreon: https://patreon.com/TheBlokeAI
 * Ko-Fi: https://ko-fi.com/TheBlokeAI
 
-**Special thanks to**: Luke from CarbonQuill, Aemon Algiz, Dmitriy Samsonov.
+**Special thanks to**: Aemon Algiz.
+
+**Patreon special mentions**: Sam, theTransient, Jonathan Leane, Steven Wood, webtim, Johann-Peter Hartmann, Geoffrey Montalvo, Gabriel Tamborski, Willem Michiel, John Villwock, Derek Yates, Mesiah Bishop, Eugene Pentland, Pieter, Chadd, Stephen Murray, Daniel P. Andersen, terasurfer, Brandon Frisco, Thomas Belote, Sid, Nathan LeClaire, Magnesian, Alps Aficionado, Stanislav Ovsiannikov, Alex, Joseph William Delisle, Nikolai Manek, Michael Davis, Junyu Yang, K, J, Spencer Kim, Stefan Sabev, Olusegun Samson, transmissions 11, Michael Levine, Cory Kujawski, Rainer Wilmers, zynix, Kalila, Luke @flexchar, Ajan Kanaga, Mandus, vamX, Ai Maven, Mano Prime, Matthew Berman, subjectnull, Vitor Caleffi, Clay Pascal, biorpg, alfie_i, 阿明, Jeffrey Morgan, ya boyyy, Raymond Fosdick, knownsqashed, Olakabola, Leonard Tan, ReadyPlayerEmma, Enrico Ros, Dave, Talal Aujan, Illia Dulskyi, Sean Connelly, senxiiz, Artur Olbinski, Elle, Raven Klaugh, Fen Risland, Deep Realms, Imad Khwaja, Fred von Graf, Will Dee, usrbinkat, SuperWojo, Alexandros Triantafyllidis, Swaroop Kallakuri, Dan Guido, John Detwiler, Pedro Madruga, Iucharbius, Viktor Bowallius, Asp the Wyvern, Edmond Seymore, Trenton Dambrowitz, Space Cruiser, Spiking Neurons AB, Pyrater, LangChain4j, Tony Hughes, Kacper Wikieł, Rishabh Srivastava, David Ziegler, Luke Pendergrass, Andrey, Gabriel Puliatti, Lone Striker, Sebastain Graf, Pierre Kircher, Randy H, NimbleBox.ai, Vadim, danny, Deo Leter
 
-**Patreon special mentions**: Pyrater, WelcomeToTheClub, Kalila, Mano Prime, Trenton Dambrowitz, Spiking Neurons AB, Pierre Kircher, Fen Risland, Kevin Schuppel, Luke, Rainer Wilmers, vamX, Gabriel Puliatti, Alex , Karl Bernard, Ajan Kanaga, Talal Aujan, Space Cruiser, ya boyyy, biorpg, Johann-Peter Hartmann, Asp the Wyvern, Ai Maven, Ghost , Preetika Verma, Nikolai Manek, trip7s trip, John Detwiler, Fred von Graf, Artur Olbinski, subjectnull, John Villwock, Junyu Yang, Rod A, Lone Striker, Chris McCloskey, Iucharbius , Matthew Berman, Illia Dulskyi, Khalefa Al-Ahmad, Imad Khwaja, chris gileta, Willem Michiel, Greatston Gnanesh, Derek Yates, K, Alps Aficionado, Oscar Rangel, David Flickinger, Luke Pendergrass, Deep Realms, Eugene Pentland, Cory Kujawski, terasurfer , Jonathan Leane, senxiiz, Joseph William Delisle, Sean Connelly, webtim, zynix , Nathan LeClaire.
 
 Thank you to all my generous patrons and donaters!
 
+And thank you again to a16z for their generous grant.
+
 <!-- footer end -->
 
 # Original model card: Kaio Ken's SuperHOT 8K
@@ -196,9 +203,9 @@ You will need to **use either the monkeypatch** or, if you are already using the
 
 
 #### Training Details
-I trained the LoRA with the following configuration: 
+I trained the LoRA with the following configuration:
 - 1200 samples (~400 samples over 2048 sequence length)
-- learning rate of 3e-4 
+- learning rate of 3e-4
 - 3 epochs
 - The exported modules are:
     - q_proj
@@ -235,7 +242,7 @@ Your instruction or question here.
 [4bit GPTQ Version provided by @TheBloke](https://huggingface.co/TheBloke/chronos-33b-GPTQ)
 
 <!--**Support My Development of New Models**
-<a href='https://ko-fi.com/Q5Q6MB734' target='_blank'><img height='36' style='border:0px;height:36px;' 
+<a href='https://ko-fi.com/Q5Q6MB734' target='_blank'><img height='36' style='border:0px;height:36px;'
 src='https://storage.ko-fi.com/cdn/kofi1.png?v=3' border='0' alt='Support Development' /></a>-->
 
 --
@@ -318,11 +325,11 @@ Hyperparameters for the model architecture
             </tr>
             <tr>
             <th>Number of parameters</th><th>dimension</th><th>n heads</th><th>n layers</th><th>Learn rate</th><th>Batch size</th><th>n tokens</th>
-            </tr>           
+            </tr>
         </thead>
-    <tbody>       
+    <tbody>
         <tr>
-            <th>7B</th> <th>4096</th> <th>32</th> <th>32</th> <th>3.0E-04</th><th>4M</th><th>1T 
+            <th>7B</th> <th>4096</th> <th>32</th> <th>32</th> <th>3.0E-04</th><th>4M</th><th>1T
         </tr>
         <tr>
             <th>13B</th><th>5120</th><th>40</th><th>40</th><th>3.0E-04</th><th>4M</th><th>1T
@@ -332,13 +339,13 @@ Hyperparameters for the model architecture
         </tr>
         <tr>
             <th>65B</th><th>8192</th><th>64</th><th>80</th><th>1.5.E-04</th><th>4M</th><th>1.4T
-        </tr>     
+        </tr>
     </tbody>
 </table>
 
 *Table 1 - Summary of LLama Model Hyperparameters*
 
-We present our results on eight standard common sense reasoning benchmarks in the table below. 
+We present our results on eight standard common sense reasoning benchmarks in the table below.
 <table>
     <thead>
             <tr>
@@ -346,23 +353,23 @@ We present our results on eight standard common sense reasoning benchmarks in th
             </tr>
             <tr>
             <th>Number of parameters</th> <th>BoolQ</th><th>PIQA</th><th>SIQA</th><th>HellaSwag</th><th>WinoGrande</th><th>ARC-e</th><th>ARC-c</th><th>OBQA</th><th>COPA</th>
-            </tr>           
+            </tr>
         </thead>
-    <tbody>    
-    <tr>   
+    <tbody>
+    <tr>
         <th>7B</th><th>76.5</th><th>79.8</th><th>48.9</th><th>76.1</th><th>70.1</th><th>76.7</th><th>47.6</th><th>57.2</th><th>93
-        </th>   
+        </th>
     <tr><th>13B</th><th>78.1</th><th>80.1</th><th>50.4</th><th>79.2</th><th>73</th><th>78.1</th><th>52.7</th><th>56.4</th><th>94
 </th>
     <tr><th>33B</th><th>83.1</th><th>82.3</th><th>50.4</th><th>82.8</th><th>76</th><th>81.4</th><th>57.8</th><th>58.6</th><th>92
 </th>
-    <tr><th>65B</th><th>85.3</th><th>82.8</th><th>52.3</th><th>84.2</th><th>77</th><th>81.5</th><th>56</th><th>60.2</th><th>94</th></tr> 
+    <tr><th>65B</th><th>85.3</th><th>82.8</th><th>52.3</th><th>84.2</th><th>77</th><th>81.5</th><th>56</th><th>60.2</th><th>94</th></tr>
     </tbody>
 </table>
 *Table 2 - Summary of LLama Model Performance on Reasoning tasks*
 
 
-We present our results on bias in the table below. Note that lower value is better indicating lower bias. 
+We present our results on bias in the table below. Note that lower value is better indicating lower bias.
 
 
 | No  | Category             | FAIR LLM |

config.json

@@ -1,28 +1,38 @@
 {
-    "_name_or_path": "/workspace/process/lora_base/elinas_chronos-33b",
-    "architectures": [
-        "LlamaForCausalLM"
-    ],
-    "auto_map": {
+  "_name_or_path": "/workspace/process/lora_base/elinas_chronos-33b",
+  "architectures": [
+    "LlamaForCausalLM"
+  ],
+  "auto_map": {
     "AutoModel": "modelling_llama.LlamaModel",
     "AutoModelForCausalLM": "modelling_llama.LlamaForCausalLM",
     "AutoModelForSequenceClassification": "modelling_llama.LlamaForSequenceClassification"
-    },
-    "bos_token_id": 1,
-    "eos_token_id": 2,
-    "hidden_act": "silu",
-    "hidden_size": 6656,
-    "initializer_range": 0.02,
-    "intermediate_size": 17920,
-    "max_position_embeddings": 8192,
-    "model_type": "llama",
-    "num_attention_heads": 52,
-    "num_hidden_layers": 60,
-    "pad_token_id": 0,
-    "rms_norm_eps": 1e-06,
-    "tie_word_embeddings": false,
-    "torch_dtype": "float16",
-    "transformers_version": "4.30.0.dev0",
-    "use_cache": true,
-    "vocab_size": 32000
-}
+  },
+  "bos_token_id": 1,
+  "eos_token_id": 2,
+  "hidden_act": "silu",
+  "hidden_size": 6656,
+  "initializer_range": 0.02,
+  "intermediate_size": 17920,
+  "max_position_embeddings": 8192,
+  "model_type": "llama",
+  "num_attention_heads": 52,
+  "num_hidden_layers": 60,
+  "pad_token_id": 0,
+  "rms_norm_eps": 1e-06,
+  "tie_word_embeddings": false,
+  "torch_dtype": "float16",
+  "transformers_version": "4.30.0.dev0",
+  "use_cache": true,
+  "vocab_size": 32000,
+  "quantization_config": {
+    "bits": 4,
+    "group_size": -1,
+    "damp_percent": 0.01,
+    "desc_act": true,
+    "sym": true,
+    "true_sequential": true,
+    "model_file_base_name": "model",
+    "quant_method": "gptq"
+  }
+}

chronos-33b-superhot-8k-GPTQ-4bit--1g.act.order.safetensors → model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:7a1fb04e7b180d95c2c768d566ce72196014a50a6ff1f73731eab3e70938f36a
-size 16940128408
+oid sha256:108ad378d2b63f9107021c3a94158085c3b00733bd2c041936c6a0b2c35f5950
+size 16940128464

quantize_config.json

@@ -1,8 +1,9 @@
 {
-    "bits": 4,
-    "group_size": -1,
-    "damp_percent": 0.01,
-    "desc_act": true,
-    "sym": true,
-    "true_sequential": true
+  "bits": 4,
+  "group_size": -1,
+  "damp_percent": 0.01,
+  "desc_act": true,
+  "sym": true,
+  "true_sequential": true,
+  "model_file_base_name": "model"
 }