Initial GPTQ model upload

README.md

@@ -17,16 +17,11 @@ license: other
 </div>
 <!-- header end -->
 
-# Camel AI's CAMEL 33B Combined Data GGML
+# Camel AI's CAMEL 33B Combined Data GPTQ
 
-These files are GGML format model files for [Camel AI's CAMEL 33B Combined Data](https://huggingface.co/camel-ai/CAMEL-33B-Combined-Data).
+These files are GPTQ 4bit model files for [Camel AI's CAMEL 33B Combined Data](https://huggingface.co/camel-ai/CAMEL-33B-Combined-Data).
 
-GGML files are for CPU + GPU inference using [llama.cpp](https://github.com/ggerganov/llama.cpp) and libraries and UIs which support this format, such as:
-* [text-generation-webui](https://github.com/oobabooga/text-generation-webui)
-* [KoboldCpp](https://github.com/LostRuins/koboldcpp)
-* [ParisNeo/GPT4All-UI](https://github.com/ParisNeo/gpt4all-ui)
-* [llama-cpp-python](https://github.com/abetlen/llama-cpp-python)
-* [ctransformers](https://github.com/marella/ctransformers)
+It is the result of quantising to 4bit using [GPTQ-for-LLaMa](https://github.com/qwopqwop200/GPTQ-for-LLaMa).
 
 ## Repositories available
 
@@ -42,71 +37,92 @@ USER: prompt
 ASSISTANT:
 ```
 
-<!-- compatibility_ggml start -->
-## Compatibility
+## How to easily download and use this model in text-generation-webui
 
-### Original llama.cpp quant methods: `q4_0, q4_1, q5_0, q5_1, q8_0`
+Please make sure you're using the latest version of text-generation-webui
 
-I have quantized these 'original' quantisation methods using an older version of llama.cpp so that they remain compatible with llama.cpp as of May 19th, commit `2d5db48`.
+1. Click the **Model tab**.
+2. Under **Download custom model or LoRA**, enter `TheBloke/CAMEL-33B-Combined-Data-GPTQ`.
+3. Click **Download**.
+4. The model will start downloading. Once it's finished it will say "Done"
+5. In the top left, click the refresh icon next to **Model**.
+6. In the **Model** dropdown, choose the model you just downloaded: `CAMEL-33B-Combined-Data-GPTQ`
+7. The model will automatically load, and is now ready for use!
+8. If you want any custom settings, set them and then click **Save settings for this model** followed by **Reload the Model** in the top right.
+  * Note that you do not need to and should not set manual GPTQ parameters any more. These are set automatically from the file `quantize_config.json`.
+9. Once you're ready, click the **Text Generation tab** and enter a prompt to get started!
 
-These are guaranteed to be compatbile with any UIs, tools and libraries released since late May.
+## How to use this GPTQ model from Python code
 
-### New k-quant methods: `q2_K, q3_K_S, q3_K_M, q3_K_L, q4_K_S, q4_K_M, q5_K_S, q6_K`
+First make sure you have [AutoGPTQ](https://github.com/PanQiWei/AutoGPTQ) installed:
 
-These new quantisation methods are compatible with llama.cpp as of June 6th, commit `2d43387`.
+`pip install auto-gptq`
 
-They are now also compatible with recent releases of text-generation-webui, KoboldCpp, llama-cpp-python and ctransformers. Other tools and libraries may or may not be compatible - check their documentation if in doubt.
+Then try the following example code:
 
-## Explanation of the new k-quant methods
+```python
+from transformers import AutoTokenizer, pipeline, logging
+from auto_gptq import AutoGPTQForCausalLM, BaseQuantizeConfig
+import argparse
 
-The new methods available are:
-* GGML_TYPE_Q2_K - "type-1" 2-bit quantization in super-blocks containing 16 blocks, each block having 16 weight. Block scales and mins are quantized with 4 bits. This ends up effectively using 2.5625 bits per weight (bpw)
-* GGML_TYPE_Q3_K - "type-0" 3-bit quantization in super-blocks containing 16 blocks, each block having 16 weights. Scales are quantized with 6 bits. This end up using 3.4375 bpw.
-* GGML_TYPE_Q4_K - "type-1" 4-bit quantization in super-blocks containing 8 blocks, each block having 32 weights. Scales and mins are quantized with 6 bits. This ends up using 4.5 bpw.
-* GGML_TYPE_Q5_K - "type-1" 5-bit quantization. Same super-block structure as GGML_TYPE_Q4_K resulting in 5.5 bpw
-* GGML_TYPE_Q6_K - "type-0" 6-bit quantization. Super-blocks with 16 blocks, each block having 16 weights. Scales are quantized with 8 bits. This ends up using 6.5625 bpw
-* GGML_TYPE_Q8_K - "type-0" 8-bit quantization. Only used for quantizing intermediate results. The difference to the existing Q8_0 is that the block size is 256. All 2-6 bit dot products are implemented for this quantization type.
+model_name_or_path = "TheBloke/CAMEL-33B-Combined-Data-GPTQ"
+model_basename = "camel-33B-combined-data-GPTQ-4bit--1g.act.order"
 
-Refer to the Provided Files table below to see what files use which methods, and how.
-<!-- compatibility_ggml end -->
+use_triton = False
 
-## Provided files
-| Name | Quant method | Bits | Size | Max RAM required | Use case |
-| ---- | ---- | ---- | ---- | ---- | ----- |
-| camel-33B-combined-data.ggmlv3.q2_K.bin | q2_K | 2 | 13.71 GB | 16.21 GB | New k-quant method. Uses GGML_TYPE_Q4_K for the attention.vw and feed_forward.w2 tensors, GGML_TYPE_Q2_K for the other tensors. |
-| camel-33B-combined-data.ggmlv3.q3_K_L.bin | q3_K_L | 3 | 17.28 GB | 19.78 GB | New k-quant method. Uses GGML_TYPE_Q5_K for the attention.wv, attention.wo, and feed_forward.w2 tensors, else GGML_TYPE_Q3_K |
-| camel-33B-combined-data.ggmlv3.q3_K_M.bin | q3_K_M | 3 | 15.72 GB | 18.22 GB | New k-quant method. Uses GGML_TYPE_Q4_K for the attention.wv, attention.wo, and feed_forward.w2 tensors, else GGML_TYPE_Q3_K |
-| camel-33B-combined-data.ggmlv3.q3_K_S.bin | q3_K_S | 3 | 14.06 GB | 16.56 GB | New k-quant method. Uses GGML_TYPE_Q3_K for all tensors |
-| camel-33B-combined-data.ggmlv3.q4_0.bin | q4_0 | 4 | 18.30 GB | 20.80 GB | Original llama.cpp quant method, 4-bit. |
-| camel-33B-combined-data.ggmlv3.q4_1.bin | q4_1 | 4 | 20.33 GB | 22.83 GB | Original llama.cpp quant method, 4-bit. Higher accuracy than q4_0 but not as high as q5_0. However has quicker inference than q5 models. |
-| camel-33B-combined-data.ggmlv3.q4_K_M.bin | q4_K_M | 4 | 19.62 GB | 22.12 GB | New k-quant method. Uses GGML_TYPE_Q6_K for half of the attention.wv and feed_forward.w2 tensors, else GGML_TYPE_Q4_K |
-| camel-33B-combined-data.ggmlv3.q4_K_S.bin | q4_K_S | 4 | 18.36 GB | 20.86 GB | New k-quant method. Uses GGML_TYPE_Q4_K for all tensors |
-| camel-33B-combined-data.ggmlv3.q5_0.bin | q5_0 | 5 | 22.37 GB | 24.87 GB | Original llama.cpp quant method, 5-bit. Higher accuracy, higher resource usage and slower inference. |
-| camel-33B-combined-data.ggmlv3.q5_1.bin | q5_1 | 5 | 24.40 GB | 26.90 GB | Original llama.cpp quant method, 5-bit. Even higher accuracy, resource usage and slower inference. |
-| camel-33B-combined-data.ggmlv3.q5_K_M.bin | q5_K_M | 5 | 23.05 GB | 25.55 GB | New k-quant method. Uses GGML_TYPE_Q6_K for half of the attention.wv and feed_forward.w2 tensors, else GGML_TYPE_Q5_K |
-| camel-33B-combined-data.ggmlv3.q5_K_S.bin | q5_K_S | 5 | 22.40 GB | 24.90 GB | New k-quant method. Uses GGML_TYPE_Q5_K for all tensors |
-| camel-33B-combined-data.ggmlv3.q6_K.bin | q6_K | 6 | 26.69 GB | 29.19 GB | New k-quant method. Uses GGML_TYPE_Q8_K - 6-bit quantization - for all tensors |
-| camel-33B-combined-data.ggmlv3.q8_0.bin | q8_0 | 8 | 34.56 GB | 37.06 GB | Original llama.cpp quant method, 8-bit. Almost indistinguishable from float16. High resource use and slow. Not recommended for most users. |
-
-
-**Note**: the above RAM figures assume no GPU offloading. If layers are offloaded to the GPU, this will reduce RAM usage and use VRAM instead.
-
-## How to run in `llama.cpp`
-
-I use the following command line; adjust for your tastes and needs:
+tokenizer = AutoTokenizer.from_pretrained(model_name_or_path, use_fast=True)
 
+model = AutoGPTQForCausalLM.from_quantized(model_name_or_path,
+        model_basename=model_basename,
+        use_safetensors=True,
+        trust_remote_code=False,
+        device="cuda:0",
+        use_triton=use_triton,
+        quantize_config=None)
+
+# Note: check the prompt template is correct for this model.
+prompt = "Tell me about AI"
+prompt_template=f'''### Human: {prompt}
+### Assistant:'''
+
+print("\n\n*** Generate:")
+
+input_ids = tokenizer(prompt_template, return_tensors='pt').input_ids.cuda()
+output = model.generate(inputs=input_ids, temperature=0.7, max_new_tokens=512)
+print(tokenizer.decode(output[0]))
+
+# Inference can also be done using transformers' pipeline
+
+# Prevent printing spurious transformers error when using pipeline with AutoGPTQ
+logging.set_verbosity(logging.CRITICAL)
+
+print("*** Pipeline:")
+pipe = pipeline(
+    "text-generation",
+    model=model,
+    tokenizer=tokenizer,
+    max_new_tokens=512,
+    temperature=0.7,
+    top_p=0.95,
+    repetition_penalty=1.15
+)
+
+print(pipe(prompt_template)[0]['generated_text'])
 ```
-./main -t 10 -ngl 32 -m camel-33B-combined-data.ggmlv3.q5_0.bin --color -c 2048 --temp 0.7 --repeat_penalty 1.1 -n -1 -p "### Instruction: Write a story about llamas\n### Response:"
-```
-Change `-t 10` to the number of physical CPU cores you have. For example if your system has 8 cores/16 threads, use `-t 8`.
 
-Change `-ngl 32` to the number of layers to offload to GPU. Remove it if you don't have GPU acceleration.
+## Provided files
+
+**camel-33B-combined-data-GPTQ-4bit--1g.act.order.safetensors**
 
-If you want to have a chat-style conversation, replace the `-p <PROMPT>` argument with `-i -ins`
+This will work with AutoGPTQ and CUDA versions of GPTQ-for-LLaMa. There are reports of issues with Triton mode of recent GPTQ-for-LLaMa. If you have issues, please use AutoGPTQ instead.
 
-## How to run in `text-generation-webui`
+It was created without group_size to lower VRAM requirements, and with --act-order (desc_act) to boost inference accuracy as much as possible.
 
-Further instructions here: [text-generation-webui/docs/llama.cpp-models.md](https://github.com/oobabooga/text-generation-webui/blob/main/docs/llama.cpp-models.md).
+* `camel-33B-combined-data-GPTQ-4bit--1g.act.order.safetensors`
+  * Works with AutoGPTQ in CUDA or Triton modes.
+  * Works with GPTQ-for-LLaMa in CUDA mode.  May have issues with GPTQ-for-LLaMa Triton mode.
+  * Works with text-generation-webui, including one-click-installers.
+  * Parameters: Groupsize = -1. Act Order / desc_act = True.
 
 <!-- footer start -->
 ## Discord