Upload README.md

README.md

@@ -6,7 +6,7 @@ inference: false
 language:
 - en
 library_name: transformers
-license: llama2
+license: cc-by-nc-4.0
 model_creator: Open-Orca
 model_link: https://huggingface.co/Open-Orca/OpenOrca-Platypus2-13B
 model_name: OpenOrca Platypus2 13B
@@ -66,7 +66,15 @@ Multiple GPTQ parameter permutations are provided; see Provided Files below for
 ```
 
 <!-- prompt-template end -->
+<!-- licensing start -->
+## Licensing
 
+The creator of the source model has listed its license as `cc-by-nc-4.0`, and this quantization has therefore used that same license.
+
+As this model is based on Llama 2, it is also subject to the Meta Llama 2 license terms, and the license files for that are additionally included. It should therefore be considered as being claimed to be licensed under both licenses. I contacted Hugging Face for clarification on dual licensing but they do not yet have an official position. Should this change, or should Meta provide any feedback on this situation, I will update this section accordingly.
+
+In the meantime, any questions regarding licensing, and in particular how these two licenses might interact, should be directed to the original model repository: [Open-Orca's OpenOrca Platypus2 13B](https://huggingface.co/Open-Orca/OpenOrca-Platypus2-13B).
+<!-- licensing end -->
 <!-- README_GPTQ.md-provided-files start -->
 ## Provided files and GPTQ parameters
 
@@ -91,22 +99,22 @@ All recent GPTQ files are made with AutoGPTQ, and all files in non-main branches
 
 | Branch | Bits | GS | Act Order | Damp % | GPTQ Dataset | Seq Len | Size | ExLlama | Desc |
 | ------ | ---- | -- | --------- | ------ | ------------ | ------- | ---- | ------- | ---- |
-| [main](https://huggingface.co/TheBloke/OpenOrca-Platypus2-13B-GPTQ/tree/main) | 4 | 128 | No | 0.1 | [wikitext](https://huggingface.co/datasets/wikitext/viewer/wikitext-2-v1/test) | 4096 | 7.26 GB | Yes | Most compatible option. Good inference speed in AutoGPTQ and GPTQ-for-LLaMa. Lower inference quality than other options. | 
-| [gptq-4bit-32g-actorder_True](https://huggingface.co/TheBloke/OpenOrca-Platypus2-13B-GPTQ/tree/gptq-4bit-32g-actorder_True) | 4 | 32 | Yes | 0.1 | [wikitext](https://huggingface.co/datasets/wikitext/viewer/wikitext-2-v1/test) | 4096 | 8.00 GB | Yes | 4-bit, with Act Order and group size 32g. Gives highest possible inference quality, with maximum VRAM usage. Poor AutoGPTQ CUDA speed. | 
-| [gptq-4bit-64g-actorder_True](https://huggingface.co/TheBloke/OpenOrca-Platypus2-13B-GPTQ/tree/gptq-4bit-64g-actorder_True) | 4 | 64 | Yes | 0.1 | [wikitext](https://huggingface.co/datasets/wikitext/viewer/wikitext-2-v1/test) | 4096 | 7.51 GB | Yes | 4-bit, with Act Order and group size 64g. Uses less VRAM than 32g, but with slightly lower accuracy. Poor AutoGPTQ CUDA speed. | 
-| [gptq-4bit-128g-actorder_True](https://huggingface.co/TheBloke/OpenOrca-Platypus2-13B-GPTQ/tree/gptq-4bit-128g-actorder_True) | 4 | 128 | Yes | 0.1 | [wikitext](https://huggingface.co/datasets/wikitext/viewer/wikitext-2-v1/test) | 4096 | 7.26 GB | Yes | 4-bit, with Act Order and group size 128g. Uses even less VRAM than 64g, but with slightly lower accuracy. Poor AutoGPTQ CUDA speed. | 
-| [gptq-8bit--1g-actorder_True](https://huggingface.co/TheBloke/OpenOrca-Platypus2-13B-GPTQ/tree/gptq-8bit--1g-actorder_True) | 8 | None | Yes | 0.1 | [wikitext](https://huggingface.co/datasets/wikitext/viewer/wikitext-2-v1/test) | 4096 | 13.36 GB | No | 8-bit, with Act Order. No group size, to lower VRAM requirements and to improve AutoGPTQ speed. | 
-| [gptq-8bit-128g-actorder_True](https://huggingface.co/TheBloke/OpenOrca-Platypus2-13B-GPTQ/tree/gptq-8bit-128g-actorder_True) | 8 | 128 | Yes | 0.1 | [wikitext](https://huggingface.co/datasets/wikitext/viewer/wikitext-2-v1/test) | 4096 | 13.65 GB | No | 8-bit, with group size 128g for higher inference quality and with Act Order for even higher accuracy. Poor AutoGPTQ CUDA speed. |
+| [main](https://huggingface.co/TheBloke/OpenOrca-Platypus2-13B-GPTQ/tree/main) | 4 | 128 | No | 0.1 | [wikitext](https://huggingface.co/datasets/wikitext/viewer/wikitext-2-v1/test) | 4096 | 7.26 GB | Yes | 4-bit, without Act Order and group size 128g. | 
+| [gptq-4bit-32g-actorder_True](https://huggingface.co/TheBloke/OpenOrca-Platypus2-13B-GPTQ/tree/gptq-4bit-32g-actorder_True) | 4 | 32 | Yes | 0.1 | [wikitext](https://huggingface.co/datasets/wikitext/viewer/wikitext-2-v1/test) | 4096 | 8.00 GB | Yes | 4-bit, with Act Order and group size 32g. Gives highest possible inference quality, with maximum VRAM usage. | 
+| [gptq-4bit-64g-actorder_True](https://huggingface.co/TheBloke/OpenOrca-Platypus2-13B-GPTQ/tree/gptq-4bit-64g-actorder_True) | 4 | 64 | Yes | 0.1 | [wikitext](https://huggingface.co/datasets/wikitext/viewer/wikitext-2-v1/test) | 4096 | 7.51 GB | Yes | 4-bit, with Act Order and group size 64g. Uses less VRAM than 32g, but with slightly lower accuracy. | 
+| [gptq-4bit-128g-actorder_True](https://huggingface.co/TheBloke/OpenOrca-Platypus2-13B-GPTQ/tree/gptq-4bit-128g-actorder_True) | 4 | 128 | Yes | 0.1 | [wikitext](https://huggingface.co/datasets/wikitext/viewer/wikitext-2-v1/test) | 4096 | 7.26 GB | Yes | 4-bit, with Act Order and group size 128g. Uses even less VRAM than 64g, but with slightly lower accuracy. | 
+| [gptq-8bit--1g-actorder_True](https://huggingface.co/TheBloke/OpenOrca-Platypus2-13B-GPTQ/tree/gptq-8bit--1g-actorder_True) | 8 | None | Yes | 0.1 | [wikitext](https://huggingface.co/datasets/wikitext/viewer/wikitext-2-v1/test) | 4096 | 13.36 GB | No | 8-bit, with Act Order. No group size, to lower VRAM requirements. | 
+| [gptq-8bit-128g-actorder_True](https://huggingface.co/TheBloke/OpenOrca-Platypus2-13B-GPTQ/tree/gptq-8bit-128g-actorder_True) | 8 | 128 | Yes | 0.1 | [wikitext](https://huggingface.co/datasets/wikitext/viewer/wikitext-2-v1/test) | 4096 | 13.65 GB | No | 8-bit, with group size 128g for higher inference quality and with Act Order for even higher accuracy. |
 
 <!-- README_GPTQ.md-provided-files end -->
 
 <!-- README_GPTQ.md-download-from-branches start -->
 ## How to download from branches
 
-- In text-generation-webui, you can add `:branch` to the end of the download name, eg `TheBloke/OpenOrca-Platypus2-13B-GPTQ:gptq-4bit-32g-actorder_True`
+- In text-generation-webui, you can add `:branch` to the end of the download name, eg `TheBloke/OpenOrca-Platypus2-13B-GPTQ:main`
 - With Git, you can clone a branch with:
 ```
-git clone --single-branch --branch gptq-4bit-32g-actorder_True https://huggingface.co/TheBloke/OpenOrca-Platypus2-13B-GPTQ
+git clone --single-branch --branch main https://huggingface.co/TheBloke/OpenOrca-Platypus2-13B-GPTQ
 ```
 - In Python Transformers code, the branch is the `revision` parameter; see below.
 <!-- README_GPTQ.md-download-from-branches end -->
@@ -119,7 +127,7 @@ It is strongly recommended to use the text-generation-webui one-click-installers
 
 1. Click the **Model tab**.
 2. Under **Download custom model or LoRA**, enter `TheBloke/OpenOrca-Platypus2-13B-GPTQ`.
-  - To download from a specific branch, enter for example `TheBloke/OpenOrca-Platypus2-13B-GPTQ:gptq-4bit-32g-actorder_True`
+  - To download from a specific branch, enter for example `TheBloke/OpenOrca-Platypus2-13B-GPTQ:main`
   - see Provided Files above for the list of branches for each option.
 3. Click **Download**.
 4. The model will start downloading. Once it's finished it will say "Done".
@@ -167,10 +175,10 @@ from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline
 
 model_name_or_path = "TheBloke/OpenOrca-Platypus2-13B-GPTQ"
 # To use a different branch, change revision
-# For example: revision="gptq-4bit-32g-actorder_True"
+# For example: revision="main"
 model = AutoModelForCausalLM.from_pretrained(model_name_or_path,
-                                             torch_dtype=torch.float16,
                                              device_map="auto",
+                                             trust_remote_code=False,
                                              revision="main")
 
 tokenizer = AutoTokenizer.from_pretrained(model_name_or_path, use_fast=True)
@@ -187,7 +195,7 @@ prompt_template=f'''### Instruction:
 print("\n\n*** Generate:")
 
 input_ids = tokenizer(prompt_template, return_tensors='pt').input_ids.cuda()
-output = model.generate(inputs=input_ids, temperature=0.7, max_new_tokens=512)
+output = model.generate(inputs=input_ids, temperature=0.7, do_sample=True, top_p=0.95, top_k=40, max_new_tokens=512)
 print(tokenizer.decode(output[0]))
 
 # Inference can also be done using transformers' pipeline
@@ -198,9 +206,11 @@ pipe = pipeline(
     model=model,
     tokenizer=tokenizer,
     max_new_tokens=512,
+    do_sample=True,
     temperature=0.7,
     top_p=0.95,
-    repetition_penalty=1.15
+    top_k=40,
+    repetition_penalty=1.1
 )
 
 print(pipe(prompt_template)[0]['generated_text'])
@@ -225,10 +235,12 @@ For further support, and discussions on these models and AI in general, join us
 
 [TheBloke AI's Discord server](https://discord.gg/theblokeai)
 
-## Thanks, and how to contribute.
+## Thanks, and how to contribute
 
 Thanks to the [chirper.ai](https://chirper.ai) team!
 
+Thanks to Clay from [gpus.llm-utils.org](llm-utils)!
+
 I've had a lot of people ask if they can contribute. I enjoy providing models and helping people, and would love to be able to spend even more time doing it, as well as expanding into new projects like fine tuning/training.
 
 If you're able and willing to contribute it will be most gratefully received and will help me to keep providing more models, and to start work on new AI projects.