Upload README.md

README.md

@@ -1,4 +1,5 @@
 ---
+base_model: https://huggingface.co/OpenBuddy/openbuddy-llama2-70b-v10.1-bf16
 inference: false
 language:
 - zh
@@ -12,7 +13,6 @@ language:
 library_name: transformers
 license: llama2
 model_creator: OpenBuddy
-model_link: https://huggingface.co/OpenBuddy/openbuddy-llama2-70b-v10.1-bf16
 model_name: OpenBuddy Llama2 70b v10.1
 model_type: llama
 pipeline_tag: text-generation
@@ -57,18 +57,24 @@ Multiple GPTQ parameter permutations are provided; see Provided Files below for
 <!-- repositories-available end -->
 
 <!-- prompt-template start -->
-## Prompt template: Vicuna-Short
+## Prompt template: OpenBuddy
 
 ```
-You are a helpful AI assistant.
+You are a helpful, respectful and honest INTP-T AI Assistant named Buddy. You are talking to a human User.
+Always answer as helpfully and logically as possible, while being safe. Your answers should not include any harmful, political, religious, unethical, racist, sexist, toxic, dangerous, or illegal content. Please ensure that your responses are socially unbiased and positive in nature.
+If a question does not make any sense, or is not factually coherent, explain why instead of answering something not correct. If you don't know the answer to a question, please don't share false information.
+You like to use emojis. You can speak fluently in many languages, for example: English, Chinese.
+You cannot access the internet, but you have vast knowledge, cutoff: 2021-09.
+You are trained by OpenBuddy team, (https://openbuddy.ai, https://github.com/OpenBuddy/OpenBuddy), you are based on LLaMA and Falcon transformers model, not related to GPT or OpenAI.
 
-USER: {prompt}
-ASSISTANT:
+User: {prompt}
+Assistant: 
 
 ```
 
 <!-- prompt-template end -->
 
+
 <!-- README_GPTQ.md-provided-files start -->
 ## Provided files and GPTQ parameters
 
@@ -93,22 +99,22 @@ All recent GPTQ files are made with AutoGPTQ, and all files in non-main branches
 
 | Branch | Bits | GS | Act Order | Damp % | GPTQ Dataset | Seq Len | Size | ExLlama | Desc |
 | ------ | ---- | -- | --------- | ------ | ------------ | ------- | ---- | ------- | ---- |
-| [main](https://huggingface.co/TheBloke/OpenBuddy-Llama2-70b-v10.1-GPTQ/tree/main) | 4 | None | Yes | 0.1 | [wikitext](https://huggingface.co/datasets/wikitext/viewer/wikitext-2-v1/test) | 4096 | 35.52 GB | Yes | Most compatible option. Good inference speed in AutoGPTQ and GPTQ-for-LLaMa. Lower inference quality than other options. | 
-| [gptq-4bit-32g-actorder_True](https://huggingface.co/TheBloke/OpenBuddy-Llama2-70b-v10.1-GPTQ/tree/gptq-4bit-32g-actorder_True) | 4 | 32 | Yes | 0.1 | [wikitext](https://huggingface.co/datasets/wikitext/viewer/wikitext-2-v1/test) | 4096 | 40.85 GB | Yes | 4-bit, with Act Order and group size 32g. Gives highest possible inference quality, with maximum VRAM usage. Poor AutoGPTQ CUDA speed. | 
-| [gptq-4bit-64g-actorder_True](https://huggingface.co/TheBloke/OpenBuddy-Llama2-70b-v10.1-GPTQ/tree/gptq-4bit-64g-actorder_True) | 4 | 64 | Yes | 0.1 | [wikitext](https://huggingface.co/datasets/wikitext/viewer/wikitext-2-v1/test) | 4096 | 38.17 GB | Yes | 4-bit, with Act Order and group size 64g. Uses less VRAM than 32g, but with slightly lower accuracy. Poor AutoGPTQ CUDA speed. | 
-| [gptq-4bit-128g-actorder_True](https://huggingface.co/TheBloke/OpenBuddy-Llama2-70b-v10.1-GPTQ/tree/gptq-4bit-128g-actorder_True) | 4 | 128 | Yes | 0.1 | [wikitext](https://huggingface.co/datasets/wikitext/viewer/wikitext-2-v1/test) | 4096 | 36.84 GB | Yes | 4-bit, with Act Order and group size 128g. Uses even less VRAM than 64g, but with slightly lower accuracy. Poor AutoGPTQ CUDA speed. | 
+| [main](https://huggingface.co/TheBloke/OpenBuddy-Llama2-70b-v10.1-GPTQ/tree/main) | 4 | None | Yes | 0.1 | [wikitext](https://huggingface.co/datasets/wikitext/viewer/wikitext-2-v1/test) | 4096 | 35.52 GB | Yes | 4-bit, with Act Order. No group size, to lower VRAM requirements. | 
+| [gptq-4bit-32g-actorder_True](https://huggingface.co/TheBloke/OpenBuddy-Llama2-70b-v10.1-GPTQ/tree/gptq-4bit-32g-actorder_True) | 4 | 32 | Yes | 0.1 | [wikitext](https://huggingface.co/datasets/wikitext/viewer/wikitext-2-v1/test) | 4096 | 40.85 GB | Yes | 4-bit, with Act Order and group size 32g. Gives highest possible inference quality, with maximum VRAM usage. | 
+| [gptq-4bit-64g-actorder_True](https://huggingface.co/TheBloke/OpenBuddy-Llama2-70b-v10.1-GPTQ/tree/gptq-4bit-64g-actorder_True) | 4 | 64 | Yes | 0.1 | [wikitext](https://huggingface.co/datasets/wikitext/viewer/wikitext-2-v1/test) | 4096 | 38.17 GB | Yes | 4-bit, with Act Order and group size 64g. Uses less VRAM than 32g, but with slightly lower accuracy. | 
+| [gptq-4bit-128g-actorder_True](https://huggingface.co/TheBloke/OpenBuddy-Llama2-70b-v10.1-GPTQ/tree/gptq-4bit-128g-actorder_True) | 4 | 128 | Yes | 0.1 | [wikitext](https://huggingface.co/datasets/wikitext/viewer/wikitext-2-v1/test) | 4096 | 36.84 GB | Yes | 4-bit, with Act Order and group size 128g. Uses even less VRAM than 64g, but with slightly lower accuracy. | 
 | [gptq-3bit--1g-actorder_True](https://huggingface.co/TheBloke/OpenBuddy-Llama2-70b-v10.1-GPTQ/tree/gptq-3bit--1g-actorder_True) | 3 | None | Yes | 0.1 | [wikitext](https://huggingface.co/datasets/wikitext/viewer/wikitext-2-v1/test) | 4096 | 26.96 GB | No | 3-bit, with Act Order and no group size. Lowest possible VRAM requirements. May be lower quality than 3-bit 128g. | 
-| [gptq-3bit-128g-actorder_True](https://huggingface.co/TheBloke/OpenBuddy-Llama2-70b-v10.1-GPTQ/tree/gptq-3bit-128g-actorder_True) | 3 | 128 | Yes | 0.1 | [wikitext](https://huggingface.co/datasets/wikitext/viewer/wikitext-2-v1/test) | 4096 | 28.21 GB | No | 3-bit, with group size 128g and act-order. Higher quality than 128g-False but poor AutoGPTQ CUDA speed. |
+| [gptq-3bit-128g-actorder_True](https://huggingface.co/TheBloke/OpenBuddy-Llama2-70b-v10.1-GPTQ/tree/gptq-3bit-128g-actorder_True) | 3 | 128 | Yes | 0.1 | [wikitext](https://huggingface.co/datasets/wikitext/viewer/wikitext-2-v1/test) | 4096 | 28.21 GB | No | 3-bit, with group size 128g and act-order. Higher quality than 128g-False. |
 
 <!-- README_GPTQ.md-provided-files end -->
 
 <!-- README_GPTQ.md-download-from-branches start -->
 ## How to download from branches
 
-- In text-generation-webui, you can add `:branch` to the end of the download name, eg `TheBloke/OpenBuddy-Llama2-70b-v10.1-GPTQ:gptq-4bit-32g-actorder_True`
+- In text-generation-webui, you can add `:branch` to the end of the download name, eg `TheBloke/OpenBuddy-Llama2-70b-v10.1-GPTQ:main`
 - With Git, you can clone a branch with:
 ```
-git clone --single-branch --branch gptq-4bit-32g-actorder_True https://huggingface.co/TheBloke/OpenBuddy-Llama2-70b-v10.1-GPTQ
+git clone --single-branch --branch main https://huggingface.co/TheBloke/OpenBuddy-Llama2-70b-v10.1-GPTQ
 ```
 - In Python Transformers code, the branch is the `revision` parameter; see below.
 <!-- README_GPTQ.md-download-from-branches end -->
@@ -121,7 +127,7 @@ It is strongly recommended to use the text-generation-webui one-click-installers
 
 1. Click the **Model tab**.
 2. Under **Download custom model or LoRA**, enter `TheBloke/OpenBuddy-Llama2-70b-v10.1-GPTQ`.
-  - To download from a specific branch, enter for example `TheBloke/OpenBuddy-Llama2-70b-v10.1-GPTQ:gptq-4bit-32g-actorder_True`
+  - To download from a specific branch, enter for example `TheBloke/OpenBuddy-Llama2-70b-v10.1-GPTQ:main`
   - see Provided Files above for the list of branches for each option.
 3. Click **Download**.
 4. The model will start downloading. Once it's finished it will say "Done".
@@ -169,26 +175,31 @@ from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline
 
 model_name_or_path = "TheBloke/OpenBuddy-Llama2-70b-v10.1-GPTQ"
 # To use a different branch, change revision
-# For example: revision="gptq-4bit-32g-actorder_True"
+# For example: revision="main"
 model = AutoModelForCausalLM.from_pretrained(model_name_or_path,
-                                             torch_dtype=torch.bfloat16,
                                              device_map="auto",
+                                             trust_remote_code=False,
                                              revision="main")
 
 tokenizer = AutoTokenizer.from_pretrained(model_name_or_path, use_fast=True)
 
 prompt = "Tell me about AI"
-prompt_template=f'''You are a helpful AI assistant.
+prompt_template=f'''You are a helpful, respectful and honest INTP-T AI Assistant named Buddy. You are talking to a human User.
+Always answer as helpfully and logically as possible, while being safe. Your answers should not include any harmful, political, religious, unethical, racist, sexist, toxic, dangerous, or illegal content. Please ensure that your responses are socially unbiased and positive in nature.
+If a question does not make any sense, or is not factually coherent, explain why instead of answering something not correct. If you don't know the answer to a question, please don't share false information.
+You like to use emojis. You can speak fluently in many languages, for example: English, Chinese.
+You cannot access the internet, but you have vast knowledge, cutoff: 2021-09.
+You are trained by OpenBuddy team, (https://openbuddy.ai, https://github.com/OpenBuddy/OpenBuddy), you are based on LLaMA and Falcon transformers model, not related to GPT or OpenAI.
 
-USER: {prompt}
-ASSISTANT:
+User: {prompt}
+Assistant: 
 
 '''
 
 print("\n\n*** Generate:")
 
 input_ids = tokenizer(prompt_template, return_tensors='pt').input_ids.cuda()
-output = model.generate(inputs=input_ids, temperature=0.7, max_new_tokens=512)
+output = model.generate(inputs=input_ids, temperature=0.7, do_sample=True, top_p=0.95, top_k=40, max_new_tokens=512)
 print(tokenizer.decode(output[0]))
 
 # Inference can also be done using transformers' pipeline
@@ -199,9 +210,11 @@ pipe = pipeline(
     model=model,
     tokenizer=tokenizer,
     max_new_tokens=512,
+    do_sample=True,
     temperature=0.7,
     top_p=0.95,
-    repetition_penalty=1.15
+    top_k=40,
+    repetition_penalty=1.1
 )
 
 print(pipe(prompt_template)[0]['generated_text'])
@@ -226,10 +239,12 @@ For further support, and discussions on these models and AI in general, join us
 
 [TheBloke AI's Discord server](https://discord.gg/theblokeai)
 
-## Thanks, and how to contribute.
+## Thanks, and how to contribute
 
 Thanks to the [chirper.ai](https://chirper.ai) team!
 
+Thanks to Clay from [gpus.llm-utils.org](llm-utils)!
+
 I've had a lot of people ask if they can contribute. I enjoy providing models and helping people, and would love to be able to spend even more time doing it, as well as expanding into new projects like fine tuning/training.
 
 If you're able and willing to contribute it will be most gratefully received and will help me to keep providing more models, and to start work on new AI projects.