Initial GPTQ model commit.

README.md

@@ -1,3 +1,186 @@
 ---
 license: other
+datasets:
+- c-s-ale/alpaca-gpt4-data
+pipeline_tag: text2text-generation
 ---
+
+## GPT4-Alpaca-LoRA_MLP-65B GPTQ
+
+These files are the result of merging the [LoRA weights of chtan's gpt4-alpaca-lora_mlp-65B](https://huggingface.co/chtan/gpt4-alpaca-lora_mlp-65b) with the original Llama 65B model.
+
+It is the result of quantising to 4bit using [GPTQ-for-LLaMa](https://github.com/qwopqwop200/GPTQ-for-LLaMa).
+
+## Repositories available
+
+* [4bit GPTQ models for GPU inference](https://huggingface.co/TheBloke/gpt4-alpaca-lora_mlp-65B-GPTQ)
+* [4bit and 5bit GGML models for CPU inference in llama.cpp](https://huggingface.co/TheBloke/gpt4-alpaca-lora_mlp-65B-GGML)
+* [float16 unquantised model for GPU inference and further conversions](https://huggingface.co/TheBloke/gpt4-alpaca-lora_mlp-65B-HF)
+
+## VRAM
+
+I tested this model with 2 x 24GB 4090 GPUs, and it was able to return 1500 tokens before one card went OOM.
+
+So you may need to preload a few layers on to CPU RAM, or else run on a system with more than 48GB VRAM.
+
+Or, if you can limit responses to <1500 tokens (eg for single prompts rather than chats), you should be fine with 48GB VRAM.
+
+## How to easily download and use this model in text-generation-webui
+
+Open the text-generation-webui UI as normal.
+
+1. Click the **Model tab**.
+2. Under **Download custom model or LoRA**, enter `TheBloke/gpt4-alpaca-lora_mlp-65B-GPTQ`.
+3. Click **Download**.
+4. Wait until it says it's finished downloading.
+5. Click the **Refresh** icon next to **Model** in the top left.
+6. In the **Model drop-down**: choose the model you just downloaded, `gpt4-alpaca-lora_mlp-65B-GPTQ`.
+7. If you see an error in the bottom right, ignore it - it's temporary.
+8. Fill out the `GPTQ parameters` on the right: `Bits = 4`, `Groupsize = None`, `model_type = Llama`
+9. Click **Save settings for this model** in the top right.
+10. Click **Reload the Model** in the top right.
+11. Once it says it's loaded, click the **Text Generation tab** and enter a prompt!
+
+## Provided files
+
+**`gpt4-alpaca-lora_mlp-65B-GPTQ-4bit.safetensors`**
+
+You will need at least 48GB VRAM to use this model, either on one GPU or multiple.
+
+This will work with all versions of GPTQ-for-LLaMa. It has maximum compatibility.
+
+It was created with `--act-order` to increase quantisation quality, but without groupsize so as to reduce VRAM requirements.
+
+* `gpt4-alpaca-lora_mlp-65B-GPTQ-4bit.safetensors`
+  * Works with all versions of GPTQ-for-LLaMa code, both Triton and CUDA branches
+  * Works with text-generation-webui one-click-installers
+  * Parameters: Groupsize = None. act-order.
+  * Command used to create the GPTQ:
+    ```
+    python llama.py gpt4-alpaca-lora_mlp-65B/HF/ c4 --wbits 4 --true-sequential --act-order  --save_safetensors gpt4-alpaca-lora_mlp-65B/gptq/gpt4-alpaca-lora_mlp-65B-GPTQ-4bit.safetensors
+    ```
+
+# Original model card
+
+This repo provides the training checkpoint of LLaMA on the alpaca_data_gpt4 dataset via LoRA [MLP] on 8xA100(80G). 
+
+He et al. 2022 gave an insight that FFN can better utilize modification at larger capacities.
+
+The codes is provided by [tloen/alpaca-lora: Instruct-tune LLaMA on consumer hardware (github.com)](https://github.com/tloen/alpaca-lora).
+
+We modify the running scripts to 
+```bash
+torchrun --nproc_per_node=8 finetune.py \
+    --base_model '/cache1/chtan/large_models/llama-hf/llama-65b' \
+    --data_path './alpaca_data_gpt4.json' \
+    --output_dir './gpt4-alpaca-lora_mlp-65b' \
+    --batch_size 128 \
+    --micro_batch_size 2 \
+    --num_epochs 10 \
+    --learning_rate 1e-4 \
+    --cutoff_len 512 \
+    --val_set_size 2000 \
+    --lora_r 16 \
+    --lora_alpha 16 \
+    --lora_dropout 0.05 \
+    --lora_target_modules '[gate_proj,down_proj,up_proj]' \
+    --train_on_inputs \
+    --group_by_length
+```
+
+### Examples
+
+**Instruction**: Tell me about alpacas.
+
+**gpt4-alpaca-lora_mlp-65b**: 
+
+Alpacas are small, domesticated mammals that are closely related to llamas. They are native to the Andes Mountains of South America, primarily in Peru, Bolivia, and Chile. These animals have been domesticated for thousands of years and were used by the Incas for their fleece, meat, and as pack animals.
+
+Alpacas are known for their soft and warm fleece, which comes in over 22 natural colors. Their fleece is highly valued and is used in the textile industry to make clothing and other products. Alpacas are also known for their friendly and gentle nature, making them popular pets and great additions to farms.
+
+These animals are social and live in herds, typically with one dominant male. They communicate through a series of hums, clucks, and other vocalizations. Alpacas are also known for their distinctive "humming" sound, which they make when they are content or curious.
+
+In terms of physical appearance, alpacas have a small, stocky build, with an average height of around 36 inches at the withers. They have large, expressive eyes, and their ears are perked forward. Alpacas are herbivores and primarily eat grasses and other vegetation. They are also known for their efficient digestive system, which allows them to thrive on a diet of low-quality forage.
+
+Overall, alpacas are unique and fascinating animals that have been an important part of Andean culture for centuries. They continue to be valued for their fleece, meat, and as pets, and their popularity continues to grow around the world.
+
+**Alpaca-LoRA**: Alpacas are members of the camelid family and are native to the Andes Mountains of South America. They are known for their soft, luxurious fleece, which is used to make clothing, blankets, and other items. Alpacas are herbivores and graze on grasses and other plants. They are social animals and live in herds of up to 20 individuals.
+
+**Stanford Alpaca**: Alpacas are small, fluffy animals related to camels and llamas. They are native to Peru and Bolivia, and were first domesticated around 5,000 years ago. They are kept mainly for their fine, soft fleece, which is used to make knitwear and other garments. Alpacas are herd animals and live in small family groups, led by an older male. They are highly social animals and can form strong bonds with their owners.
+
+**text-davinci-003**: Alpacas are a domesticated species of South American camelid, similar to llamas. They are native to the Andes Mountains in South America and are kept as livestock for their soft, luxurious wool. Alpacas are social animals and live in herds of up to 20 individuals. They are typically kept in small herds of two to five animals, and are relatively easy to care for. Alpacas are herbivores and feed on grass, hay, and other vegetation. They are also known for their gentle and friendly nature, making them popular as pets.
+
+---
+
+**Instruction**: Write a Python program that prints the first 10 Fibonacci numbers.
+
+**gpt4-alpaca-lora_mlp-65b**:
+
+Here is a Python program that prints the first 10 Fibonacci numbers:
+
+```python
+# initialize variables
+a = 0
+b = 1
+
+# loop to print the first 10 Fibonacci numbers
+for i in range(10):
+    print(a, end=" ")
+    a, b = b, a + b
+```
+
+In this program, we initialize two variables `a` and `b` with the first two Fibonacci numbers, which are 0 and 1. Then, we use a for loop to iterate 10 times. In each iteration, we print the current value of `a` and update the values of `a` and `b` using the formula `a = b` and `b = a + b`. This formula is the definition of the Fibonacci sequence.
+
+**Alpaca-LoRA**:
+
+```python
+def fibonacci(n):
+    if n == 0:
+        return 0
+    elif n == 1:
+        return 1
+    else:
+        return fibonacci(n-1) + fibonacci(n-2)
+
+print(fibonacci(10))
+```
+
+**Stanford Alpaca**:
+
+```python
+numbers = [0, 1]
+
+for i in range(2, 11):
+    numbers.append(numbers[i-2] + numbers[i-1])
+
+print(numbers)
+# Output: [0, 1, 1, 2, 3, 5, 8, 13, 21, 34]
+```
+
+**text-davinci-003**:
+
+```python
+# Fibonacci numbers are the numbers in the following integer sequence, called the Fibonacci sequence, and characterized by the fact that every number after the first two is the sum of the two preceding ones.
+
+# Fibonacci sequence: 0, 1, 1, 2, 3, 5, 8, 13, 21, 34, ...
+
+# Program:
+
+# Initialize the first two numbers of the Fibonacci sequence
+a = 0
+b = 1
+
+# Print the first 10 numbers of the Fibonacci sequence
+print(a)
+print(b)
+
+for i in range(2,10):
+    c = a + b
+    a = b
+    b = c
+    print(c)
+```
+
+---
+
+> [1] Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpatrick, Graham Neubig: Towards a Unified View of Parameter-Efficient Transfer Learning. ICLR 2022

config.json

@@ -0,0 +1,24 @@
+{
+  "_name_or_path": "/workspace/models/huggyllama_llama-65b",
+  "architectures": [
+    "LlamaForCausalLM"
+  ],
+  "bos_token_id": 1,
+  "eos_token_id": 2,
+  "hidden_act": "silu",
+  "hidden_size": 8192,
+  "initializer_range": 0.02,
+  "intermediate_size": 22016,
+  "max_position_embeddings": 2048,
+  "max_sequence_length": 2048,
+  "model_type": "llama",
+  "num_attention_heads": 64,
+  "num_hidden_layers": 80,
+  "pad_token_id": 0,
+  "rms_norm_eps": 1e-05,
+  "tie_word_embeddings": false,
+  "torch_dtype": "float16",
+  "transformers_version": "4.28.1",
+  "use_cache": true,
+  "vocab_size": 32000
+}

generation_config.json

@@ -0,0 +1,7 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 1,
+  "eos_token_id": 2,
+  "pad_token_id": 0,
+  "transformers_version": "4.28.1"
+}

gpt4-alpaca-lora_mlp-65B-GPTQ-4bit.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:072e471cd025a2e9d804481375550567afe07d82307e955c7bce40da4a5f458b
+size 33489332352

quantize_config.json

@@ -0,0 +1,5 @@
+{
+  "bits": 4,
+  "desc_act": true,
+  "group_size": -1
+}

special_tokens_map.json

@@ -0,0 +1,23 @@
+{
+  "bos_token": {
+    "content": "<s>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9e556afd44213b6bd1be2b850ebbbd98f5481437a8021afaf58ee7fb1818d347
+size 499723

tokenizer_config.json

@@ -0,0 +1,33 @@
+{
+  "add_bos_token": true,
+  "add_eos_token": false,
+  "bos_token": {
+    "__type": "AddedToken",
+    "content": "<s>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "clean_up_tokenization_spaces": false,
+  "eos_token": {
+    "__type": "AddedToken",
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "model_max_length": 2048,
+  "pad_token": null,
+  "sp_model_kwargs": {},
+  "tokenizer_class": "LlamaTokenizer",
+  "unk_token": {
+    "__type": "AddedToken",
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  }
+}