Update README.md

README.md

@@ -25,54 +25,97 @@ This repo includes two types of quantized models: **GGUF** and **AWQ**, for our
 
 
 # GGUF Qauntization
-Run with [Ollama](https://github.com/ollama/ollama)
+## (Recommended) Run with [llama.cpp](https://github.com/ggerganov/llama.cpp)
 
+1. **Clone and compile:**
+
+```bash
+   git clone https://github.com/ggerganov/llama.cpp
+   cd llama.cpp
+   # Compile the source code:
+   make
+```
+
+2. **Prepare the Input Prompt File:**
+   
+Navigate to the `prompt` folder inside the `llama.cpp`, and create a new file named `chat-with-octopus.txt`.
+
+   `chat-with-octopus.txt`:
+
+```bash
+   User: 
+```
+  
+3. **Execute the Model:**
+
+Run the following command in the terminal:
+
+```bash
+   ./main -m ./path/to/octopus-v2-Q4_K_M.gguf -c 512 -b 2048 -n 256 -t 1 --repeat_penalty 1.0 --top_k 0 --top_p 1.0 --color -i -r "User:" -f prompts/chat-with-octopus.txt
+```
+
+Example prompt to interact  
+```bash
+  <|system|>You are a router. Below is the query from the users, please call the correct function and generate the parameters to call the function.<|end|><|user|>Query: Take a selfie for me with front camera<|end|><|assistant|>
+```
+
+## Run with [Ollama](https://github.com/ollama/ollama)
+1. Create a `Modelfile` in your directory and include a `FROM` statement with the path to your local model:
 ```bash
-ollama run NexaAIDev/octopus-v2-Q4_K_M
+FROM ./path/to/octopus-v2-Q4_K_M.gguf
+```
+
+2. Use the following command to add the model to Ollama:
+```bash
+ollama create octopus-v2-Q4_K_M -f Modelfile
+```
+
+3. Verify that the model has been successfully imported:
+```bash
+ollama ls
+```
+
+### Run the model
+```bash
+ollama run octopus-v2-Q4_K_M "<|system|>You are a router. Below is the query from the users, please call the correct function and generate the parameters to call the function.<|end|><|user|>Query: Take a selfie for me with front camera<|end|><|assistant|>"
 ```
 
 # AWQ Quantization
 Python example:
 
 ```python
+from transformers import AutoTokenizer
 from awq import AutoAWQForCausalLM
-from transformers import AutoTokenizer, GemmaForCausalLM
 import torch
 import time
 import numpy as np
 
 def inference(input_text):
-
-    tokens = tokenizer(
-        input_text,
-        return_tensors='pt'
-    ).input_ids.cuda()
-
     start_time = time.time()
+    input_ids = tokenizer(input_text, return_tensors="pt").to('cuda')
+    input_length = input_ids["input_ids"].shape[1]
     generation_output = model.generate(
-        tokens,
-        do_sample=True,
-        temperature=0.7,
-        top_p=0.95,
-        top_k=40,
-        max_new_tokens=512
+        input_ids["input_ids"],
+        do_sample=False,
+        max_length=1024
     )
     end_time = time.time()
 
-    res = tokenizer.decode(generation_output[0])
-    res = res.split(input_text)
+    # Decode only the generated part
+    generated_sequence = generation_output[:, input_length:].tolist()
+    res = tokenizer.decode(generated_sequence[0])
+
     latency = end_time - start_time
-    output_tokens = tokenizer.encode(res)
-    num_output_tokens = len(output_tokens)
+    num_output_tokens = len(generated_sequence[0])
     throughput = num_output_tokens / latency
 
-    return {"output": res[-1], "latency": latency, "throughput": throughput}
+    return {"output": res, "latency": latency, "throughput": throughput}
 
-
-model_id = "path/to/Octopus-v2-AWQ"
+# Initialize tokenizer and model
+model_id = "/home/mingyuanma/Octopus-v2-AWQ-NexaAIDev"
+tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=False)
 model = AutoAWQForCausalLM.from_quantized(model_id, fuse_layers=True,
                                           trust_remote_code=False, safetensors=True)
-tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=False)
 
 prompts = ["Below is the query from the users, please call the correct function and generate the parameters to call the function.\n\nQuery: Can you take a photo using the back camera and save it to the default location? \n\nResponse:"]
 
@@ -115,4 +158,3 @@ _Quantized with llama.cpp_
 
 **Acknowledgement**:  
 We sincerely thank our community members, [Mingyuan](https://huggingface.co/ThunderBeee), [Zoey](https://huggingface.co/ZY6), [Brian](https://huggingface.co/JoyboyBrian), [Perry](https://huggingface.co/PerryCheng614), [Qi](https://huggingface.co/qiqiWav), [David](https://huggingface.co/Davidqian123) for their extraordinary contributions to this quantization effort.
-