Upload README.md
Browse files
README.md
CHANGED
@@ -50,7 +50,7 @@ This repo contains GGUF format model files for [Henk717's Airochronos 33B](https
|
|
50 |
<!-- README_GGUF.md-about-gguf start -->
|
51 |
### About GGUF
|
52 |
|
53 |
-
GGUF is a new format introduced by the llama.cpp team on August 21st 2023. It is a replacement for GGML, which is no longer supported by llama.cpp.
|
54 |
|
55 |
Here is an incomplate list of clients and libraries that are known to support GGUF:
|
56 |
|
@@ -93,7 +93,7 @@ Below is an instruction that describes a task. Write a response that appropriate
|
|
93 |
<!-- compatibility_gguf start -->
|
94 |
## Compatibility
|
95 |
|
96 |
-
These quantised GGUFv2 files are compatible with llama.cpp from August 27th onwards, as of commit [
|
97 |
|
98 |
They are also compatible with many third party UIs and libraries - please see the list at the top of this README.
|
99 |
|
@@ -157,7 +157,7 @@ Then click Download.
|
|
157 |
I recommend using the `huggingface-hub` Python library:
|
158 |
|
159 |
```shell
|
160 |
-
pip3 install huggingface-hub
|
161 |
```
|
162 |
|
163 |
Then you can download any individual model file to the current directory, at high speed, with a command like this:
|
@@ -186,25 +186,25 @@ pip3 install hf_transfer
|
|
186 |
And set environment variable `HF_HUB_ENABLE_HF_TRANSFER` to `1`:
|
187 |
|
188 |
```shell
|
189 |
-
|
190 |
```
|
191 |
|
192 |
-
Windows
|
193 |
</details>
|
194 |
<!-- README_GGUF.md-how-to-download end -->
|
195 |
|
196 |
<!-- README_GGUF.md-how-to-run start -->
|
197 |
## Example `llama.cpp` command
|
198 |
|
199 |
-
Make sure you are using `llama.cpp` from commit [
|
200 |
|
201 |
```shell
|
202 |
-
./main -ngl 32 -m airochronos-33b.Q4_K_M.gguf --color -c
|
203 |
```
|
204 |
|
205 |
Change `-ngl 32` to the number of layers to offload to GPU. Remove it if you don't have GPU acceleration.
|
206 |
|
207 |
-
Change `-c
|
208 |
|
209 |
If you want to have a chat-style conversation, replace the `-p <PROMPT>` argument with `-i -ins`
|
210 |
|
@@ -218,22 +218,24 @@ Further instructions here: [text-generation-webui/docs/llama.cpp.md](https://git
|
|
218 |
|
219 |
You can use GGUF models from Python using the [llama-cpp-python](https://github.com/abetlen/llama-cpp-python) or [ctransformers](https://github.com/marella/ctransformers) libraries.
|
220 |
|
221 |
-
### How to load this model
|
222 |
|
223 |
#### First install the package
|
224 |
|
225 |
-
|
|
|
|
|
226 |
# Base ctransformers with no GPU acceleration
|
227 |
-
pip install ctransformers
|
228 |
# Or with CUDA GPU acceleration
|
229 |
-
pip install ctransformers[cuda]
|
230 |
-
# Or with ROCm GPU acceleration
|
231 |
-
CT_HIPBLAS=1 pip install ctransformers
|
232 |
-
# Or with Metal GPU acceleration for macOS systems
|
233 |
-
CT_METAL=1 pip install ctransformers
|
234 |
```
|
235 |
|
236 |
-
#### Simple example code
|
237 |
|
238 |
```python
|
239 |
from ctransformers import AutoModelForCausalLM
|
@@ -246,7 +248,7 @@ print(llm("AI is going to"))
|
|
246 |
|
247 |
## How to use with LangChain
|
248 |
|
249 |
-
Here
|
250 |
|
251 |
* [LangChain + llama-cpp-python](https://python.langchain.com/docs/integrations/llms/llamacpp)
|
252 |
* [LangChain + ctransformers](https://python.langchain.com/docs/integrations/providers/ctransformers)
|
|
|
50 |
<!-- README_GGUF.md-about-gguf start -->
|
51 |
### About GGUF
|
52 |
|
53 |
+
GGUF is a new format introduced by the llama.cpp team on August 21st 2023. It is a replacement for GGML, which is no longer supported by llama.cpp.
|
54 |
|
55 |
Here is an incomplate list of clients and libraries that are known to support GGUF:
|
56 |
|
|
|
93 |
<!-- compatibility_gguf start -->
|
94 |
## Compatibility
|
95 |
|
96 |
+
These quantised GGUFv2 files are compatible with llama.cpp from August 27th onwards, as of commit [d0cee0d](https://github.com/ggerganov/llama.cpp/commit/d0cee0d36d5be95a0d9088b674dbb27354107221)
|
97 |
|
98 |
They are also compatible with many third party UIs and libraries - please see the list at the top of this README.
|
99 |
|
|
|
157 |
I recommend using the `huggingface-hub` Python library:
|
158 |
|
159 |
```shell
|
160 |
+
pip3 install huggingface-hub
|
161 |
```
|
162 |
|
163 |
Then you can download any individual model file to the current directory, at high speed, with a command like this:
|
|
|
186 |
And set environment variable `HF_HUB_ENABLE_HF_TRANSFER` to `1`:
|
187 |
|
188 |
```shell
|
189 |
+
HF_HUB_ENABLE_HF_TRANSFER=1 huggingface-cli download TheBloke/airochronos-33B-GGUF airochronos-33b.Q4_K_M.gguf --local-dir . --local-dir-use-symlinks False
|
190 |
```
|
191 |
|
192 |
+
Windows Command Line users: You can set the environment variable by running `set HF_HUB_ENABLE_HF_TRANSFER=1` before the download command.
|
193 |
</details>
|
194 |
<!-- README_GGUF.md-how-to-download end -->
|
195 |
|
196 |
<!-- README_GGUF.md-how-to-run start -->
|
197 |
## Example `llama.cpp` command
|
198 |
|
199 |
+
Make sure you are using `llama.cpp` from commit [d0cee0d](https://github.com/ggerganov/llama.cpp/commit/d0cee0d36d5be95a0d9088b674dbb27354107221) or later.
|
200 |
|
201 |
```shell
|
202 |
+
./main -ngl 32 -m airochronos-33b.Q4_K_M.gguf --color -c 2048 --temp 0.7 --repeat_penalty 1.1 -n -1 -p "Below is an instruction that describes a task. Write a response that appropriately completes the request.\n\n### Instruction:\n{prompt}\n\n### Response:"
|
203 |
```
|
204 |
|
205 |
Change `-ngl 32` to the number of layers to offload to GPU. Remove it if you don't have GPU acceleration.
|
206 |
|
207 |
+
Change `-c 2048` to the desired sequence length. For extended sequence models - eg 8K, 16K, 32K - the necessary RoPE scaling parameters are read from the GGUF file and set by llama.cpp automatically.
|
208 |
|
209 |
If you want to have a chat-style conversation, replace the `-p <PROMPT>` argument with `-i -ins`
|
210 |
|
|
|
218 |
|
219 |
You can use GGUF models from Python using the [llama-cpp-python](https://github.com/abetlen/llama-cpp-python) or [ctransformers](https://github.com/marella/ctransformers) libraries.
|
220 |
|
221 |
+
### How to load this model in Python code, using ctransformers
|
222 |
|
223 |
#### First install the package
|
224 |
|
225 |
+
Run one of the following commands, according to your system:
|
226 |
+
|
227 |
+
```shell
|
228 |
# Base ctransformers with no GPU acceleration
|
229 |
+
pip install ctransformers
|
230 |
# Or with CUDA GPU acceleration
|
231 |
+
pip install ctransformers[cuda]
|
232 |
+
# Or with AMD ROCm GPU acceleration (Linux only)
|
233 |
+
CT_HIPBLAS=1 pip install ctransformers --no-binary ctransformers
|
234 |
+
# Or with Metal GPU acceleration for macOS systems only
|
235 |
+
CT_METAL=1 pip install ctransformers --no-binary ctransformers
|
236 |
```
|
237 |
|
238 |
+
#### Simple ctransformers example code
|
239 |
|
240 |
```python
|
241 |
from ctransformers import AutoModelForCausalLM
|
|
|
248 |
|
249 |
## How to use with LangChain
|
250 |
|
251 |
+
Here are guides on using llama-cpp-python and ctransformers with LangChain:
|
252 |
|
253 |
* [LangChain + llama-cpp-python](https://python.langchain.com/docs/integrations/llms/llamacpp)
|
254 |
* [LangChain + ctransformers](https://python.langchain.com/docs/integrations/providers/ctransformers)
|