asedmammad commited on
Commit
81ac2a3
1 Parent(s): e497e73

Create README.md

Browse files
Files changed (1) hide show
  1. README.md +48 -0
README.md ADDED
@@ -0,0 +1,48 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ inference: false
3
+ ---
4
+
5
+ # Ejafa's Vicuna Vanilla 1.1 7B GGML
6
+
7
+
8
+ These files are GGML format model files for [Ejafa's Vicuna Vanilla 1.1 7B](https://huggingface.co/Ejafa/vicuna_7B_vanilla_1.1).
9
+
10
+ GGML files are for CPU + GPU inference using [llama.cpp](https://github.com/ggerganov/llama.cpp) and libraries and UIs which support this format, such as:
11
+ * [text-generation-webui](https://github.com/oobabooga/text-generation-webui)
12
+ * [KoboldCpp](https://github.com/LostRuins/koboldcpp)
13
+ * [ParisNeo/GPT4All-UI](https://github.com/ParisNeo/gpt4all-ui)
14
+ * [llama-cpp-python](https://github.com/abetlen/llama-cpp-python)
15
+ * [ctransformers](https://github.com/marella/ctransformers)
16
+
17
+ ## How to run in `llama.cpp`
18
+
19
+ I use the following command line; adjust for your tastes and needs:
20
+
21
+ ```
22
+ ./main -t 8 -ngl 32 -m vicuna_7B_vanilla_1.1.ggmlv3.q5_0.bin --color -c 2048 --temp 0.7 --repeat_penalty 1.1 -n -1 -p "prompt goes here"
23
+ ```
24
+ Change `-t 8` to the number of physical CPU cores you have. For example if your system has 8 cores/16 threads, use `-t 8`.
25
+
26
+ Change `-ngl 32` to the number of layers to offload to GPU. Remove it if you don't have GPU acceleration.
27
+
28
+ If you want to have a chat-style conversation, replace the `-p <PROMPT>` argument with `-i -ins`
29
+
30
+ ## Compatibility
31
+
32
+ I have uploded bothe the original llama.cpp quant methods (`q4_0, q4_1, q5_0, q5_1, q8_0`) as well as the new k-quant methods (`q2_K, q3_K_S, q3_K_M, q3_K_L, q4_K_S, q4_K_M, q5_K_S, q6_K`).
33
+
34
+ Please refer to [llama.cpp](https://github.com/ggerganov/llama.cpp) and [TheBloke](https://huggingface.co/TheBloke)'s GGML models for further explanation.
35
+
36
+ ## How to run in `text-generation-webui`
37
+
38
+ Further instructions here: [text-generation-webui/docs/llama.cpp-models.md](https://github.com/oobabooga/text-generation-webui/blob/main/docs/llama.cpp-models.md).
39
+
40
+ <!-- footer start -->
41
+ ## Thanks
42
+
43
+ Thanks to the [TheBloke](https://huggingface.co/TheBloke) for inspiring and providing almost all of the readme here!
44
+
45
+ Thanks to the [Ejafa](https://huggingface.co/Ejafa) for providing checkpoints of the model.
46
+
47
+ Thanks to the [Georgi Gerganov](https://github.com/ggerganov) and all of the awesome people in the AI community.
48
+