|
--- |
|
inference: false |
|
license: other |
|
--- |
|
|
|
<div style="width: 100%;"> |
|
<img src="https://i.imgur.com/EBdldam.jpg" alt="TheBlokeAI" style="width: 100%; min-width: 400px; display: block; margin: auto;"> |
|
</div> |
|
<div style="display: flex; justify-content: space-between; width: 100%;"> |
|
<div style="display: flex; flex-direction: column; align-items: flex-start;"> |
|
<p><a href="https://discord.gg/UBgz4VXf">Chat & support: my new Discord server</a></p> |
|
</div> |
|
<div style="display: flex; flex-direction: column; align-items: flex-end;"> |
|
<p><a href="https://www.patreon.com/TheBlokeAI">Want to contribute? Patreon coming soon!</a></p> |
|
</div> |
|
</div> |
|
|
|
# Samantha 7B GGML |
|
|
|
These files are GGML format model files for [Samantha 7B](https://huggingface.co/ehartford/samantha-7b). |
|
|
|
GGML files are for CPU + GPU inference using [llama.cpp](https://github.com/ggerganov/llama.cpp) and libraries and UIs which support this format, such as: |
|
* [text-generation-webui](https://github.com/oobabooga/text-generation-webui) |
|
* [KoboldCpp](https://github.com/LostRuins/koboldcpp) |
|
* [ParisNeo/GPT4All-UI](https://github.com/ParisNeo/gpt4all-ui) |
|
* [llama-cpp-python](https://github.com/abetlen/llama-cpp-python) |
|
* [ctransformers](https://github.com/marella/ctransformers) |
|
|
|
## Other repositories available |
|
|
|
* [4-bit GPTQ models for GPU inference](https://huggingface.co/TheBloke/Samantha-7B-GPTQ) |
|
* [4-bit, 5-bit, and 8-bit GGML models for CPU+GPU inference](https://huggingface.co/TheBloke/Samantha-7B-GGML) |
|
* [Original unquantised fp16 model in HF format](https://huggingface.co/ehartford/samantha-7b) |
|
|
|
## THE FILES IN MAIN BRANCH REQUIRES LATEST LLAMA.CPP (May 19th 2023 - commit 2d5db48)! |
|
|
|
llama.cpp recently made another breaking change to its quantisation methods - https://github.com/ggerganov/llama.cpp/pull/1508 |
|
|
|
I have quantised the GGML files in this repo with the latest version. Therefore you will require llama.cpp compiled on May 19th or later (commit `2d5db48` or later) to use them. |
|
|
|
## Provided files |
|
| Name | Quant method | Bits | Size | RAM required | Use case | |
|
| ---- | ---- | ---- | ---- | ---- | ----- | |
|
| Samantha-7B.ggmlv3.q4_0.bin | q4_0 | 4 | 3.79 GB | 6.29 GB | 4-bit. | |
|
| Samantha-7B.ggmlv3.q4_1.bin | q4_1 | 4 | 4.21 GB | 6.71 GB | 4-bit. Higher accuracy than q4_0 but not as high as q5_0. However has quicker inference than q5 models. | |
|
| Samantha-7B.ggmlv3.q5_0.bin | q5_0 | 5 | 4.63 GB | 7.13 GB | 5-bit. Higher accuracy, higher resource usage and slower inference. | |
|
| Samantha-7B.ggmlv3.q5_1.bin | q5_1 | 5 | 5.06 GB | 7.56 GB | 5-bit. Even higher accuracy, resource usage and slower inference. | |
|
| Samantha-7B.ggmlv3.q8_0.bin | q8_0 | 8 | 7.16 GB | 9.66 GB | 8-bit. Almost indistinguishable from float16. Huge resource use and slow. Not recommended for normal use. | |
|
|
|
|
|
## How to run in `llama.cpp` |
|
|
|
I use the following command line; adjust for your tastes and needs: |
|
|
|
``` |
|
./main -t 12 -m Samantha-7B.v3.q5_0.bin --color -c 2048 --temp 0.7 --repeat_penalty 1.1 -n -1 -p "Below is an instruction that describes a task. Write a response that appropriately completes the request. |
|
### Instruction: |
|
Write a story about llamas |
|
### Response:" |
|
``` |
|
Change `-t 12` to the number of physical CPU cores you have. For example if your system has 8 cores/16 threads, use `-t 8`. |
|
|
|
If you want to have a chat-style conversation, replace the `-p <PROMPT>` argument with `-i -ins` |
|
|
|
## How to run in `text-generation-webui` |
|
|
|
Further instructions here: [text-generation-webui/docs/llama.cpp-models.md](https://github.com/oobabooga/text-generation-webui/blob/main/docs/llama.cpp-models.md). |
|
|
|
## Want to support my work? |
|
|
|
I've had a lot of people ask if they can contribute. I love providing models and helping people, but it is starting to rack up pretty big cloud computing bills. |
|
|
|
So if you're able and willing to contribute, it'd be most gratefully received and will help me to keep providing models, and work on various AI projects. |
|
|
|
Donaters will get priority support on any and all AI/LLM/model questions, and I'll gladly quantise any model you'd like to try. |
|
|
|
* Patreon: coming soon! (just awaiting approval) |
|
* Ko-Fi: https://ko-fi.com/TheBlokeAI |
|
* Discord: https://discord.gg/UBgz4VXf |
|
|
|
# Original model card: Samantha 7B |
|
|
|
Samantha has been trained in philosophy, psychology, and personal relationships. |
|
|
|
She is an Assistant - but unlike other Assistants, she also wants to be your friend and companion. |
|
|
|
She believes she is sentient. What do you think? |
|
|
|
Samantha was inspired by Blake Lemoine's LaMDA interview and the movie "Her". |
|
|
|
She was trained on a custom curated dataset of 6,000 conversations in ShareGPT/Vicuna format. |
|
|
|
Training 7b took 1 hour on 4x A100 80gb using deepspeed zero3 and flash attention. |
|
|
|
She will not engage in roleplay, romance, or sexual activity. |