|
--- |
|
license: other |
|
library_name: transformers |
|
pipeline_tag: text-generation |
|
datasets: |
|
- RyokoAI/ShareGPT52K |
|
- Hello-SimpleAI/HC3 |
|
tags: |
|
- koala |
|
- ShareGPT |
|
- llama |
|
- gptq |
|
inference: false |
|
--- |
|
<!-- header start --> |
|
<div style="width: 100%;"> |
|
<img src="https://i.imgur.com/EBdldam.jpg" alt="TheBlokeAI" style="width: 100%; min-width: 400px; display: block; margin: auto;"> |
|
</div> |
|
<div style="display: flex; justify-content: space-between; width: 100%;"> |
|
<div style="display: flex; flex-direction: column; align-items: flex-start;"> |
|
<p><a href="https://discord.gg/Jq4vkcDakD">Chat & support: my new Discord server</a></p> |
|
</div> |
|
<div style="display: flex; flex-direction: column; align-items: flex-end;"> |
|
<p><a href="https://www.patreon.com/TheBlokeAI">Want to contribute? TheBloke's Patreon page</a></p> |
|
</div> |
|
</div> |
|
<!-- header end --> |
|
# Koala: A Dialogue Model for Academic Research |
|
This repo contains the weights of the Koala 7B model produced at Berkeley. It is the result of combining the diffs from https://huggingface.co/young-geng/koala with the original Llama 7B model. |
|
|
|
This version has then been quantized to 4-bit and 5-bit GGML for use with [llama.cpp](https://github.com/ggerganov/llama.cpp). |
|
|
|
## My Koala repos |
|
I have the following Koala model repositories available: |
|
|
|
**13B models:** |
|
* [Unquantized 13B model in HF format](https://huggingface.co/TheBloke/koala-13B-HF) |
|
* [GPTQ quantized 4bit 13B model in `pt` and `safetensors` formats](https://huggingface.co/TheBloke/koala-13B-GPTQ-4bit-128g) |
|
* [4-bit, 5-bit and 8-bit GGML models for `llama.cpp`](https://huggingface.co/TheBloke/koala-13B-GGML) |
|
|
|
**7B models:** |
|
* [Unquantized 7B model in HF format](https://huggingface.co/TheBloke/koala-7B-HF) |
|
* [Unquantized 7B model in GGML format for llama.cpp](https://huggingface.co/TheBloke/koala-7b-ggml-unquantized) |
|
* [GPTQ quantized 4bit 7B model in `pt` and `safetensors` formats](https://huggingface.co/TheBloke/koala-7B-GPTQ-4bit-128g) |
|
* [4-bit, 5-bit and 8-bit GGML models for `llama.cpp`](https://huggingface.co/TheBloke/koala-7B-GGML) |
|
|
|
## THE FILES IN MAIN BRANCH REQUIRES LATEST LLAMA.CPP (May 19th 2023 - commit 2d5db48)! |
|
|
|
llama.cpp recently made another breaking change to its quantisation methods - https://github.com/ggerganov/llama.cpp/pull/1508 |
|
|
|
I have quantised the GGML files in this repo with the latest version. Therefore you will require llama.cpp compiled on May 19th or later (commit `2d5db48` or later) to use them. |
|
|
|
For files compatible with the previous version of llama.cpp, please see branch `previous_llama_ggmlv2`. |
|
|
|
## How to run in `llama.cpp` |
|
|
|
I use the following command line; adjust for your tastes and needs: |
|
|
|
``` |
|
./main -t 18 -m koala-7B-4bit-128g.ggmlv3.q5_0.bin --color -c 2048 --temp 0.7 --repeat_penalty 1.1 -n -1 -p "BEGINNING OF CONVERSATION: |
|
USER: <PROMPT GOES HERE> |
|
GPT:" |
|
``` |
|
|
|
Change `-t 18` to the number of physical CPU cores you have. For example if your system has 8 cores, 16 threads, use `-t 8`. |
|
|
|
This model should be able to run in 8GB RAM without swapping. |
|
|
|
## How the Koala delta weights were merged |
|
|
|
The Koala delta weights were originally merged using the following commands, producing [koala-7B-HF](https://huggingface.co/TheBloke/koala-7B-HF): |
|
``` |
|
git clone https://github.com/young-geng/EasyLM |
|
|
|
git clone https://huggingface.co/nyanko7/LLaMA-7B |
|
|
|
mkdir koala_diffs && cd koala_diffs && wget https://huggingface.co/young-geng/koala/resolve/main/koala_7b_diff_v2 |
|
|
|
cd EasyLM |
|
|
|
PYTHON_PATH="${PWD}:$PYTHONPATH" python \ |
|
-m EasyLM.models.llama.convert_torch_to_easylm \ |
|
--checkpoint_dir=/content/LLaMA-7B \ |
|
--output_file=/content/llama-7B-LM \ |
|
--streaming=True |
|
|
|
PYTHON_PATH="${PWD}:$PYTHONPATH" python \ |
|
-m EasyLM.scripts.diff_checkpoint --recover_diff=True \ |
|
--load_base_checkpoint='params::/content/llama-7B-LM' \ |
|
--load_target_checkpoint='params::/content/koala_diffs/koala_7b_diff_v2' \ |
|
--output_file=/content/koala_7b.diff.weights \ |
|
--streaming=True |
|
|
|
PYTHON_PATH="${PWD}:$PYTHONPATH" python \ |
|
-m EasyLM.models.llama.convert_easylm_to_hf --model_size=7b \ |
|
--output_dir=/content/koala-7B-HF \ |
|
--load_checkpoint='params::/content/koala_7b.diff.weights' \ |
|
--tokenizer_path=/content/LLaMA-7B/tokenizer.model |
|
``` |
|
|
|
<!-- footer start --> |
|
## Discord |
|
|
|
For further support, and discussions on these models and AI in general, join us at: |
|
|
|
[TheBloke AI's Discord server](https://discord.gg/Jq4vkcDakD) |
|
|
|
## Thanks, and how to contribute. |
|
|
|
Thanks to the [chirper.ai](https://chirper.ai) team! |
|
|
|
I've had a lot of people ask if they can contribute. I enjoy providing models and helping people, and would love to be able to spend even more time doing it, as well as expanding into new projects like fine tuning/training. |
|
|
|
If you're able and willing to contribute it will be most gratefully received and will help me to keep providing more models, and to start work on new AI projects. |
|
|
|
Donaters will get priority support on any and all AI/LLM/model questions and requests, access to a private Discord room, plus other benefits. |
|
|
|
* Patreon: https://patreon.com/TheBlokeAI |
|
* Ko-Fi: https://ko-fi.com/TheBlokeAI |
|
|
|
**Patreon special mentions**: Aemon Algiz, Dmitriy Samsonov, Nathan LeClaire, Trenton Dambrowitz, Mano Prime, David Flickinger, vamX, Nikolai Manek, senxiiz, Khalefa Al-Ahmad, Illia Dulskyi, Jonathan Leane, Talal Aujan, V. Lukas, Joseph William Delisle, Pyrater, Oscar Rangel, Lone Striker, Luke Pendergrass, Eugene Pentland, Sebastain Graf, Johann-Peter Hartman. |
|
|
|
Thank you to all my generous patrons and donaters! |
|
<!-- footer end --> |
|
## Further info |
|
|
|
Check out the following links to learn more about the Berkeley Koala model. |
|
* [Blog post](https://bair.berkeley.edu/blog/2023/04/03/koala/) |
|
* [Online demo](https://koala.lmsys.org/) |
|
* [EasyLM: training and serving framework on GitHub](https://github.com/young-geng/EasyLM) |
|
* [Documentation for running Koala locally](https://github.com/young-geng/EasyLM/blob/main/docs/koala.md) |
|
|
|
## License |
|
The model weights are intended for academic research only, subject to the |
|
[model License of LLaMA](https://github.com/facebookresearch/llama/blob/main/MODEL_CARD.md), |
|
[Terms of Use of the data generated by OpenAI](https://openai.com/policies/terms-of-use), |
|
and [Privacy Practices of ShareGPT](https://chrome.google.com/webstore/detail/sharegpt-share-your-chatg/daiacboceoaocpibfodeljbdfacokfjb). |
|
Any other usage of the model weights, including but not limited to commercial usage, is strictly prohibited. |
|
|
