--- license: other library_name: transformers pipeline_tag: text-generation datasets: - RyokoAI/ShareGPT52K - Hello-SimpleAI/HC3 tags: - koala - ShareGPT - llama - gptq inference: false --- # Koala: A Dialogue Model for Academic Research This repo contains the weights of the Koala 7B model produced at Berkeley. It is the result of combining the diffs from https://huggingface.co/young-geng/koala with the original Llama 7B model. This version has then been quantized to 4-bit and 5-bit GGML for use with [llama.cpp](https://github.com/ggerganov/llama.cpp). ## My Koala repos I have the following Koala model repositories available: **13B models:** * [Unquantized 13B model in HF format](https://huggingface.co/TheBloke/koala-13B-HF) * [GPTQ quantized 4bit 13B model in `pt` and `safetensors` formats](https://huggingface.co/TheBloke/koala-13B-GPTQ-4bit-128g) * [4-bit, 5-bit and 8-bit GGML models for `llama.cpp`](https://huggingface.co/TheBloke/koala-13B-GGML) **7B models:** * [Unquantized 7B model in HF format](https://huggingface.co/TheBloke/koala-7B-HF) * [Unquantized 7B model in GGML format for llama.cpp](https://huggingface.co/TheBloke/koala-7b-ggml-unquantized) * [GPTQ quantized 4bit 7B model in `pt` and `safetensors` formats](https://huggingface.co/TheBloke/koala-7B-GPTQ-4bit-128g) * [4-bit, 5-bit and 8-bit GGML models for `llama.cpp`](https://huggingface.co/TheBloke/koala-7B-GGML) ## THE FILES IN MAIN BRANCH REQUIRES LATEST LLAMA.CPP (May 19th 2023 - commit 2d5db48)! llama.cpp recently made another breaking change to its quantisation methods - https://github.com/ggerganov/llama.cpp/pull/1508 I have quantised the GGML files in this repo with the latest version. Therefore you will require llama.cpp compiled on May 19th or later (commit `2d5db48` or later) to use them. For files compatible with the previous version of llama.cpp, please see branch `previous_llama_ggmlv2`. ## How to run in `llama.cpp` I use the following command line; adjust for your tastes and needs: ``` ./main -t 18 -m koala-7B-4bit-128g.ggmlv3.q5_0.bin --color -c 2048 --temp 0.7 --repeat_penalty 1.1 -n -1 -p "BEGINNING OF CONVERSATION: USER: GPT:" ``` Change `-t 18` to the number of physical CPU cores you have. For example if your system has 8 cores, 16 threads, use `-t 8`. This model should be able to run in 8GB RAM without swapping. ## How the Koala delta weights were merged The Koala delta weights were originally merged using the following commands, producing [koala-7B-HF](https://huggingface.co/TheBloke/koala-7B-HF): ``` git clone https://github.com/young-geng/EasyLM git clone https://huggingface.co/nyanko7/LLaMA-7B mkdir koala_diffs && cd koala_diffs && wget https://huggingface.co/young-geng/koala/resolve/main/koala_7b_diff_v2 cd EasyLM PYTHON_PATH="${PWD}:$PYTHONPATH" python \ -m EasyLM.models.llama.convert_torch_to_easylm \ --checkpoint_dir=/content/LLaMA-7B \ --output_file=/content/llama-7B-LM \ --streaming=True PYTHON_PATH="${PWD}:$PYTHONPATH" python \ -m EasyLM.scripts.diff_checkpoint --recover_diff=True \ --load_base_checkpoint='params::/content/llama-7B-LM' \ --load_target_checkpoint='params::/content/koala_diffs/koala_7b_diff_v2' \ --output_file=/content/koala_7b.diff.weights \ --streaming=True PYTHON_PATH="${PWD}:$PYTHONPATH" python \ -m EasyLM.models.llama.convert_easylm_to_hf --model_size=7b \ --output_dir=/content/koala-7B-HF \ --load_checkpoint='params::/content/koala_7b.diff.weights' \ --tokenizer_path=/content/LLaMA-7B/tokenizer.model ``` ## Further info Check out the following links to learn more about the Berkeley Koala model. * [Blog post](https://bair.berkeley.edu/blog/2023/04/03/koala/) * [Online demo](https://koala.lmsys.org/) * [EasyLM: training and serving framework on GitHub](https://github.com/young-geng/EasyLM) * [Documentation for running Koala locally](https://github.com/young-geng/EasyLM/blob/main/docs/koala.md) ## License The model weights are intended for academic research only, subject to the [model License of LLaMA](https://github.com/facebookresearch/llama/blob/main/MODEL_CARD.md), [Terms of Use of the data generated by OpenAI](https://openai.com/policies/terms-of-use), and [Privacy Practices of ShareGPT](https://chrome.google.com/webstore/detail/sharegpt-share-your-chatg/daiacboceoaocpibfodeljbdfacokfjb). Any other usage of the model weights, including but not limited to commercial usage, is strictly prohibited.