README.md

---
license: other
---

# OpenAssistant LLaMa-Based Models

Due to the license attached to LLaMa models by Meta AI it is not possible to directly distribute LLaMa-based models. Instead we provide XOR weights for the OA models.

Thanks to Mick for writing the `xor_codec.py` script which enables this process

## The Process

Note: This process applies to `oasst-sft-6-llama-30b` model. The same process can be applied to other models in future, but the checksums will be different..

To use OpenAssistant LLaMa-Based Models, you need to have a copy of the original LLaMa model weights and add them to a `llama` subdirectory here.

Ensure your LLaMa 30B checkpoint matches the correct md5sums:

```
f856e9d99c30855d6ead4d00cc3a5573  consolidated.00.pth
d9dbfbea61309dc1e087f5081e98331a  consolidated.01.pth
2b2bed47912ceb828c0a37aac4b99073  consolidated.02.pth
ea0405cdb5bc638fee12de614f729ebc  consolidated.03.pth
4babdbd05b8923226a9e9622492054b6  params.json
```

**Important: Follow these exact steps to convert your original LLaMa checkpoint to a HuggingFace Transformers-compatible format. If you use the wrong versions of any dependency, you risk ending up with weights which are not compatible with the XOR files.**

1. Create a clean Python **3.10** virtual environment & activate it:

```
python3.10 -m venv xor_venv
source xor_venv/bin/activate
```

2. Clone transformers repo and switch to tested version:

```
git clone https://github.com/huggingface/transformers.git
cd transformers
git checkout d04ec99bec8a0b432fc03ed60cea9a1a20ebaf3c
pip install .
```

3. Install **exactly** these dependency versions:

```
pip install torch==1.13.1 accelerate==0.18.0 sentencepiece==0.1.98 protobuf==3.20.1
```

4. Check `pip freeze` output:

```
accelerate==0.18.0
certifi==2022.12.7
charset-normalizer==3.1.0
filelock==3.12.0
huggingface-hub==0.13.4
idna==3.4
numpy==1.24.2
nvidia-cublas-cu11==11.10.3.66
nvidia-cuda-nvrtc-cu11==11.7.99
nvidia-cuda-runtime-cu11==11.7.99
nvidia-cudnn-cu11==8.5.0.96
packaging==23.1
protobuf==3.20.1
psutil==5.9.5
PyYAML==6.0
regex==2023.3.23
requests==2.28.2
sentencepiece==0.1.98
tokenizers==0.13.3
torch==1.13.1
tqdm==4.65.0
transformers @ file:///mnt/data/koepf/transformers
typing_extensions==4.5.0
urllib3==1.26.15
```

5. While in `transformers` repo root, run HF LLaMA conversion script:

```
python src/transformers/models/llama/convert_llama_weights_to_hf.py --input_dir <input_path_llama_base>  --output_dir <output_path_llama30b_hf> --model_size 30B
```

6. Run `find . -type f -exec md5sum "{}" +` in the conversion target directory (`output_dir`). This should produce exactly the following checksums if your files are correct:

```
462a2d07f65776f27c0facfa2affb9f9  ./pytorch_model-00007-of-00007.bin
e1dc8c48a65279fb1fbccff14562e6a3  ./pytorch_model-00003-of-00007.bin
9cffb1aeba11b16da84b56abb773d099  ./pytorch_model-00001-of-00007.bin
aee09e21813368c49baaece120125ae3  ./generation_config.json
92754d6c6f291819ffc3dfcaf470f541  ./pytorch_model-00005-of-00007.bin
3eddc6fc02c0172d38727e5826181adb  ./pytorch_model-00004-of-00007.bin
eeec4125e9c7560836b4873b6f8e3025  ./tokenizer.model
99762d59efa6b96599e863893cf2da02  ./pytorch_model-00006-of-00007.bin
598538f18fed1877b41f77de034c0c8a  ./config.json
fdb311c39b8659a5d5c1991339bafc09  ./tokenizer.json
fecfda4fba7bfd911e187a85db5fa2ef  ./pytorch_model.bin.index.json
edd1a5897748864768b1fab645b31491  ./tokenizer_config.json
6b2e0a735969660e720c27061ef3f3d3  ./special_tokens_map.json
5cfcb78b908ffa02e681cce69dbe4303  ./pytorch_model-00002-of-00007.bin
```

**Important: You should now have the correct LLaMa weights and be ready to apply the XORs. If the checksums above do not match yours, there is a problem.**

7. Once you have LLaMa weights in the correct format, you can apply the XOR decoding:

```
python xor_codec.py oasst-sft-6-llama-30b/ oasst-sft-6-llama-30b-xor/ llama30b_hf/
```

You should **expect to see one warning message** during execution:

`Exception when processing 'added_tokens.json'`

This is normal. **If similar messages appear for other files, something has gone wrong**.

8. Now run `find . -type f -exec md5sum "{}" +` in the output directory (here `oasst-sft-6-llama-30b`). You should get a file with exactly these checksums:

```
970e99665d66ba3fad6fdf9b4910acc5  ./pytorch_model-00007-of-00007.bin
659fcb7598dcd22e7d008189ecb2bb42  ./pytorch_model-00003-of-00007.bin
ff6e4cf43ddf02fb5d3960f850af1220  ./pytorch_model-00001-of-00007.bin
27b0dc092f99aa2efaf467b2d8026c3f  ./added_tokens.json
2917a1cafb895cf57e746cfd7696bfe5  ./generation_config.json
740c324ae65b1ec25976643cda79e479  ./pytorch_model-00005-of-00007.bin
f7aefb4c63be2ac512fd905b45295235  ./pytorch_model-00004-of-00007.bin
eeec4125e9c7560836b4873b6f8e3025  ./tokenizer.model
369df2f0e38bda0d9629a12a77c10dfc  ./pytorch_model-00006-of-00007.bin
cc9dbf56b68b68a585cc7367696e06a7  ./config.json
76d47e4f51a8df1d703c6f594981fcab  ./pytorch_model.bin.index.json
fd9452959d711be29ccf04a97598e8d1  ./tokenizer_config.json
785905630a0fe583122a8446a5abe287  ./special_tokens_map.json
ae48c4c68e4e171d502dd0896aa19a84  ./pytorch_model-00002-of-00007.bin
```

If so you have successfully decoded the weights and should be able to use the model with HuggingFace Transformers. **If your checksums do not match those above, there is a problem.**