File size: 2,382 Bytes
a4ac538 3f0da90 98775ac dd633c5 98775ac e0a1997 98775ac 132eae3 98775ac |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 |
---
license: llama2
---
EXL2 quant of alpindale/goliath-120b (https://huggingface.co/alpindale/goliath-120b), to be used on exllamav2. 4.25bpw to being to able to use CFG comfortably on 72GB VRAM. (20,21,22 for gpu split)
Update 06/01/2024: Updated with new quant method after some time, thanks for the measurement [here](https://github.com/turboderp/exllamav2/files/13846439/goliath-120b-rpcal-measurement.json)
Calibration dataset is a cleaned, fixed pippa RP dataset, which does affect the results (in favor) for RP usage.
You can find the calibration dataset [here](https://huggingface.co/datasets/royallab/PIPPA-cleaned)
I've added a measurement.json file if you want to do your own quants.
# Original model card
# Goliath 120B
An auto-regressive causal LM created by combining 2x finetuned [Llama-2 70B](https://huggingface.co/meta-llama/llama-2-70b-hf) into one.
Please check out the quantized formats provided by [@TheBloke](https:///huggingface.co/TheBloke) and [@Panchovix](https://huggingface.co/Panchovix):
- [GGUF](https://huggingface.co/TheBloke/goliath-120b-GGUF) (llama.cpp)
- [GPTQ](https://huggingface.co/TheBloke/goliath-120b-GPTQ) (KoboldAI, TGW, Aphrodite)
- [AWQ](https://huggingface.co/TheBloke/goliath-120b-AWQ) (TGW, Aphrodite, vLLM)
- [Exllamav2](https://huggingface.co/Panchovix/goliath-120b-exl2) (TGW, KoboldAI)
# Prompting Format
Both Vicuna and Alpaca will work, but due the initial and final layers belonging primarily to Xwin, I expect Vicuna to work the best.
# Merge process
The models used in the merge are [Xwin](https://huggingface.co/Xwin-LM/Xwin-LM-70B-V0.1) and [Euryale](https://huggingface.co/Sao10K/Euryale-1.3-L2-70B).
The layer ranges used are as follows:
```yaml
- range 0, 16
Xwin
- range 8, 24
Euryale
- range 17, 32
Xwin
- range 25, 40
Euryale
- range 33, 48
Xwin
- range 41, 56
Euryale
- range 49, 64
Xwin
- range 57, 72
Euryale
- range 65, 80
Xwin
```
# Screenshots
# Benchmarks
Coming soon.
# Acknowledgements
Credits goes to [@chargoddard](https://huggingface.co/chargoddard) for developing the framework used to merge the model - [mergekit](https://github.com/cg123/mergekit).
Special thanks to [@Undi95](https://huggingface.co/Undi95) for helping with the merge ratios. |