Update README.md

README.md

@@ -10,6 +10,22 @@ model-index:
 - name: starchat2-15b-v0.1
   results: []
 ---
+# Exllama v2 Quantizations of starchat2-15b-v0.1
+
+Using <a href="https://github.com/turboderp/exllamav2/releases/tag/v0.0.21">turboderp's ExLlamaV2 v0.0.21</a> for quantization.
+
+<b>The "main" branch only contains the measurement.json, download one of the other branches for the model</b>
+
+Each branch contains an individual bits per weight, with the main one containing only the meaurement.json for further conversions.
+
+Original model: <a href="https://huggingface.co/HuggingFaceH4/starchat2-15b-v0.1">HuggingFaceH4/starchat2-15b-v0.1</a><br>
+Calibration dataset based on: <a href="https://huggingface.co/datasets/bigcode/self-oss-instruct-sc2-exec-filter-50k">bigcode/self-oss-instruct-sc2-exec-filter-50k</a>
+
+| Branch | Bits | lm_head bits | VRAM (4k) | VRAM (16k) | VRAM (32k) | Description |
+| ----- | ---- | ------- | ------ | ------ | ------ | ------------ |
+| [8_0](https://huggingface.co/Apel-sin/starchat2-15b-v0.1-exl2/tree/8_0) | 8.0 | 8.0 | 16.6 GB | 17.5 GB | 18.8 GB | Maximum quality that ExLlamaV2 can produce, near unquantized performance. |
+| [6_5](https://huggingface.co/Apel-sin/starchat2-15b-v0.1-exl2/tree/6_5) | 6.5  | 8.0 | 13.9 GB | 14.9 GB | 16.2 GB | Near unquantized performance at vastly reduced size, **recommended**. |
+| [5_5](https://huggingface.co/Apel-sin/starchat2-15b-v0.1-exl2/tree/5_5) | 5.5  | 8.0 | 12.6 GB | 13.6 GB | 14.9 GB | Slightly lower quality vs 6.5. |
 
 <img src="https://huggingface.co/HuggingFaceH4/starchat2-15b-v0.1/resolve/main/model_logo.png" alt="StarChat2 15B Logo" width="800" style="margin-left:'auto' margin-right:'auto' display:'block'"/>