Text Generation
Edit model card

Llamacpp Quantizations of airoboros-34b-3.2

Using llama.cpp release b2334 for quantization.

Original model: https://huggingface.co/jondurbin/airoboros-34b-3.2

Download a file (not the whole branch) from below:

Filename Quant type File Size Description
airoboros-34b-3.2-Q8_0.gguf Q8_0 36.54GB Extremely high quality, generally unneeded but max available quant.
airoboros-34b-3.2-Q6_K.gguf Q6_K 28.21GB Very high quality, near perfect, recommended.
airoboros-34b-3.2-Q5_K_M.gguf Q5_K_M 24.32GB High quality, very usable.
airoboros-34b-3.2-Q5_K_S.gguf Q5_K_S 23.70GB High quality, very usable.
airoboros-34b-3.2-Q5_0.gguf Q5_0 23.70GB High quality, older format, generally not recommended.
airoboros-34b-3.2-Q4_K_M.gguf Q4_K_M 20.65GB Good quality, similar to 4.25 bpw.
airoboros-34b-3.2-Q4_K_S.gguf Q4_K_S 19.59GB Slightly lower quality with small space savings.
airoboros-34b-3.2-Q4_0.gguf Q4_0 19.46GB Decent quality, older format, generally not recommended.
airoboros-34b-3.2-Q3_K_L.gguf Q3_K_L 18.13GB Lower quality but usable, good for low RAM availability.
airoboros-34b-3.2-Q3_K_M.gguf Q3_K_M 16.65GB Even lower quality.
airoboros-34b-3.2-Q3_K_S.gguf Q3_K_S 14.96GB Low quality, not recommended.
airoboros-34b-3.2-Q2_K.gguf Q2_K 12.82GB Extremely low quality, not recommended.

Want to support my work? Visit my ko-fi page here: https://ko-fi.com/bartowski

Downloads last month
Model size
34.4B params
Inference Examples
Unable to determine this model's library. Check the docs .
Invalid base_model specified in model card metadata. Needs to be a model id from hf.co/models.

Datasets used to train bartowski/airoboros-34b-3.2-GGUF