TableLLM-13b-GGUF / README.md
bartowski's picture
Llamacpp quants
4de7e86 verified
metadata
license: llama2
datasets:
  - RUCKBReasoning/TableLLM-SFT
language:
  - en
tags:
  - Table
  - QA
  - Code
quantized_by: bartowski
pipeline_tag: text-generation

Llamacpp Quantizations of TableLLM-13b

Using llama.cpp release b2589 for quantization.

Original model: https://huggingface.co/RUCKBReasoning/TableLLM-13b

Download a file (not the whole branch) from below:

Filename Quant type File Size Description
TableLLM-13b-Q8_0.gguf Q8_0 13.83GB Extremely high quality, generally unneeded but max available quant.
TableLLM-13b-Q6_K.gguf Q6_K 10.67GB Very high quality, near perfect, recommended.
TableLLM-13b-Q5_K_M.gguf Q5_K_M 9.23GB High quality, very usable.
TableLLM-13b-Q5_K_S.gguf Q5_K_S 8.97GB High quality, very usable.
TableLLM-13b-Q5_0.gguf Q5_0 8.97GB High quality, older format, generally not recommended.
TableLLM-13b-Q4_K_M.gguf Q4_K_M 7.86GB Good quality, uses about 4.83 bits per weight.
TableLLM-13b-Q4_K_S.gguf Q4_K_S 7.42GB Slightly lower quality with small space savings.
TableLLM-13b-IQ4_NL.gguf IQ4_NL 7.41GB Decent quality, similar to Q4_K_S, new method of quanting,
TableLLM-13b-IQ4_XS.gguf IQ4_XS 7.01GB Decent quality, new method with similar performance to Q4.
TableLLM-13b-Q4_0.gguf Q4_0 7.36GB Decent quality, older format, generally not recommended.
TableLLM-13b-Q3_K_L.gguf Q3_K_L 6.92GB Lower quality but usable, good for low RAM availability.
TableLLM-13b-Q3_K_M.gguf Q3_K_M 6.33GB Even lower quality.
TableLLM-13b-IQ3_M.gguf IQ3_M 5.98GB Medium-low quality, new method with decent performance.
TableLLM-13b-IQ3_S.gguf IQ3_S 5.65GB Lower quality, new method with decent performance, recommended over Q3 quants.
TableLLM-13b-Q3_K_S.gguf Q3_K_S 5.65GB Low quality, not recommended.
TableLLM-13b-Q2_K.gguf Q2_K 4.85GB Extremely low quality, not recommended.

Want to support my work? Visit my ko-fi page here: https://ko-fi.com/bartowski