GPTQ 4-bit actor order version that works in textgen-webui with exllamav2
Generated by using scripts from https://gitee.com/yhyu13/llama_-tools
Original weight : https://huggingface.co/Xwin-LM/Xwin-Math-7B-V1.0
| Branch | Bits | GS | Act Order | Damp % | GPTQ Dataset | Seq Len | Size | ExLlama | Desc |
|---|---|---|---|---|---|---|---|---|---|
| main | 4 | 128 | Yes | 0.1 | code | 4096 | 4.4GB | Yes | 4-bit, with Act Order. 128 group size |
Here is some testing done in textgen-webui, I was using Q&A from this dataset https://huggingface.co/datasets/TIGER-Lab/MathInstruct
Basic arithmatic, the answer (A) is correct
Does not follow the instruction to write python code. And hulluciate an answer not in exists in options
Answer is correct, but hulluciate with a not existing option
- Downloads last month
- 8
This model does not have enough activity to be deployed to Inference API (serverless) yet.
Increase its social visibility and check back later, or deploy to Inference Endpoints (dedicated)
instead.