Instructions to use VikramPal/mistral-7b-instruct-v0.3-text2sql-DynQuant-3bit with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use VikramPal/mistral-7b-instruct-v0.3-text2sql-DynQuant-3bit with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="VikramPal/mistral-7b-instruct-v0.3-text2sql-DynQuant-3bit")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("VikramPal/mistral-7b-instruct-v0.3-text2sql-DynQuant-3bit")
model = AutoModelForCausalLM.from_pretrained("VikramPal/mistral-7b-instruct-v0.3-text2sql-DynQuant-3bit", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use VikramPal/mistral-7b-instruct-v0.3-text2sql-DynQuant-3bit with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "VikramPal/mistral-7b-instruct-v0.3-text2sql-DynQuant-3bit"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "VikramPal/mistral-7b-instruct-v0.3-text2sql-DynQuant-3bit",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/VikramPal/mistral-7b-instruct-v0.3-text2sql-DynQuant-3bit

SGLang

How to use VikramPal/mistral-7b-instruct-v0.3-text2sql-DynQuant-3bit with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "VikramPal/mistral-7b-instruct-v0.3-text2sql-DynQuant-3bit" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "VikramPal/mistral-7b-instruct-v0.3-text2sql-DynQuant-3bit",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "VikramPal/mistral-7b-instruct-v0.3-text2sql-DynQuant-3bit" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "VikramPal/mistral-7b-instruct-v0.3-text2sql-DynQuant-3bit",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use VikramPal/mistral-7b-instruct-v0.3-text2sql-DynQuant-3bit with Docker Model Runner:
```
docker model run hf.co/VikramPal/mistral-7b-instruct-v0.3-text2sql-DynQuant-3bit
```

Mistral-7B-Instruct-v0.3 text-to-SQL, DynQuant 3-bit

mistralai/Mistral-7B-Instruct-v0.3 fine-tuned on text-to-SQL and quantized to 3 bits with DynQuant. It is one of 9 arms in a panel where every quantized arm was allocated the same byte budget, so the accuracies below differ by method and not by size.

What this is


base model	mistralai/Mistral-7B-Instruct-v0.3
fine-tune	lora r=32, 2.0 epoch over 39,531 text-to-SQL conversations
the adapter it merged	VikramPal/mistral-7b-instruct-v0.3-text2sql-lora
training data	`gretelai/synthetic_text_to_sql`, `Salesforce/wikisql`, `b-mc2/sql-create-context`
quantization	DynQuant, 3-bit, per-module widths from a DynQuant allocation
size on disk	2.857 GiB (3.3859 bits per parameter)
loads with	`transformers` with `dynquant` installed

Results

Execution match on 2,454 held-out text-to-SQL problems: the generated query is run against the schema and compared to the reference result set.

arm	exec match	size	bits/param
bf16	78.16%	13.500 GiB	16.0000
gptq_4b	78.28%	3.692 GiB	4.3760
awq_4b	77.91%	3.692 GiB	4.3760
dq_4b	78.08%	3.692 GiB	4.3754
gptq_3b	6.68%	2.858 GiB	3.3869
awq_3b	74.16%	2.858 GiB	3.3869
dq_3b	75.22%	2.857 GiB	3.3859
gptq_3b_asym_noao	76.08%	2.858 GiB	3.3869
gptq_3b_asym	3.99%	2.858 GiB	3.3869

This arm, by evaluation source:

eval source	exec match	items
`gretel`	74.21%	818
`spider`	58.68%	818
`wikisql`	92.79%	818

How this arm compares

McNemar exact over the per-item hits, so every row is a paired test on the same problems in the same order. p (Holm) is step-down corrected within the family the panel declared, not within this card.

comparison	delta (pts)	95% CI	p	p (Holm)	verdict
3b DynQuant vs GPTQ	+68.54	[+66.66, +70.42]	0	0	separated
3b DynQuant vs AWQ	+1.06	[-0.32, +2.44]	0.149	0.744	not separated
3b DynQuant vs GPTQ asym+actorder	+71.23	[+69.39, +73.07]	0	0	separated
3b DynQuant vs GPTQ asym, no actorder	-0.86	[-2.06, +0.35]	0.186	0.745	not separated
3b DynQuant vs bf16	-2.93	[-4.16, -1.71]	3.55e-06	1.42e-05	separated

What is not claimed

The accuracy above was measured in bf16, not from this directory. A DynQuant arm is scored by encoding its allocated widths back into bf16 -- the same encoder, the same widths, the same values -- so that every arm in the panel is scored through one path and no arm's number depends on which container it was read from. The directory you are downloading holds those same values packed. What is carried across from the measurement is the arithmetic; what is not is a claim that the packed and encoded containers were separately scored.
The arms above are not all the same scheme. In this panel GPTQ runs symmetric with no activation reordering, asymmetric with no activation reordering, and asymmetric with group activation reordering; AWQ runs asymmetric with no activation reordering. DynQuant's quantizer is asymmetric and does not reorder columns, which is a property of the method rather than a recipe flag, so the panel records no scheme for its arms. Where a comparison above pairs a symmetric arm against an asymmetric one its delta spans two differences at once -- how the bits were allocated, and whether a zero point was stored per group -- so a large gap between those two arms is not on its own evidence about allocation. The comparison that isolates it is in this panel: gptq_3b and gptq_3b_asym_noao are the same method at the same byte anchor and differ in the scheme alone, so the difference between those two rows is the scheme and nothing else.
Storage, measured; throughput, not. The number reported here is bytes on disk and execution match. This card makes no claim about decode speed or peak VRAM against an fp16 baseline, because this panel did not measure either.
One task. Execution match on held-out text-to-SQL is what was scored. It says nothing about how this arm behaves on anything else, and a quantization that holds one task can lose another.

Install

This directory is packed, so transformers alone cannot open it -- it needs DynQuant's HfQuantizer, which the package registers. Prebuilt CUDA kernels come with it where a wheel exists for your platform, and it falls back to a pure-torch path where one does not.

pip install dynquant

Source, format spec, and the allocator that produced this arm's bit map: https://github.com/kambojvikram/dynquant

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

import dynquant

dynquant.register_hf_quantizer()

model = AutoModelForCausalLM.from_pretrained("VikramPal/mistral-7b-instruct-v0.3-text2sql-DynQuant-3bit", device_map="cuda")
tokenizer = AutoTokenizer.from_pretrained("VikramPal/mistral-7b-instruct-v0.3-text2sql-DynQuant-3bit")

Provenance

panel model: /workspace/runs/s4/mistral7b-v03.text2sql/merged
parameters counted: 7,248,023,552
byte target this arm was allocated against: 3,068,534,784 B
fine-tune: 2472 steps, train loss 0.0540, 3.8 h
fine-tune commit: 5959fe04b7db0512eaf7567d77a43f3df5860651
evaluation: 2,454 problems in 11 min

Downloads last month: 694

Safetensors

Model size

0.8B params

Tensor type

I32

BF16

Model tree for VikramPal/mistral-7b-instruct-v0.3-text2sql-DynQuant-3bit

Base model

mistralai/Mistral-7B-v0.3

Finetuned

mistralai/Mistral-7B-Instruct-v0.3

Quantized

(287)

this model