Instructions to use VikramPal/LFM2.5-8B-A1B-text2sql-bf16 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use VikramPal/LFM2.5-8B-A1B-text2sql-bf16 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="VikramPal/LFM2.5-8B-A1B-text2sql-bf16")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("VikramPal/LFM2.5-8B-A1B-text2sql-bf16")
model = AutoModelForCausalLM.from_pretrained("VikramPal/LFM2.5-8B-A1B-text2sql-bf16", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use VikramPal/LFM2.5-8B-A1B-text2sql-bf16 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "VikramPal/LFM2.5-8B-A1B-text2sql-bf16"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "VikramPal/LFM2.5-8B-A1B-text2sql-bf16",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/VikramPal/LFM2.5-8B-A1B-text2sql-bf16

SGLang

How to use VikramPal/LFM2.5-8B-A1B-text2sql-bf16 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "VikramPal/LFM2.5-8B-A1B-text2sql-bf16" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "VikramPal/LFM2.5-8B-A1B-text2sql-bf16",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "VikramPal/LFM2.5-8B-A1B-text2sql-bf16" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "VikramPal/LFM2.5-8B-A1B-text2sql-bf16",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use VikramPal/LFM2.5-8B-A1B-text2sql-bf16 with Docker Model Runner:
```
docker model run hf.co/VikramPal/LFM2.5-8B-A1B-text2sql-bf16
```

LFM2.5-8B-A1B text-to-SQL, bf16

LiquidAI/LFM2.5-8B-A1B fine-tuned on text-to-SQL, merged and left in bf16. It is the ceiling arm of a panel of 7 arms: every quantized arm below was made from this checkpoint and allocated the same byte budget, so their accuracies differ by method and not by size.

What this is


base model	LiquidAI/LFM2.5-8B-A1B
fine-tune	lora r=32, 1.0 epoch over 49,905 text-to-SQL conversations
training data	`gretelai/synthetic_text_to_sql`, `Salesforce/wikisql`, `b-mc2/sql-create-context`
quantization	none -- this is the bf16 fine-tune every quantized arm was made from
size on disk	15.773 GiB (16.0000 bits per parameter)
loads with	`transformers`

Results

Execution match on 12,000 held-out text-to-SQL problems: the generated query is run against the schema and compared to the reference result set.

arm	exec match	size	bits/param
bf16	84.29%	15.773 GiB	16.0000
gptq_4b	82.07%	4.097 GiB	4.1565
awq_4b	81.77%	4.097 GiB	4.1565
dq_4b	82.84%	4.096 GiB	4.1547
gptq_3b	60.76%	3.104 GiB	3.1488
awq_3b	57.92%	3.104 GiB	3.1488
dq_3b	79.89%	3.103 GiB	3.1475

This arm, by evaluation source:

eval source	exec match	items
`gretel`	72.02%	3,063
`wikisql`	88.50%	8,937

How this arm compares

McNemar exact over the per-item hits, so every row is a paired test on the same problems in the same order. p (Holm) is step-down corrected within the family the panel declared, not within this card.

comparison	delta (pts)	95% CI	p	p (Holm)	verdict
4b GPTQ vs bf16	-2.22	[-2.66, -1.79]	1.61e-23	3.23e-23	separated
4b AWQ vs bf16	-2.52	[-2.98, -2.07]	2.52e-28	7.55e-28	separated
4b DynQuant vs bf16	-1.45	[-1.84, -1.06]	2.09e-13	2.09e-13	separated
3b GPTQ vs bf16	-23.53	[-24.38, -22.69]	0	0	separated
3b AWQ vs bf16	-26.37	[-27.25, -25.50]	0	0	separated
3b DynQuant vs bf16	-4.40	[-4.96, -3.84]	1.82e-53	7.29e-53	separated

What is not claimed

Storage, measured; throughput, not. The number reported here is bytes on disk and execution match. This card makes no claim about decode speed or peak VRAM against an fp16 baseline, because this panel did not measure either.
One task. Execution match on held-out text-to-SQL is what was scored. It says nothing about how this arm behaves on anything else, and a quantization that holds one task can lose another.

Install

This checkpoint is plain bf16 and loads with transformers alone. It is the ceiling arm of a DynQuant panel: the quantized arms in the table above are this same fine-tune at a fraction of the size, and those need the package.

pip install dynquant

Source, format spec, and the allocator that produced their bit maps: https://github.com/kambojvikram/dynquant

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("VikramPal/LFM2.5-8B-A1B-text2sql-bf16", device_map="cuda")
tokenizer = AutoTokenizer.from_pretrained("VikramPal/LFM2.5-8B-A1B-text2sql-bf16")

Provenance

panel model: /workspace/runs/s4/lfm25-8b-a1b.text2sql/merged
parameters counted: 8,467,856,128
byte target: --
fine-tune: 1560 steps, train loss 0.1114, 6.7 h
fine-tune commit: d0d33f3bce6f3f59359ce704b16040c7e9ba78f5
evaluation: 12,000 problems in 59 min

Downloads last month: -

Safetensors

Model size

8B params

Tensor type

F32

BF16

Model tree for VikramPal/LFM2.5-8B-A1B-text2sql-bf16

Base model

LiquidAI/LFM2.5-8B-A1B-Base

Finetuned

LiquidAI/LFM2.5-8B-A1B

Finetuned

(39)

this model