PointGuardAI/LFM2.5-1.2B-Instruct-INT8

LFM2.5-1.2B-Instruct INT8 W8A8

This repository contains a post-training-quantized checkpoint of LiquidAI/LFM2.5-1.2B-Instruct, prepared for efficient vLLM inference and per-request LoRA serving.

Base model

• Model: LiquidAI/LFM2.5-1.2B-Instruct
• Model revision: 868df74dd56ff8a0c2ac5dbf281690c2dbebe4c9
• Tokenizer revision: 868df74dd56ff8a0c2ac5dbf281690c2dbebe4c9
• Upstream license: Liquid AI Model License 1.0

Quantization

• Scheme: W8A8 INT8
• Weight quantization: symmetric, per-channel INT8
• Activation quantization: symmetric, dynamic per-token INT8
• Format: compressed-tensors
• Pipeline: data-free dynamic-activation quantization
• Quantized modules: 72 attention and feed-forward projections
• Kept unquantized: embeddings, LM head, normalization layers, and 20 Liquid convolution projections

This is therefore a mixed-precision checkpoint: the targeted attention and feed-forward projections use W8A8 INT8, while unsupported Liquid convolution projections remain BF16. They were intentionally excluded because Transformers names them conv, while vLLM 0.24.0 uses short_conv and does not remap the compressed scale names.

The full target-module list, exclusions, source revision, and compatibility rationale are recorded in recipe.yaml and quantization_manifest.json.

Integrity

SHA-256 for model.safetensors:

1071099fafdda7400ac9624b916ff45c56cb276eb061b22ea6c0ba4343f34edc

vLLM

Verified with vLLM 0.24.0 using the native compressed-tensors W8A8 INT8 kernels.

vllm serve PointGuardAI/LFM2.5-1.2B-Instruct-INT8 \
  --served-model-name lfm25-1.2b-int8 \
  --max-model-len 5120 \
  --gpu-memory-utilization 0.90 \
  --performance-mode throughput \
  --max-num-batched-tokens 8192

For per-request LoRA serving:

vllm serve PointGuardAI/LFM2.5-1.2B-Instruct-INT8 \
  --enable-lora \
  --max-loras 4 \
  --max-lora-rank 16 \
  --lora-modules application-policy-match=/path/to/adapter

Chat-completion example:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "lfm25-1.2b-int8",
    "messages": [
      {"role": "system", "content": "You are a concise assistant."},
      {"role": "user", "content": "Explain dynamic INT8 quantization in one sentence."}
    ],
    "max_tokens": 64,
    "temperature": 0
  }'

Guardrail validation

When paired with the separately trained PointGuard application-policy LoRA (not included in this repository), this checkpoint achieved the following on a balanced 558-row held-out development-validation set:

• Accuracy: 96.42%
• Macro F1: 96.41%
• AUPRC: 99.44%
• MATCH recall: 98.57%
• False negatives: 4 of 279 MATCH examples

Using vLLM generative scoring with one prompt per HTTP request:

Concurrency	Median throughput	Median p95 latency
16	62.4 requests/second	303 ms
32	67.7 requests/second	543 ms

These measurements are specific to the evaluated prompt distribution, LoRA adapter, NVIDIA A10G GPU, and vLLM configuration; they are not general language-model benchmarks.

Intended use and limitations

This repository contains the quantized general-purpose base checkpoint, not a standalone guardrail. It does not include a task-specific LoRA adapter, policy engine, calibrated decision thresholds, or safety guarantees.

Quantization can change model quality and probability calibration. Evaluate the exact production adapter, prompt format, serving runtime, thresholds, and deployment data before use.