Cyrag/LFM2-2.6B-Longevity-NVFP4

Docs • Discord

LFM-2.6B-Longevity-NVFP4

This repository provides an NVFP4 (NVIDIA Floating Point 4) quantized checkpoint of LiquidAI/LFM2-2.6B-Longevity, specifically optimized for native tensor-core acceleration on NVIDIA Blackwell architecture GPUs (e.g., RTX PRO 6000 Blackwell, B200, GB200) served via vLLM.

The base model, LFM2-2.6B-Longevity, was developed jointly by Insilico Medicine and Liquid AI and accompanies the study "An Open Benchmark and Language Models for AI in Aging Biology" (Zhavoronkov et al., Cell, 2026).

---

1. Quantization Architecture & Precision Preservation

Because LFM2-2.6B is a compact hybrid model combining Mamba-style short convolutions and multi-head self-attention, standard uniform 4-bit quantization across all layers can degrade biological reasoning and token stability.

This checkpoint utilizes a selective mixed-precision strategy:

Layer Type	Precision	Rationale
Feed-Forward Blocks (feed_forward.w1, w2, w3)	ModelOpt NVFP4 (W4A4 / W4A16, block_size=16)	Accounts for the vast majority of parameters; accelerated natively via Blackwell NVFP4 CUTLASS GEMM kernels.
Normalization Layers (embedding_norm, operator_norm, ffn_norm, q_layernorm, k_layernorm)	Float32 (torch.float32)	Preserves full dynamic range and numerical stability during layer normalization.
SSM Conv Filter Kernels (conv.conv.weight / short_conv)	Float32 (torch.float32)	Retains precision for state-space temporal filter dynamics.
Token Embeddings (embed_tokens) & LM Head (lm_head)	Bfloat16 (torch.bfloat16)	Protects vocabulary representation fidelity and logit calibration.
Attention Projections (self_attn.q_proj, k_proj, v_proj, out_proj)	Bfloat16 (torch.bfloat16)	Retains precision for key-value projections and attention routing.
SSM Projections (short_conv.in_proj, out_proj)	Bfloat16 (torch.bfloat16)	Retains precision for SSM input/output gates.

• Original BF16 Checkpoint Size: 5.14 GB
• NVFP4 Checkpoint Size: 2.29 GB (44.6% of original)
• 0 NaN / Inf verified across all 536 tensors.

---

2. Serving with vLLM (Docker / CLI)

This checkpoint is natively supported in vLLM $\ge$ 0.28.0 using --quantization modelopt.

Quick Start via Docker (Recommended for Blackwell)

docker run --rm -it \
  --name vllm-lfm-2.6b-longevity-nvfp4 \
  --gpus all \
  --ipc=host \
  -p 7000:8000 \
  -e VLLM_WSL2_ENABLE_PIN_MEMORY=1 \
  -e CUDA_DEVICE_MAX_CONNECTIONS=1 \
  -e VLLM_USE_SPINLOOP_EXT=1 \
  -v "/path/to/models:/models" \
  vllm/vllm-openai:v0.28.0 \
  /models/LFM-2.6B-Longevity-NVFP4 \
  --served-model-name LFM-2.6B-Longevity LFM-2.6B-Longevity-NVFP4 \
  --quantization modelopt \
  --host 0.0.0.0 \
  --port 8000 \
  --tensor-parallel-size 1 \
  --gpu-memory-utilization 0.90 \
  --max-model-len 32768 \
  --max-num-batched-tokens 32768 \
  --max-num-seqs 16 \
  --enable-prefix-caching \
  --compilation-config '{"pass_config": {"fuse_norm_quant": true}, "cudagraph_num_of_warmups": 2}' \
  --tool-call-parser lfm2 \
  --enable-auto-tool-choice

Key Performance Flags Explained

• --quantization modelopt: Activates ModelOpt NVFP4 unpacking and dispatch to FlashInferCutlassNvFp4LinearKernel.
• --max-num-batched-tokens 32768: Eliminates prefill chunking splits for long inputs, enabling single-pass tensor core prefill up to 32k tokens.
• pass_config.fuse_norm_quant: true: Fuses RMSNorm and FP4 activation quantization into a single kernel pass, eliminating intermediate global memory round-trips.
• --enable-prefix-caching: Caches recurrent SSM and attention KV states for near-instant multi-turn conversation and agentic harness serving.
• --tool-call-parser lfm2 & --enable-auto-tool-choice: Enables native structured tool / function calling.

---

3. Benchmarks & Serving Performance

Benchmarked on NVIDIA RTX PRO 6000 Blackwell (96 GB VRAM, sm_120) using the llm_context_benchmarks suite across context lengths from 0.5k to 16k tokens:

Power Limit Configuration: These benchmark numbers were collected on a test system with the GPU Power Limit capped at 300W (verified via nvidia-smi against the default 600W factory limit). Systems running at the standard/default factory TDP or higher power caps may achieve even higher prompt ingestion throughput and token generation speeds.

A. Context Sweep Throughput (128 generated tokens)

Context Size	Cold Prompt Ingestion	Warm Cached Ingestion	Generation Speed (Decode)	Time Per Token (TPOT)
0.5k	22,738 tok/s	18,997 tok/s	348.7 tok/s	2.89 ms
1.0k	34,673 tok/s	24,053 tok/s	349.5 tok/s	2.89 ms
2.0k	76,597 tok/s	45,084 tok/s	347.6 tok/s	2.90 ms
4.0k	95,581 tok/s	91,063 tok/s	343.3 tok/s	2.94 ms
8.0k	92,970 tok/s	131,778 tok/s	339.8 tok/s	2.97 ms
16.0k	65,019 tok/s	85,485 tok/s	336.5 tok/s	3.00 ms

• Peak Cold Prefill: 106,083 tokens/sec
• Peak Cached Prefill: 131,778 tokens/sec
• Generation Speed: Sustained ~340 – 350 tokens/sec across all context lengths.
• Single-token generation latency (TPOT): ~2.88 – 2.95 ms.

B. Concurrency Scaling (Batch Sizes 1 to 16)

Prompt: ~2048 tokens, Generation: 128 tokens:

Concurrency (Streams)	Time to First Token (TTFT)	Aggregate Generation Throughput
1	27 ms	~348 tok/s
2	23 ms	~690 tok/s
4	40 ms	~1,350 tok/s
8	53 ms	~2,500 tok/s
16	74 ms	~4,000+ tok/s

---

4. Python API Example (OpenAI SDK)

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:7000/v1",
    api_key="none"
)

response = client.chat.completions.create(
    model="LFM-2.6B-Longevity-NVFP4",
    messages=[
        {"role": "system", "content": "You are a biomedical AI specialized in aging biology and clinical longevity research."},
        {"role": "user", "content": "What are the primary molecular hallmarks of cellular senescence, and how do senolytics target them?"}
    ],
    temperature=0.3,
    max_tokens=500
)

print(response.choices[0].message.content)

---

5. Tool Calling Example

tools = [
    {
        "type": "function",
        "function": {
            "name": "lookup_gene_aging_association",
            "description": "Look up documented associations between a gene and aging hallmarks.",
            "parameters": {
                "type": "object",
                "properties": {
                    "gene_symbol": {"type": "string", "description": "HGNC gene symbol, e.g. CDKN2A, SIRT1"}
                },
                "required": ["gene_symbol"]
            }
        }
    }
]

response = client.chat.completions.create(
    model="LFM-2.6B-Longevity-NVFP4",
    messages=[{"role": "user", "content": "Check the aging role for SIRT1"}],
    tools=tools,
    temperature=0.0
)

print(response.choices[0].message.tool_calls)

---

6. Intended Use and Domain Scope

This model is intended for research in aging biology, omics data interpretation, and life extension science. The outputs represent machine-generated hypotheses and predictions, not formal medical advice, and should be validated experimentally.

---

7. Citations

If you use this model in your research, please cite:

@article{zhavoronkov2026longevitybench,
  title   = {An Open Benchmark and Language Models for AI in Aging Biology},
  author  = {Zhavoronkov, Alex and Naumov, Vladimir and Sidorenko, Denis and Aliper, Alex and Aladinskiy, Vladimir and Hasani, Ramin and Amini, Alexander and Nasto, Katerina and Reymond, Mathieu and Shayakhmetov, Rim and Miftakhutdinov, Zulfat and Gladyshev, Vadim N. and Galkin, Fedor},
  journal = {Cell},
  volume  = {189},
  pages   = {5980--5994},
  year    = {2026},
  doi     = {10.1016/j.cell.2026.08.026},
  url     = {https://www.cell.com/cell/fulltext/S0092-8674(26)00999-2}
}

@article{liquidai2025lfm2,
  title   = {LFM2 Technical Report},
  author  = {Liquid AI},
  journal = {arXiv preprint arXiv:2511.23404},
  year    = {2025}
}