Azaz666/LFM2-VL-1.6B-torchao-W4A8

LiquidAI__LFM2-VL-1.6B__torchao_w4a8

This is a torchao W4A8 (4-bit) quantized version of LiquidAI/LFM2-VL-1.6B.

Quantization Details

• Method: torchao W4A8
• Bits: 4
• Base model: LiquidAI/LFM2-VL-1.6B
• Weight bits: 4
• Activation bits: 8 (dynamic)
• Group size: 128
• Backend: torchao (Int8DynamicActivationInt4Weight)

Usage

from transformers import AutoProcessor, AutoModelForImageTextToText
import torch

model = AutoModelForImageTextToText.from_pretrained(
    "{REPO_ID}",
    torch_dtype=torch.float16,
    device_map="auto",
    trust_remote_code=True,
)
processor = AutoProcessor.from_pretrained("{REPO_ID}", trust_remote_code=True)

Replace {REPO_ID} with the repo ID of this model.

Original Model

See LiquidAI/LFM2-VL-1.6B for the original FP16 model.