Azaz666/InternVL2_5-4B-AWQ-INT4

OpenGVLab__InternVL2_5-4B__awq_w4_merged

This is a AWQ (4-bit) quantized version of OpenGVLab/InternVL2_5-4B.

Quantization Details

• Method: AWQ
• Bits: 4
• Base model: OpenGVLab/InternVL2_5-4B
• Group size: 128
• Quantized modules: language_model (InternLM2)
• FP16 modules: vision_model, mlp1 (connector)
• Approach: Extract inner InternLM2 -> AWQ -> re-attach to InternVLChatModel

Usage

from transformers import AutoProcessor, AutoModelForImageTextToText
import torch

model = AutoModelForImageTextToText.from_pretrained(
    "{REPO_ID}",
    torch_dtype=torch.float16,
    device_map="auto",
    trust_remote_code=True,
)
processor = AutoProcessor.from_pretrained("{REPO_ID}", trust_remote_code=True)

Replace {REPO_ID} with the repo ID of this model.

Original Model

See OpenGVLab/InternVL2_5-4B for the original FP16 model.