antontuzovAI/QWE-VL-3B-ImageMining

Qwen-VL-3B-ImageMining

Model Description

Qwen-VL-3B-ImageMining is a fine-tuned Vision-Language Model specialized in complex visual reasoning, multi-step deduction, and knowledge discovery from images.

It is built upon the powerful Qwen2.5-VL-3B-Instruct base model and fine-tuned using QLoRA (4-bit) on the high-quality zai-org/ImageMining dataset.

Architecture & Training Details

• Base Model: Qwen/Qwen2.5-VL-3B-Instruct
• Fine-tuning Method: QLoRA (4-bit NF4 quantization)
• Rank (r): 16 | Alpha: 32 | Dropout: 0.05
• Target Modules: q_proj, v_proj, k_proj, o_proj, gate_proj, up_proj, down_proj
• Optimizer: Paged AdamW 8-bit
• Learning Rate: 2e-4 with Cosine decay
• Trainable Parameters: ~37M (0.98% of total 3.79B parameters)

How to Use This Model

Installation

pip install transformers accelerate peft bitsandbytes pillow qwen-vl-utils

Inference Code

import torch
from PIL import Image
from transformers import AutoProcessor, AutoModelForImageTextToText, BitsAndBytesConfig
from peft import PeftModel

model_id = 'antontuzovAI/QWE-VL-3B-ImageMining'
base_model_id = 'Qwen/Qwen2.5-VL-3B-Instruct'

bnb_config = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_compute_dtype=torch.bfloat16, bnb_4bit_quant_type='nf4')
base_model = AutoModelForImageTextToText.from_pretrained(base_model_id, quantization_config=bnb_config, device_map='cuda', trust_remote_code=True)
model = PeftModel.from_pretrained(base_model, model_id)
processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)

image = Image.open('your_image.jpg').resize((384, 384))
messages = [{'role': 'user', 'content': [{'type': 'image', 'image': image}, {'type': 'text', 'text': 'Analyze this image.'}]}]
text_prompt = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = processor(text=[text_prompt], images=[image], return_tensors='pt').to('cuda')

with torch.no_grad():
    generated_ids = model.generate(**inputs, max_new_tokens=256)
print(processor.batch_decode(generated_ids, skip_special_tokens=True)[0])