Aditya8595/Qwen2.5-VL-7B-DocInfographic-QA

Qwen2.5-VL-7B Fine-tuned on Document/Infographic/Chart QA

Fine-tuned Qwen/Qwen2.5-VL-7B-Instruct on document, infographic, and chart visual question answering using TRL SFTTrainer with QLoRA.

Training Recipe

Component	Detail
Base Model	Qwen/Qwen2.5-VL-7B-Instruct
Method	SFT + QLoRA (r=16, alpha=32)
Dataset	HuggingFaceM4/the_cauldron (docvqa + chartqa + ai2d) - ~50K samples
Epochs	3
Learning Rate	1e-4 (LoRA), cosine schedule
Warmup	3% of steps
Batch Size	2 per device x 8 grad accum = 16 effective
Precision	bf16, 4-bit NF4 quantization
max_pixels	1280x28x28 (~1M pixels for high-res docs)
Reference	Chart-RVR (arxiv:2510.10973) + TRL v1.2.0 SFT VLM docs

How to Run

pip install torch transformers trl datasets peft bitsandbytes accelerate trackio qwen-vl-utils
python train.py

Hardware Requirements

• 7B model: Minimum 24GB VRAM with 4-bit quantization (L4/A10G/a100)
• Full precision: 80GB VRAM (A100/H100)

Evaluation Benchmarks

• DocVQA (document QA): lmms-lab/DocVQA
• InfoVQA (infographic QA): lmms-lab/DocVQA, config=InfographicVQA
• ChartQA (chart QA): HuggingFaceM4/the_cauldron, config=chartqa

SOTA Baselines

• Qwen2.5-VL-7B-Instruct base: DocVQA=95.7, InfoVQA=82.6, ChartQA=87.3
• Chart-RVR-3B-Hard (fine-tuned): ChartQA=85.8