Luanneee/ominiVLM

omniVLM

Lightweight multimodal VLM training & alignment project. Trained on a single RTX 4090/4090D (24 GB) with bf16 precision, spanning pretraining, SFT, GRPO and DPO of a 65M MiniMind-V VLM and Qwen2.5-VL-7B adapters.

Models

Model	Description
minimind-v-65m-multitask	65M from-scratch Pretrain → multitask SFT (caption + VQAv2 + OK-VQA + MMBench)
qwen2.5-vl-7b-dpo-v6	Qwen2.5-VL-7B DPO v6 on 11,093 balanced preference pairs
omniVLM-checkpoints	All 65M checkpoints & ablations (GRPO, VQA-mix, hallucination SFT, architecture variants)

Evaluation highlights

Model	VQAv2	MMBench	OK-VQA	COCO CIDEr	COCO BLEU-4	POPE
65M multitask	32.8%	26.0%	3.2%	0.6395	0.2271	37.4%
7B DPO v6	47.7%*	87.32%	45.9%–47.7%	0.9973	—	95.2%

*OK-VQA / CIDEr reported for the DPO (balanced preferences, β=0.1) stage.

Usage

minimind-v-65m-multitask

Load the checkpoint with the MiniMind-V model definition in this repo:

import torch
from model_vlm import MiniMindVLM  # see minimind-v-65m-multitask/model_vlm.py

model = MiniMindVLM(...)
model.load_state_dict(torch.load("multitask_final_vlm_768.pth", map_location="cpu"))

Tokenizer files (tokenizer.json, tokenizer_config.json) are included in the model folder.

qwen2.5-vl-7b-dpo-v6

This is a LoRA adapter for Qwen/Qwen2.5-VL-7B-Instruct. Load the base model and apply the adapter with PEFT:

from peft import PeftModel
from transformers import Qwen2_5_VLForConditionalGeneration

base = Qwen2_5_VLForConditionalGeneration.from_pretrained("Qwen/Qwen2.5-VL-7B-Instruct")
model = PeftModel.from_pretrained(base, "qwen2.5-vl-7b-dpo-v6")

Training & evaluation

Code, configs, data builders and evaluation scripts are in the source repository Icecream102/ominiVLM.

License

Apache-2.0