IVC-liuyuan/M3Diff

M³Diff

M³Diff is the official model for fine-grained Image Difference Captioning (IDC) introduced in OmniDiff: A Comprehensive Benchmark for Fine-grained Image Difference Captioning (ICCV 2025). Given a pair of similar images, the model generates a detailed natural-language description of their visual differences.

This repository contains the merged 7B model weights. The Hugging Face model ID is IVC-liuyuan/M3Diff.

Model architecture

M³Diff extends LLaVA-OneVision-7B with a plug-and-play Multi-scale Differential Perception (MDP) module. MDP computes feature-level differences between the two images and adaptively fuses information from several vision encoder depths. This combines low-level visual detail with high-level semantics while retaining the general capabilities of the foundation model.

• Language model: Qwen2, 7B scale
• Vision encoder: SigLIP SO400M, 27 transformer layers
• Multi-scale vision features: layers 17, 20, 23, and 26
• MDP: two stacked refinement layers
• Projector: two-layer MLP
• Checkpoint format: merged FP16 safetensors

The model uses the M³Diff/LLaVA code because the MDP module is a custom model component; loading it with stock transformers.AutoModel alone is not supported.

Training

The model was instruction-tuned on 896K question-answer samples assembled from OmniDiff and public IDC datasets, including Spot-the-Diff, IEdit, Birds-to-Words, CLEVR-Change, and CLEVR-DC. The language model was adapted with LoRA (r=128, alpha=256), while the vision encoder, multimodal projector, and MDP module were fully fine-tuned. The reported run used eight NVIDIA A100 40GB GPUs, a global batch size of 256, and approximately 26 hours of training.

OmniDiff contains 15,598 human-captioned image pairs from 324 diverse indoor and outdoor scenarios. It covers 12 change types: viewpoint, illumination, addition, disappearance, removal, substitution, size, color, orientation, pose, OCR, and counting. Captions contain 60 words on average.

Usage

Install the official M³Diff code and its dependencies, then use its LLaVA-compatible loader:

from llava.model.builder import load_pretrained_model
from llava.mm_utils import get_model_name_from_path

model_path = "IVC-liuyuan/M3Diff"
tokenizer, model, image_processor, context_len = load_pretrained_model(
    model_path=model_path,
    model_base=None,
    model_name=get_model_name_from_path(model_path),
)

For paired-image captioning and benchmark evaluation, use the evaluation launchers supplied with the official M³Diff source release. They construct a two-image prompt, preprocess both images with SigLIP, and generate the difference caption with the qwen_2 conversation template.

Intended use and limitations

M³Diff is intended for research on image difference captioning, paired-image comparison, and fine-grained multimodal understanding. Potential applications include environmental monitoring and visual surveillance research.

The model may hallucinate, omit, or mislocalize subtle changes, especially for domains or image distributions not represented during training. Its outputs must not be treated as a sole source of truth in safety-critical settings. Users are responsible for evaluating bias, privacy, and dataset licensing for their deployment context. The current model operates on image pairs rather than continuous temporal changes or video.

Citation

@inproceedings{liu2025omnidiff,
  title={OmniDiff: A Comprehensive Benchmark for Fine-grained Image Difference Captioning},
  author={Liu, Yuan and Hou, Saihui and Hou, Saijie and Du, Jiabao and Meng, Shibei and Huang, Yongzhen},
  booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision},
  pages={21440--21449},
  year={2025}
}

License and acknowledgements

The released model artifacts are provided under the Apache License 2.0. This work builds on LLaVA, LLaVA-NeXT, LLaVA-OneVision, Qwen2, SigLIP, Hugging Face Transformers, PEFT, and DeepSpeed. Users must also comply with the applicable licenses and terms of the base model, upstream projects, and datasets.