Hugging Face model: nkkbr/ViCA2
Author: nkkbr
Created: 2025-04-21 07:05:27
Last modified: 2025-12-15 13:59:02
Pipeline: video-text-to-text
Library: transformers
Likes: 0 | Downloads: 13
Tags: transformers, safetensors, vica_qwen, text-generation, multimodal, vision-language, video understanding, visuospatial cognition, spatial reasoning, vlm, llava, qwen, siglip, hiera, sam2, dual-encoder, video-text-to-text, en, dataset:liuhaotian/LLaVA-CC3M-Pretrain-595K, dataset:lmms-lab/LLaVA-OneVision-Data, dataset:nkkbr/ViCA-322K, arxiv:2505.12363, arxiv:2505.12312, doi:10.57967/hf/7544, license:apache-2.0, model-index, endpoints_compatible, region:us
Card metadata:
license: apache-2.0
tags: multimodal, vision-language, video understanding, visuospatial cognition, spatial reasoning, vlm, llava, qwen, siglip, hiera, sam2, dual-encoder
datasets: liuhaotian/LLaVA-CC3M-Pretrain-595K, lmms-lab/LLaVA-OneVision-Data, nkkbr/ViCA-322K
language: en
library_name: transformers
pipeline_tag: video-text-to-text
model_name: ViCA2-7B
model_description: ViCA2 (Visuospatial Cognitive Assistant 2) is a state-of-the-art large multimodal model tailored for fine-grained visuospatial reasoning in indoor video and image environments.
It builds upon the LLaVA-OneVision framework, and introduces a novel dual vision encoder architecture that integrates:
  - **SigLIP** for high-level semantic abstraction, and
  - **Hiera** (from SAM2) for detailed spatial structure modeling.

This dual-stream design enables robust performance in tasks involving object layouts, relative positioning, temporal order, and geometric reasoning.
Trained with a multi-stage strategy on over **322K video-based QA pairs**, ViCA2 significantly surpasses LLaVA-NeXT-Video and Gemini-1.5 Pro.

ViCA2 is built with modularity and efficiency in mind, leveraging:
  - Token ratio control for balancing semantic and spatial token contributions
  - Hiera stage-specific sampling and projection
  - Multi-stage DeepSpeed fine-tuning with frozen vision backbones