Model: nkkbr/ViCA2
Author: nkkbr
Pipeline: video-text-to-text
Library: transformers
Tags: transformers, safetensors, vica_qwen, text-generation, multimodal, vision-language, video understanding, visuospatial cognition, spatial reasoning, vlm, llava, qwen, siglip, hiera, sam2, dual-encoder, video-text-to-text, en, dataset:liuhaotian/LLaVA-CC3M-Pretrain-595K, dataset:lmms-lab/LLaVA-OneVision-Data, dataset:nkkbr/ViCA-322K, arxiv:2505.12363, arxiv:2505.12312, license:apache-2.0, model-index, endpoints_compatible, region:us
Card metadata: license: apache-2.0
tags: multimodal, vision-language, video understanding, visuospatial cognition, spatial reasoning, vlm, llava, qwen, siglip, hiera, sam2, dual-encoder
datasets: liuhaotian/LLaVA-CC3M-Pretrain-595K, lmms-lab/LLaVA-OneVision-Data, nkkbr/ViCA-322K
language: en
library_name: transformers
pipeline_tag: video-text-to-text
model_name: ViCA2-7B
model_description: ViCA2 (Visuospatial Cognitive Assistant 2) is a state-of-the-art large multimodal model tailored for fine-grained visuospatial reasoning in indoor video and image environments.
It builds upon the LLaVA-OneVision framework, and introduces a novel dual vision encoder architecture that integrates:
  - **SigLIP** for high-level semantic abstraction, and
  - **Hiera** (from SAM2) for detailed spatial structure modeling.

This dual-stream design enables robust performance in tasks involving object layouts, relative positioning, temporal order, and geometric reasoning.
Trained with a multi-stage strategy on over **322K video-based QA pairs**, ViCA2 significantly surpasses LLaVA-NeXT-Video and Gemini-1.5 Pro.

ViCA2 is built with modularity and efficiency in mind, leveraging:
  - Token ratio control for balancing semantic and spatial token contributions
  - Hiera stage-specific sampling and projection
  - Multi-stage DeepSpeed fine-tuning with frozen vision backbones

model-index: name: ViCA2-7B
results: task: type: visual-question-answering
dataset: name: VSI-Bench
type: vsi-bench
metrics: type: score
value: 56.81
name: Average
verified: no, type: MRA
value: 65.73
name: Object Count
verified: no, type: MRA
value: 50.98
name: Absolute Distance
verified: no, type: MRA
value: 75.54
name: Object Size
verified: no, type: MRA
value: 71.42
name: Room Size
verified: no, type: accuracy
value: 51.55
name: Relative Distance
verified: no, type: accuracy
value: 34.61
name: Relative Direction
verified: no, type: accuracy
value: 38.14
name: Route Plan
verified: no, type: accuracy
value: 66.5
name: Appearance Order
verified: no
Likes: 0 | All-time downloads: 378
Created: 2025-04-21 07:05:27