MoonViT-V2

MoonViT-V2 is the native-resolution vision backbone of Kimi K3.

Attribute	Specification
Parameters	~0.4B
Architecture	27-layer ViT · 1024 hidden · 12 heads · 1536 QKV dim
Patch Size	14 × 14
Token Compression	2×2 pixel-shuffle + temporal pooling
Max Resolution	~3584 × 3584
Initialization	From scratch (not SigLIP)
Source Model	moonshotai/Kimi-K3
Technical Report	Kimi K3 (arXiv:2607.24653)

Overview

MoonViT-V2 encodes images and videos into visual tokens for Kimi K3’s shared multimodal embedding space (via a lightweight MLP projector in the full checkpoint).

Unlike MoonViT-SO-400M (SigLIP-initialized), MoonViT-V2 is randomly initialized and jointly trained with the language model under next-token prediction.

Unified image/video path — shared parameters with factorized spatial (intra-frame) and temporal (inter-frame) attention.
Token compression — 2×2 pixel-shuffle (~4× fewer spatial tokens) plus temporal pooling, for high-res images and long videos within Kimi K3’s up-to-1M-token context.
Training stability — RMSNorm and bias-free linear/attention projections; from-scratch training avoided gradient spikes seen with contrastive vision initializations at this scale.

Usage

import torch
from transformers import AutoModel, AutoImageProcessor

model_id = "AI4Industry/MoonViT-V2"

model = AutoModel.from_pretrained(
    model_id,
    dtype=torch.bfloat16,
    trust_remote_code=True,
)
processor = AutoImageProcessor.from_pretrained(model_id, trust_remote_code=True)

device = "cuda" if torch.cuda.is_available() else "cpu"
model = model.to(device)

Image input

from PIL import Image

image = Image.open("your_image.png").convert("RGB")
inputs = processor(image, return_tensors="pt")  # T=1
pixel_values = inputs["pixel_values"].to(device=device, dtype=model.dtype)
grid_thws = inputs["grid_thws"].to(device=device)

with torch.no_grad():
    image_features = model(pixel_values, grid_thws)

print(image_features[0].dtype, image_features[0].shape)
# e.g. torch.bfloat16, torch.Size([N, 4, 1024])

Video input

Pass an ordered list of frames. All frames share the first frame’s resize/pad so spatial grids align. T must be ≤ max_num_frames (default 4, matching temporal position embedding).

from PIL import Image

frames = [
    Image.open("frame_0.png").convert("RGB"),
    Image.open("frame_1.png").convert("RGB"),
    Image.open("frame_2.png").convert("RGB"),
    Image.open("frame_3.png").convert("RGB"),
]
inputs = processor.preprocess_video(frames, return_tensors="pt")  # T=len(frames)
pixel_values = inputs["pixel_values"].to(device=device, dtype=model.dtype)
grid_thws = inputs["grid_thws"].to(device=device)  # [[T, H_patches, W_patches]]

with torch.no_grad():
    video_features = model(pixel_values, grid_thws)

print(grid_thws.tolist(), video_features[0].shape)
# e.g. [[4, H, W]], torch.Size([N, 4, 1024])  # N after temporal pool + 2×2 merge

Notes

pixel_values: packed patches (num_patches, 3, 14, 14)
grid_thws: (batch, 3) = (T, H_patches, W_patches); images use T=1
Normalization: mean/std (0.5, 0.5, 0.5)
Optional: attn_implementation="flash_attention_2" if FlashAttention-2 is installed (default is SDPA)

Weight Provenance

Weights (~401M params, bfloat16) are taken from model-00096-of-000096.safetensors in moonshotai/Kimi-K3.

Original Kimi K3 Prefix	This Repo
`vision_tower.*`	`*` (prefix stripped)
`mm_projector.*`	omitted
`language_model.*`	omitted

Training Data & Infrastructure

Per the technical report, pretraining covers captions, interleaved documents, OCR, video, visual code, grounding (absolute + normalized coords), SVG, 3D assets, web pages, games, and CAD. Training uses context-parallel KV gather, image sharding across devices, and hiding ViT compute in pipeline bubbles. Typical use cases: OCR, visual UI iteration, and video understanding.