apiantonio/vjepa2.1-vit-giant-384

V-JEPA 2.1 ViT-g/16 384 — HuggingFace port

A HuggingFace-format conversion of Meta AI's V-JEPA 2.1 ViT-g/16 video encoder and predictor, operating at 384x384 resolution.

The weights are Meta's, copied without modification. This repository provides the transformers-compatible packaging plus a documented numerical validation against the original implementation.

No prior HuggingFace port of this variant existed at the time of upload.

Provenance

Original weights	Meta AI — facebookresearch/vjepa2
Source checkpoint	vjepa2_1_vitg_384.pt
State dict key	target_encoder
Conversion script	convert_vjepa21_to_hf.py from Dev-Jahn/vjepa2-hf
Modeling code	modeling_vjepa21.py / configuration_vjepa21.py, same repository
Weight modifications	None. Every tensor is a bit-exact copy of the original.

The only structural change is that the fused QKV projection of each attention block is split into separate query / key / value matrices, following the convention used by transformers. This is a re-parameterization, not a change of weights. It is also convenient downstream: PEFT adapters apply to independent q, k, v projections as in the original LoRA formulation, rather than to a fused projection.

Architecture

	Encoder	Predictor
Hidden size	1408	384
Layers	40	24
Attention heads	22	12
MLP ratio	48/11 ≈ 4.3636	4.0
Patch size	16 x 16, tubelet 2	—
Input resolution	384 x 384	—
Position encoding	3D RoPE	3D RoPE
Parameters	1.01 B	55.4 M

Total: 1.07 B.

Not distilled. Hierarchical self-distillation over the last 4 encoder representations (n_output_distillation = 4). The predictor input is a 2-layer MLP over the concatenated hierarchical features and projects back to encoder space (5632 = 1408 x 4).

The encoder carries a separate patch embedding for single images (img_temporal_dim_size = 1), learnable image/video modality embeddings, and four per-layer normalizations used for hierarchical feature extraction.

The four per-layer normalizations sit at encoder layers [9, 19, 29, 39] and all four carry trained affine parameters, since this checkpoint was trained with four distillation outputs. This makes it the right choice when intermediate multi-level features matter.

RoPE is interpolatable: positions are rescaled by (pretrained_grid_size - 1) / (n_patches - 1) with pretrained_grid_size = 256 / patch_size, following app/vjepa_2_1/models/utils/modules.py in the reference repository. This is what allows the model to accept resolutions other than 384 and non-square inputs.

Usage

Feature extraction

import torch
from transformers import AutoModel, AutoVideoProcessor

CKPT = "apiantonio/vjepa2.1-vit-giant-384"

processor = AutoVideoProcessor.from_pretrained(CKPT, trust_remote_code=True)
model = AutoModel.from_pretrained(CKPT, trust_remote_code=True, dtype=torch.float32).eval().cuda()

# list_of_clips: list of clips, each a list of HxWx3 uint8 frames
inputs = processor(list_of_clips, return_tensors="pt").to("cuda")

with torch.no_grad():
    out = model(**inputs, skip_predictor=True)

feats = out.last_hidden_state       # (B, T/2 * 24 * 24, 1408)

pixel_values_videos accepts (B, C, T, H, W), (B, T, C, H, W) — the layout produced by the video processor and by the transformers V-JEPA 2 classes — and (B, T, H, W, C). The channel axis is identified by matching config.in_chans; when more than one axis could match, the channels-first reading wins.

Token grid: (frames / 2) x 24 x 24. A 64-frame clip yields 32 x 24 x 24 = 18,432 tokens, so attention cost grows quadratically with clip length — use model.gradient_checkpointing_enable() and small batches for long clips.

Intermediate features

out = model(
    **inputs,
    skip_predictor=True,
    out_layers=model.config.encoder_hierarchical_layers,   # [9, 19, 29, 39]
)
levels = out.multilevel_hidden_states   # 4 tensors, each normalized with its own LayerNorm

This reproduces the out_layers recipe of the reference frozen-evaluation probes. output_hidden_states=True is also available and returns the raw, un-normalized output of the embedding layer plus every transformer layer.

Predictor

out = model(**inputs)                       # runs the predictor
pred = out.predictor_output.last_hidden_state      # (B, N_target, 5632)
ctx = out.predictor_output.context_hidden_state    # (B, N_context, 5632)

Without explicit masks, context_mask and target_mask both default to all tokens. Pass lists of (B, K) index tensors for a real masked-prediction setup. A single (context_mask, target_mask) pair is supported; more raises NotImplementedError rather than silently producing wrong shapes.

Classification head

from transformers import AutoModelForVideoClassification

model = AutoModelForVideoClassification.from_pretrained(
    CKPT, trust_remote_code=True, num_labels=101, label2id=label2id, id2label=id2label
)

The attentive pooler and the classifier are initialized from scratch, so the warning about newly initialized weights is expected and the video-classification pipeline tag is for discoverability only — the head must be trained before the model predicts anything meaningful.

Parameter-efficient fine-tuning

LoRA and DoRA work out of the box with peft. Scope target_modules explicitly, because matching on bare names such as ["query", "key", "value", "proj"] also adapts the predictor, which VJEPA21ForVideoClassification never executes:

from peft import LoraConfig, get_peft_model

config = LoraConfig(
    r=16, lora_alpha=32, lora_dropout=0.05,
    target_modules=r".*vjepa21\.encoder\.layer\.\d+\.attention\.(query|key|value|proj)$",
    modules_to_save=["classifier", "pooler"],
)
model = get_peft_model(model, config)

Preprocessing

VJEPA21VideoProcessor reproduces EvalVideoTransform from evals/video_classification_frozen/utils.py: the short side is resized to 384, a 384x384 crop is taken, pixels are scaled to [0, 1] and normalized with ImageNet statistics — mean (0.485, 0.456, 0.406), std (0.229, 0.224, 0.225), confirmed at evals/video_classification_frozen/eval.py:411.

This differs from the stock VJEPA2VideoProcessor, which resizes the short side to crop_size * 256 / 224 before cropping; the V-JEPA 2.1 evaluations resize to exactly the crop size, hence a dedicated class.

Frame sampling is left to the caller. The reference configs use 16 frames with frame_step = 4 for Kinetics-400 and Something-Something v2, and 32 frames with frame_step = 2 for Jester. config.frames_per_clip = 64 records the pre-training clip length and is not an inference constraint.

Validation

All tests use float32 and torch.no_grad().

1. Weight-level provenance

Every tensor in the published safetensors was traced back to a tensor in the official checkpoint and compared element-wise.

Check	Result
max|Δ| across all tensors	0.000e+00
Parameters with no origin in the checkpoint	0

The second row matters: the conversion script calls load_state_dict(..., strict=False), which silently leaves unmatched parameters randomly initialized. Zero orphans means no such parameter survived — for the predictor as well as the encoder.

2. Encoder forward parity, published weights

The same input tensor was passed through Meta's official encoder (loaded via torch.hub) and through this port, at three clip lengths and full 384x384 resolution.

T	Tokens	max|Δ|	mean|Δ|	mean|Δ| / mean|a|	cos-sim (min)
16	4,608	0.000e+00	0.000e+00	0.000e+00	1.000000
32	9,216	0.000e+00	0.000e+00	0.000e+00	1.000000
64	18,432	0.000e+00	0.000e+00	0.000e+00	1.000000

Summary. Bit-exact agreement with the reference implementation at every tested clip length. Varying T from 16 to 64 exercises the RoPE position interpolation, which is the most likely silent failure mode in a port of this architecture; no growth of the residual with sequence length was observed.

3. Implementation parity, code paths not covered above

The checks in §2 only exercise the encoder video branch through skip_predictor=True. To cover the rest, reference modules were built directly from app/vjepa_2_1/models/ and their weights copied into the port, then outputs compared. These runs use reduced hidden sizes so the suite stays fast: they establish that the code paths are equivalent, which is width-independent, and complement the full-resolution published-weight results in §2.

Path	max|Δ|
Encoder, video branch, n_output_distillation = 1	0.000e+00
Encoder, video branch, n_output_distillation = 4	0.000e+00
Encoder, image branch (T = 1)	0.000e+00
Encoder, non-square input (96x128)	0.000e+00
out_layers per-level features, all 4 levels, both regimes	0.000e+00
Concatenated hierarchical output	0.000e+00
Predictor, target / context tokens	2.4e-06 / 9.5e-07
Hierarchical predictor, target / context tokens	2.1e-06 / 1.0e-06

Both distillation regimes are covered: n_output_distillation = 1 is the ViT-B and ViT-L configuration, 4 the ViT-g and ViT-G one, where the encoder emits concatenated levels and the predictor fuses them through a 2-layer MLP. The predictor residual is float32 accumulation noise from a different gather ordering — the port reorders tokens with gather where the reference uses stack — and sits at roughly twenty times the float32 epsilon of 1.2e-07 after twelve layers, an order of magnitude inside the 1e-3 tolerance Meta uses to validate its own ports. The reproduction script is shipped in this repository as tests/test_parity_official.py; point VJEPA2_REPO at a local clone of facebookresearch/vjepa2 and run it with pytest.

4. Functional test suite

30 tests covering configuration properties and validation, input layouts, hidden states and attentions, multi-level extraction, image branch, variable resolution, sdpa-vs-eager agreement, predictor with default and explicit masks, classification loss and backward pass, gradient checkpointing parity, the video processor's geometry and normalization constants, and a full save_pretrained / from_pretrained round-trip through the Auto classes, plus low-precision weight loading in bf16 and fp16. Shipped as tests/test_vjepa21.py; requires no weights and no reference repository. Green on both transformers==4.57.1 and transformers==5.14.1.

5. Properties relevant to downstream pipelines

tests/test_thesis_robustness.py checks assumptions an experimental pipeline tends to make implicitly. Point it at this repository with VJEPA21_CKPT to re-measure on the published weights.

Measured on this checkpoint, float32 on an NVIDIA GPU unless stated otherwise.

Property	Result
Two identical forwards in eval mode	bit-identical
Same clip at batch 1 vs batch 8	rel 5.397e-06, cos-sim 1.000000 (abs 1.111e-02)
bfloat16 inference vs float32	rel 1.543e-02, min cos-sim 0.998124
float16 inference vs float32	rel 1.960e-03, min cos-sim 0.999957
Storing float32 features as fp16	rel 1.760e-04, min cos-sim 1.000000
merge_and_unload changes the function	max|Δ| = 5.7e-09
Merged model vs baseline parameter count	identical, 0 residual adapter tensors

Repeated forwards of the same shape are bit-identical, so a cached feature is reproducible from its input. Changing the batch size is not bit-exact on GPU: cuBLAS and the fused attention kernels select tiling and reduction order as a function of the input shape, so the same clip accumulates differently at batch 1 and batch 8. The effect is ~1e-05 relative and irrelevant to a probe, but it means a feature store should be built with a fixed batch size. On CPU the same comparison is exact.

Read the absolute column with care: it tracks the magnitude of the activations, not the quality of the agreement, which is why the relative error is the meaningful number.

The last two rows are the reparameterisation property that makes LoRA and DoRA free at inference time. They are measured on a small model, since exact merging follows from the algebra rather than from the weights.

Precision guidance

float16 is roughly an order of magnitude more accurate than bfloat16 for inference with this model — ten mantissa bits against eight — and inference activations stay well inside the float16 range.

For feature extraction, prefer float32 or float16. bfloat16 costs a worst-case per-token cosine similarity of 0.998124, about four degrees of angular noise; if features feed a cosine-similarity or nearest-neighbour decision rule, that noise enters the decision directly. Storing float32 features as float16 is nearly free by comparison and halves the cache.

For training, the usual bfloat16 argument still applies: the wider exponent range is safer for gradients. These are two separate decisions and it is worth making them separately.

Compatibility

Requires transformers >= 4.50; tested on 4.57.1 and 5.14.1. sdpa and flash_attention_2 are supported; requesting output_attentions=True falls back to the eager kernel, the only one that materializes attention probabilities. peft 0.19.1 was verified for LoRA, DoRA and merge_and_unload. Weights held in bfloat16 or float16 are supported; see the precision note in section 5 of the validation before choosing one.

Not verified

Stated explicitly so the scope of the validation above is not overread:

• Downstream benchmarks. No evaluation numbers from the V-JEPA 2.1 release were reproduced. This port is validated for numerical equivalence, not for task performance.
• Frame sampling. The video processor reproduces the spatial transform; temporal sampling (clip length, frame step, multi-clip aggregation) is left to the caller and was not validated against any specific benchmark protocol.
• Multiple mask pairs in the predictor. Only a single (context_mask, target_mask) pair is supported.
• Reduced precision downstream. bf16 and fp16 deviations are measured above, but their effect on any downstream metric was not assessed.

License and attribution

The weights are released by Meta AI under the license of the original V-JEPA 2 release (MIT). This repository redistributes them unmodified and adds no additional restrictions.

Credit where it is due:

• Meta AI — the V-JEPA 2 / V-JEPA 2.1 research and the pretrained weights.
• Dev-Jahn — the original transformers-compatible modeling code and the conversion script, which this repository builds on.

The contribution specific to this repository is the conversion of the g variant, the revised modeling code and video processor described above, and the numerical validation documented here.

Citation

@article{assran2025vjepa2,
  title={V-JEPA~2: Self-Supervised Video Models Enable Understanding, Prediction and Planning},
  author={Assran, Mahmoud and Bardes, Adrien and Fan, David and Garrido, Quentin and Howes, Russell and
Komeili, Mojtaba and Muckley, Matthew and Rizvi, Ammar and Roberts, Claire and Sinha, Koustuv and Zholus, Artem and
Arnaud, Sergio and Gejji, Abha and Martin, Ada and Robert Hogan, Francois and Dugas, Daniel and
Bojanowski, Piotr and Khalidov, Vasil and Labatut, Patrick and Massa, Francisco and Szafraniec, Marc and
Krishnakumar, Kapil and Li, Yong and Ma, Xiaodong and Chandar, Sarath and Meier, Franziska and LeCun, Yann and
Rabbat, Michael and Ballas, Nicolas},
  journal={arXiv preprint arXiv:2506.09985},
  year={2025}
}

@article{murlabadia2026vjepa2_1,
  title={V-JEPA 2.1: Unlocking Dense Features in Video Self-Supervised Learning},
  author={Mur-Labadia, Lorenzo and Muckley, Matthew and Bar, Amir and Assran, Mahmoud and
Sinha, Koustuv and Rabbat, Michael and LeCun, Yann and Ballas, Nicolas and Bardes, Adrien},
  journal={arXiv preprint arXiv:2603.14482},
  year={2026}
}

Please cite the original V-JEPA 2 work. If the format conversion itself was useful, a link back to Dev-Jahn/vjepa2-hf is appreciated.