apiantonio/vjepa2.1-vit-gigantic-384

V-JEPA 2.1 ViT-G/16 384 — HuggingFace port

A HuggingFace-format conversion of Meta AI's V-JEPA 2.1 ViT-G/16 video encoder and predictor, operating at 384x384 resolution.

The weights are Meta's, copied without modification. This repository provides the transformers-compatible packaging plus a documented numerical validation against the original implementation.

No prior HuggingFace port of this variant existed at the time of upload.

Provenance

Original weights	Meta AI — facebookresearch/vjepa2
Source checkpoint	vjepa2_1_vitG_384.pt
State dict key	target_encoder
Conversion script	convert_vjepa21_to_hf.py from Dev-Jahn/vjepa2-hf
Modeling code	modeling_vjepa21.py / configuration_vjepa21.py, same repository
Weight modifications	None. Every tensor is a bit-exact copy of the original.

The only structural change is that the fused QKV projection of each attention block is split into separate query / key / value matrices, following the convention used by transformers. This is a re-parameterization, not a change of weights. It is also convenient downstream: PEFT adapters apply to independent q, k, v projections as in the original LoRA formulation, rather than to a fused projection.

Architecture

	Encoder	Predictor
Hidden size	1664	384
Layers	48	24
Attention heads	26	12
MLP ratio	64/13 ≈ 4.9231	4.0
Patch size	16 x 16, tubelet 2	—
Input resolution	384 x 384	—
Position encoding	3D RoPE	3D RoPE
Parameters	1.85 B	59.4 M

Total: 1.90 B.

Not distilled. Hierarchical self-distillation over the last 4 encoder representations (n_output_distillation = 4). The predictor input is a 2-layer MLP over the concatenated hierarchical features and projects back to encoder space (6656 = 1664 x 4).

The encoder carries a separate patch embedding for single images (img_temporal_dim_size = 1), learnable image/video modality embeddings, and four per-layer normalizations used for hierarchical feature extraction.

The four per-layer normalizations sit at encoder layers [11, 23, 37, 47] and all four carry trained affine parameters, since this checkpoint was trained with four distillation outputs. This makes it the right choice when intermediate multi-level features matter.

RoPE is interpolatable: positions are rescaled by (pretrained_grid_size - 1) / (n_patches - 1) with pretrained_grid_size = 256 / patch_size, following app/vjepa_2_1/models/utils/modules.py in the reference repository. This is what allows the model to accept resolutions other than 384 and non-square inputs.

Usage

Feature extraction

import torch
from transformers import AutoModel, AutoVideoProcessor

CKPT = "apiantonio/vjepa2.1-vit-gigantic-384"

processor = AutoVideoProcessor.from_pretrained(CKPT, trust_remote_code=True)
model = AutoModel.from_pretrained(CKPT, trust_remote_code=True, dtype=torch.float32).eval().cuda()

# list_of_clips: list of clips, each a list of HxWx3 uint8 frames
inputs = processor(list_of_clips, return_tensors="pt").to("cuda")

with torch.no_grad():
    out = model(**inputs, skip_predictor=True)

feats = out.last_hidden_state       # (B, T/2 * 24 * 24, 1664)

pixel_values_videos accepts (B, C, T, H, W), (B, T, C, H, W) — the layout produced by the video processor and by the transformers V-JEPA 2 classes — and (B, T, H, W, C). The channel axis is identified by matching config.in_chans; when more than one axis could match, the channels-first reading wins.

Token grid: (frames / 2) x 24 x 24. A 64-frame clip yields 32 x 24 x 24 = 18,432 tokens, so attention cost grows quadratically with clip length — use model.gradient_checkpointing_enable() and small batches for long clips.

Intermediate features

out = model(
    **inputs,
    skip_predictor=True,
    out_layers=model.config.encoder_hierarchical_layers,   # [11, 23, 37, 47]
)
levels = out.multilevel_hidden_states   # 4 tensors, each normalized with its own LayerNorm

This reproduces the out_layers recipe of the reference frozen-evaluation probes. output_hidden_states=True is also available and returns the raw, un-normalized output of the embedding layer plus every transformer layer.

Predictor

out = model(**inputs)                       # runs the predictor
pred = out.predictor_output.last_hidden_state      # (B, N_target, 6656)
ctx = out.predictor_output.context_hidden_state    # (B, N_context, 6656)

Without explicit masks, context_mask and target_mask both default to all tokens. Pass lists of (B, K) index tensors for a real masked-prediction setup. A single (context_mask, target_mask) pair is supported; more raises NotImplementedError rather than silently producing wrong shapes.

Classification head

from transformers import AutoModelForVideoClassification

model = AutoModelForVideoClassification.from_pretrained(
    CKPT, trust_remote_code=True, num_labels=101, label2id=label2id, id2label=id2label
)

The attentive pooler and the classifier are initialized from scratch, so the warning about newly initialized weights is expected and the video-classification pipeline tag is for discoverability only — the head must be trained before the model predicts anything meaningful.

Parameter-efficient fine-tuning

LoRA and DoRA work out of the box with peft. Scope target_modules explicitly, because matching on bare names such as ["query", "key", "value", "proj"] also adapts the predictor, which VJEPA21ForVideoClassification never executes:

from peft import LoraConfig, get_peft_model

config = LoraConfig(
    r=16, lora_alpha=32, lora_dropout=0.05,
    target_modules=r".*vjepa21\.encoder\.layer\.\d+\.attention\.(query|key|value|proj)$",
    modules_to_save=["classifier", "pooler"],
)
model = get_peft_model(model, config)

Preprocessing

VJEPA21VideoProcessor reproduces EvalVideoTransform from evals/video_classification_frozen/utils.py: the short side is resized to 384, a 384x384 crop is taken, pixels are scaled to [0, 1] and normalized with ImageNet statistics — mean (0.485, 0.456, 0.406), std (0.229, 0.224, 0.225), confirmed at evals/video_classification_frozen/eval.py:411.

This differs from the stock VJEPA2VideoProcessor in two ways, both of which require a dedicated class. It resizes the short side to exactly the crop size rather than to crop_size * 256 / 224, and it disables torchvision's antialiasing so the resize matches the reference cv2.resize(..., cv2.INTER_LINEAR). The second point only shows up when downscaling: on a 1080p frame the antialiased and non-antialiased results differ by up to 150/255 per pixel, mean 33/255, which is a visibly different image fed to a model that never saw antialiased inputs. With antialiasing disabled the two pipelines agree to 1/255, the 8-bit quantisation floor, at every resolution tested.

The reference transform takes several sliding crops along the long axis and averages the predictions; this processor takes a single centre crop. Multi-view aggregation, if you want it, is the caller's job.

Frame sampling is left to the caller. The reference configs use 16 frames with frame_step = 4 for Kinetics-400 and Something-Something v2, and 32 frames with frame_step = 2 for Jester. config.frames_per_clip = 64 records the pre-training clip length and is not an inference constraint.

Data pipeline

The processor covers the spatial transform. Which frames a clip is built from, and what a decoder returns for them, are the caller's responsibility and both fail silently, so video_io.py ships a reference implementation of each and tests/test_frame_sampling.py checks any implementation against it.

from video_io import clip_indices, decode_frames

indices = clip_indices(video_len=len(video), frames_per_clip=16, frame_step=4, num_clips=1)
frames = decode_frames(path, indices[0])      # (16, H, W, 3) uint8, RGB
inputs = processor([list(frames)], return_tensors="pt")

Two details are worth stating because they are easy to get wrong and produce no error.

The reference places the frames of a clip with np.linspace across a window of frames_per_clip * frame_step, so the effective stride is fpc*fstp/(fpc-1), not frame_step. Writing range(0, fpc*fstp, fstp) is the intuitive reading of "16 frames, step 4" and it selects a different frame 12 times out of 16. The resulting clip looks entirely plausible.

The decoder must return RGB. cv2.VideoCapture returns BGR, and decode_frames converts explicitly; it also decodes sequentially rather than seeking with CAP_PROP_POS_FRAMES, which snaps to the nearest keyframe on several codecs and quietly returns a neighbouring frame. decord, the backend the reference pipeline uses, is preferred when installed.

Long videos and multiple clips

The reference evaluation splits a video into num_segments partitions, takes one clip and num_views_per_segment spatial crops from each, and averages the resulting softmax distributions into a single prediction per video. That is variance reduction for classifying a short video, and it does not transfer to a long one: with the reference settings for this checkpoint, eight segments cover 98.7% of a ten-second video but only 14.2% of a two-minute one, leaving a blind gap of 386 frames — thirteen seconds during which nothing is observed.

clip_indices reproduces that partitioned sampling. When you need a score per position in time rather than one prediction per video, use dense_clip_indices instead, and map the results back onto frames:

from video_io import dense_clip_indices, temporal_coverage, clip_scores_to_frame_scores

clips = dense_clip_indices(video_len, frames_per_clip=16, frame_step=4)   # stride=window
print(temporal_coverage(clips, video_len))     # {'covered_fraction': 1.0, 'max_gap': 0, ...}

frame_scores = clip_scores_to_frame_scores(clips, scores, video_len, reduce="max")

stride controls overlap and therefore temporal resolution. temporal_coverage reports what a set of clips actually looks at, which is worth checking before trusting any per-frame metric.

For video-level predictions, aggregate_predictions reproduces the reference combination — the mean of the softmax outputs, not of the logits. The two are different estimators and can rank classes differently; on a two-view example they give [0.52, 0.19, 0.29] against [0.77, 0.10, 0.13].

Note that dense extraction has a storage cost. A 16-frame clip at 384 is 4608 tokens, so caching token-level features for a two-minute video is about 1.8 GB at this hidden size. Pooling to one vector per clip is four orders of magnitude smaller, but forecloses any probe that consumes the token sequence.

Validation

All tests use float32 and torch.no_grad().

1. Weight-level provenance

Every tensor in the published safetensors was traced back to a tensor in the official checkpoint and compared element-wise.

Check	Result
max|Δ| across all tensors	0.000e+00
Parameters with no origin in the checkpoint	0

The second row matters: the conversion script calls load_state_dict(..., strict=False), which silently leaves unmatched parameters randomly initialized. Zero orphans means no such parameter survived — for the predictor as well as the encoder.

2. Encoder forward parity, published weights

The same input tensor was passed through Meta's official encoder (loaded via torch.hub) and through this port, at three clip lengths and full 384x384 resolution.

T	Tokens	max|Δ|	mean|Δ|	mean|Δ| / mean|a|	cos-sim (min)
16	4,608	0.000e+00	0.000e+00	0.000e+00	1.000000
32	9,216	0.000e+00	0.000e+00	0.000e+00	1.000000
64	18,432	0.000e+00	0.000e+00	0.000e+00	1.000000

Summary. Bit-exact agreement with the reference implementation at every tested clip length. Varying T from 16 to 64 exercises the RoPE position interpolation, which is the most likely silent failure mode in a port of this architecture; no growth of the residual with sequence length was observed.

3. Implementation parity, code paths not covered above

The checks in §2 only exercise the encoder video branch through skip_predictor=True. To cover the rest, reference modules were built directly from app/vjepa_2_1/models/ and their weights copied into the port, then outputs compared. These runs use reduced hidden sizes so the suite stays fast: they establish that the code paths are equivalent, which is width-independent, and complement the full-resolution published-weight results in §2.

Path	max|Δ|
Encoder, video branch, n_output_distillation = 1	0.000e+00
Encoder, video branch, n_output_distillation = 4	0.000e+00
Encoder, image branch (T = 1)	0.000e+00
Encoder, non-square input (96x128)	0.000e+00
out_layers per-level features, all 4 levels, both regimes	0.000e+00
Concatenated hierarchical output	0.000e+00
Predictor, target / context tokens	2.4e-06 / 9.5e-07
Hierarchical predictor, target / context tokens	2.1e-06 / 1.0e-06

Both distillation regimes are covered: n_output_distillation = 1 is the ViT-B and ViT-L configuration, 4 the ViT-g and ViT-G one, where the encoder emits concatenated levels and the predictor fuses them through a 2-layer MLP. The predictor residual is float32 accumulation noise from a different gather ordering — the port reorders tokens with gather where the reference uses stack — and sits at roughly twenty times the float32 epsilon of 1.2e-07 after twelve layers, an order of magnitude inside the 1e-3 tolerance Meta uses to validate its own ports.

tests/test_end_to_end_pipeline.py closes the loop between the two: one clip is pushed through the reference Resize -> crop -> ClipToTensor -> Normalize and through VJEPA21VideoProcessor, at 240p, 480p, 720p and 1080p. All four land on max|Δ| = 1.75e-02, which is one 8-bit grey level divided by the smallest normalisation std — the floor two different bilinear implementations can reach. Both reproduction scripts are shipped here; point VJEPA2_REPO at a local clone of facebookresearch/vjepa2 and run them with pytest.

4. Functional test suite

88 tests ship with the repository, green on both transformers==4.57.1 and transformers==5.14.1.

File	Tests	Needs
tests/test_vjepa21.py	30	nothing
tests/test_frame_sampling.py	32	ffmpeg
tests/test_parity_official.py	9	VJEPA2_REPO
tests/test_end_to_end_pipeline.py	9	VJEPA2_REPO, optionally VJEPA21_CKPT
tests/test_thesis_robustness.py	8	optionally VJEPA21_CKPT

The first file covers configuration properties and validation, input layouts, hidden states and attentions, multi-level extraction, the image branch, variable resolution, sdpa-vs-eager agreement, the predictor with default and explicit masks, classification loss and backward pass, gradient checkpointing parity, low-precision weight loading in bf16 and fp16, the video processor's geometry and normalization constants, and a full save_pretrained / from_pretrained round-trip through the Auto classes. It needs neither weights nor the reference repository.

5. Properties relevant to downstream pipelines

tests/test_thesis_robustness.py checks assumptions an experimental pipeline tends to make implicitly. Point it at this repository with VJEPA21_CKPT to re-measure on the published weights.

Measured on this checkpoint, float32 on an NVIDIA GPU unless stated otherwise.

Property	Result
Two identical forwards in eval mode	bit-identical
Same clip at batch 1 vs batch 8	rel 0.000e+00, cos-sim 1.000000 (abs 0.000e+00)
bfloat16 inference vs float32	rel 1.721e-02, min cos-sim 0.999011
float16 inference vs float32	rel 2.254e-03, min cos-sim 0.999964
Storing float32 features as fp16	rel 1.768e-04, min cos-sim 1.000000
merge_and_unload changes the function	max|Δ| = 5.7e-09
Merged model vs baseline parameter count	identical, 0 residual adapter tensors

Repeated forwards of the same shape are bit-identical, so a cached feature is reproducible from its input. On this checkpoint batching happened to be bit-exact as well, but that is not guaranteed: at other shapes or on other hardware the kernels may split the reduction differently. Smaller variants of this family do show a residual of about 1e-05 relative.

Read the absolute column with care: it tracks the magnitude of the activations, not the quality of the agreement, which is why the relative error is the meaningful number.

The last two rows are the reparameterisation property that makes LoRA and DoRA free at inference time. They are measured on a small model, since exact merging follows from the algebra rather than from the weights.

Precision guidance

float16 is roughly an order of magnitude more accurate than bfloat16 for inference with this model — ten mantissa bits against eight — and inference activations stay well inside the float16 range.

For feature extraction, prefer float32 or float16. bfloat16 costs a worst-case per-token cosine similarity of 0.999011, about four degrees of angular noise; if features feed a cosine-similarity or nearest-neighbour decision rule, that noise enters the decision directly. Storing float32 features as float16 is nearly free by comparison and halves the cache.

For training, the usual bfloat16 argument still applies: the wider exponent range is safer for gradients. These are two separate decisions and it is worth making them separately.

Compatibility

Requires transformers >= 4.50; tested on 4.57.1 and 5.14.1. sdpa and flash_attention_2 are supported; requesting output_attentions=True falls back to the eager kernel, the only one that materializes attention probabilities. peft 0.19.1 was verified for LoRA, DoRA and merge_and_unload. Weights held in bfloat16 or float16 are supported; see the precision note in section 5 of the validation before choosing one.

Not verified

Stated explicitly so the scope of the validation above is not overread:

• Downstream benchmarks. No evaluation numbers from the V-JEPA 2.1 release were reproduced. This port is validated for numerical equivalence, not for task performance.
• Your own data pipeline. video_io.py reproduces the reference sampler and decoder and is tested against them, but if you write your own, only the contract tests in tests/test_frame_sampling.py stand between you and a silent mismatch.
• Aggregation policy. aggregate_predictions reproduces the reference combination and dense_clip_indices covers the whole timeline, but which policy suits a given task — mean over clips, max, top-k — is a modelling decision that is neither made nor evaluated here.
• Multiple mask pairs in the predictor. Only a single (context_mask, target_mask) pair is supported.
• Reduced precision downstream. bf16 and fp16 deviations are measured above, but their effect on any downstream metric was not assessed.

License and attribution

The weights are released by Meta AI under the license of the original V-JEPA 2 release (MIT). This repository redistributes them unmodified and adds no additional restrictions.

Credit where it is due:

• Meta AI — the V-JEPA 2 / V-JEPA 2.1 research and the pretrained weights.
• Dev-Jahn — the original transformers-compatible modeling code and the conversion script, which this repository builds on.

The contribution specific to this repository is the conversion of the G variant, the revised modeling code and video processor described above, and the numerical validation documented here.

Citation

@article{assran2025vjepa2,
  title={V-JEPA~2: Self-Supervised Video Models Enable Understanding, Prediction and Planning},
  author={Assran, Mahmoud and Bardes, Adrien and Fan, David and Garrido, Quentin and Howes, Russell and
Komeili, Mojtaba and Muckley, Matthew and Rizvi, Ammar and Roberts, Claire and Sinha, Koustuv and Zholus, Artem and
Arnaud, Sergio and Gejji, Abha and Martin, Ada and Robert Hogan, Francois and Dugas, Daniel and
Bojanowski, Piotr and Khalidov, Vasil and Labatut, Patrick and Massa, Francisco and Szafraniec, Marc and
Krishnakumar, Kapil and Li, Yong and Ma, Xiaodong and Chandar, Sarath and Meier, Franziska and LeCun, Yann and
Rabbat, Michael and Ballas, Nicolas},
  journal={arXiv preprint arXiv:2506.09985},
  year={2025}
}

@article{murlabadia2026vjepa2_1,
  title={V-JEPA 2.1: Unlocking Dense Features in Video Self-Supervised Learning},
  author={Mur-Labadia, Lorenzo and Muckley, Matthew and Bar, Amir and Assran, Mahmoud and
Sinha, Koustuv and Rabbat, Michael and LeCun, Yann and Ballas, Nicolas and Bardes, Adrien},
  journal={arXiv preprint arXiv:2603.14482},
  year={2026}
}

Please cite the original V-JEPA 2 work. If the format conversion itself was useful, a link back to Dev-Jahn/vjepa2-hf is appreciated.