LookWhen ViT-B/16

Pre-trained checkpoint for "LookWhen? Fast Video Recognition by Learning When, Where, and What to Compute" (arXiv:2605.06809).

LookWhen splits video recognition in two: a shallow selector scores every patch of a 2x-downscaled (2x in each dimension) clip, and a deep extractor processes only the top-K of them while still predicting features of the whole video.

This is the model used throughout the paper: ViT-B/16, pre-trained for 20 epochs on Kinetics-400 and SSv2 by distilling InternVideo2's video token and DINOv3's frame and patch tokens, with selection supervised by top1-distance token uniqueness. It was trained over sparsity 70-95%, so it runs anywhere in that range without retraining.

Usage

pip install git+https://github.com/alisalamatian1/LookWhen.git

import torch
from lookwhen import LookWhen
from lookwhen.data.video import load_clip

model = LookWhen.from_pretrained().cuda().eval()   # downloads this checkpoint
clip = load_clip("video.mp4", num_frames=16, img_size=224).unsqueeze(0).cuda()

out = model(clip, keep_ratio=0.1)   # keep_ratio = 1 - sparsity
out["video"]   # (1, 768) video embedding

Code, training, and evaluation: https://github.com/alisalamatian1/LookWhen

Citation

@article{salamatian2026lookwhen,
  title   = {LookWhen? Fast Video Recognition by Learning When, Where, and What to Compute},
  author  = {Salamatian, Ali and Fuller, Anthony and Sarkar, Pritam and
             Green, James R. and Sigal, Leonid and Shelhamer, Evan},
  journal = {arXiv preprint arXiv:2605.06809},
  year    = {2026}
}