Foliation-B

A self-supervised video encoder in the line of TT-VidT: every frame keeps its own spatial tokens, and a few motion tokens per frame exchange information across time through block-causal Temporal Transfer (TT3D) attention.

Parameters	121.0M
Architecture	12 blocks, width 768, 12 heads, patch 16; per frame 8 motion tokens and 4 class tokens
Input	RGB clips [B, T, 3, H, W], pixels in [-1, 1] (mean = std = 0.5); trained on 32-frame clips of 256 x 256 frames at 6 fps
Weights	fp32 safetensors, encoder only

Quick start

The model code ships in this repository (needs only torch and transformers):

import torch
from transformers import AutoModel

model = AutoModel.from_pretrained("KBlueLeaf/Foliation-B", trust_remote_code=True).cuda().eval()
video = torch.rand(1, 32, 3, 256, 256, device="cuda") * 2 - 1    # [B, T, C, H, W] in [-1, 1]
with torch.no_grad(), torch.autocast("cuda", dtype=torch.float16):
    out = model(video)
out.motion          # [B, T, 8, 768] motion tokens per frame
out.frames          # [B, T, H/16 * W/16, 768] patch tokens, row-major
out.cls             # [B, T, 768] per-frame CLS (mean of the class tokens)

Frame t depends on frames 0..t only, so a prefix of a clip gives the same outputs as the clip itself.

Citation

This model builds on TT-VidT: Decoupling the Temporal Axis for Efficient Motion-Centric Video Pretraining, NeurIPS 2026 (arXiv:2609.33419).

License

Apache-2.0.