You need to agree to share your Hugging Face information to access this model
This repository is publicly accessible, but you have to accept the conditions to access its files and content.
The NativeCodec repositories use manual approval. Weights, configs, tokenizers and dataset archives are available after your access request is approved. Please describe your intended use; the model card and dataset card remain readable without approval.
Log in or Sign Up to review the conditions and access this model content.
NativeCodec
Encoding every video frame with variable-length causal visual tokens.
NativeCodec is a causal visual encoder that feeds every video frame to a multimodal LLM at up to 16 fps without exceeding the visual-token budget. It does not sample frames or prune tokens after encoding. It learns variable-length causal visual tokens: full I-frame anchors plus compact P-frame suffixes whose length adapts to how much new information each frame carries relative to its history.
This repository holds the official code and the released codec checkpoints (weights/). The aligned 4B language models live in two companion model repositories, and the evaluation dataset lives in a companion dataset repository:
| Repository | Content |
|---|---|
Tennineee/NativeCodec-4B-PredLen |
4B LLM aligned with the learned length predictor (dynamic suffix ratio) |
Tennineee/NativeCodec-4B-FixedRatio |
4B LLM aligned for a constant P-frame suffix ratio (1/16 recommended) |
Tennineee/NativeCodec-BallBench |
BallBench high-frame-rate identity-reasoning dataset |
The code is released under the MIT License. The released weights are distributed under Apache-2.0, inherited from the Qwen3.5-4B-Base base model. See License.
News
| Date | Update |
|---|---|
| 2026/09/23 | Initial release: two-stage codec training, downstream LLM alignment, five-benchmark evaluation pipeline, and BallBench. |
Access request
All four repositories are public but gated with manual approval. Anonymous visitors can read this model card and the model cards of the companion repositories, and can see file names and sizes. Downloading any file other than the README requires access. Access is granted to individual users, not to organizations.
To request access, open the repository page on huggingface.co, sign in, click Agree and access repository / Request access, and submit the form. The form collects your Hub username and primary email by default; the model card asks for your full name, affiliation and intended use. You also confirm that you will use the materials in accordance with the license terms.
Requests are reviewed manually by the maintainers. There is no automatic approval and no published turnaround guarantee. The Hub notifies you when a decision is made. A rejected request cannot be resubmitted from the same account on the Hub; if you believe the decision was a mistake, contact the maintainers through the repository page.
Once approved, install huggingface_hub and download with:
# Code and codec checkpoints (about 9.47 GiB)
hf download Tennineee/NativeCodec --local-dir NativeCodec
# Aligned LLM, predicted-length variant (about 8.7 GiB)
hf download Tennineee/NativeCodec-4B-PredLen --local-dir NativeCodec-4B-PredLen
# Aligned LLM, fixed-ratio variant (about 8.7 GiB, optional)
hf download Tennineee/NativeCodec-4B-FixedRatio --local-dir NativeCodec-4B-FixedRatio
# BallBench dataset (archive about 4.0 GB, optional)
hf download Tennineee/NativeCodec-BallBench --repo-type dataset --local-dir NativeCodec-BallBench
Every download above needs an authenticated session. Log in with a token that can see gated repositories:
hf auth login
# older huggingface_hub versions: huggingface-cli login
From mainland China, huggingface.co is often slow or unreachable. The commands also work through the community mirror, which proxies gated downloads after you log in:
HF_ENDPOINT=https://hf-mirror.com hf download Tennineee/NativeCodec --local-dir NativeCodec
The mirror does not host the login flow, so authenticate against huggingface.co first and only set HF_ENDPOINT for the download command. Use --include to fetch a subset, for example --include "weights/*". If you use a fine-grained token, enable access to public gated repositories in the token settings, otherwise the Hub returns 403 for gated files even after approval.
What is NativeCodec?
Video MLLMs face a dilemma: motion and identity evidence often lives in brief interactions that sparse frame sampling destroys. Raising the frame rate helps, yet encoding every frame independently floods the LLM with redundant tokens: the second frame of a static scene barely adds anything over the first, and still costs the same H/32 x W/32 visual tokens.
The practical question is how much capacity each frame deserves given what came before. NativeCodec answers it with a predictive-coding-style interface:
- Every admitted frame keeps its temporal position, so transient events such as contacts, rebounds and identity swaps stay addressable.
- Each P frame is encoded causally against its history, and only a suffix of its representation is retained.
- The suffix length is predicted per frame, so static continuation costs a few tokens while scene changes cost many.
- Anchor (I-frame) placement follows the stream itself: once accumulated suffixes fill a frame's worth of budget, the next frame resets the causal context as a new anchor.
At inference the budget is enforced exactly: if a stream exceeds max_video_tokens, frames are uniformly subsampled across the clip and the stream is re-planned. A frame is never truncated mid-frame.
Our solution
| Component | What it does |
|---|---|
| Incremental Encoder | A causal transformer over per-frame token grids (initialized from frozen LLM layers). Frame t attends to all history and itself, producing a full causal representation H_t. |
| Truncatable Suffixes | Train with random suffix ratios {1, 1/2, ..., 1/64}; pixel and feature reconstruction at every truncation length pushes each frame's residual information into the tail of its representation. |
| Oracle-Guided Length Predictor | An offline reconstruction teacher searches, per frame and in temporal order, the smallest ratio that stays within delta dB of full-ratio quality; a feed-forward head distills this search into one parallel pass. |
| Adaptive GOP Stream | Cumulative suffix budget determines anchor placement. Every retained token enters the LLM at its true (t, y, x) coordinates, with no pooling and no reordering. |
| BallBench | A new benchmark for high-frame-rate identity reasoning: 4 to 10 visually identical balls collide at 30 fps; answering requires tracking trajectories through collisions where appearance cues are useless. |
Results
Main comparison (open-source video MLLMs, 64k visual-token budget; NativeCodec uses a constant suffix ratio of 1/16, † = predicted suffix lengths):
| Model | BallBench | TL-AN | TL-CH | TL-QV | TL-All | MS | VH | MME-v2 |
|---|---|---|---|---|---|---|---|---|
| Qwen3-VL-8B | 14.3 | 48.0 | 47.2 | 59.4 | 49.6 | 9.8 | 42.5 | 27.9 |
| Qwen3.5-9B | 14.7 | 49.9 | 45.6 | 64.0 | 50.7 | 12.8 | 46.3 | 26.0 |
| LLaVA-OV-2-8B | 17.1 | 53.8 | 53.5 | 66.4 | 55.8 | 19.6 | 32.5 | 19.9 |
| Qwen3.5-4B (base) | 12.2 | 49.4 | 46.0 | 64.6 | 50.7 | 18.5 | 39.5 | 24.1 |
| Qwen3.5-4B-SFT | 15.3 | 51.7 | 49.5 | 66.6 | 53.2 | 18.8 | 44.0 | 23.3 |
| NativeCodec-4B | 26.1 | 54.5 | 53.8 | 67.7 | 56.4 | 25.0 | 47.1 | 24.6 |
| NativeCodec-4B† | 28.2 | 52.7 | 51.4 | 66.2 | 54.4 | 24.7 | 48.7 | 23.7 |
Column keys: BallBench accuracy, TimeLens-Bench (TL) average/action (AN)/change (CH)/query (QV) mIoU, MomentSeeker (MS) mIoU, Video-Holmes (VH), Video-MME-v2 (MME-v2).
NativeCodec-4B is the best open-source model on TimeLens-Bench (56.4 mIoU) and every subset, best on MomentSeeker (25.0), best on Video-Holmes with predicted lengths (48.7), and +13.9 over its own Qwen3.5-4B base on BallBench. On BallBench every other open-source baseline sits within 4 points of the 13.9 random level.
Tight budgets show the largest margin over the native visual pathway of the same Qwen3.5-4B base (TimeLens-Bench mIoU):
| Visual-token budget | 4k | 8k | 16k | 32k | 64k |
|---|---|---|---|---|---|
| Delta mIoU vs. native pathway | +14.2 | +12.8 | +9.9 | +7.6 | +5.1 |
At 16k tokens NativeCodec reaches 54.4 mIoU, above the 50.9 the native pathway reaches with four times the budget (64k).
BallBench vs. frame rate (accuracy, 490 questions; suffix ratio 1/16):
| Model | 16 fps | 8 fps | 4 fps | 2 fps |
|---|---|---|---|---|
| Random | 13.9 | 13.9 | 13.9 | 13.9 |
| Qwen3.5-4B | 12.2 | 13.2 | 8.6 | 10.4 |
| LLaVA-OV-2-8B | 17.1 | 17.8 | 16.3 | 17.8 |
| NativeCodec-4B | 26.1 | 24.3 | 23.5 | 22.9 |
Every open-source baseline stays within four points of random at all frame rates. NativeCodec improves monotonically with temporal resolution: it admits high-frame-rate input and uses that temporal resolution.
Repository structure
NativeCodec/
|-- README.md
|-- LICENSE # MIT, for the code
|-- LICENSE-APACHE # Apache-2.0, for the released weights
|-- NOTICE.md # Qwen attribution and change statement
|-- requirements.txt
|-- weights/ # released codec checkpoints (download from this Hub repository)
| |-- inc_enc_stage1_step18750.pt
| |-- pred_len_delta00_step9000.pt
| `-- pred_len_delta08_step3150.pt
|-- stage1_recodec_pretrain/ # Stage 1: truncatable representations
| |-- train_frame_token.py # training entry (pixel + feature reconstruction)
| |-- config_real_phase1_4b.yaml # 4B config (9 size x window curricula)
| |-- run.sh
| |-- recodec/ # model / encoder / decoder / vision / rope / data
| `-- analysis/ # reconstruction sweeps and visualizations
|-- stage2_len_pred/ # Stage 2: oracle-guided length training
| |-- train_len_pred.py # frozen-oracle sequential GT search + CE head
| |-- config_*.yaml # delta = 0.0 / 0.8, multi-pooling and soft-RD variants
| |-- PHASE2_TRAINING_PIPELINE_4B.md
| |-- recodec/
| `-- analysis/
|-- stage3_llm_sft/ # Downstream alignment (LLM trained, codec frozen)
| |-- train.py # HF Trainer + DeepSpeed ZeRO-2 entry
| |-- modeling_recodec_qwen35.py # dynamic GOP + two-pass budget enforcement
| |-- dataset.py # tensor-cache / decord loading, random fps
| |-- recodec_budget.py # token-budget math (bisection frame planning)
| |-- demo_recodec_inference.py # minimal inference example
| |-- configs/ deepspeed/ tools/
| `-- start_predlen_tsfix150_training.sh
`-- stage4_benchmark_eval/ # Five-benchmark evaluation
|-- evaluate_*.py # per-benchmark entry points
|-- run_predlen_delta08_3150_5bench.sh # serial 8-shard launcher (resumable)
|-- watch_delta08_3150_5bench.py # live leaderboard with subset breakdown
|-- configs/ # per-benchmark evaluation configs
`-- EVAL_GUIDE.md
The code is identical to the source tree. weights/ is added by the Hub upload; the Stage 1 and Stage 2 checks are released there so that the pipeline can be used without retraining.
Released checkpoints
| File | Size (bytes) | Content |
|---|---|---|
weights/inc_enc_stage1_step18750.pt |
7,907,749,717 | Stage 1 incremental encoder, merge norm and reconstruction decoders (ckpt["model"]: inc_enc, merge_norm, decoders, aux_predictors) |
weights/pred_len_delta00_step9000.pt |
1,132,687,741 | Stage 2 length predictor trained with teacher tolerance delta = 0.0 (strict, more tokens) |
weights/pred_len_delta08_step3150.pt |
1,132,687,677 | Stage 2 length predictor trained with teacher tolerance delta = 0.8 (aggressive, about 45% fewer tokens) |
Both length predictors share the same architecture (7-class suffix-ratio head over the last spatial token of each P frame) and are interchangeable; delta only affects how aggressive the predicted compression is. The codec checkpoints are loaded by the codec front-end in stage3_llm_sft/modeling_recodec_qwen35.py and by the evaluation entry points in stage4_benchmark_eval/.
Installation
# Create the environment (Python 3.11)
conda create -n nativacodec python=3.11 -y
conda activate nativacodec
# Install dependencies (PyTorch >= 2.4 with CUDA required)
pip install -r requirements.txt
# FlashAttention for the Qwen3.5 backbone
pip install flash-attn --no-build-isolation
Data preparation
Training and evaluation rows are ShareGPT-style JSONL (id, video_url, conversations), one QA per line:
{
"id": "videoseek_268014_1",
"video_url": "/path/to/videos/ytb_50l5HAWTjwE.mp4",
"meta_info": {"source": "videoseek", "is_mcq": true},
"conversations": [
{"role": "user", "content": "<video>\nWhat is the final text shown in the video?\nOptions:\nA. ... D. ..."},
{"role": "assistant", "content": "</think>\n\nTherefore, the final answer is: D"}
]
}
BallBench is available in the Tennineee/NativeCodec-BallBench dataset repository; its card documents the qa.jsonl fields, the archive layout and the generation scripts.
Optionally pre-decode videos into a uint8 tensor cache (fps 16, long side <= 512, 32-aligned) for fast training I/O:
cd stage3_llm_sft
python tools/convert_video_tensor_cache.py --input /path/to/videos --output /path/to/video_tensor
Quick start
Paths inside the configs (model directories, checkpoints, data roots) must be adapted to your machine first; every stage validates them at startup.
1. Use the released checkpoints
Download this repository and one of the companion LLM repositories (see Access request). The codec front-end lives in stage3_llm_sft/modeling_recodec_qwen35.py; it takes the aligned LLM directory plus the two codec checkpoints:
from modeling_recodec_qwen35 import ReCodecQwen35ForConditionalGeneration
model = ReCodecQwen35ForConditionalGeneration(
# Released HF checkpoint: a complete Qwen3.5-4B model with the adapter merged
model_name_or_path="NativeCodec-4B-PredLen",
recodec_root="recodec",
phase1_ckpt="NativeCodec/weights/inc_enc_stage1_step18750.pt",
phase2_ckpt="NativeCodec/weights/pred_len_delta08_step3150.pt",
video_pad_token_id=video_pad_id,
max_gop_frames=64,
max_frames=12288,
max_video_tokens=64000,
fixed_p_ratio=None, # None = learned length predictor (PredLen variant)
lora_cfg=None, # released checkpoints have the adapter merged; do not wrap them again
)
The full inference path (frame sampling, codec encoding, prompt assembly, generation) is implemented in stage3_llm_sft/demo_recodec_inference.py. To run the demo on the released checkpoint, set QWEN_DIR to the downloaded NativeCodec-4B-PredLen directory, point PHASE1_CKPT and PHASE2_CKPT at the files under NativeCodec/weights/, pass lora_cfg=None in build_model, and skip the load_safetensors / load_state_dict block there (those lines load a Stage 3 trainer checkpoint, while the released files already contain the merged weights):
cd stage3_llm_sft
python demo_recodec_inference.py --video /path/to/video.mp4 --question "What happens in the video?"
For the fixed-ratio variant, load Tennineee/NativeCodec-4B-FixedRatio instead and pass fixed_p_ratio=1/16; the length predictor is then not used.
2. Training stages (optional)
The released checkpoints make inference possible without training. The original training pipeline is kept complete and runs on 8 GPUs.
STAGE 1: truncatable representations (needs the Qwen3.5-4B base):
cd stage1_recodec_pretrain
bash run.sh # 8-GPU DDP; writes the Phase 1 checkpoint (inc_enc, decoders, aux)
STAGE 2: oracle-guided length training (needs the Phase 1 checkpoint):
cd stage2_len_pred
bash run.sh # default: delta = 0.8, random fps {16,8,4,2}
STAGE 3: downstream LLM alignment (needs Phase 1 + Phase 2 checkpoints):
cd stage3_llm_sft
bash start_predlen_tsfix150_training.sh # pred-len dynamic mode, 8-GPU DeepSpeed ZeRO-2
# Export an HF-format checkpoint for inference / evaluation
python tools/convert_full_ckpt_to_hf.py --ckpt <output>/checkpoint-150 --out <hf_dir>
3. Evaluation
cd stage4_benchmark_eval
bash run_predlen_delta08_3150_5bench.sh # full chain: 01 to 05, 8 shards, 1 shard/GPU
bash run_predlen_delta08_3150_5bench.sh 02 # resume from stage 02
# Live progress and subset-level metrics while it runs
python watch_delta08_3150_5bench.py
The eval configs under stage4_benchmark_eval/configs/ reference paths from the training cluster. Adapt recodec.phase1_ckpt, recodec.phase2_ckpt, checkpoint_path, data_path and video_dir to your environment. For BallBench, use the archive from Tennineee/NativeCodec-BallBench; the evaluation entry point is evaluate_ballbench_predlen_hf.py.
Configuration
Key parameters (see each config_*.yaml for the full list):
| Parameter | Description | Default |
|---|---|---|
video.max_video_tokens |
Hard visual-token budget; oversize streams are re-planned with uniformly subsampled frames until they fit | 64000 |
recodec.phase2_ckpt |
Trained length predictor (Stage 2), defines the per-frame length policy | not set |
video.fixed_p_ratio |
Constant suffix ratio mode (null = use the learned length predictor) |
null |
recodec.target_delta_db |
Stage-2 teacher quality tolerance in dB; larger values mean more aggressive compression | 0.8 |
video.video_fps |
Codec input frame rate | 16 |
video.max_gop_frames |
Max frames per GOP (anchor + suffixes) | 64 |
video.max_side |
Frame long side after resize (32-aligned) | 512 |
video.random_fps_choices |
Training-time fps augmentation pool | [2,4,8,16] |
License
This repository contains two kinds of material with different licenses.
The code in stage1_recodec_pretrain/, stage2_len_pred/, stage3_llm_sft/, stage4_benchmark_eval/ and the accompanying scripts is released under the MIT License; see LICENSE.
The released weights in weights/, and the aligned models in the two companion repositories, are distributed under the Apache License, Version 2.0, following the license of Qwen/Qwen3.5-4B-Base. The full text is in LICENSE-APACHE, and NOTICE.md records the upstream attribution for the Qwen3.5-4B-Base model and the changes made here (codec front-end added, language model fine-tuned). Redistribution must keep both license texts and the notice, and must state any further modifications.
Data materials (BallBench) carry their own terms; see the dataset card of Tennineee/NativeCodec-BallBench.
Acknowledgements
- Qwen for the Qwen3.5 base multimodal LLM and its visual tower.
- TimeLens-Bench, MomentSeeker, Video-Holmes and Video-MME, the public video understanding benchmarks used in the evaluation.
- The open-source video-LLM community's HF-native inference tooling.
NativeCodec Team