SurgRec — Surgical Video Foundation Models

Pretrained backbones from Scaling Video Pretraining for Surgical Foundation Models.

Each dataset is compared against whichever third-party baseline is strongest on it — DINOv3, DINOv3-SurgeNetXL, VideoMAE, or V-JEPA (colour-coded; VideoMAE is never the strongest). Numbers are Table 2 of the paper. cataract-1k is the one dataset where a baseline leads, and cat-21 is a tie where every method scores 15.69.

hf download SichengLu/SurgRec --local-dir checkpoints

These are headless backbones — no classification head. Fine-tune them on a downstream surgical video dataset with the code repository.

Sicheng Lu, Zikai Xiao, Jianhui Wei, Danyu Sun, Qi Lu, Keli Hu, Yang Feng, Jian Wu, Zongxin Yang, Zuozhu Liu

Highlights

• Pretrained on a multi-source surgical corpus of 10,535 videos / 214.5M frames spanning endoscopy, laparoscopy, cataract, and robotic surgery.
• Evaluated on a standardized 16-dataset downstream benchmark across four clinical domains with consistent splits.
• SurgRec-MAE is strictly the best of the seven compared methods on 11 of 16 datasets, and ties for best on 3 more. On the mean it leads the strongest third-party baseline (JEPA) by 11.3 points.

Results

Top-1 accuracy (%) on the 16-dataset downstream benchmark, from Table 2 of the paper. Baselines are DINOv3, DINOv3-Surg (SurgeNetXL), V-MAE (VideoMAE), SR-MAE without balanced sampling, and JEPA; bold marks the best method on each dataset.

Dataset	DINOv3	DINOv3-Surg	V-MAE	SR-MAE (w/o bal.)	JEPA	SR-MAE	SR-JEPA
AIxSuture	32.61	39.13	39.13	39.13	39.13	43.48	39.13
AutoLaparo	17.74	19.35	19.35	17.74	20.97	22.58	17.74
cat-21	15.69	15.69	15.69	15.69	15.69	15.69	15.69
cataract-101	17.97	19.53	19.53	21.09	24.22	35.16	23.44
cataract-1k	9.90	14.85	11.88	16.83	19.80	18.81	17.82
Cholec80	22.34	16.12	14.65	24.18	26.01	29.30	31.14
Colonoscopic	53.33	53.33	53.33	53.33	53.33	73.33	53.33
Hyper-Kvasir	43.03	41.81	25.11	36.91	41.65	55.28	42.27
JIGSAWS	18.12	17.11	15.67	21.23	29.72	47.49	32.24
Kvasir-Capsule	46.34	47.56	19.51	46.34	48.78	73.17	60.98
LapGyn4	62.68	65.63	62.68	67.07	60.66	68.13	64.13
LDPolypVideo	90.35	90.35	90.35	90.35	90.35	90.76	90.42
M2CAI16	23.01	25.66	12.39	30.09	26.55	38.05	38.05
MultiBypass140	38.62	42.63	20.54	51.56	12.88	56.92	56.47
SurgicalActions160	6.25	6.25	6.25	18.75	12.50	18.75	6.25
SAR-RARP50	23.01	26.88	25.97	24.37	22.10	37.59	36.67
Mean	32.56	33.87	28.25	35.92	34.02	45.28	39.11

SR-MAE (w/o bal.) is an ablation of our own model, not a third-party method. Every method scores exactly 15.69 on cat-21, consistent with all of them collapsing to a single class, so that row carries no signal. JEPA leads on cataract-1k and SurgRec-JEPA on Cholec80.

Models

File	Model	Architecture	Params	Pretraining objective	Load with
surgrec_mae.pth	SurgRec-MAE	ViT-B/16	86.5 M encoder	Masked video reconstruction (VideoMAE-style)	model_key=model
surgrec_jepa.pt	SurgRec-JEPA	ViT-L/16	303.9 M encoder	Latent predictive (V-JEPA-style)	model_key=encoder

Checkpoint internals, training configuration, and checksums

surgrec_mae.pth — configuration recorded inside the checkpoint:

• model = pretrain_videomae_base_patch16_224
• mask_type = tube, mask_ratio = 0.9, decoder_depth = 4
• input_size = 224, num_frames = 16, sampling_rate = 4
• lr = 3e-4, batch_size = 64, taken at epoch = 149

The file stores 94.2 M parameters in total including the MAE decoder; the encoder used for fine-tuning is 86.5 M. The archive holds model, optimizer, scaler, epoch, and args. The model entry contains both the encoder and the MAE decoder; fine-tuning loads the encoder and discards decoder.*.

surgrec_jepa.pt — holds encoder, predictor, and target_encoder, each ViT-L/16; taken at epoch = 131, loss = 0.4225, trained with batch_size = 24 across world_size = 8. Load the encoder entry; its weights sit under a module.backbone. prefix, which the loader in the code repository strips.

SHA256, for verifying downloads:

ce2c0298292d636f95c77a12c26fcdeb93de8b4a78e515a68d247de5e61759c5  surgrec_mae.pth
84040aa2e2fc015d03165903e6eec06282507f1d275e76efa97292457cd1bd3e  surgrec_jepa.pt

Usage

git clone https://github.com/Siiichenggg/SurgRec.git
cd SurgRec
pip install -r requirements.txt
hf download SichengLu/SurgRec --local-dir checkpoints

Fine-tune on a downstream dataset:

DATASET_ROOT=data/splits_local CKPT_ROOT=checkpoints GPUS=8 \
  bash scripts/train/finetune_surgrec_video.sh cholec80 --backbone surgrec_mae

Or reproduce a paper run with its original hyperparameters:

CKPT_ROOT=checkpoints DATA_ROOT=/path/to/split GPUS=8 \
  bash scripts/paper/finetune/jepa_v3_finetune_8gpu.sh

License and intended use

Released for non-commercial research. These weights are a research artifact, not a medical device, and must not be used for clinical diagnosis or treatment decisions.

CC BY-NC 4.0 — attribution required, no commercial use. The code derives in part from VideoMAE, which carries the same obligation; see LICENSE and NOTICE in the code repository.

Citation

@article{lu2026surgrec,
  title   = {Scaling Video Pretraining for Surgical Foundation Models},
  author  = {Lu, Sicheng and Xiao, Zikai and Wei, Jianhui and Sun, Danyu and
             Lu, Qi and Hu, Keli and Feng, Yang and Wu, Jian and Yang, Zongxin
             and Liu, Zuozhu},
  journal = {arXiv preprint arXiv:2603.29966},
  year    = {2026},
  url     = {https://arxiv.org/abs/2603.29966}
}