Clarification on Reproducing SurgMotion Attentive Probing on Cholec80
The paper says that 16 probing heads are trained simultaneously, but the released Cholec80 YAML contains 20
multihead_kwargs
entries. Did the Table 2 result use 16 or 20 probing heads? Which exact hyperparameter combinations were used?Is
num_heads: 16the number of cross-attention heads inside each probe, while everymultihead_kwargsentry represents one
independent probing model?What is the exact Cholec80 preprocessing protocol: train/test split, sampling FPS, causal or centered 64-frame clips, padding
method, label timestamp, resolution, and data augmentation? Are 86,304 training samples and 98,194 test samples correct?Did the reported experiment use one epoch, batch size 4, AdamW with cosine schedules, and balanced class-weighted cross-entropy?
Was it run in FP16 or BF16? Was there any additional gradient normalization or gradient accumulation?How was the best probe selected? Was it selected by macro F1 on a separate validation set, or directly on the 40-video test set?
How exactly were Accuracy, F1, and Jaccard calculated: globally over all frames or per video and then averaged? Are F1 and Jaccard
unweighted macro averages over the seven phases? Was any temporal smoothing applied?Were the probing heads initialized sequentially using one global seed? If the probes are trained separately to reduce GPU memory,
how should we preserve the original probe initialization and DataLoader random sequence?Could you provide the exact checkpoint, YAML, repository commit, preprocessing command, and training command used for the Cholec80
result in Table 2?
Using the released configuration, our current best result is 84.17 Accuracy, 75.90 macro F1, and 65.26 macro IoU, compared with 91.05,
84.17, and 77.95 in the paper. We would like to confirm which protocol difference may explain this gap.
Thanks for looking into this.
One thing we ran into during reproduction that's worth sharing: for ViT-G, make sure the config uses model_name: vit_giant_xformers β setting it as vit_giant (without _xformers) causes a mismatch between model architecture and checkpoint parameter names, https://github.com/CAIR-HKISI/SurgMotion/blob/main/configs/foundation_model_probing/surgmotion/Cholec80/surgmotion_vitg_64f_cholec80.yaml#L176 , which leads to a noticeable performance drop.
Hope this helps.
Feel free to reach out at jinlin.wu@cair-cas.org.hk, if you have any other questions.