SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Training details. We use learning rates  $10^{-5}$  and  $10^{-6}$  for models STCN-BL30K/ XMem -BL30K and STCN-DY/ XMem -DY, respectively since their training data differ significantly. Jump step  $s$  for sampling training frames is set to 10. For each test example, we train the models with  $tt$ -MCC and  $tt$ -Ent for 100 iterations and with  $tt$ -AE for 20, using the Adam [23] optimizer and a batch size of 4 sequences for STCN and 1 for XMem. We consider all target objects in the ground truth mask during TTT, and the raw videos are used with no augmentations. We develop our methods on top of the publicly available  $\mathrm{STCN}^2$  and  $\mathrm{XMem}^3$  implementations.
4.1 Datasets and evaluation
We report results on the two most commonly used benchmarks for video object segmentation, the DAVIS-2017 validation set [38] and the YouTubeVOS-2018 validation set [51]. We further report results on the validation set of the recent MOSE [10] dataset, which includes challenging examples with heavy occlusion and crowded real-world scenes. Finally, we introduce DAVIS-C, a variant of the DAVIS test set that represents distribution shifts via corruptions and stylization. We evaluate all methods using the widely established  $\mathcal{J}\&\mathcal{F}$  measure similar to prior work [9, 6].
DAVIS-C: corrupted and stylized DAVIS. To model distribution shifts during the testing, we process the set of test videos of the original DAVIS dataset to create DAVIS-C. This newly created test set offers a test bed for measuring robustness to corrupted input and generalization to previously unseen appearance. It is the outcome of processing the original videos to introduce image-level and video-level corruption and image-level appearance modification. We perform 14 different transformations to each video, each one applied at three different strengths, namely low, medium, and high strength, making it a total of 42 different variants of DAVIS.
$^{2}$ https://github.com/hkchengrex/STCN
3https://github.com/hkchengrex/XMem
(a) Sim-to-real transfer
(b) Corrupted test examples
Figure 5: VOS performance under distribution shifts. Left: performance of STCN-BL30K before and after test-time training for the sim-to-real transfer case on four datasets. Right: performance of STCN-DY on DAVIS-C for input corruptions with different strength levels.
For the image-level corruptions, we follow the paradigm of ImageNet-C and adopt 7 of their corruptions, namely gaussian noise, brightness, contrast, saturate, pixelate, defocus blur, and glass blur. The different transformation strengths are obtained by using severity values 3, 4, and 5 from ImageNet-C. For the video-level corruptions, we introduce constant rate factor (CRF) compression for three increasing values of compression and motion blur by averaging every 2, 3, and 4 consecutive frames. For the image-level transformations, we create cartoonization using the mean shift algorithm by increasing the amount of smoothing and image-stylization [13] using four different styles. For stylization, we do not vary the strength, but we preserve the original video colors (low and medium strength) while switching to those of the style image (high strength), making it a more drastic appearance shift. Frame examples for all cases are shown in Figure 4.
Some of the transformations in DAVIS-C do not perfectly represent video distortions. Nevertheless, several of the transformations constitute common real-world video edits (contrast, brightness, crf compression, cartoonization, stylization). We believe DAVIS-C provides a valuable benchmark to study video understanding under distribution shift.
Sim-to-real transfer. In Figure 5a, we report results with and without test-time training on four datasets for the case of sim-to-real transfer. All three TTT variants bring gains consistently across datasets, with  $tt$ -MCC improving performance significantly, i.e. a relative gain of more than  $39\%$ ,  $15\%$  and  $15\%$  for YouTube-VOS, DAVIS and MOSE, respectively. Moreover, we see that naively applying task-agnostic entropy minimization test-time training brings only a small percentage of the gains one can get. It is also worth noting that the gains from TTT are so high that performance using the mask cycle consistency variant becomes comparable to the performance of a model trained on video data. Specifically,  $tt$ -MCC recovers  $82\%$ ,  $72\%$  and  $44\%$  of the performance gains that training with all the training videos from YouTube-VOS and DAVIS combined brings. This is very promising in cases where annotated data are scarce but synthetic data is widely available.
Corrupted test examples. We report results for DAVIS-C in Figures 5b, 6, and the right side of Table 1. Figure 5b clearly shows how tt-MCC outperforms STCN for all corruption levels, with the gains increasing more as corruption strength increases. The entropy minimization approach does not offer any gains, while tt-AE is effective but worse than tt-MCC. In Figure 6, we report the  $\mathcal{J}\&\mathcal{F}$  score separately per video of DAVIS-C at the highest strength. We see that the gains come from many transformations and examples, while we see that TTT is able to improve most videos where the original  $\mathcal{J}\&\mathcal{F}$  score was in the lower side of the spectrum. Finally, on the right side of Table 1, we further report results on DAVIS-C for XMem and see that gains persist for one more state-of-the-art method. Interestingly, from Table 1, we also see that the performance gain of XMem over STCN disappears for the sim-to-real transfer case for YouTube-VOS, as well as for the extreme case a model trained only on synthetic videos is tested on corrupted inputs.
Figure 6: Test-time training on DAVIS-C. We plot the  $\mathcal{J}\&\mathcal{F}$  score separately per video before (triangles) and after tt-MCC (circles) for the STCN-DY model on the 14 corruptions of the proposed DAVIS-C benchmark. We report results for the variants with the highest corruption strength. A red (grey) vertical line denotes that performance drops (increases) with test-time training.
Table 1: Comparison to the state-of-the-art for two cases of test-time distribution shift. Left part: Results when using models trained without real videos. The HODOR model is trained on COCO [27] images, while STCN and XMem on BL-30K. Right part: Results on DAVIS-C for different levels of corruption. HODOR, STCN and XMem are fine-tuned with DAVIS and YouTube-VOS videos. † Results from [1].
Method	Training without real videos	Corrupted test examples (DAVIS-C)
DAVIS	DAVIS-C	YouTube-VOS	MOSE	no corr.	low	med	high	avg
HODOR [1]	77.5†	-	71.7†	-	81.3†	69.0	64.5	55.2	65.0
STCN [9]	70.4	41.7	57.3	38.9	85.3	76.6	72.6	58.8	73.3
STCN + tt-MCC (ours)	81.1	70.1	79.4	44.9	86.7	78.3	75.6	67.3	77.0
XMem [6]	78.1	53.9	65.6	40.9	87.7	80.4	77.3	69.4	78.7
XMem + tt-MCC (ours)	82.1	70.1	78.9	44.7	88.1	81.7	78.9	72.2	80.2
Comparison to the state of the art. In Table 1, we additionally report results for our  $tt$ -MCC variant when starting from the state-of-the-art XMem [6] method. Once again, the gains are consistent, for both cases. It is worth noting that from the fourth column of the table, we can see that  $tt$ -MCC is also able to slightly improve the case when there is no test-time distribution shift, i.e. we report 1.4 and 0.4 higher  $\mathcal{I} \& \mathcal{F}$  score on DAVIS for STCN-DY and XMem -DY, respectively. In the table, we also compare our method to HODOR [1] method that uses cycle consistency during the offline training stage and reports results when training without real videos. Our STCN-BL30K model with TTT outperforms HODOR by +3.6 and +8.3  $\mathcal{I} \& \mathcal{F}$  score, on DAVIS and YouTube-VOS respectively.
Impact of sampling longer sequences. Regarding the sequence length used during test-time training, we adopt the setup used by the base methods during their training phase, i.e. frame triplets for STCN and octuplets for XMem. We experiment with quadruplets or quintuplets for mask cycle consistency on top of STCN which achieve performance of 81.7 and 82.4, respectively, on DAVIS versus 81.1 for triplets. This gain in performance, however, comes with a significant increase in test-time training time, i.e.  $33\%$  and  $66\%$ , respectively.
Impact of changing the sampling strategy. Additionally to changing the sequence length, another way to affect the size of the temporal context is by the value of the jump step, i.e. the interval used to sample frames  $x_{i}$  and  $x_{j}$  for a training triplet. Test-time training with tt-MCC over the STCN-BL30K model achieves 81.1 on DAVIS, with the jump step  $s$  parameter set to 10 by default. Varying the jump step from 1, 2, 25, and 50, achieves 80.6, 81.0, 80.8, and 79.0, respectively. Neither sampling too close to the first frame nor sampling frames too far from each other is the optimal choice. Switching to random sampling, without any step as a restriction, causes a performance drop to 79.7, while sampling only within the last  $5\%$  or  $10\%$  of the video also causes a drop to 79.8 and 79.7, respectively.
Across all strategies and hyperparameter values, we see noticeable improvements using  $tt$ -MCC; the base model without TTT, merely achieves 70.4  $\mathcal{J} \& \mathcal{F}$  score on DAVIS.
Impact of training iterations. For all results we use a fixed number of 100 iterations during TTT. In Figure 7, we present the performance improvement of  $tt$ -MCC for all 30 DAVIS videos (one curve per video) for a varying number of iterations. One can clearly see that optimal performance is achieved at different iterations per video, and we believe that adaptive early stopping is a promising future direction that may boost the overall improvement even further.
Impact on inference time. We discuss the effect that test-time training has on inference speed, by measuring average inference time over all DAVIS validation videos. The vanilla STCN model requires roughly 2 seconds per video. For tt-MCC, tt-Ent, and tt-AE, it takes 67, 29, and 5.3 seconds, respectively, for 100 iterations. Since TTT is run per video and its speed is only
affected by the number of iterations, per frame timings vary with video length, i.e. the TTT overhead reduces as video length increases. It is worth noting here that all timings presented above are estimated with a non-optimized TTT implementation that leaves a lot of space for further optimization.
Figure 7: Change in  $\mathcal{I}\&\mathcal{F}$  score during TTT compared to the score before TTT. Each curve is one video and is normalized wrt. the maximum improvement/decline observed over 200 iterations.
Reducing TTT overhead. As discussed, TTT induces a significant overhead at test time, which is an aspect not often discussed in the relevant literature. The TTT overhead is reduced by training for less iterations with only a minor decrease in performance. As we show Table 3 of the Appendix, using 20 instead of 100 iterations is even advantageous in cases, particularly for larger datasets like YouTube-VOS and MOSE with a model trained on real videos and without a distribution shift. In a real-world application, applying  $tt$ -MCC on all test examples might seem infeasible due to time overheads. Nevertheless, it is very useful for cases of extreme test-time distribution shifts. In scenarios involving a few out-of-distribution examples, where rapid adaptation of the current model is desired for improved performance,  $tt$ -MCC constitutes a straightforward and cost-effective solution, that is far more efficient than retraining the base model, which typically requires approximately 12.5 hours on 2 A100 GPUs to train STCN.
Memory requirements. During inference, STCN requires roughly 8GB of GPU RAM. Test-time training on top of STCN with tt-AE, tt-Ent and tt-MCC requires 8,16 and 23.5GB, respectively, i.e. it can still fit in a modest GPU. Note that the calculations for TTT are computed when using a batch size of 4 (the default). Memory requirements can be further reduced by using a smaller batch size, if needed, without significant change in performance.
In this work we show that test-time training is a very effective way of boosting the performance of state-of-the-art matching-based video object segmentation methods in the case of distribution shifts. We propose a mask cycle consistency loss that is tailored to matching-based VOS and achieves top performance. Its applicability goes beyond the two methods used in this work, as it is compatible with the general family of matching-based VOS methods [36, 60, 56, 1]. We report very strong gains over top performing methods for both sim-to-real transfer and the case of corrupted test examples. We also show that achieving such gains is not trivial and that it is important to tailor the test-time training method to the task and method at hand. A limitation of the proposed approach is the lack of a way for performing early stopping, and selecting the best iteration to stop training, a very promising direction for future work.
This work was supported by Naver Labs Europe, by Junior Star GACR GM 21-28830M, and by student grant SGS23/173/OHK3/3T/13. The authors would like to sincerely thank Gabriela Csurka for providing valuable feedback.
[1] A. Athar, J. Luiten, A. Hermans, D. Ramanan, and B. Leibe. Hodor: High-level object descriptors for object re-segmentation in video learned from static images. In CVPR, 2022.
[2] F. Azimi, S. Palacio, F. Raue, J. Hees, L. Bertinetto, and A. Dengel. Self-supervised test-time adaptation on video data. In WACV, 2022.
[3] H. S. Behl, M. Naja, A. Arnab, and P. H. Torr. Meta-learning deep visual words for fast video object segmentation. In IROS, 2020.
[4] S. Caelles, K.-K. Maninis, J. Pont-Tuset, L. Leal-Taixe, D. Cremers, and L. Van Gool. One-shot video object segmentation. In CVPR, 2017.
[5] Y. Chen, J. Pont-Tuset, A. Montes, and L. Van Gool. Blazingly fast video object segmentation with pixel-wise metric learning. In CVPR, 2018.
[6] H. K. Cheng and A. G. Schwing. Xmem: Long-term video object segmentation with an atkinson-shiffrin memory model. In ECCV, 2022.
[7] H. K. Cheng, J. Chung, Y.-W. Tai, and C.-K. Tang. Cascadepsp: Toward class-agnostic and very high-resolution segmentation via global and local refinement. In CVPR, 2020.
[8] H. K. Cheng, Y.-W. Tai, and C.-K. Tang. Modular interactive video object segmentation: Interaction-to-mask, propagation and difference-aware fusion. In CVPR, 2021.
[9] H. K. Cheng, Y.-W. Tai, and C.-K. Tang. Rethinking space-time networks with improved memory coverage for efficient video object segmentation. NeurIPS, 2021.
[10] H. Ding, C. Liu, S. He, X. Jiang, P. H. Torr, and S. Bai. Mose: A new dataset for video object segmentation in complex scenes. arXiv, 2023.
[11] B. Duke, A. Ahmed, C. Wolf, P. Aarabi, and G. W. Taylor. Sstvos: Sparse spatiotemporal transformers for video object segmentation. In CVPR, 2021.
[12] Y. Gandelsman, Y. Sun, X. Chen, and A. Efros. Test-time training with masked autoencoders. NeurIPS, 2022.
[13] G. Ghiasi, H. Lee, M. Kudlur, V. Dumoulin, and J. Shlens. Exploring the structure of a real-time, arbitrary neural artistic stylization network. In BMVC, 2018.
[14] D. Hendrycks and T. Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. In ICLR, 2019.
[15] P. Hu, G. Wang, X. Kong, J. Kuen, and Y.-P. Tan. Motion-guided cascaded refinement network for video object segmentation. In CVPR, 2018.
[16] Y.-T. Hu, J.-B. Huang, and A. Schwing. Masknn: Instance level video object segmentation. NeurIPS, 2017.
[17] Y.-T. Hu, J.-B. Huang, and A. G. Schwing. Unsupervised video object segmentation using motion saliency-guided spatio-temporal propagation. In ECCV, 2018.
[18] Y.-T. Hu, J.-B. Huang, and A. G. Schwing. Videomatch: Matching based video object segmentation. In ECCV, 2018.
[19] A. Jabri, A. Owens, and A. Efros. Space-time correspondence as a contrastive random walk. NeurIPS, 2020.
[20] V. Jampani, R. Gadde, and P. V. Gehler. Video propagation networks. In CVPR, 2017.
[21] A. Khoreva, R. Benenson, E. Ilg, T. Brox, and B. Schiele. Lucid data dreaming for video object segmentation. *IJCV*, 2019.
[48] Y. Wang, J. Choi, Y. Chen, S. Li, Q. Huang, K. Zhang, M.-S. Lee, and C.-C. J. Kuo. Unsupervised video object segmentation with distractor-aware online adaptation. Journal of Visual Communication and Image Representation, 74:102953, 2021.
[49] H. Xiao, B. Kang, Y. Liu, M. Zhang, and J. Feng. Online meta adaptation for fast video object segmentation. PAMI, 42(5):1205-1217, 2019.
[50] Z. Xiao, X. Zhen, S. Liao, and C. G. Snoek. Energy-based test sample adaptation for domain generalization. *ICLR*, 2023.
[51] N. Xu, L. Yang, Y. Fan, D. Yue, Y. Liang, J. Yang, and T. Huang. Youtube-vos: A large-scale video object segmentation benchmark. In ECCV, 2018.
[52] L. Yang, Y. Wang, X. Xiong, J. Yang, and A. Katsaggelos. Efficient video object segmentation via network modulation. In CVPR, 2018.
[53] S. Yang, X. Wang, Y. Li, Y. Fang, J. Fang, W. Liu, X. Zhao, and Y. Shan. Temporally efficient vision transformer for video instance segmentation. In CVPR, 2022.
[54] Z. Yang and Y. Yang. Decoupling features in hierarchical propagation for video object segmentation. NeurIPS, 2022.
[55] Z. Yang, Y. Wei, and Y. Yang. Collaborative video object segmentation by foreground-background integration. In ECCV, 2020.
[56] Z. Yang, Y. Wei, and Y. Yang. Associating objects with transformers for video object segmentation. NeurIPS, 2021.
[57] C. Yi, S. Yang, Y. Wang, H. Li, Y.-P. Tan, and A. C. Kot. Temporal coherent test-time optimization for robust video classification. In ICLR, 2023.
[58] Y. Zeng, P. Zhang, J. Zhang, Z. Lin, and H. Lu. Towards high-resolution salient object detection. In ICCV, 2019.
[59] M. Zhang, S. Levine, and C. Finn. Memo: Test time robustness via adaptation and augmentation. NeurIPS, 2022.
[60] Y. Zhang, Z. Wu, H. Peng, and S. Lin. A transductive approach for video object segmentation. In CVPR, 2020.
[61] Y. Zhang, L. Li, W. Wang, R. Xie, L. Song, and W. Zhang. Boosting video object segmentation via space-time correspondence learning. In CVPR, 2023.
A Datasets and additional implementation details 14
A.1 Datasets 14
A.2 Additional implementation details 15
B Additional experimental results and analysis 15
B.1 Measuring the standard deviation of TTT 15
B.2 Extended results on the MOSE dataset 15
B.3 Results with TTT on top of XMem 15
B.4 Varying the number of real video used for training the base model 16
B.5 Additional qualitative results 17
B.6 Results in the case of no distribution shift 17
B.7 Results in the case of training with static images 18
B.8 Performance analysis on a per-video and per-object basis 18
B.9 Impact of longer test-time training 19
B.10 Assessing temporal stability 19
B.11 Detailed results on DAVIS-C 19
B.12 Failure cases 19
C Broader impact 20
D Figures for all TTT losses 20
A Datasets and additional implementation details
DAVIS The validation split of the DAVIS-2017 [38] dataset contains 30 videos covering a variety of real-world scenarios, including indoor and outdoor scenes, different lighting conditions, occlusions, and complex motion patterns. Each video contains one to three annotated objects of interest.
YouTubeVOS-2018 The validation split of the YouTubeVOS-2018 [51] dataset contains 474 high-quality videos downloaded from YouTube, including indoor and outdoor scenes, different lighting conditions, occlusions, and complex motion patterns. Each video contains one to five annotated objects of interest.
style 3
Figure 8: The four styles in DAVIS-C.
Table 2: Multiple runs of the proposed method on top of STCN and XMem for two cases of test-time distribution shift. Mean and standard deviation over 3 different seeds is reported. Left part: Results for STCN-BL30K and XMem-BL30K. Right part: Results for STCN-DY and XMem-DY on DAVIS-C for different levels of corruption.
Method	Training without real videos	Corrupted test examples (DAVIS-C)
DAVIS	DAVIS-C	YT-VOS	MOSE	no corr.	low	med	high	avg
STCN [9]	70.4	41.7	57.3	38.9	85.3	76.6	72.6	58.8	73.3
STCN + tt-MCC (ours)	81.1±0.1	70.1 ±0.1	79.4 ±0.2	44.9 ±0.2	86.7 ±0.2	78.3 ±0.1	75.6 ±0.1	67.3 ±0.1	77.0
XMem [6]	78.1	53.9	65.6	40.9	87.7	80.4	77.3	69.4	78.7
XMem + tt-MCC (ours)	82.1 ±0.2	70.1 ±0.3	78.9 ±0.2	44.7 ±0.2	88.1 ±0.2	81.7±0.1	78.9 ±0.2	72.2±0.1	80.2
DAVIS-C We refer the reader to the main paper for details. Here, we highlight the 14 different transformations to each video applied at three different strengths, namely low, medium, and high strength in Figure 17. The four original images that we use to compute the four stylizations style 1, style 2, style 3, and style 4 are displayed in Figure 8.
MOSE We further report results on a newer, much larger dataset, i.e. the MOSE dataset [10], which contains challenging examples with heavy occlusions and crowded real-world scenes. The validation split of the MOSE dataset contains 311 high-quality videos. Each video contains one to fifteen annotated objects of interest. In addition, to show the robustness of our method, we also ran our models on the training split of the MOSE dataset, which contains 1507 high-quality videos with up to twenty annotated objects of interest. Because the STCN and XMem models are not trained on the MOSE-train split before, we consider this split as a larger dataset for testing our method. Because of limited space, we present results on MOSE-train only in the supplementary, and as we show in Section B, the observations of the main paper also hold in this case.
A.2 Additional implementation details
The model is in evaluation mode during test-time training, i.e. the running statistics of the Batch Normalization layers remain fixed.
Because the last frame  $m_j$  of the frame sequence used in our tt-MCC method is the initial frame of the backward pass, all the target objects in  $m_0$  must also be depicted in  $m_j$ . To guarantee this, we select only the frames where the network can detect all the target objects, i.e. there is at least one pixel for each object where it has the largest probability. We do so by extracting pseudo masks for the video frames and resetting the optimizer every ten iterations.
In the MOSE dataset, some videos have up to twenty annotated objects. Because our method is linear with the number of annotated objects in the video, we randomly select up to six objects at each iteration.
B Additional experimental results and analysis
B.1 Measuring the standard deviation of TTT
We report mean and standard deviation after running TTT with 3 different seeds in Table 2 for four datasets, namely DAVIS, DAVIS-C, YouTube-VOS, and MOSE.
B.2 Extended results on the MOSE dataset
In Figure 9, we report STCN results with and without test-time training on the two splits of the MOSE dataset for the case of sim-to-real transfer. We can see that on both splits of the MOSE dataset, the three TTT variants boost the performance, and  $tt$ -MCC brings a relative gain of  $15\%$  on  $18\%$  on the MOSE-valid and MOSE-train datasets, respectively.
B.3 Results with TTT on top of XMem
In Figure 10, we show the performance comparison with and without TTT on top of XMem. We observe that the performance benefits are similar to those of using TTT on top of STCN.
Figure 9: STCN performance on the MOSE dataset for the sim-to-real transfer case. We report results on both the validation set, as well as the much larger training set of 1507 videos; used as another test set since it is not used as a training set by the models used in this work.
Figure 10: XMem performance under distribution shifts. Left: performance of XMem -BL30K before and after test-time training for the sim-to-real transfer case on four datasets. Right: performance of XMem -DY on DAVIS-C for input corruptions with different strength levels.
B.4 Varying the number of real video used for training the base model
We increase the number of real videos used to train the STCN model from 0 (BL-30K) to 3531 (DAVIS and YouTube-VOS training sets) and present results in Figure 11. When the model is trained with little to no real videos, test-time training recovers the bulk of the performance achieved using models trained on larger annotated datasets while requiring little to no annotation.