SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Figure 11: Performance on DAVIS with respect to the number of real videos used during training. Performance before and after test-time training for models trained using only synthetic videos (BL-30K) and up to 3531 real videos (DAVIS and YouTube-VOS together).
Figure 12: Additional qualitative examples using a model trained on synthetic videos from BL-30K [8] and tested on a real video from DAVIS [38]. Second-to-bottom row: Results obtained using the STCN [9] approach. Bottom row: Results after test-time training using the proposed mask cycle consistency loss (tt-MCC) on the single ground-truth mask provided for the first video frame.
Figure 13: Additional qualitative examples using a model trained on DAVIS [38] and YouTube-VOS [51] and tested on a corrupted video from the DAVIS-C benchmark. Left: the test example is corrupted using the glass-blur corruption with medium strength. Right: the test video is stylised using style 1 with medium strength. Second-to-bottom row: Results obtained using the STCN [9] approach. Bottom row: Results after test-time training using the proposed mask cycle consistency loss (tt-MCC) on the single ground-truth mask provided for the first video frame.
B.5 Additional qualitative results
We show more qualitative examples in Figures 12 and 13.
B.6 Results in the case of no distribution shift
In the main part of the manuscript, we focus on scenarios involving test-time distribution shifts. There, we report results based on 100 iterations, which we've found to be optimal for all cases with such distribution shifts. When a model is trained on real videos and there is no distribution shift in the test videos, as demonstrated in the left part of Table 3, using fewer iterations is advantageous, particularly for larger datasets like YouTube-VOS and MOSE. Overall, we observe that  $tt$ -MCC does
Table 3: Additional results for STCN and Xmem. The number of TTT iterations is reported next to tt-MCC. Left part: Results when starting from a model trained with real videos. Right part: Results when starting from a model trained only on static images.
Method	Training with real videos	Training with static images
DAVIS	DAVIS-C	YT-VOS	MOSE	DAVIS	DAVIS-C	YT-VOS	MOSE
STCN	85.3	69.3	84.3	52.5	75.7	59.3	76.3	42.0
STCN + tt-MCC-20 (ours)	86.0	72.1	84.6	53.3	78.7	68.8	78.2	43.6
STCN + tt-MCC-100 (ours)	86.7	73.8	84.0	51.8	79.7	70.7	79.4	43.4
XMem	87.7	75.7	86.1	60.4	72.8	56.2	77.0	42.3
XMem + tt-MCC-20 (ours)	88.0	76.8	85.8	60.7	75.9	63.0	76.5	41.7
XMem + tt-MCC-100 (ours)	88.1	77.6	85.1	59.8	78.8	67.2	78.4	42.9
Figure 14: Performance gain of tt-MCC for the sim-to-real case on the MOSE-train dataset. We plot the performance gain vs. a) the video length in number of frames, b) the object area in the first frame normalized by the frame area, c) the object area in the first frame normalized by the maximum object area over all frames, d) the percentage of the video length where the object is visible.
not negatively impact the performance of state-of-the-art methods in these cases. Instead, it offers a method to either maintain or enhance state-of-the-art performance across all test-time video scenarios, whether extreme distribution shifts are present or not.
B.7 Results in the case of training with static images
In this section, we focus on the scenario where the model is trained using only static images. Following STCN, we employ five datasets of static images [47, 41, 58, 7, 25] for the offline training of the networks. Given a training image, several deformed versions are generated to compose an artificial video sequence, which is used for the model training. The corresponding results are presented in the right part of Table 3 for both STCN and XMem across the four datasets. Notably, for STCN, tt-MCC yields relative performance gains exceeding  $5\%$ ,  $4\%$ , and  $3\%$  on DAVIS, YouTube-VOS, and MOSE, respectively. These gains represent substantial recovery of the performance achieved by a model trained with real videos, i.e. amounting to  $42\%$ ,  $39\%$ , and  $13\%$ , respectively. Performance improvement is even more significant on the DAVIS-C dataset, with a  $19\%$  relative performance gain, surpassing the STCN model trained on real videos (70.7 vs. 69.3).
B.8 Performance analysis on a per-video and per-object basis
We conduct a comprehensive analysis using the MOSE-train dataset on a per-video and per-object basis and present results in Figure 14. Figure 14 (a) examines the performance gain using TTT in relation to video length, revealing that video length does not significantly affect performance. Figure 14 (b) demonstrates a subtle correlation between performance gain and the size of objects in the first frame, with larger objects showing positive gains and no negative impact. In Figure 14 (c), we explore how the object's size in the initial frame relative to its maximum size in the video affects performance. We observe a slight negative impact when the object is less visible in the first frame compared to subsequent frames. This is attributed to  $tt$ -MCC overfitting to the partially annotated first frame and often segmenting the partial object in future frames. Note that extreme cases of severe occlusion in the first frame can make the video object segmentation task particularly ambiguous. Finally, Figure 14 (d) assesses the impact of object visibility duration within the video sequence,
Figure 15: Change in  $\mathcal{J}\&\mathcal{F}$  score during TTT compared to the score before TTT. Each curve is one video and is normalized wrt. the maximum improvement/decline observed over 500 iterations.
confirming that  $tt$ -MCC is adversely affected when the object is briefly visible. This suggests that our cycle consistency loss, emphasizing long-term temporal consistency, may introduce bias towards objects being visible for extended periods.
B.9 Impact of longer test-time training
To better study the effect of longer training at test-time, we run our method for up to 500 iterations on DAVIS for the sim-to-real case and presents results in Figure 15. In summary, we do not observe any significant further decrease in performance nor any degenerate outputs after longer training. More specifically, the number of DAVIS videos for which VOS performance decreases remained the same from 50 TTT iterations up to approximately 400, i.e. the  $\mathcal{J}\&\mathcal{F}$  score decreases in 7 out of 30 videos. Only one extra video shows a decrease in  $\mathcal{J}\&\mathcal{F}$  score by 500 iterations. Visually observing segmentation results, we notice that a common issue is some background pixels of similar appearance to the object are wrongly segmented. It is also worth noting that increasing the iterations only leads to a minor overall decrease in VOS performance, i.e.  $\mathcal{J}\&\mathcal{F}$  score drops from 81.1 to 81.0/80.4 for 200/500 iterations, respectively.
B.10 Assessing temporal stability
In Figure 16, we present a performance comparison on a per-video basis, both with and without tt-MCC. The graph illustrates consistent and lasting performance improvements. Importantly, it highlights the enhanced stability of performance when tt-MCC is employed. In the case of a briefly visible object (as seen in video 3bf a8379), our use of tt-MCC results in incorrectly segmenting out the background as the object, significantly impacting the metric in a negative manner.
B.11 Detailed results on DAVIS-C
In Figures 18 and 19, we present the  $\mathcal{J}\&\mathcal{F}$  score change before and after test-time training for the STCN-BL30K and STCN-DY models, respectively, for all three corruption strengths.
B.12 Failure cases
We notice a couple of patterns when inspecting failure cases of our method, and TTT in general. The most important is that TTT might make the model more confident about a dubious prediction and propagate it wrongly. Moreover, we observe that TTT might overfit the appearance module and wrongly add background pixels. E.g. we observe that a red shirt from the background in the breakdancer video confuses the model more than the STCN case. We refer the reader to the video we provide on the project webpage, where we visualize a number of such failure cases.
(a) DAVIS
Figure 16: Performance per frame of STCN and our tt-MCC for the sim-to-real case on selected videos from DAVIS and MOSE-train. We plot the performance of each method on each frame of the test video averaged over all objects. The colors of the highlighted areas indicate the winning method. The average performance for the whole video is reported in the legend per method.
(b) MOSE-train
The proposed method is effective for the two types of distribution shifts evaluated in our work. We expect it to be valuable for knowledge transfer to very different domains with limited or no annotated data. For instance, in the bio-medical domain, for a task such as tracking cells or organisms captured under a microscope. Due to the lack of appropriate datasets, we do not evaluate the proposed approach in such a use case.
Test-time training methods like ours can also be used to improve the semi-automatic annotation process for large video collections in the presence of distribution shifts. Such data can then be used to train and fine-tune other models in other domains and tasks. It is worth noting that whatever the annotation, societal, or otherwise biases existing in the dataset the base model was trained on will also be present in the automatically annotated frames and propagated through this process. Test-time training on a single annotated example is not expected to successfully correct such biases; understanding the limitations of the base model is therefore crucial for any real-world application of our method.
D Figures for all TTT losses
We illustrate the  $tt$ -Ent and  $tt$ -AE losses, as well as the regular cross-entropy loss for the supervised offline training stage in Figures 20a, 20b, and 20c, respectively.
Figure 17: Impact of the strength of the corruption for the 14 types of corruption in DAVIS-C
(a) Low strength.
(b) Medium strength
(c) High strength
Figure 18: Test-time training on DAVIS-C for the STCN-BL30K model. We plot the  $\mathcal{I}\&\mathcal{F}$  score separately per video before (triangles) and after  $tt$ -MCC (circles) for the STCN-BL30K model on the 14 corruptions of the proposed DAVIS-C benchmark. We report results for the variants with the highest corruption strength. A red vertical line denotes that performance drops with test-time training.
(b) Medium strength.
(c) High strength.
Figure 19: Test-time training on DAVIS-C for the STCN-DY model. We plot the  $\mathcal{I} \& \mathcal{F}$  score separately per video before (triangles) and after tt-MCC (circles) for the STCN-DY model on the 14 corruptions of the proposed DAVIS-C benchmark. We report results for the variants with the highest corruption strength. A red vertical line denotes that performance drops with test-time training.
time-forward / step 1
time-forward / step 2
time-forward / step 2
Figure 20: The  $tt$ -Ent,  $tt$ -AE, and the cross-entropy loss. The function  $f$  represents the overall prediction model. It takes as input the current test frame and a memory  $M$  of predicted masks from the previous frames and outputs the predicted mask for the current frame.