Test-time Training for Matching-based Video Object Segmentation
Authors: Juliette Bertrand
Giorgos Kordopatis-Zilos
Yannis Kalantidis
Giorgos Tolias
Venue: NeurIPS 2023 poster (2023)
Keywords: VOS
video object segmentation
test-time training
test-time adaptation

Abstract: The video object segmentation (VOS) task involves the segmentation of an object over time based on a single initial mask. Current state-of-the-art approaches use a memory of previously processed frames and rely on matching to estimate segmentation masks of subsequent frames. Lacking any adaptation mechanism, such methods are prone to test-time distribution shifts. This work focuses on matching-based VOS under distribution shifts such as video corruptions, stylization, and sim-to-real transfer. We explore test-time training strategies that are agnostic to the specific task as well as strategies that are designed specifically for VOS. This includes a variant based on mask cycle consistency tailored to matching-based VOS methods. The experimental results on common benchmarks demonstrate that the proposed test-time training yields significant improvements in performance. In particular for the sim-to-real scenario and despite using only a single test video, our approach manages to recover a substantial portion of the performance gain achieved through training on real videos. Additionally, we introduce DAVIS-C, an augmented version of the popular DAVIS test set, featuring extreme distribution shifts like image-/video-level corruptions and stylizations. Our results illustrate that test-time training enhances performance even in these challenging cases.

Test-time Training for Matching-based Video Object Segmentation
Juliette Bertrand
Giorgos Kordopatis-Zilon
Yannis Kalantidis²
Giorgos Tolias
VRR, FEE, Czech Technical University in Prague
NAVER LABS Europe
Sim-to-real transfer
Corrupted test examples
Frames and ground- trrth masks
Frames and ground-trun masks
STCN
STCN
STCN 1-MMC (ours)
STCN H-MCC (ours)
Figure 1: Test-time training significantly improves performance in the presence of distribution shifts for the task of video object segmentation (VOS). Left: Example using model trained on synthetic videos from BL-30K [8] and tested on real video from DAVIS [38]. Right: Example using a model trained DAVIS [38] and YouTuub-- VOS [51] and tested
cartoonized video from the corrupted DAVIS-C benchmark which we introduce in this work. Second-to to-ttotoboooommo row: Results obtained using the STCN [9] approach. Bottom row: Results after test-time training with the proposed mask cycle consistency loss (t--MCC) using the single ground-truth mask provided for the first video frame.
Abstract
The video object segmentation (VOS) task involves the segmentation of an object over time based on a single initial mask. Current caat--of-the-a approaches use a memory of previously processed frames and rely on matching to estimate segmentation masks of subsequent framess Lacking any adaptation mechanism, such methods are prone to test-time distribution shifts. This work focuses on matching-basee VOS under distribution shifts such as video corruptions, stylization, and sim-to-real transfer. We explore test-time training strategies that are agnostic to the specific task as well as strategies that are designed specifically for VOS. This includes variant based mask cycle consistency tailored to matching-basse based VOS methods. The experimental results on common benchmarks demonstrate that the proposed test-time training yields significant improvements in performance. In particular for the sim-to- to-real scenario and despite using only a single test video, approach manages to recover substantial portion of the performance gain achieved through training on real videos. Additionally, we introduce DAVIS-C an augmented version of the popular DAVIS test set, featuring extreme distribution shifts like image-Ivideo-leva corruptions and stylizations. Our results illustrate that test-time training enhances performance even in these challenging cases. Project page: https: jerjj gi gitb...... io/test- time-time--eeee wwwe--ttiiiiiei teeee-trriiinnnvineeee voa/
37th Conference Neural Information Processing Systems (NeurIPS 2023),
Equal contribution

---

Introduction
In one-shot video object segmentation (VOS), we are provided with a single segmentation mask for one or more objects in the first frame of video, which we need to segment across all video frames. is a dense prediction task over time and space, and therefore, collecting training data is highly demanding. Early VOS methods design foreground/backgroondd segmentation models that operate on single-fram framee input and require a two-stage training process [4, 29I. During the off-line stage, the model is trained to segment a variety of foreground objects. During the on-line stage for specific test video, the model is ane-tuned to adapt to the objects of interest indicated in the provided mask.
Recently, matching- nutching-based methods like STCN [9] or XMem [6] have shown impressive performance on the common VOS benchmarks. These methods propagate the segmentation mask from memory of previously predicted masks to the current frame. They only involve an off-line training stage where the model learns how to perform matching and propagation. Lacking any test-time adaptation mechanism, however, such methods are highly prone to test-time distribution shifts.
Our goal is to improve the generalization of matching natching-based VOS methods under distribution shifts by fine-tuning the model at test time through a single test video. Such one-shot adaptation is form of test-time training (TTT), a research direction that lately attracting much attention classification tasks and a promising way for generalizing to previously unseen distributions. We argue that TTT is a great fit for matching-based based VOS, not only because of the additional temporal dimension to exploit, but also because we are given a ground-truth label for a frame of the test example.
In this work, we focus on two cases of test-time distribution shift for VOS: a) when training on synthetic data and testing on real data, also known as sim-to-real transfer, and b) when the test data contain image/video corruption stylization. We explore three different ways of performing TTT: i) a task approach that borrows from the image classification literature and performs entropy minimization [34, 46], ii) an approach tailored the STCN architecture that employs an auto-encoder loss on the mask provided for the first frame and does not exploit any temporal information, and iii) a mask cycle consistency approach that is tailored matching- based VOS, and utilizes temporal cycle consistency of segmentation masks to update the model. As we show, these different test-time training strategies do not work equally well, and tailoring TTT to the task and the methods important and not trivial.
Our experiments demonstrate that some of the proposed approaches constitute a powerful way of generalizing to unseen distributions. When starting from a model trained only on synthetic videos, test-time training based on mask cycle consistency improves STCN by +22 and +11 points in terms of J&FF score on the two most popular VOS benchmarks, YouTube- VOS and DAVIS, respectively. What is even more impressive is that by simply using test-time training, we recover the bulk of the performance gain that training the original model on real videos brings: for You VOS and DAVIS, TTT recovers 82% and 72% of the performance gains, respectively, that training on real videos brings compared to synthetic ones, but without requiring any video annotations during off-line training.
We also study the performance of TTT in the presence of image-level video-level corruptions and stylization. To that end, we follow a process similar to ImageNet-C [14] and create DAVIS-C, version of the DAVIS test set with 14 corruptions and style changes at three strength levels. We evaluate models with and without TTT on DAVIS-C and analyze how performance changes as the distribution shift increases. Our results show that TTT significantly improves the performance of STCN by more than 8 points in terms of J&F score for the hardest case of corruption/tyllizttinn A qualitative example for both types of distribution shift is shown in Figure 1.
2 Related Work
Foreground/background models with online fine-tuning for VOS. Early approaches train a seg- mentation model to separate the foreground object from the background. The offline training stage is followed by an online supervised fine-tuning stage that uses the mask of the first frame for adapting the model [4, 29]. The fine-tuning process is improved by including pseudo-masks masks derived from highly confident predictions [44] or by additionally including hard-negative examples [48]. Maninis et al. [33] additionally incorporate semantic information about the underlined object categories via an exter- nal instance-aware semantic segmentation model. Other methods first produce a coarse segmentation result and then refine it by the guidance of optical flow [15], the appearance cue only [39], or temporal
2

---

consistency [32]. Hu et al. [16] utilize an RNN model to exploit the long-term temporal structures, while others additionally incorporate optical flow information [16]. The heavy cost of fine-tuning is reduced by meta-learning in the work of Xiao et al. [49] or Meinhardt and Leal-TTaxxé [35].
Most recent approaches build upon matching-basee prediction model, which is trained offline and neither requires nor incorporates any fine-tuning stage. The current test frame is matched to either the first annotated frame [52, 18, 5, 45, 3], propagated mask [20, 17, 28), or a memory containing both [31, 36, 9]. Early methods use simple matching process [18] or are inspired by classical label propagation [60], while different methods improve the design of the matching [1, 9, 40, 53, 55, 54], the type and extent of the memory used [6], or exploit interactions between objects or frames [37, 56], among other aspects.
Matching- based VOS.
Hybrid and other VOS methods. Mao et al. [34] borrow from both families of approaches, ie. matching-based and those that require online fine-tuning, and jointly integrate two models, but , unlike our work, the matching model is not updated at test time. Some methods differ from matching-based in the way they capture the evolution over time, e.g. by RNNs [43] or by spatio-temporal object-level attention [11].
Cycle consistency in VOS. Matching-baad based methods require dense video annotations for train- ing, which is limitation. To dispense with the need for mask annotations during training, cycle consistency of pixel or region correspondences over space and time has been successfully used by prior work [19, 24, 61]. This is an unsupervised loss that is also appropriate for test-time training. In our case, we tailor cycle consistency to the task of matching- basdd VOS, and propose a supervised consistency loss that operates on masks by taking advantage of the provided mask at test-time in order to better adapt to the object of interest. Mask consistency based on the first frame is used by prior work during model training. Khoreva al. [21] synthesize future frames using appearance and motion priors, while in our case, instead of synthesizing, we use mask predictions of the existing future video frames. Li et al. [26] and HODÓR [1] use a temporal cycle consistency loss on a the first frame mask during training that allows learning from videos with a single segmentation mask. In contrast, we utilize the mask frame that is provided at test-time for the first frame of the test video and we are able to outperform HODOR and other state-of-the-art methods for multiple test-time distribution shifts.
Test-time training. A family of approaches adapts the test example to the training domain. Kim et al. [22] chose the appropriate image augmentation that maximizes the loss according to loss prediction function, while Xiao et al. [50] updates the features of the test sample by energy minimization with stochastic gradient Langevin dynamics. Another family of approaches adapts the model to the test do- main. Entropy minimization is a common way to update either batch-normalizztionn statistics only [46] or the whole model [59]. Other self-supervised objectives include rotation prediction [42], contrastive learning [30], or spatial auto-encoding [12]. In our work, we move beyond the image domain and introduce mask cycle consistency as an objective to adapt the model specifically for video object segmentation applications. Azimi et al. [2] evaluate some of the aforementioned TTT techniques on top of self- supervieddd dense tracking methods under several distribution shifts on videos. Another recent work [57]. concurrent to ours, also studies TTT in the video domain but for a classification task; we instead fully tailor the TTT on the VOS task and use temporal mask consistency as our loss.
3 Test-time training for matching-based VOS
In this section, we first formulate the task of Video Object Segmentation (VOS) and briefly describe
the basic components of the matching- VOS methods we build on [9, 6]. We then present three ways for test-time training: a task-agnostic baseline based on entropy minimization that has been highly successful in other tasks and two VOS-specific variants that leverage the fact that we are provided with a grouhd-truth mask at test time; one using an auto-encoder loss and another a temporal cycle-consistency loss on the masks.
3

---

time-forward step
time fore-forward / step
M
mig
M
Zig
mi
f/t
m
M
3zj
2;
M
Zo
tos
lime-backward step
time-backward step
Figure 2: The mask cycle consistency loss for a given frame triplet {zo, 24, .ry} and mask mo- Frames Tp. Ty and 2, are shown with red, green and blue border, respectively. Function f represents the overall prediction model. It takes as input the current test frame and memory M of predicted masks from the previous frames and outputs the predicted mask for the current frame. For the first frame, the memory contains the ground-truth mask.
3.1 Preliminaries
The goal of VOS is to segment set of particular objects in a video provided their ground-truth seg- mentation mask my in the first video frame xo Space-Time Correspondence Networks (STCN) [9] sequentially predict the mask each frame separately, while the prediction conditioned on the pro- vided mask and the predicted masks of earlier frames that stored in memory queue. XMem [6] is an extension of STCN that features a long-term memory and top performance for long-term prediction. Although provide here the necessary parts of STCN required for presenting our contribution, we refer the reader to Cheng et al. [9] for complete description. Our formulation is generic and applicable to wide range of matching-based VOS methods, including the ones that require only single forward pass for multiple objects such as AOT [56].
We denote the model that performs the overall process by function f. The predicted mask of frame Tj is given by m, = fxz M), where M = { m,)} is the memory of previous predictions consisting of frame-mask tuples (I,, The memory always includes (Iu, my), while any other tuple for > is formed by using the corresponding predicted mask m, since no other ground-truth mask is available. Note that fround-truth masks for the first frame binary, while the predicted ones consist of object probabilities in [0, 1).
The STCN model contains frame encoder ez that extracts frame representation u; = Gx (x(x) R xD and mask encoder em that extracts mask representation 6; = cm (m,) E R x D...
The latter can take either ground-truth (binary) or predicted (real-value) masks as input. Frame and mask representations of earlier frames are added to the memory.
During the prediction for the current frame denoted by Zy, representation uj is matched (each -e-dimensional vector separately) the representations of frames in the memory, and the most similar items are identified. In particular, the full similarity matrix is formed between the current frame items and the memory items, but only the top-k values per row (rows correspond to the current frame) are maintained, and the rest are set to 0. By the term item we refer to each D..dimmensional vector, with each frame having W most similar items is aggregated to obtain ij xx In this way we compose the mask representation for frame zj via matching since the mask itself not available. This process is
of them. The corresponding mask representation of these
IIn the following, we assume the presence of a single object for simplicity. In the case of multiple objects. the prediction process repeated for each object independently, similar to [6, 9].

---

equivalent to a cross- aooss-tttntionn operation between the test frame features (queries) and the memory frame features (keys) that aggregates memory mask features (values). Finally, a mask decoder dmm used obtain the predicted mask my = dm (lū,, uz]), where [0,, uy] IR (WWxXxDDeDD. the concatenation of the two representations along the depth dimension.
In this work, we assume that have access to matching-based based VOS model, trained real or synthetic data with mask annotations. These models do not generalize well to extreme distribution shifts not encountered during supervised training. Our goal is to improve their performance in such cases by fine-tuning the parameters of the model at test time.
3.2 Test-time training
We explore three different losses to perform TTT for the case of matching-based VOS: (i) based on entropy minimization (II-Ent), which forms a paradigm transferred from the image domain (image classification in particular) and is a task- and method-agnostic loss, (ii) based on an auto-encoder (U-AE) that operates on segmentation masks, which is an approach tailored for the STCN architecture, (iii) based on mask cycle consistency (tt-MCC), which more general and appropriate for a variety of matching-base methods. tt-MC exploits the matching- -hased nature of STCN and includes mask propagation via a memory of masks, while the tt-AE variant does not include the matching step between multiple frames.
Given test video, we optimize the parameters of the model using the provided mask and either only the first frame (for tt-AE) or all its frames (tt-Ent, tt-MCC). The test-time loss is optimized, the model parameters are updated, and the model is used provide predictions for the specific test video. Then, the model is re-initialized back to its original parameters before another test video is processed.
For the two TTT variants that multiple frames, i.e. tt-MCC and tt-Ent, follow the process used during the off-line training phase of STCN and XMem and operate on randomly sampled frame triplcts and octuplets, respectively. In the following, we describe the case of triplets which is extended to larger sequences in a straightforward way. All the triplets include the first video frame, while the second (third) triplet frame is uniformly sampled among the frames that follow the first (second) triplet frame in the video. For triplet Zo, Ij. Ty with 0 < i < J. the first predicted mask is given by m. (a;;; {z mo))), and the second by m f(zy (20, mo). (x,, m.))). i.e. the second frame is added to the memory before predicting the last mask of the triplet. Multiple triplets are sampled during test-time training. The per-pixel losses averaged form the frame/mask loss. We denote the cross entropy loss between a ground-truth mask m and a predicted mask mn by fce(m, (m, mi).
3.2.1 Entropy (ft-Ent) loss
We start from task- and method-agnostic loss based on Entropy (tt-Ent) minimization, common choice for test-time training on image classification [46]. In this case, we force the model to provide confident predictions for the triplets we randomly sample. The entropy of mask rn is denoted by H (fiz and the loss for a particular triplet is LH = 0.5 * (m,)) + H (m,,) which is minimized over multiple triplets. Prior work image classification avoids the trivial solution by using a batch of test examples [46] or augmenting the input example and minimizing the entropy over the average prediction of the augmented frames [59]. In the task of semantic segmentation, batches augmenta- tions are not required since it is a dense prediction task and we have multiple outputs to optimize, i.e. prediction per pixel. In the case of VOS, we have even more outputs due the temporal dimension and because we are sampling triplets among many frames. However, similar to Wang et al. [46], we observe that training only the batch normalization parameters is a requirement for tn-Ent work.
3.2.2 Mask auto-encoder (U-AE) loss
The STCN model includes a mask encoder and mask decoder to decode the mask representation coming from the cross-attention process. We repurpose these components for test-time training and compose an auto-encoder that reconstructs the input mask. The auto auto-en input is the provided ground-truth mask, i.e. mg, and the loss is given by LAF = (e ( dm (m.(ee..()))) eg (xr))) Note that, together with the mask encoder that gets optimized to better represent the specific object shape, we also optimize the frame encoder with the fI-AE loss, since the mask representation is concatenated with the frame representation in the input of the mask encoder [9].
5

---

50
33 (a) frame triplet
3j
Ma4
(b) provided mask
m; (step 1) (c) Mask predictions before test-time training
my (step 2)
m (step 3)
my (step
( m))
(d) loss before TTT
Pluce//on
|8Laee/emi
|
|Lxcee/0mg
(e) Gradients of mask predictions before test-time training
mi (step 1)
my (step 2)
my (step 3)
mi (step
( mb) (g) loss after TTT
(f) Mask predictions after test-time training
Figure 3: Unravelling the I-MCC loss. (a) a triplet from video with gaussian noise corruption; (b) the provided mask; (c) mask predictions after each of the four steps of the cycle, before test-time training (TTT); (d) the pixel- wise loss before TTT; (e) gradients of mask predictions for each of the steps of the cycle; (f) predictions after test-time training with the mask cycle consistency loss; (f) the loss after TTT. We see that tt-MCC significantly improves all predicted masks and reduces the loss. We binarize the predicted masks for visualization purposes; they are otherwise probability maps.
3.2.3 Mask eycle consistency (#+-MCC) loss
After sampling triplet, we make predictions in forward and bine-backward manner to form a cycle that allows us to use the provided ground-truth mask as a label with a cross-entropy loss. The cycle consists of four steps. We first obtain masks mh and m, with time-forward predictions. Then, we consider the triplet reverse order going from xy to x; and then to &o, and obtain time-backward predictions. In particular, my is treated the only mask in the memory enable prediction for frame I, which is given by m fx; (x),,,)))) m)) In the last step, the mask of 2n is given by m f(o, {aa, mn (I4; m)))). We compare this prediction the provided ground-truth mask and the loss to optimize is LMCC lc (mp. mo). The overall process and the four prediction steps shown Figure 2. During the optimization, pack-propagation is performed through all the predicted masks mentioned above. In this way, earlier predictions are also optimized in indirect way.
For successful object segmentation răg masks ri,, rijy, and vi in the whole sequence of predic- tions, are required to have successful object segmentation too. If for example an object part is not properly segmented in the intermediate predictions, it will be missed in mi too. We further experi- mented with adding extra mask consistency loss between the forward and backward predictions for the middle//ntermmdiite frame, i.e. Eo (m, m2.) but did not result any additional gains.
Figure 3 shows the four predictions during cycle using particular triplet before and after tt-MCC We see that initially, the object is not properly segmented during internal steps of the cycle and is fully missed in the last step, of the cycle resulting in large loss values for most of its pixels. After TTT is performed, not only the mask which is given as input to the loss, i.e. rig gets improved, but so are the predictions during intermediate steps. Note that the prediction of step 4 is not part of the inference process, but is rather proxy to optimize predictions for steps and that are part of The gradients of the internal masks are indicating how the loss is affecting their predictions per pixel.

---

original
brightness
contrast
saturate
pixclate
gaussian noise
defocus blur
glass blur
erf compression
motion blur
cartoon
style
style 2
style 3
style 4
Figure 4: The 14 types of corruption in DAVIS-C. Corruptions are depicted at "medium" strength.
Experiments
In this section, first describe the training and implementation details. Then, introduce the DAVIS-C benchmark and report our experimental results on four datasets and two distribution shifts: sim-to-real transfer and corrupted/stylized test data.
Initial models. We start from two publicly available models for STCN. STCN-BL30K is the model trained on synthetic data from BL-30K [8]. STCN-DY is model initialized by STCN-BL30K and further trained using the training videos of DAVIS and YouTube- which include approximately 100K frames annotated with segmentation masks. We the same models for XMem, denoted by XMem -LL3K and XMem -DY, respectively.
Training details. We use learning rates 10-3 and 10 for models STCN-BL30K/ XMem -BL30K and STCN-DY/ XMem -DY, respectively since their training data differ significantly. Jump step 10. For each test example,
for sampling training frames is and tt-Ent for 100 iterations and with tt At-AE for 20, using the Adam [23] optimizer and a batch size of 4 sequences for STCN and for XMem. We consider all target objects in the ground truth mask during TTT, and the raw videos are used with no augmentations. We develop our methods on top of the publicly available STCN2 and XMem implementations.
train the models with (I-MCC
4.1 Datasets and evaluation
We report results on the two most commonly used benchmarks for video object segmentation, the DAVIS-2017 validation set [38] and the MouTubeVVS-222 validation set [51]. We further report results on the validation set of the recent MOSE [10] dataset, which includes challenging examples with heavy occlusion and crowded real-world scenes. Finally, we introduce DAVIS-C. variant of the DAVIS test set that represents distribution shifts via corruptions and stylization. We evaluate all methods using the widely established J & measure similar prior work [9, 6].
DAVIS-C: corrupted and stylized DAVIS. To model distribution shifts during the testing, we process the set of test videos of the original DAVIS dataset to create DAVIS-C. This newly created test set offers a test bed for measuring robustness to corrupted input and generalization to previously unseen appearance. It is the outcome of processing the original videos to introduce image-level and video-level corruption and image-level appearance modification. We perform 14 different transformations to each video, each one applied three different strengths, namely low, medium, and high strength, making it total of 42 different variants of DAVIS.
hnttpe://github.com httpp////ggthbbbccm///cchhggggg/ hhttps://gi httpa//..ooooccccegeeeee//ee/eee

---

STCN
U--Ent U-AR
M-MCC
80
+10.7
+22.1
90
J&.F score
J&&F score
80
60
+6.0
40
38,9
60
DAVIS DAVIS-C
YouTube
MOSE
no corruption low med high
(a) Sim-to-real transfer
(b) Corrupted test examples
Figure 5: VOS performance under distribution shifts. Left: performance of STCN-BL30K before and after test-time training for the sim-to-real transfer case on four datasets. Right: performance of STCN-DY on DAVIS-C for input corruptions with different strength levels.
For the image-level corruptions, we follow the paradigm of ImageNet-C and adopt of their cor- ruptions, namely gaussian noise, brightness, contrast, saturate, pixelate, defocus blur, and glass blur. The different transformation strengths are obtained by using severity values 3, 4, and 5 from ImageNet-C. For the video-level corruptions, we introduce constant rate factor (CRF) compression for three increasing values of compression and motion blur by averaging every 2, 3, and 4 consec- utive frames. For the image-level transformations, we create cartoonization using the mean shift algorithm by increasing the amount of smoothing and image-stylization [13] using four different styles. For stylization, we do not vary the strength, but we preserve the original video colors (low and medium strength) while switching to those of the style image (high strength), making it more drastic appearance shift. Frame examples for all cases are shown in Figure 4.
Some of the transformations in DAVIS-C do not perfectly represent video distortions. Nevertheless, several of the transformations constitute common real-world video edits (contrast, brightness, erf compression, cartoonization, stylization). We believe DAVIS-C provides a valuable benchmark to study video understanding under distribution shift.
4.2 Results
Sim-to-real transfer. In Figure 5a, we report results with and without test-time training on four datasets for the case of sim-to-real transfer. All three TTT variants bring gains consistently across datasets, with tt-MCC improving performance significantly, i.e. relative gain of more than 39%, 15% and 15% for YouTube- VOS, DAVIS and MOSE, respectively. Moreover, we see that naively applying task-agnostic entropy minimization test-time training brings only a small percentage of the gains one can get. is also worth noting that gains from TTT are so high that performance using the mask cycle consistency variant becomes comparable to the performance of model trained on video data. Specifically, I/MMC recovers 82%, 72% and 44% of the performance gains that training with all the training videos from YouTube VOS and DAVIS combined brings. This is very promising in cases where annotated data are scarce but synthetic data widely available.
Corrupted test examples. We report results for DAVIS-C in Figures 5b, 6, and the right side of Table Figure 5b clearly shows how ttMCCC outperforms STCN for all corruption levels, with the gains increasing more corruption strength increases. The entropy minimization approach does not offer any gains, while tt-AE effective but worse than ftMCC. In Figure 6, we report the J &FF score separately per video of DAVIS-C at the highest strength. We see that the gains come from many transformations and examples, while we see that TTT is able to improve most videos where the original J & score was in the lower side of the spectrum. Finally, on the right side of Table 1, we further report results on DAVIS-C for XMem and see that gains persist for one more state-of-the-an method. Interestingly, from Table 1, we also see that the performance gain of XMMm over STCN disappears for the sim-to-real transfer case for YouTube- VOS, as well as for the extreme case a model trained only synthetic videos is tested corrupted inputs.

---

100
80
J&. score
60
40
20
brightness
contrast
saturate
pixelate
gaussian noise defocus blur
glass blur
crf compression motion blur
cartoon
style1
style2
style3
style4
Figure 6: Test-time training on DAVIS-C. We plot the J&F score separately per video before (triangles) and after tt-MCC (circles) for the STCN-DY model on the 14 corruptions of the proposed DAVIS-C benchmark. We report results for the variants with the highest corruption strength. A red (grey) vertical line denotes that performance drops (increases) with test-time training.
Table 1: Comparison to the state-of-the-art for two cases of test-time distribution shift. Left part: Results when using models trained without real videos. The HODOR model is trained on COCO [27] images, while STCN and XMem on BL-30K. Right part: Results on DAVIS-C for different levels of corruption. HODOR, STCN and XMem fine-tuned with DAVIS and YouTube- VOS videos. Results from [1].
Training without real videos DAVIS-C
Corrupted test examples (DAVIS-C) no corr.
Method
DAVIS
YouTube-
MOSE
low
med
high
avg
HODOR [1]
77.51
71.71
81.3"
69.0
64.5
55.2
65.0
STCN [9]
70.4
41.7
57.3
38.9
85.3
76.6
72.6
58.8
73.3
STCN + #-MCC (ours)
81.1
70.1
79.4
44.9
86.7
78.3
75.6
67.3
77.0
XMem [6]
78.1
53.9
65.6
40.9
87.7
80.4
77.3
69.4
78.7
XMem + M-MCC (ours)
82.1
70.1
78.9
44.7
88.1
81.7
78.9
72.2
80.2
Comparison to the state of the art. In Table 1, we additionally report results for H-MCC variant when starting from the state-of-the-ar XMem [6] method. Once again, the gains are consistent, for both cases. is worth noting that from the fourth column of the table, we can see that ttMCC is also able to slightly improve the case when there is no test-time distribution shift, i.e. we report 1.4 and 0.4 higher J&.F score DAVIS for STCN-DY and XMem -DY, respectively. In the table, we also compare our method to HODOR [1] method that uses cycle consistency during the offline training stage and reports results when training without real videos. Our STCN-BL30S model with TTT outperforms HODOR by +3.6 and +8.3 J &FF score, on DAVIS and YouTube-VOR respectively.
Impact of sampling longer sequences. Regarding the sequence length used during test-time training, we adopt the setup used by the base methods during their training phase, i.e. frame triplets for STCN and octuplets for XMem. We experiment with quadruplets or quintuplets for mask cycle consistency on top of STCN which achieve performance of 81.7 and 82.4, respectively, on DAVIS versus 81.1 for triplets. This gain in performance, however, comes with significant increase in test-time training time, i.e. 33% and 66%, respectively.
Impact of changing the sampling strategy. Additionally to changing the sequence length, another way to affect the size of the temporal context is by the value of the jump step, i.e. the interval used to sample frames z; and 2y for training triplet. Test-time training with tt-MCC over the STCN-BL30K model achieves 81.1 on DAVIS, with the jump step s parameter set to 10 by default. Varying the jump step from 1, 2, 25, and 50, achieves 80.6, 81.0, 80.8, and 79.0, respectively. Neither sampling too close to the first frame nor sampling frames too far from each other is the optimal choice. Switching to random sampling, without any step as restriction, causes performance drop to 79.7, while sampling only within the last 5% or 10% of the vidco also causes drop to 79.8 and 79.7, respectively.

---

Across all strategies and hyperparameter values, we see noticeable improvements using #tMCCC the base model without TTT, merely achieves 70.4 J &&F score on DAVIS.
Impact of training iterations. For all results fixed number of 100 iterations during
100
norm. Jk.F improvement
TTT. In Figure 7, we present the performance improvement of tt-MCC for all 30 DAVIS videos (one curve per video) for a varying num- ber of iterations. One can clearly see that op-
50
timal performance is achieved at different iter- ations per video, and we believe that adaptive early stopping is promising future direction that may boost the overall improvement even further.
-50
100
Impact inference time. We discuss the effect that test-time training has on inference speed, by measuring average inference time all DAVIS validation vidcos. The vanilla STCN model requires roughly 2 seconds per video. For tt-MCC, tt-Ent, and tt-E,, it takes 67, 29, and 5.3 seconds, respectively, for 100 iterations. Since TTT is run per video and its speed is only affected by the number of iterations, per frame timings vary with video length, iee. the TTT overhead reduces as video length increases. It is worth noting here that all timings presented above are estimated with non-optimized TTT implementation that leaves lot of space further optimization.
0
50
100
150
200
number of training iterations
Figure 7: Change in J&F score during TTT compared the score before TTT. Each curve one video and is normalized wrt. the maximum improvement/decline observed over 200 iterations.
Reducing TTT overhead. As discussed, TTT induces a significant overhead at test time, which an aspect often discussed in the relevant literature. The TTT overhead is reduced by training for less iterations with only a minor decrease in performance. As we show Table of the Appendix, using 20 instead of 100 iterations even advantageous in cases, particularly for larger datasets like YouTube- VOS and MOSE with a model trained on real videos and without a distribution shift. In a real-world application, applying #tMCC on all test examples might seem infeasible due to time overheads. Nevertheless, it is very useful for cases of extreme test-time distribution shifts. In scenarios involving few desired for improved performance, (IMCC constitutes straightforward and cost-effective solution, that is far more efficient than retraining the base model, which typically requires approximately 12.5 hours on 2 A100 GPUs to train STCN.
examples, where rapid adaptation of the current model is
Memory requirements. During inference, STCN requires roughly 8GB of GPU RAM. Test-time training on top of STCN with tt-AE, tt-Ent and tt-MCC requires 8,16 and 23.5GB, respectively, i.e. it can still fit modest GPU. Note that the calculations for TTT are computed when using a batch size of 4 (the default), Memory requirements can be further reduced by using smaller batch size, if needed, without significant change in performance.
5 Conclusions
In this work we show that test-time training is very effective way boosting the performance of state-of-the-art matching-bart based video object segmentation methods in the case of distribution shifts. We propose a mask cycle consistency loss that tailored matching-basse performance. Its applicability goes beyond the two methods used in this work, as it is compatible with the general family of matching-bart over top performing methods for both sim-to-real transfer and the case of corrupted test examples. We also show that achieving such gains not trivial and that is important to tailor the test-time training method to the task and method at hand. A limitation of the proposed approach is the lack of way for performing early stopping, and selecting the best iteration to stop training, very promising direction for future work.
VOS and achieves top
VOS methods [36, 60, 56, 1]. We report very strong gains
10

---

Acknowledgments and Disclosure of Funding
This work was supported by Naver Labs Europe, by Junior Star GACR GM 21-28830M, and by student grant SGS22/133/HK3/33//3 The authors would like to sincerely thank Gabriela Csurka for providing valuable feedback.
References
A. Athar, Luiten, A. Hermans, D. Ramanan, and Leibe. Hodor: High-level object descriptors for object re-segmentation in vidco learned from static images. In CVPR, 2022.
F. Azimi, Palacio, Raue, Hees, I... Bertinetto, and A. Dengel. Self-supervised test-time adaptation on video data. WACV, 2022.
[3] H. S. Behl, M. Naja, A. Amab, and H. Torr. Meta-Icarning deep visual words for fast video object segmentation. In IROS, 2020.
[4] S. Caclles, K.-K. Maninis, J. Pont-Tuset, L. Leal-Taixé, D. Cremers, and L. Van Gool. One-shot vidco object segmentation. In CVPR, 2017.
[5] Y. Chen, J. Pont- Tuset, A. Montes, and L. Van Gool. Blazingly fast video object segmentation with pixel-wise metric learning. In CVPR, 2018.
[6] H. K. Cheng and A. G. Schwing. Xmem: Long-term video object segmentation with an atkinson-shiffrin memory model. In ECCV, 2022.
[7] H. K. Cheng, J. Chung, Y.W. Tai, and C.-K. Tang. Cascadepsp: Toward class-agnostic and very high- resolution segmentation via global and local refinement. In CVPR, 2020.
[8] K. Cheng, Y.W. Tai, and C.-K. Tang, Modular interactive video object segmentation: Interaction-to- mask, propagation and difference-aware fusion. In CVPR, 2021.
[9] K. Cheng, Y.-W. Tai, and C.-K. Tang. Rethinking space-time networks with improved memory coverage for efficient video object segmentation. NewrIPS, 2021.
[10] H. Ding, Liu, S. He, X. Jiang, Torr, and Bai. Mose: A new dataset for video object segmentation in complex seenes. arXiv, 2023.
[11] B. Duke, A. Ahmed, C. Wolf, Aarabi, and G. W. Taylor. Sstvos: Sparse spatiotemporal transformers for video object segmentation. In CVPR, 2021.
[12] Y. Gandelsman, Y. Sun, X. Chen, and A. Efros. Test-time training with masked autoencoders. NeurIPS. 2022.
[13] G. Ghiasi, H. Lee, M. Kudlur, V. Dumoulin, and J. Shlens. Exploring the structure of real-time, arbitrary neural artistic stylization network. In BMVC, 2018.
[14] D. Hendrycks and T. Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. In ICLR, 2019.
[15] P. Hu, G. Wang, X. Kong, Kuen, and Y.PP. Tan. Motion-guided cascaded refinement network for video object segmentation. In CVPR, 2018.
[16] Y.T. Hu, J.-B. Huang, and A. Schwing. Maskmn: Instance level vidco object segmentation. NeurIPS, 2017.
[17] Y.TT. Hu, J.BB. Huang, and A. Schwing. Unsupervised video object segmentation using motion saliency-guided spatio-temporal propagation. In ECCV, 2018.
[18] Y.-T. J.-B. Huang. and Schwing. Videomatch: Matching based video object segmentation. In ECCV, 2018.
[19] A. Jabri, A. Owens, and Efros. Space-time correspondence contrastive random walk. NeurIPS, 2020.
[20] V. Jampani, Gadde, and V. Gehler. Video propagation networks. In CVPR, 2017.
[21] A. Khoreva, R. Benenson, E. 1lg. T. Brox, and B. Schiele. Lucid data dreaming for video object segmenta- tion. IJCV, 2019.

---

[22] I. Kim, Y. Kim, and S. Kim. Learning loss for test-time augmentation. NewrIPS, 2020.
[23] D. Kingma and J. Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
[24] Z. Lai, Lu, and W. Xic. Mast:
self-supervised tracker. CVPR, 2020.
[25] X. Li, T. Wei, Y. P. Chen, Y.W. Tai, and C.-K. Tang, Fss-1000: A 1000-class dataset for few-shot segmentation. In CVPR, 2020.
[26] Y. Li, N. Xu, J. Peng. J. See, and W. Lin. Delving into the cyclic mechanism in semi-supervised video object segmentation. NeurIPS, 2020.
[27] T.Y.. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick. Microsoft Common objects in context. In ECCV, 2014.
coco:
[28] S. Liu, G. Zhong, S. De Mello, J. Gu, V. Jampani, M.-H. Yang. and J. Kautz. Switchable temporal propagation network. In ECCV, 2018.
[29] Y. Liu, Y. Dai, A.-D. Doan, Liu, and Reid. defense osvos. arXiv, 2019.
[30] Y. Liu, P. Kothari, B. Van Delft, B. Bellot-Gurlet, T. Mordan, and A. Alahi. Tu++: When does self- supervised test-time training fail thrive? NeurIPS, 2021.
[31] X. Lu, W. Wang, M. Danelljan, T. Zhou, J. Shen, and L. Van Gool, Video object segmentation with episodic graph memory networks. ECCV, 2020.
[32] J. Luiten, P. Voigtlaender, and B. Leibe. Premvos: Proposal-generatte refinement and merging for video object segmentation. In ACCV, 2019.
[33] K.-K. Maninis, S. Caclles, Y. Chen, J. Pont-Tuset, L. Leal- Maix-T D. Cremers, and L. Van Gool. Video object segmentation without temporal information. PAMI, 41(6):5515 153--1500, 2018.
[34] Y. Mao, N. Wang, W. Zhou, and H. Li. Joint inductive and transductive learning for video object segmentation. In ICCV, 2021.
[35] T. Meinhardt and -aix-. Make one-shot video object segmentation efficient again. NeurlPS, 2020.
[36] S. Oh, J.-Y. Lec, N. Xu, and S. Kim. Video object segmentation using space-time memory networks. In ICCV, 2019.
[37] K. Park, S. Woo, S. W. Oh, S. Kweon, and J.-Y. Lee. Per-clip video object segmentation. In CVPR, 2022.
[38] J. Pont-Tuset, F. Perazzi, S. Caelles, P. Arbeláez, A. Sorkine-Hornung, and L. Van Gool, The 2017 davis challenge video object segmentation. arXiv, 2017.
[39] A. Robinson, F. J. Lawin, M. Danelljan, F. S. Khan, and M. Felsberg. Discriminative online learning for fast video object segmentation. arXiv, 2019.
[40] Н. Seong, J. Hyun, and E. Kim. Kernlizedd memory network for video object segmentation. In ECCV, 2020.
[41] Shi, Q. Yan, L. Xu, and J. Jia. Hierarchical image saliency detection on extended essd. PAMI, 38(4): 717-729, 2015.
[42] Y. Sun, Wang, Liu. J. Miller, A. Efros, and Hardt. Test-time training with self-supervision for gencralization under distribution shifts. In ICML, 2020.
[43] C. Ventura, M. Bellver, Girbau, A. Salvador, Marques, and X. Giro-i Nicto. Rvos: End-to-end recurrent network for video object segmentation. CVPR, 2019.
[44] P. Voigtlaender and B. Leibe. Online adaptation of convolutional neural networks for video object segmentation. In BMVC. 2017.
[45] P. Voigtlaender, Y. Chai, F. Schroff, H. Adam, B. Leibe, and L.-C. Chen. Feelvos: Fast end-to-end embedding learning for video object segmentation. In CVPR, 2019.
[46] D. Wang, E. Shelhamer. S. Liu, B. Olshausen, and T. Darrell. Tent: Fully test-time adaptation by entropy minimization. In ICLR. 2021.
[47] L. Wang. H. Lu, Y. Wang, M. Feng, D. Wang, B. Yin, and X. Ruan. Learning to detect salient objects with levee-level supervision. In CVPR, 2017.
12

---

[48] Y. Wang, J. Choi, Y. Chen, S. Li. Q. Huang. K. Zhang, M.-S, Lee, and C.-C. J. Kuo. Unsupervised video object segmentation with distractor-awar online adaptation. Journal of Visual Communication and Image Representation, 74:102953, 2021.
[49] H. Xiao, B. Kang. Y. Liu, M. Zhang, and J. Feng. Online meta adaptation for fast video object segmentation. PAMI, 42(5):1205-1217, 2019.
[50] Z. Xiao, X. Zhen, Liao, and C. G. Snoek. Energy- test sample adaptation for domain generalization. ICLR, 2023.
[51] N. Xu, Yang. Y. Fan, D. Yue, Y. Liang, J. Yang, and T. Huang. Youtube-vos: large-scale video object segmentation benchmark. In ECCV, 2018.
[52] L. Yang, Y. Wang, X. Xiong, J. Yang, and A. K. Katsaggelos. Efficient video object segmentation via network modulation. In CVPR, 2018
[53] S. Yang, X. Wang. Y. Li, Y. Fang. J. Fang, W. Liu, X. Zhao, and Y. Shan. Temporally efficient vision transformer for video instance segmentation. In CVPR, 2022.
[54] 7. Yang and Y. Yang. Decoupling features hierarchical propagation for video object segmentation. NeurIPS, 2022.
[55] Z. Yang, Y. Wei, and Y. Yang. Collaborative video object segmentation by foreground-background integration. ECCV, 2020.
[56] Z. Yang, Wei, and Y. Yang. Associating objects with transformers for video object segmentation. NeurIPS, 2021.
[57] C. Yi, S. Yang, Y. Wang. H. Li, Y.-P. Tan, and A. C. Kot. Temporal coherent test-time optimization for robust video classification. In ICLR, 2023.
[58] Y. Zeng, P. Zhang, Zhang, Z. Lin, and H. Lu. Towards high-resolution salient object detcction. ICCV, 2019.
[59] M. Zhang, Levine, and C. Finn. Memo: Test time robustness via adaptation and augmentation. NeurlPS, 2022.
[60] Y. Zhang, Z. Wu, H. Peng, and Lin. transductive approach for video object segmentation. In CVPR, 2020.
[61] Y. Zhang, L. Li, W. Wang, Xic, L. Song, and W. Zhang. Boosting video object segmentation via space-time correspondence leamning CVPR, 2023.
13

---

Appendix
Contents
Datasets and additional implementation details
14
A.1 Datasets
14
A.2
Additional implementation details
15
B Additional experimental results and analysis
15
B.1 Measuring the standard deviation of TTT
15
B.2 Extended results on the MOSE dataset
15
B.3 Results with TTT on top of XMem
15
B.4 Varying the number of real video used for training the base model
16
B.5 Additional qualitative results
17
B.6 Results in the case of no distribution shift
17
B.7 Results in the case of training with static images
18
B.8 Performance analysis on per-video and per-object basis
................................
18
B.9 Impact of longer test-time training
19
B.10 Assessing temporal stability
19
B.11 Detailed results DAVIS-C
19
B.12 Failure cases
19
Broader impact
20
Figures for all TTT losses
20
A Datasets and additional implementation details
A.1 Datasets
The validation split of the DAVIS- 201IS-2011// [38] dataset contains 30 videos covering a variety of real-world scenarios, including indoor and outdoor scenes, different lighting conditions, occlusions, and complex motion patterns. Each video contains one to three annotated objects of interest.
DAVIS
YouTube VOS-2013 The validation split the YouTube VOS-2018 [51] dataset contains 474 high- quality videos downloaded from YouTube, including indoor and outdoor scenes, different lighting conditions, occlusions, and complex motion patterns. Each video contains one to five annotated objects of interest.
style
style 2
style
style 4
Figure 8: The four styles in DAVIS-C.
14

---

Table 2: Multiple runs of the proposed method on top of STCN and XMem for two cases of test-time distribution shift. Mean and standard deviation over 3 different seeds is reported. Left part:
Results for STCN-BL30K and XMMm-BL330K Right part: Results for STCN-DY and XMem-DY on DAVIS-C for different levels of corruption.
Training without real videos
Coruppedd test examples (DAVIS C)
Method
DAVIS
DAVIS-C
YT-VOS
MOSE
no con.
low
med
high
avg
STON [9]
70.4
85.3
72.6
58.8
73.3
STCN U-MCC (ours)
81.
70.1 #0.1
79.4 s0.2
44.9 #0.2
86.7 40.2
78.3 40.1
75.6
=0.1
67.3
77.0
XMem [6]
78.1
53.9
65.6
40.9
87.7
80.4
77.3
69.4
78.7
XMem f-MMC (ours)
82.1
20.2
70.1
20.3
78.9
20.2
44.7 20.2
88.1 20.2
81.7.00
78.9
2022
72.2.0.1
80.2
We refer the reader to the main paper for details. Here, we highlight the 14 different transformations each video applied three different strengths, namely low, medium, and high strength in Figure 17. The four original images that we use to compute the four stylizations style I, style 2, style 3, and style 4 are displayed in Figure 8.
DAVIS-C
We further report results on a newer, much larger dataset, i.e. the MOSE dataset [10], which contains challenging examples with heavy occlusions and crowded real-world scenes. The validation split of the MOSE dataset contains 311 high-qualit, videos. Each video contains one to fifteen annotated objects of interest. In addition, show the robustness of our method, we also ran models on the training split of the MOSE dataset, which contains 1507 high-quality videos with up to twenty annotated objects of interest. Because the STCN and XMem models MOSE-train split before, we consider this split as a larger dataset for testing our method, Because of limited space, we present results on MOSE-train only in the supplementary, and as show in Section B, the observations of the main paper also hold in this case.
MOSE
trained on the
A.2 Additional implementation details
The model is in evaluation mode during test-time training, i.e. the running statistics of the Batch Normalization layers remain fixed.
Because the last frame m, of the frame sequence used in our MtCMCC method is the initial frame of the backward pass, all the target objects mo must also be depicted in my. To guarantce this, we select only the frames where the network can detect all the target objects, i.e. there is at least one pixel for each object where it has the largest probability. We do by extracting pseudo masks for the video frames and resetting the optimizer every ten iterations.
In the MOSE dataset, some videos have up twenty annotated objects. Because method is linear with the number of annotated objects in the video, we randomly select up to six objects at each iteration.
B
Additional experimental results and analysis
B.1 Measuring the standard deviation of TTT
We report mean and standard deviation after running TTT with 3 different seeds in Table 2 for four datasets, namely DAVIS, DAVIS-C, YouTube- VOS,, and MOSE.
B.2 Extended results on the MOSE dataset
In Figure 9, we report STCN results with and without test-time training on the two splits of the MOSE dataset for the case of sim-to-real transfer. We see that both splits of the MOSE dataset, the three TTT variants boost the performance, and tt-MCC brings relative gain of 15% on 18% on the MOSE-valid and MOSE-train datasets, respectively.
B.3 Results with TTT on top of XMem
In Figure 10, we show the performance comparison with and without TTT on top of XMem. We observe that the performance benefits are similar to those of using TTT top of STCN.
15

---

STCN DE tt-Ent Ltt-AE I tt-MCC
J&F score
+6.0
MOSE-vall
MOSE-train
Figure 9: STCN performance the MOSE dataset for the sim-to-real transfer case. We report results on both the validation set, as well as the much larger training set of 1507 videos; used as another test set since it not used training by the models used in this work.
XMem
tt-Ent
tt-AE
tt-MCC
+4.0
80
76.3
90
J&FF score
+16.2
J&F score
60
80
13.8
40
40,9
70
no corruption low med high
DAVIS
DAVIS-C
YouTube MOSE
(a) Sim-to-real transfer
(b) Conrupted test examples
Figure 10: XMem performance under distribution shifts. Left: performance of XMem -BL30K before and after test-time training for the sim-to-real transfer case four datasets. Right: performance of XMem -DY on DAVIS-C for input corruptions with different strength levels.
B.4 Varying the number of real video used for training the base model
We increase the number of real videos used to train the STCN model from (BL-30K) to 3531 (DAVIS and YouTube-VOS training sets) and present results in Figure 11. When the model is trained with little to real videos, test-time training recovers the bulk of the performance achieved using models trained on larger annotated datasets while requiring little to no annotation.
STCN
IT W-Ennt
90
tt-AAE tt-CC
J&F score
+1.7
80
76.0
70.4
70
O (BL30K)
60
3471
3531 (DAVIS+YouTubel
(DAVIS)
(YouTube)
Number of real videos used during training
Figure 11: Performance on DAVIS with respect to the number of real videos used during training. Performance before and after test-time training for models trained using only synthetic videos (BL-30K) and to 3531 real videos (DAVIS and YouTube- VOS together).
16

---

Frames and ground-truth masks
Frames and ground ground-truth -ruth masks
STCN
STCN
STCN + t-MCC (ours)
STCN #-MMC (ours)
Figure 12: Additional qualitative examples using a model trained on synthetic videos from
BL-30K [8] and tested on real video from DAVIS [38].
row: Results obtained
using the STCN [9] approach. Bottom row: Results after test-time training using the proposed mask cycle consistency loss (tt-MCC) the single ground-truth mask provided for the first video frame.
Frames and ground- truth masks
Frames ground- truth masks
STCN
STCN
STCN + t-MCC (ours)
STCN #--MC (ours)
Figure 13: Additional qualitative examples using model trained on DAVIS [38] and YouTube- VÓS [51] and tested on corrupted video from the DAVIS-C benchmark. Left: the test example corrupted using the glass-blut corruption with medium strength. Right: the test video is stylised using style I with medium strength. Second-to-botton row: Results obtained using the STCN [9] approach. Bottom row: Results after test-time training using the proposed mask cycle consistency loss (ft-MCC) on the single ground-truth mask provided for the first video frame.
B,5 Additional qualitative results
We show more qualitative examples in Figures 12 and 13.
B.6 Results in the case of no distribution shift
In the main part of the manuscript, we focus on scenarios involving test-time distribution shifts. There, we report results based on 100 iterations, which we' ve found to optimal for cases with such distribution shifts. When model is trained on real videos and there is no distribution shift in the test videos, as demonstrated the left part of Table 3, using fewer iterations is advantageous, particularly for larger datasets like YouTube- VOS and MOSE. Overall, observe that ttMCC does
17

---

Table Additional results for STCN and Xmem. The number TTT iterations reported next to
tt-CC. Left part: Results when starting from model trained with real videos. Right part: Results when starting from model trained only on static images.
Training with real vidcos YT-VOS
Training with static images
Method
DAVIS
DAVIS-C
MOSE
DAVIS
DAVIS-C
YT-VOS
MOSE
STCN
85.3
69.3
84.3
52.5
75.7
59.3
76.3
42.0
STCN E4-MCC-20 (ours)
86.0
72.1
84.6
53.3
78.2
STCN 11-M---M----- (ours)
86.7
73.8
84.0
51.8
79.7
70.7
79.4
43.4
XMem
87.7
75.7
60.4
72.8
77.0
42.3
XMem t--MCC-20 (ours)
88.0
76.8
60.7
75.9
76.5
41.7
XMem ft-MCC-100 (ours)
88.1
77.6
85.1
59.8
78.8
67.2
78.4
42.9
(b)
(c)
(d)
100
100
100
100
Improvement
Inprrvveennt
80
Mr.F improvement
80
Inprovementt
80
60
40
0
U
0
20
20
20
100
100
100
200
400
video length
object area/ frame area
objoct area max object area.
visibility duration
Figure 14: Performance gain of tt-MCC for the sim-to-real case the MOSE-train dataset. We plot the performance gain vs. a) the video length in number of frames, b) the object area the first frame normalized by the frame area, the object in the first frame normalized by the maximum object area over all frames, d) the percentage the video length where the object is visible.
not negatively impact the performance of state-of- ffate-of-the-at the-art methods in these cases. Instead, it offers method to either maintain or enhance state-of-the-art performance across all test-time video scenarios, whether extreme distribution shifts are present or not.
B.7 Results in the case of training with static images
In this section, we focus on the scenario where the model is trained using only static images. Following STCN, we employ five datasets of static images [47, 41, 58, 7, 25] for the offline training of the networks. Given training image, several deformed versions are generated compose artificial video sequence, which is used for the model training. The corresponding results are presented in the right part of Table 3 for both STCN and XMem across the four datasets. Notably, for STCN, tt-MCC yields relative performance gains exceeding 5%, 4%, and 3% on DAVIS, YouTube- VOS, and MOSE, respectively. These gains represent substantial recovery of the performance achieved by a model trained with real videos, i.e. amounting to 42%, 39%, and 13%, respectively. Performance improvement is more significant the DAVIS-C dataset, with 19% relative performance gain, surpassing the STCN model trained on real videos (70.7 vs. 69.3).
B.8 Performance analysis on per-video and per-object basis
We conduct a comprehensive analysis using the MOSE-train dataset on a per-video and per-object basis and present results in Figure 14. Figure 14 (a) examines the performance gain using TTT in relation to video length, revealing that video length does not significantly affect performance. Figure 14 (b) demonstrates subtle correlation between performance gain and the size of objects in the first frame, with larger objects showing positive gains and no negative impact. In Figure 14 (c). we explore how the object's ' size in the initial frame relative to its maximum size the video affects performance. We observe slight negative impact when the object is less visible in the first frame compared subsequent frames. This attributed to IIMCC overfitting to the partially annotated first frame and often segmenting the partial object in future frames. Note that extreme cases of severe occlusion in the first frame can make the video object segmentation task particularly ambiguous. Finally, Figure 14 (d) assesses the impact of object visibility duration within the video sequence,
18

---

100
nom.. J&JF improvement
50
0
-50
100
100
200
300
400
500
number of training iterations
Figure 15: Change in J&F score during TTT compared the score before TTT. Each curve is one video and is normalized wrt. the maximum improvemen/ddecline observed over 500 iterations.
confirming that II-MC is adversely affected when the object is briefly visible. This suggests that our cycle consistency loss, emphasizing long-term temporal consistency, may introduce bias towards objects being visible for extended periods.
B.9 Impact of longer test-time training
To better study the effect of longer training test-time, DAVIS for the sim-to-real case and presents results in Figure 15. summary, we do not observe any significant further decrease in performance More specifically, the number of DAVIS videos for which VOS performance decreases remained the same from 50 TTT iterations up approximately 400, i.e. the J J&F score decreases in 7 out of 30 videos. Only one extra video shows a decrease in &&F score by 500 iterations. Visually observing segmentation results, notice that common issue is some background pixels of similar appearance to the object are wrongly segmented. It is also worth noting that increasing the iterations only leads minor overall decrease in VOS performance, i.e. J&F score drops from 81.1 81.0 $1.0/80.4 for 200/500 iterations, respectively.
method for up to 500 iterations
any degenerate outputs after longer training.
B.10 Assessing temporal stability
In Figure 16, present performance comparison per-video basis, both with and without tt-MCC. The graph illustrates consistent and lasting performance improvements. Importantly, highlights the enhanced stability of performance when I/-CC employed. In the case of briefly visible object (as seen video 3bf98379), our use of /I-MCC results in incorrectly segmenting out the background the object, significantly impacting the metric in negative manner.
B.11 Detailed results on DAVIS-C
In Figures 18 and 19, we present the J&F score change before and after test-time training for the STCN-BL30K and STCN-DY models, respectively, for all three corruption strengths.
B.12 Failure cases
We notice couple of patterns when inspecting failure cases of our method, and TTT in general. The most important is that TTT might make the model more confident about dubious prediction and propagate it wrongly. Moreover, we observe that TTT might overfit the appearance module and wrongly add background pixels. E.g. we observe that red shirt from the background in the breakdancer video confuses the model more than the STCN case. We refer the reader to the video we provide on the project webpage, where we visualize a number of such failure cases.
19

---

gold-fish
pige
dog
breakdance
loading
101
01
frane
fraaee
fraane
fromne
(a) DAVIS
Recda066
1c6086dd
37aBac1
3bfa8879
06eall 72c
2014
1..55 ocore
ST0C I2C
STICK 90.3
5005 201
frumne idb
franze REx
Srumte idb
fncane kdx
frume idx
(b) MOSE-train
Figure 16: Performance per frame of STCN and ttMMCC for the sim-to-real case selected videos from DAVIS and MOSE-train. We plot the performance of each method on each frame of the test video averaged over all objects. The colors of the highlighted areas indicate the winning method. The average performance for the whole video is reported in the legend per method.
C Broader impact
The proposed method is effective for the two types of distribution shifts evaluated in our work. We expect to be valuable for knowledge transfer to very different domains with limited or no annotated data. For instance, in the bio dio-medical domain, for task such tracking cells organisms captured under a microscope. Due to the lack of appropriate datasets, we do not evaluate the proposed approach in such case.
Test-time training methods like ours can also be used to improve the semi-automatic annotation process for large video collections in the presence of distribution shifts. Such data can then be used to train and fine-tune other models in other domains and tasks. is worth noting that whatever the annotation, societal, or otherwise biases existing in the dataset the base model was trained on will also present in the automatically annotated frames and propagated through this process. Test-time training single annotated example is not expected to successfully correct such biases; understanding the limitations of the base model is therefore crucial for any real-world application of method.
D Figures for all TTT losses
We illustrate the l/EEnt and Il-AE losses, as well as the regular cross-entropy loss for the supervised offline training stage in Figures 20a, 20b, and 20c, respectively.
20

---

original
low
medium
high
brightness
contrast
saturate
pixelate
gaussian noise
defocus blur
glass blur
crf compression
motion blur
cartoon
style
style 2
style 3
style 4
Figure 17: Impact of the strength of the corruption for the 14 types of corruption in DAVIS-C -
21

---

100
80
J&. score
60
20
brightness
contrast
saturate
pixelate
gaussian noise
defocus blur
glass blur
crf compression motion blur cartoon
style1
style2
style3
style4
(a) Low strength.
100
80
J&P score
brightness
contrast
saturate
pixelate
gaussian noise defocus blur
glass blur
crf compression motion blur
cartoon
style1
style2
style3
style4
(b) Medium strength
100
80
JoLF score
60
40
20
brightness
contrast
saturate
pixelate
gaussian noise
defocus blur
glass blur
crf compression
motion blur
cartoon
style1
style2
style3
style4
(c) High strength
Figure 18: Test-time training on DAVIS-C for the STCN-BL330 model. We plot the J&F score separately per video before (triangles) and after #tMCC (circles) for the STCN-BL30K model on the 14 corruptions of the proposed DAVIS-C benchmark. We report results for the variants with the highest corruption strength. A red vertical line denotes that performance drops with test-time training.
22

---

100
90
80
J&.F score
70
60
50
40
30
brightness
contrast
saturate
pixelate
gaussian noise
defocus blur
glass blur
crf compression motion blur
cartoon
style1
style2
style3
style4
(a) Low strength.
100
nnn m
J&P score
brightness
contrast
saturate
pixelate
gaussian noise defocus blur glass blur
crf compression motion blur
cartoon
style1
style2
style3
style4
(b) Medium strength.
100
80
Jo.F score
60
40
20
brightness
contrast
saturate
pixelate
gaussian noise
defocus blur
glass blur
crf compression
motion blur
cartoon
style1
style2
style3
style4
(c) High strength.
Figure 19: Test-time training on DAVIS-C for the STCN-DY model. We plot the J&F score separately per video before (triangles) and after ft-MCC (circles) for the STCN-DY model on the 14 corruptions of the proposed DAVIS-C benchmark. We report results for the variants with the highest corruption strength. A red vertical line denotes that performance drops with test-time training.
23

---

time-forward step
time-forward / step
M
450
774g
M
-2.0
m7g
m,
Ti
Mi
l Ent
Ent
(a) The t-Ent loss given frame triplet { m. The frames zo zy and my are shown with red, green and blue border, respectively.
22g
CxP
172
dm
f
Exr
Eyn
eyn
22
ty
Ex
Cross attention
dm
LCE
(b) The tt-AE loss for a given first frame zp and mask ma- The functions e and em represent the frame and mask encoders. The function d., represents the mask decoder. They are components of function f which is detailed on the right for STCN.
time-forward step
time-forward / step 2
M
Z0
77g
M
Z0
7710
2;
mg
m;
f
mi
mi
tCE
tCE
(c) The regular cross-entropy loss for given frame triplet {æo, z,, 2, and mask pair {m,, my }. The frames z0, Ii and æj shown with red, green and blue border, respectively.
Figure 20: The tt-Ent, tt-AE, and the cross-entropy loss. The function represents the overall prediction model. It takes as input the current test frame and memory of predicted masks from the previous frames and outputs the predicted mask for the current frame.
24


Reviews:
[{"review_id": "uFlJkBp6sF", "reviewer": "Reviewer_3wfH", "summary": "The paper introduces an adaptation/test-time training algorithm for matching-based video object segmentation, specifically addressing the challenges posed by out-of-distribution videos(corruptions, stylization, and sim-to-real transfer). The proposed framework incorporates test-time training with various losses (entropy, mask consistency, etc.) The experimental evaluation includes benchmark datasets like DAVIS and youtube-VOS, where the authors demonstrate performance improvements for out-of-distribution scenarios.", "questions": "Please discuss the questions raised above.", "limitations": "It has been discussed however not fully clear what the failure mode of the approach would be, might be helpful to conduct more analysis on this.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The proposed direction is highly interesting and carries significant practical implications. \n\n- The results obtained are quite promising. Notably, there are substantial improvements observed across multiple datasets, particularly in the case of corrupted DAVIS. These outcomes highlight the effectiveness and value of test-time training in this context.", "weaknesses": "The technical aspects of the paper may appear somewhat limited. While concepts such as test-time training and loss are well-known techniques, the actual technical contribution seems marginal. It would be beneficial to explicitly highlight the distinctions between this work and previous approaches.\n\nThe validation process also seems somewhat limited, which undermines the full convincing power of the proposed method, in particular:\n- The absence of qualitative results in video formats makes it difficult to assess the practical impact and temporal stability of the improvements achieved. \n- The resemblance of DAVIS-C to real-world noise is uncertain, as the augmentation techniques seem quite artificial. Additionally, it may be difficult to reproduce the study on DAVIS-C, and the argumentation process is not fully clear.\n- Providing more in-depth analysis on intermediate/breakdown results would be valuable. For instance, exploring the performance relative to video length, examining corner cases where the method fails, and identifying scenarios where the proposed approach is most beneficial. Currently it's hard to gain further insights from the results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> __Distinctions between this work and previous approaches__\n\nAlthough Cycle Consistency (CC) has been used in the past, our method is the first on to have such a loss tailored for the task of one-shot VOS: Unlike methods like “Space-time correspondence as a contrastive random walk” [16] or the concurrent work “Boosting Video Object Segmentation via Space-time Correspondence Learning” from CVPR 23, we do not apply the cycle-consistency loss on RGB pixel values, but instead on the values of the mask in the first frame.  This means that, unlike space-time correspondence CC losses, ours is not self-supervised. Together with the fact that our method is the first to apply this _at test-time_ we believe makes our tt-MCC method a solid technical contribution.\n\nUnlike other methods like HODOR [1] that might employ a cycle consistency loss during training, our method tailors this to test time training and uses the GT frame that is provided at test-time. Concurrent work [49] also studies test time training (TTT) in the video domain but for a classification task; we instead fully tailor the TTT on the VOS task and use temporal _mask consistency_ as our loss. We hope that this answer covers the differences between the proposed and the closest related works, if the reviewer has another in mind, please let us know.\n\nBeside the technical contribution of the tt-MCC method, we want to further note here that our paper is the first one to study the effect of extreme test-time distribution shifts like sim-to-real and frame-level corruptions for the task of one-shot VOS in a principled way. Additionally, our paper is the first to study test time training for VOS and includes benchmarking of a number of TTT baselines to offer valuable insights and a new benchmark for the community to build on and evaluate. Until now TTT has been mostly studied on image classification.\n\n> __Absence of qualitative results in video formats makes it difficult to assess the practical impact and temporal stability__\n\nThis is a great point. We are providing multiple qualitative results in the rebuttal video shared with the AC. It enables comparing results with and without TTT visually for a number of corner cases. From the rebuttal video and after extensive quantitative analysis, the key failure modes can be summarized as: a) With TTT, the model might get more/overly confident about a dubious prediction and propagate it wrongly (e.g. see monkey clip in the video). b)  TTT might overfit the appearance module and wrongly add background pixels (e.g. see case of breakdancer, where a red shirt from the background confuses the model more than the STCN case). \n\nRegarding temporal stability, we include in Figure B of the rebuttal PDF a performance comparison over time on a per video basis, with and without TTT. It demonstrates enduring performance gains. Notably, it also shows that the performance with TTT is more stable compared to that without TTT. \n\n\n> __The resemblance of DAVIS-C to real-world noise is uncertain__\n\nThank you for a valuable comment. We agree that these transformations might not perfectly represent the outcome of video recording in the real-world. Nevertheless, we believe that several of the transformations do constitute common real-world video edits (contrast, brightness, crf compression, cartoonization, stylization).\n\nWe follow the footsteps of ImageNet-C which has been a valuable benchmark to study image classification under distribution shift. We believe that this benchmark can lead to valuable insights, similar to what ImageNet-C achieved for image understanding. \n\n\n> __it may be difficult to reproduce the study on DAVIS-C__\n\nWe are committed to releasing the DAVIS-C dataset, the code to regenerate the dataset, and the code for this study, which will make it possible to not only reproduce this study, but also make it easy to evaluate any future method on the same benchmark in a fair way.\n\n\n> __More in-depth analysis on intermediate/breakdown results would be valuable__\n\nGreat suggestion. In Figure A of the rebuttal PDF, we provide some more in-depth analyses and a breakdown of the performance per video/object for the large MOSE-train dataset. In Figure A.a we evaluate performance gain with TTT vs the length of the video, as the reviewer suggested. We can observe that the video length has no significant impact on the performance. In Figure A.b, we present the performance gain vs the object size within the first frame and observe a slight correlation; for larger area there is no negative gain.\n\nInspired by the fact that the MOSE dataset contains examples where only a portion of the object is visible and annotated in the first frame due to occlusions, in Figure A.c we measure the impact of the size of the object in the first frame relative to the maximum size of the object within the video.  A minor observation is that the gain is negatively impacted to a small extent when the object is less visible in the first frame compared to the subsequent frames. Unsurprisingly, the proposed tt-MCC overfits more to that partially annotated first frame, and usually segments the partial object in future frames instead of the full object. Note that extreme cases of severe occlusion in the first frame make such a VOS task ambiguous.\n\nFinally, we evaluate the impact of the duration of the object visibility within the video sequence in Figure A.d and confirm that tt-MCC is negatively impacted when the object is only briefly visible in the sequence. This could mean that by requesting long term temporal consistency, our CC loss imposes an inductive bias that the target object should be generally visible for a long period of time.\n"}, {"review_id": "dwOJzfaZ89", "reviewer": "Reviewer_iPsr", "summary": "This paper revisits test-time training in video object segmentation (VOS) and introduces three losses (entropy loss, mask auto-encoder loss, and mask cycle consistency loss) that significantly enhance top-performing methods, particularly under extreme distribution shifts between training and testing sets. Additionally, it introduces DAVIS-C, a variant of the DAVIS test set with extreme distribution shifts, demonstrating great performance on this challenging dataset.", "questions": "1. It would be beneficial to clarify how the proposed model can be extended to handle multiple objects. Specifically, it is important to address whether multiple object segmentation can be achieved in a single pass during the inference step or if multiple forwards are required.\n\n2. I kindly request that the authors report the finetuning time alongside Figure 7. It would be more informative to have the exact time instead of just the number of training iterations. This information is valuable since one of the main drawbacks of test-time training methods is the potentially long finetuning time during test runs. Additionally, please clarify if the reported average inference time with TTT per frame includes the finetuning time, as mentioned in lines 227-228.\n\n3. Please kindly provide information regarding the memory requirements of the proposed method. Since it stores both mask and frame representations, it would be helpful to know if there is a threshold or mechanism to manage memory usage and if any data is dropped when reaching that threshold.\n\n4. In lines 129-131, please provide clarification regarding the similarity matching metric utilized to identify similar items.\n\n5. In line 160-166, the last frame prediction is dependent on the second frame prediction, I am wondering if this type of \"dependency\" is necessary? For example, is it possible to predict both the last mask and the second mask based solely on the first mask?", "limitations": "NA", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The proposed method enhances test-time training for matching-based video object segmentation (VOS) and achieves significant performance improvements for state-of-the-art methods in scenarios with extreme distribution shifts (e.g., synthetic to real data, stylization, and video corruptions).\n2. The paper introduces the DAVIS-C dataset, specifically designed to evaluate model performance under extreme distribution shifts during test time.\n3. The proposed method achieves a 70%-80% recovery in performance gain by training solely on synthetic data, without the need for video annotations during offline training.", "weaknesses": "Please see my comments in below Questions section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> __Clarify how the proposed model can be extended to handle multiple objects__\n\nThe proposed test-time training (TTT) method is a way of performing test-time adaptation over a given base method (STCN/XMem in our paper). It does not change the way the base method does inference, and for STCN and XMem, for example, multiple objects are handled sequentially. Nevertheless, our formulation is generic and applicable to a wide range of matching-based VOS methods, including ones that require only a single forward pass for multiple objects, e.g. AOT [48]. In such a case, multiple objects would be handled via a single forward pass for TTT as well.\n\n\n> __Finetuning/TTT time__\n\nWe thank the reviewer for bringing attention to this. The average inference time reported in lines 227-228 of the main paper is _after_ TTT. We tried to clarify timings better in the supplementary, i.e. see lines 66-71; we apologize for any confusion. In the supplementary, we report that one iteration of tt-MCC takes ~814 ms for STCN (supp-line68). With this information, one can translate the x-axis of Figure 7 from iterations to seconds; we will update this in the final version. \n\nTiming estimates for a single video (on average, on DAVIS) are as follows: it takes roughly 3 seconds to run STCN per video, and 81, 10.5 and 8.5 seconds to run tt-MCC, tt-Ent, and tt-AE, respectively, for 100 iterations. Note however that: a) TTT is run per video, i.e. TTT overhead is amortized and becomes less  for longer videos; b) decent improvements can be obtained with less TTT iterations, e.g. see Table A of the rebuttal PDF; c) the timings above are estimated with our basic, non-optimized TTT implementation. During the rebuttal, for example, we managed to obtain a speed-up of 20%, while further optimization is definitely possible. \n\nAlthough applying TTT consistently might seem infeasible due to time overheads, what our paper shows is that it is highly useful for cases of extreme test-time distribution shifts. One should therefore also consider the case where we are given a few out-of-distribution examples and want to quickly adapt our current model to improve its performance on them. In such cases, TTT is a simple and cost-effective solution, far preferable to re-training the base model (it takes around 12.5 hours on 2 A100 GPUs to train STCN).\n\nOverall, we fully agree with the reviewer that long fine tuning times are a key drawback for test-time training methods in general, and one that the community does not even touch: Most TTT papers do not have any discussion on timings, although it is something important to report and discuss. We thank the reviewer for their comment, and we promise to clearly report and discuss the time needed for test time training in the paper.\n\n>__Memory requirements__\n\nDuring inference, STCN requires approx. 8GB of GPU RAM. Test-time training on top of STCN with tt-AE, tt-Ent and tt-MCC requires 8,16 and 23.5GB, respectively, i.e. it can still fit in a modest GPU. Note that the calculations for TTT are computed when using a batch size of 4 (our default). Memory requirements can be further reduced by using a smaller batch size, if needed, without significant change in performance.\n\nWe thank the reviewer for such comments on timings and memory; we will make sure to clearly report and discuss the added resources needed for test time training.\n\n> __Clarification regarding the similarity matching metric__\n\nThe matching part is inherited from the base method. The technical details are exactly the same as in STCN and are described in the original paper. In particular, the full similarity matrix is formed between the test frame items and the memory items but only the top-k values per row (rows correspond to the test frame) are maintained, the rest are set to 0. By the term item we refer to each D_e-dimensional vector, with each frame having WxH of them. \n\n> __Is it possible to predict both the last mask and the second mask based solely on the first mask?__\n\nYes, this is technically possible. We did not experiment with that, as we adopt the design choices of STCN. We expect its performance to not vary significantly. \n"}, {"review_id": "4b9tn1Mh8Q", "reviewer": "Reviewer_dPR9", "summary": "This paper focuses on test-time training for matching-based VOS methods. Three methods are presented, including entropy loss, auto-encoder loss and cycle consistency loss. The cycle-consistency loss is tailored for VOS and utilizes the first-frame mask for supervision. Two evaluation protocols are provided, the sim-to-real and DAVIS-C. Results show that the proposed tt-MCC method can help matching-based VOS methods to gain better performance under sim-real shifts and corrupted videos.", "questions": "See Weaknesses.", "limitations": "The method does not include a mechanism for implementing early stopping and determining the optimal iteration to stop training.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The proposed method appears to be straightforward to implement and potentially effective in conjunction with matching-based VOS methods.\n\n- The proposed DAVIS-C benchmark is a valuable contribution that can aid the community in advancing their research.\n", "weaknesses": "- **The proposed tt-MCC focuses solely on triplets and overlooks longer temporal consistency.** Methods that rely on short-term memory (STM) usually utilize multiple frames in their memory pools, allowing for aggregating long-term temporal information. However, tt-MCC only employs two frames for each training step, potentially limiting its exploitation of temporal consistency.\n\n- **The concept of cycle consistency has been extensively explored and widely employed in various video-related fields.** In light of this, the proposed method is not groundbreaking. Moreover, a recent study [a] shares a similar idea and asserts that their approach outperforms cycle consistency (as shown in Table 7). The authors may want to highlight the disparities and advantages of tt-MCC compared to the method proposed in [a].\n\n- **The results for XMem-YD or STCN-YD + tt-MCC under no-corruption conditions on large-scale benchmarks like YouTube-VOS and MOSE are missing.** Additionally, synthetic datasets are not as commonly used as static images for pretraining. It would be beneficial to examine whether tt-MCC aids matching-based VOS methods after static pretraining.\n\n[a] Boosting Video Object Segmentation via Space-time Correspondence Learning, CVPR2023", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> __The proposed method focuses solely on triplets and overlooks longer temporal consistency.__\n\nWe generally followed exactly the same design that each base method uses during training and adopted the use of triplets for STCN and octuplets for XMem. We briefly explore performance using longer sequences for STCN in the supplementary material (lines 60-62): Using quadruplets or quintuplets instead of triplets reaches 81.7 and 82.4, respectively, but with a corresponding increase of 33% and 64% in training time, i.e., in summary, having longer sequences does bring some minor improvements, but with the cost of additional TTT time. We can discuss this in the main paper if reviewers think we should.\n\nLet us however also note that one can achieve longer temporal consistency even using triplets, by modulating the value of the jump steps: In the supplementary, we validate its impact (supp-line 55) to observe that neither very small nor very large values are the optimal choices: \nVarying the jump steps from the default value of 10 to 1, 2, 25, and 50 achieves 80.6, 81.0, 80.8, and 79.0, respectively. See paragraph “Impact of triplet sampling” in Section C of the supplementary for further exploration of the sequence sampling. \n\n> __Cycle consistency has been extensively explored - relation to [a] “Boosting Video Object Segmentation via Space-time Correspondence Learning” (CVPR 2023)__\n\nWe thank the reviewer for pointing out an interesting related work that is concurrent to ours. The key differences between [a] and our approach are: a) unlike our paper, [a] does not study the case of test-time adaptation of a model for VOS,  and b) we use the cycle consistency on the _mask_ labels, not on the pixels or regions. [a] builds on other related papers like [16] where space-time consistency is imposed on the rgb pixels and further extends it to regions, but does not use the mask as a label. \n\n> __The results for XMem-YD or STCN-YD + tt-MCC under no-corruption conditions on large-scale benchmarks like YouTube-VOS and MOSE are missing__\n\nWe report the missing results in Table A of the rebuttal PDF, for a different number of test time training iterations. In the main paper, our core focus is on the case of test-time distribution shifts,  and we only report results for 100 iterations which we found to be best for all cases with distribution shifts. For the case where there is no distribution shift, however, we see from Table A that, unsurprisingly, using less iterations could be beneficial, especially for the larger scale YTVOS and MOSE datasets. Overall we see that TTT does not hurt the state of the art methods in this case, but instead gives us a way of either preserving or improving SoTA performance in all cases of test-time videos, with or without distribution shift, something important in practice. We thank the reviewer and we will add these results. \n\nNote that a large number of existing papers for TTT in classification ([18][35][39]) report no or insignificant improvements for the case of no test-time distribution shifts. Actually, not decreasing the performance under no shift is typically considered an achievement ([39] states “Tent reaches the least error for most corruption types without increasing the error on the original data“)\n\n\n> __examine whether tt-MCC aids matching-based VOS methods after static pretraining__ \n\nWe thank the reviewer for a valuable additional experiment. We explored this during rebuttal and report results in the right part of Table A in the rebuttal pdf, for STCN and XMem on 4 datasets. We conclude that tt-MCC provides significant improvements in this setup too. We will add these experiments.\n\n"}, {"review_id": "qxkndbpwuk", "reviewer": "Reviewer_hK29", "summary": "This paper points out that Current state-of-the-art approaches use a memory of already processed frames and estimate the segmentation masks of follow-up frames through matching. Lacking any adaptation mechanism, such methods are prone to test-time distribution shifts. To address this, this paper  explores task-agnostic and VOS-specific test-time training strategies, including a mask cycle consistency-based variant tailored for matching-based VOS methods. The authors present a new dataset DAVIS-C and the proposed method is evaluated on the most common benchmarks, and demonstrate its effectiveness.", "questions": "See weaknesses above", "limitations": "Limitations are missing", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Experimental results show that the proposed approach (MCC) significantly boosts the performance.\n\n2. Paper is very easy to read and understand.\n\n", "weaknesses": "1. Lack of technical contributions. Although mask cycle consistency usage at test time helps to boost the performance and enable more task-agnostic prediction, this seems to be a very similar technique that with minor modification (mask) from other works.  \n\n2. Just a question though. Why is XMeM not included in Fig 5? \n\n3. Figure 6 and 7 are very hard to see and understand. I understand what these figures are trying to deliver, but to me, these figures definitely do not look so informative due to its seemingly unorganized graphs.   \n\n4. Ablations and limitations are missing. It would be much more informative to include failure cases as well.\nWhat would happen if the method extends from triplets, and use more images for MCC? \n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> __Lack of technical contributions [...] this seems to be a very similar technique that with minor modification (mask) from other works__\n\nAlthough Cycle Consistency (CC) has been used in the past, our method is the first one to have such a loss tailored for the task of one-shot VOS, i.e. for a setting where a single _groundtruth_ mask is also given. In this fashion, the CC loss in our case is using privileged information and is supervised in contrast to many cases in prior work where it is self-supervised. This, together with the fact that our method is the first to apply this _at test-time_ we believe makes our tt-MCC method a solid technical contribution.\n\nLet us further note that, beside the technical contribution of the tt-MCC method, our paper is the first one to study the effect of extreme test-time distribution shifts like sim-to-real and frame-level or video-level corruptions for the task of one-shot VOS in a principled way. Additionally, our paper is the first to study test time training for VOS and includes benchmarking of a number of TTT baselines to offer valuable insights and a new benchmark for the community to build on and evaluate. Until now TTT has been mostly studied on image classification.\n\n> __Why is XMeM not included in Fig 5?__\n\nWe did not include it because it cluttered the presentation, while providing little amount of additional information. The numbers are overall slightly higher, but the insights and trends remain. We will include this. \n\n> __Ablations and limitations are missing. It would be much more informative to include failure cases as well.__\n\nWe thank the reviewer for a valid comment. We believe that the rebuttal PDF and video significantly expand our exploration and analysis on limitations and failure cases. After extensive quantitative analysis (some cases also included in the rebuttal video), the key failure modes can be summarized as: a) With TTT, the model might get more/overly confident about a dubious prediction and propagate it wrongly (e.g. see monkey clip in the video). b)  TTT might overfit the appearance module and wrongly add background pixels (e.g. see case of breakdancer, where a red shirt from the background confuses the model more than the STCN case).\n\nIn Figure A of the rebuttal PDF, we provide some more in-depth analyses and a breakdown of the performance per video/object for the large MOSE-train dataset. In Figure A.a we evaluate performance gain with TTT vs the length of the video, as the reviewer suggested. We can observe that the video length has no significant impact on the performance. In Figure A.b, we present the performance gain vs the object size within the first frame and observe a slight correlation; for larger areas there is no negative gain.\n\nInspired by the fact that the MOSE dataset contains examples where only a portion of the object is visible and annotated in the first frame due to occlusions, in Figure A.c we measure the impact of the size of the object in the first frame relative to the maximum size of the object within the video. A minor observation is that the gain is negatively impacted to a small extent when the object is less visible in the first frame compared to the subsequent frames. Unsurprisingly, the proposed tt-MCC overfits more to that partially annotated first frame, and usually segments the partial object in future frames instead of the full object. Note that extreme cases of severe occlusion in the first frame make such a VOS task ambiguous.\n\nMoreover, in Figure A.d of the rebuttal PDF we evaluate the impact of the object visibility within the video sequence, and confirm that tt-MCC is negatively impacted when the object is only briefly visible in the sequence. This could mean that by requesting long term temporal consistency, our CC loss imposes an inductive bias that the target object should be generally visible for a long period of time.\n\nFinally, Figure B of the rebuttal PDF explores temporal stability and presents a performance comparison over time on a per video basis, with and without TTT and also displays some failure cases. \n\nWe will include all these additional analyses in the paper.\n\n> __What would happen if the method extends from triplets, and use more images for MCC?__\n\nWe generally followed exactly the same design that each base method uses during training and adopted the use of triplets for STCN and octuplets for XMem. We briefly explore performance using longer sequences for STCN in the supplementary material (lines 60-62): Using quadruplets or quintuplets instead of triplets reaches 81.7 and 82.4, respectively, but with a corresponding increase of 33% and 64% in training time, i.e., in summary, having longer sequences does bring some minor improvements, but with the cost of additional TTT time. We can discuss this in the main paper if reviewers think we should.\n\nLet us also note that, instead of increasing the number of images, one can achieve longer temporal consistency by modulating the value of the jump step $s$: In the supplementary, we validate its impact (supp-line 55) to observe that neither very small nor very large values are the optimal choices: \nVarying the jump steps from the default value of 10 to 1, 2, 25, and 50 achieves 80.6, 81.0, 80.8, and 79.0, respectively. See paragraph “Impact of triplet sampling” in Section C of the supplementary for further exploration of the sequence sampling. \n"}, {"review_id": "U3MAkQ1Yny", "reviewer": "Reviewer_jwwD", "summary": "This paper investigates several test-time training methods for improving matching-based video object segmentation algorithms. It proposes and compares three types of test-time training methods and finds that the mask cycle loss works the best. The mask cycle loss forward/backward propagates a mask and requires cycle consistency. The authors find that test-time training significantly improves performance, especially in sim-to-real transfer and in videos with corruption.", "questions": "- Table 3 in the supplementary material: the two bars with 38.9 do not have the same height. Is this a typo?\n\n- Figure 3 – The mask predictions before test-time training look bad. Is that solely caused by the gaussian noise?\n", "limitations": "Test-time training is slow. Also, cycle loss can lead to degenerate output (e.g., identical mask at every time step) which might be why the authors only train for 100 steps and that some outputs become worse after finetuning (Figure 7).", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Adapting to harder cases during inference has been a challenge for video object segmentation algorithms. Test-time adaptation (i.e., online learning) can be useful but most of the current online learning methods are limited to fine-tuning on the frame-level without considering the temporal aspect. This means those methods are of limited aid when the video contains fast and complex motion of the target object. Differently, the proposed cycle loss takes the temporal information into account which is helpful.\n\n- There are good analyses about how different models behave with different levels of data corruption. \n\n- I like the inclusion of MOSE results in the supplementary material. It is a real-world challenging dataset that has much harder examples than the training set and current methods have difficulties generalizing to it. This shows that the proposed method works for real-world applications and the result is encouraging. I would recommend putting this in the main paper.\n", "weaknesses": "- There is no discussion on the time required for the fine-tuning stage and how it affects the overall running time during inference.\n\n- The main results are presented with DAVIS-C which is synthetic. The results would carry more weight if done with the MOSE dataset instead.\n\n- It would be more complete if the authors include a discussion on the use of data augmentation during the finetuning stage. Related: Lucid Data Dreaming for Video Object Segmentation, IJCV 2019.\n\n- Missing related paper: Delving into the Cyclic Mechanism in Semi-supervised Video Object Segmentation, NeurIPS 2020\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">  __Timings for the fine-tuning stage__\n\nWe thank the reviewer for bringing attention to this. In the supplementary, we report that one iteration of tt-MCC takes ~814 ms for STCN (supp-line68). Timing estimates for a single video (on average and on DAVIS) are as follows: it takes roughly 3 seconds to run STCN per video, and 81, 10.5 and 8.5 seconds to run tt-MCC, tt-Ent, and tt-AE, respectively, for 100 iterations. Note however that: a) TTT is run per video, i.e. TTT overhead is amortized and becomes less for longer videos; b) decent improvements can be obtained with less than 100 TTT iterations, e.g. see Table A of the rebuttal PDF; c) the timings above are estimated with our basic, non-optimized TTT implementation. During the rebuttal, for example, we managed to obtain a speed-up of 20%, while further optimization is definitely possible. \n\nAlthough applying TTT consistently might seem infeasible due to time overheads, what our paper shows is that it is highly useful for cases of extreme test-time distribution shifts. One should therefore also consider the case where we are given a few out-of-distribution examples and want to quickly adapt our current model to improve its performance on them. In such cases, TTT is a simple and cost-effective solution, far preferable to re-training the base model (it takes around 12.5 hours on 2 A100 GPUs to train STCN).\n\nWe thank the reviewer for their comment and we promise to clearly report and discuss the time needed for test time training in the paper.\n\n> __The main results are presented with DAVIS-C which is synthetic. The results would carry more weight if done with the MOSE dataset instead.__\n\nWe agree that MOSE is a great new dataset (intricate occlusions, densely populated real-world settings). It is however not clear to us whether the reviewer suggests (a) to test on MOSE with corruptions, (b) to test on MOSE for the sim-to-real case, or (c) to test on MOSE for the case of no test-time distribution shift.\n\nWith respect to (a), we agree that this would be valuable, but also very costly for experimentation since MOSE is many orders of magnitude larger than DAVIS and DAVIS-C contains 45 times more videos than DAVIS (for each original video we produce 15 corruptions over 3 levels). We can however consider this as a possible future update.\n\nWith respect to (b), this is something already included in the supplementary material Figure 3; TTT is very valuable there as well. We can include this in the main paper if space allows.\n\nThe case of (c) has not been the major focus of our paper, i.e. we focus on test-time distribution shifts. We did however run these experiments during rebuttal and the results are included in Table A of the rebuttal PDF with figures we submitted. There, we explore the use of a smaller number of iterations for TTT, which appears to be a better option for this setup and works for the case of testing on corruptions too. Let us further note here that it is common in the TTT literature to not report improvements for the case of no test-time distribution shift [18,35,39]; not decreasing the performance under no shift is typically considered an achievement ([39] states “Tent reaches the least error for most corruption types without increasing the error on the original data“).\n\n> __Discussion on the use of data augmentation during the finetuning stage. Related: Lucid Data Dreaming for Video Object Segmentation, IJCV 2019.__\n\nWe thank the reviewer for a great comment and a related work we missed. Inspired by that work, we tested the impact of standard image augmentations (color+geometric) on our TTT method. Their use had little impact over all cases. In particular, we observed some insignificant performance increase in the sim-to-real case on DAVIS, and a similarly insignificant drop in performance for videos with corruptions from DAVIS-C. Note that the authors of that work claim that “Ideally training data should be as similar as possible to the test data, even subtle differences may affect quality” which fully aligns with the proposed MCC loss that makes use of the actual future video frames instead of trying to hallucinate them. We will add this experiment and discussion in the paper.\n\n\n> __Missing related paper: Delving into the Cyclic Mechanism in Semi-supervised Video Object Segmentation, NeurIPS 2020__\n\nThank you. We will add the missing reference and discuss it. In short, the paper is using CC during training time, which allows to perform training while requiring masks only for the first video frame. This is similar to the way the cyclical loss is used in HODOR [1]. In contrast, we use CC for TTT, which allows us to perform TTT without any extra input than the one that is required by the task itself. \n\n> __Table 3 in the supplementary material: the two bars with 38.9 do not have the same height. Is this a typo?__\n\nThank you for pointing this out. It is indeed a typo. The correct number for the MOSE train is _42.9_\n\n> __Figure 3 – The mask predictions before test-time training look bad. Is that solely caused by the gaussian noise?__\n\nThis is correct. Gaussian noise in itself is a quite challenging corruption.\n\n\n"}]

Author rebuttal:
We would like to thank all five reviewers for insightful and constructive reviews. 

We are pleased that the feedback is __overall positive__. Four out of five reviewers highlighted the significant performance improvements of our proposed method under test-time distribution shifts. We are happy that they also found our paper “highly interesting” and with “significant practical implications” (__R-3wfH__), as well as our paper “easy to read and understand” (__R-hK29__). Reviewers further appreciated the introduction of the DAVIS-C benchmark (__R-iPsr__), calling it a “valuable contribution” for the community (__R-dPR9__). They highlighted our analyses over different levels of frame-level corruptions (__R-jwwD__) and the large improvements our method can achieve in such cases (__R-3wfH__).

We are responding to each reviewer individually below, addressing all of their questions and comments.  

We will additionally send a video to the AC (referred to as __rebuttal video__ in the responses below) using the provided functionality, that summarizes the qualitative performance before and after our TTT for success, failure and other corner cases.

We further attach to this response a one-page rebuttal PDF document with additional results and plots (referred to as __rebuttal PDF__ in the responses below) that help us respond to reviewer comments. It contains:
* Table A with results for STCN and XMem on four datasets, for the cases of training from static images and real videos, as well as for a reduced number of iterations of test-time training.
* Figure A with four different statistical analyses for the sim-to-real case on the large MOSE dataset, demonstrating the performance of the proposed approach versus video length, object size as well as other key elements that help us demonstrate better where our method helps and where it fails.
* Figure B with plots that compare performance over time for a few videos, that justify the improved temporal stability of the proposed method and also demonstrate success and failure cases.

We hope that our responses below, together with all the additional qualitative and quantitative results and analysis, address the reviewers’ comments in a satisfactory way. We are more than happy to engage in further discussion and clarify any points that might still remain unclear.


Meta-review: The paper investigates test-time training in video object segmentation. The experiments show performance improvement particularly in scenarios of sim-to-real transfer and corrupted videos. 

This paper underwent a thorough review process by five reviewers. It received mixed ratings of 7, 6, 5, 4, and 4, placing it in a borderline category. The AC read the paper, the reviews, and the rebuttals. The rebuttals addressed concerns raised by the reviewers about speed, technical contributions, and discussions with prior works. Moreover, to validate the effectiveness the authors provided more qualitative results and a thorough ablation on four datasets including DAVIS, DAVIS-C, YTVOS and MOSE. The authors also promised to release the proposed DAVIS-C dataset and the code for reproduction of the research to address the reviewer's concern about reproducing the results for validation.

The AC agrees with the majority of the reviewers that the paper is technically sound and recommends it for acceptance. The authors should add all the additional results including the speed, memory consumption and the ablations in the final version.
Decision: Accept (poster)