Self-supervised Amodal Video Object Segmentation
Authors: Jian Yao
Yuxin Hong
Chiyu Wang
Tianjun Xiao
Tong He
Francesco Locatello
David Wipf
Yanwei Fu
Zheng Zhang
Venue: NeurIPS 2022 Accept (2022)
Keywords: Amodal Segmentation
Self-supervised Learning
Spatiotemporal
Video

Abstract: Amodal perception requires inferring the full shape of an object that is partially occluded. This task is particularly challenging on two levels: (1) it requires more information than what is contained in the instant retina or imaging sensor, (2) it is difficult to obtain enough well-annotated amodal labels for supervision. To this end, this paper develops a new framework of Self-supervised amodal Video object segmentation (SaVos). Our method efficiently leverages the visual information of video temporal sequences to infer the amodal mask of objects. The key intuition is that the occluded part of an object can be explained away if that part is visible in other frames, possibly deformed as long as the deformation can be reasonably learned. Accordingly, we derive a novel self-supervised learning paradigm that efficiently utilizes the visible object parts as the supervision to guide the training on videos. In addition to learning type prior to complete masks for known types, SaVos also learns the spatiotemporal prior, which is also useful for the amodal task and could generalize to unseen types. The proposed framework achieves the state-of-the-art performance on the synthetic amodal segmentation benchmark FISHBOWL and the real world benchmark KINS-Video-Car. Further, it lends itself well to being transferred to novel distributions using test-time adaptation, outperforming existing models even after the transfer to a new distribution.

Self-supervised Amodal Video Object Segmentation
Jian Yao Yuxin Hong" Chiyu Wang" Tianjun Xiao
Tong He*
Francesco Locatello*
David Wipf Yanwei Fu Zheng Zhang*
School of Management, Fudan University
School of Data Science, Fudan University
3 University of California, Berkeley
Amazon Web Services
{jianyao20, yxhong20, yanweiful edu. way_james@berkelley edu (tianjux, htong, locatelf, daviwipf, zhaz)eamazon. com
Abstract
Amodal perception requires inferring the full shape of an object that is partially occluded. This task is particularly challenging on two levels: (1) it requires more information than what is contained in the instant retina imaging sensor, (2) it is difficult to obtain enough rell-annotated amodal labels for supervision. To this end, this paper develops new framework of Self supe-supe amodal Video object segmentation Vos). Our method efficiently leverages the visual information video temporal sequences to infer the amodal mask of objects. The key intuition is that the occluded part of an object can be explained away if that part is visible in other frames, possibly deformed as long the deformation can be reasonably learned. Accordingly, we derive a novel self-supervised learning paradigm that efficiently utilizes the visible object parts the supervision guide the training on videos. In addition to learning type prior to complete masks for known types, Sa also learns the spatiotemporal prior, which is also useful for the amodal task and could generalize to unseen types. The proposed framework achieves the tfate-of-the-art performance on the synthetic amodal segmentation benchmark FISHBOWL and the real world benchmark KINS-Video- itself well being transferred novel distributions using test-time adaptation, outperforming existing models even after the transfer to a new distribution. The experiment code is available at https://github... com/amazon /aam///mm--cccccc/sseelfff wttpn///seessssevsssseeeeeeeseee amooal-vide--oheet-sgmeeete
Further, it lends
Introduction
Cognitive scientists have found human vision system contains several hierarchies. Visual percep- tion [27] first carves a scene at its physical joints, decomposing into initial object representation by grouping and simple completion. At this point, the representation is tethered into the retina sensor 13. Then, correspondence or motion on the temporal dimension is built to form the object repre- sentation that is untethered from the retinal reference frame through operations like spatiotemporal aggregation, tracking, inference and prediction [6]. The more stable untethered representation is ready to be raised from the perception system the cognitive system for higher level action and symbolic cognition 23). Machine learning, especially with artificial neural networks, has progressed tremendously tethered vision tasks like detection and modal segmentation. The natural next step is to go the higher rung of the ladder by tackling untethered vision.
Work completed during internship AWS Shanghai AI Labs. i 'Correspondence authors are Tianjun Xiao and Yanwei Fu.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

This paper studies the task of amodal segmentation which aims at inferring the whole shape of the object on both visible and occluded parts. It has critical applications on robot manipulation and autonomous driving [24] Conceptually, this task is on the bridge between tethered and untethered rep- resentations. Amodal segmentation requires prior knowledge. One option that has been explored in lit- erature is using the tethered representation and prior knowledge about object type to get amodal mask. Alternatively, we can get amodal masks using the untethered representation by building dense object motion across frames to explain away occlusion, which is referred spatiotemporal prior. We prefer to explore more on the second one since the dependence on type prior makes the first method hard to generalize, considering the frequency distribution of visual categories in daily life is long- aangttiled.
Following this direction, we propose a Self-supervised amodal Video object segmentation Vos) pipeline which simultancously models amodal mask and the dense object motion on the amodal mask. Unlike traditional optical flow or correspondence networks, our approach does not require explicit visual correspondence across pixels, which would be impossible due to occlusions. Instead, modeling motion using temporal information allows us to complete dense amodal motion predictions.
The architecture is built for spatiotemporal modeling, which has better generalization performance than using type priors. Despite that, we show that Sa automatically figures its way learn type prior as well, as learning types can help the encoder-decoder-styl architecture make prediction. This makes generalization to distribution shifts remain challenging, for example, to unseen types of objects. To address this issue, we need to suppress the type prior and amplify spatiotemporal prior to make predictions. This is achieved by combining SaVos with test-time adaptation. Critically, we found that our model is cnlaptation-frinddll adaptation techniques without any change on the self-supervised loss, achieving a significant boost in generalization performance.
as it can naturally be improved with test-time
We make several contributions in this paper:
(I) We propose a Self-supervised amodal Video object segmentation (Sa training pipeline built upon the intuition that the occluded part of an object can be explained away if that part is visible in other frames (Figure T]. possibly deformed as long as the deformation can be reasonably learned. The pipeline turns visible masks in other frames to amodal self-supervision signals.
(2) The proposed approach simultaneously models the amodal mask and the dense amodal object motion. The dense amodal object motion builds the bridge between different frames to achieve the transition from visible masks to amodal supervision. To address the challenge of predicting motion on the occluded area, we propose a novel architecture design that takes the advantage of the inductive bias from the spatiotemporal modeling and the common-fate principle of Gestalt Psychology [39]. The proposed method shows the state-of-the-art amodal segmentation performance in self-supervised setting on several simulation and real-world benchmarks.
(3) The proposed Sa Vos model shows strong generalization performance on drastic distribution shifts between training and test data after combining with one-shot test-time adaptation. We empirically demonstrate that, by applying test-time adaptation without any change on the loss, SaVos trained on synthetic fish dataset can even outperform competitor that well learned on the target real-worte driving car dataset. Interestingly, applying test-time adaptation on an image-level level baseline model doesn't 't bring the same improvement as observed on Vos. This provides an unique perspective on comparing different models by checking how effective can test-time adaptation work on them.
2 Related works
Untethered vision and amodal segmentation. Human vision forms hierarchy by grouping retina signals into initial object concept; and the representation will untether from the immediate retina sensor input grouping the spatiotemporally disjoint pieces. Such untethered representation has been
studied in various topics (23 21 28 26 34| [2]. Particularly, Amodal segmentation [46] is task
inferring shape of the object on both visible and occluded part. There are various image amodal datasets such as COCOA [46] and KINS (24). and video amodal dataset- SAIL-VOS [I] created by the GTA game engine. Unfortunately, SAIL-VOS has frequent camera view switches, not the ideal testbed to apply video tracking or motion. Several efforts are made towards amodal segmentation on these datasets 46 24
30 20 Generally speaking, most of the methods are on image level and they model type priors with shape statistics, such it is challenging to extend

---

their models to open-world applications where object category distributions are long-tail. Amodal segmentation is also related structured generative model 44 maximize the likelihood of the whole video sequences during training so as to learn a more consistent object representation and the amodal representation. However, the major tasks for those models are object discovery and presentation and they are tested on simpler datasets; self-supervvsed object discovery in real-world complex scenes like the driving scene in [| remains too challenging for these methods. Without object discovered, no proper amodal prediction can be expected.
29 18 These models attempt to 16
Dense correspondence and motion Our goal is achieve amodal using untethered process, which requires object motion signals. There have been studies [8] [ on correspondence and motion before deep learning time. FlowNet [5] and its follow-up work FlowNet2 [14] train deep networks in a supervised way using simulation videos. Truong et al. B4 proposes GLU-Net, a global-loce universal network for dense correspondences. However, motion on the occlusion area cannot be estimated with those methods. Occlusion and correspondence estimation depend on each other and it a typical chicken-and-ege problem 15]
We need to model additional priors.
Video inpainting A related but different task is video inpainting. Existing video inpainting methods fill the spatio-temporal holes by encourage the spatial and temporal coherence and smoothness 40 TO [2], rather than particularly inferring the occluded objects. The object-level knowledge was not explicitly leveraged to inform the model learning. Recently, Ke al. [7] learns object completion by contributing the large-scale dataset Youtube-VOl, where occlusion masks are generated using high-fidelitg simulation provide training signal. Nevertheless, there is still the reality gap between synthetic occlusions and the amodal masks in the real-world. Accordingly, our model is designed to learn the amodal supervision signal the easily accessible raw videos if spatiotemporal information is properly mined.
Domain generalization and test-time adaptation Transfer learning 4 and domain adaptation 25 are general approaches for improving the performance of predictive models when training and test data come from different distributions. Sun et al. (3] proposes Test-time Training. It is different from finetuning where some labeled data are available in test domain, and different from domain adaptation where there is access to both train and test samples. They design a self-supervised loss to train together with the supervised loss and in test time, apply the self-supervised loss on the test sample. Wang et al. (37) proposes a fully test time adaptation by test entropy minimization. A related topie Deep Image Prior [36] and Deep Video Prior [19] they directly optimize on test samples without training on training set. Our model is self- supervupe thus naturally fit into test-time adaptation framework. We will see how it works for our method on challenging adaptation scenarios.
3 Method
Notations. Given the input video (T))== T frames with K objects, the task is to generate the amodal "binary" segmentation mask sequences M = {M,} for each object in every frame. On the raw frames, we obtain the image patch 1 and visible modal segmentation mask V = {V=+. Further, also obtain the optical flow AV,- {AV,, AV; },y such that 4++1 + AV.,l v). y A AVy [a, y]] 1z y). These information can be retrieved from human annotation extracted with off-the-shelf models, and we use them as given input to our model.
V.k
3.1 Overview of Our Sa Learning Problem
The key insight of the amodal segmentation task is to maximally exploit and explore visual prior patterns to explain away the occluded object parts (35]. Such prior patterns include, but are not be limited to (1) type prior: the statistics of images, or the shape of certain types of objects; and (2) spatiotemporal prior: the current-oceluded part of an object might be visible in the other frames, as illustrated in Figure T Under self-supervieed setting, we exploit temporal correlation among frames relying exclusively on the data itself.
Specifically, Sa Vos generates training supervision signals by investigating the relations between the amodal masks and visible masks on neighboring frames. The key assumption is "some part is occluded at some time, but not all parts all the time". and that deformation of the past visible parts can be approximately learned. Le. gleaning visible parts over enough frames produces enough evidences to complete object. The inductive bias we can and must leverage is spatiotemporal continuity.
3

---

81 Figure 1: Though we don't see the full shape for the blue object in any frames, human can tell the full shape of that object at frame 1z. We just need to stitch seen parts in different frames.
to
f2
fg
We note that parts remain occluded all the time can not be recovered unless there are other priors such as classes and types, which also learnable our model design. Figure 2 a) shows the training pipeline.
Prediction of amodal mask and amodal motion At frame for object k. SaVos predicts the amodal mask Mt with
Mt. A Fo V AVE.)
where Fg learnable module paramterized by with more detailed introduction in Section (.3) and AM the dense motion on the amodal mask. Ideally Fn is able aggregate available visual, semantic and motion information from all the currently available frames. The intuition here is that seeing various parts of an object (i.e. V) and their motions (AV) is enough reconstruct M. However, leaving as is, there can be an infinite number of explain-away solutions.
The first obvious training signal is to check the consistency between M and V.d. The signal is weak since the model learn simple copy function. A stronger signal is assume generative component to predict the amodal mask in the next time frame and use visible V in addition. Since transformation is already predicted by AM, frame + by a warping function, i.c. M Warp(M, AM Mi4 satisfies
obtain estimation of the amodal mask at
M4pla AM LM y). AM vll [M
(2)
Now we compute the distance between MA and V with the assumption that V might includes parts occluded in V**. and define the first training loss as
LM ΣΣ dm (M Vit1)
where
dM (Mitr vo+
W w o BCE (Mtt V24.)
(4)
with BCE being the vanilla form of binary cross entropy loss function, and W being a weight matrix that masks out the occluded + for object Concretely,
K
w =
+ V
(5)
++1
where all-one matrix with the same shape as the mask tensor. With this mask, Ls only generates supervision signal on the visible parts and background between different frames, with the occluded part being masked out from providing any feedback. Our prediction can be under-complete if only LM is applied, since all that the model is foreed learn is the visible masks one frame later.
Amodal consistency loss The temporal consistency loss Lcs assuming some distance measure function d(.,.) is straightforward in its form:
K
T
Le
ΣΣ de SMC,MM k=1t=2
(6)
The intuition is that the amodal mask prediction Mt at t should be consistent with the estimation Mi warped from t 1. M includes new evidence (V.*t AV! that is not available when computing

---

Spatiotemponi Embedding Module
LSTH
Worp
A.R
Amodal Mark Complster
V?
R. 41.
Fe
AM
Mean
Mosan Prodictor
Figure 2: a) Va Training Pipeline. b) SaVos Architecture
Mr forcing the generative component in estimating M to catch up. This loss links the supervision signals in all frames to guide the prediction in each frame.
Consider an object that is made up by two parts, each is visible in only one of the two adjacent frames, good estimation is encouraged include both parts under this loss. In this work, use Diff JoU as the metric do, introduced in [3] since it's symmetric to inputs. Note that this loss has a similar form of temporal cycle-consistency
Combining Equation 4 and Equation the final loss is:
L X1 LM + X2 Lc
Analysis on necessity and sufficiency of L Define L/t, Et- dm (N Vi+1) and CG.
El= de (M D It is easy to show that Che = LbC = 0 is necessary for Mr = NT, = M
Specifically, for any and k, we have d. (M V/+) )=)== dc (Mig MM) 0 if Mr = M = M
=dc
and that directly leads to Cht c = O. We further analyzes the sufficiency of Chi, C/C = 0. The theorem statement and its proof can be found in supplementary material.
Some pixels of an object might never be visible in any frames. For example, parked cars that line up along the roadside with corners invisible, e.g. the target car in Figure We emphasize that our model can still align the correct amodal segmentation with the global optima of the proposed loss. Consequently,
those cases, our method can still work least good image-level methods since it is also able to capture type prior for known types with the architecture design in Section 3.3 The encoder-decoder architecture contains an information bottleneck, which makes it easy squeeze out type information since i's concise and beneficial to amodal prediction. In addition, the fewer pixels remain invisible all the way, the more spatiotemporal information our model can leverage to improve its performance.
Bi-directional prediction SaVos described far suffers from cold start problem, i.e. the first few frames may not be informative enough. We solve this by simply predicting backwards in time. To merge the prediction from both directions, we add alpha channel prediction together with the amodal mask and let the model decide which side to trust more:
M á +
M
where af and & are the alpha channel from each direction normalized with each other using Softmax function. Mr and Mr are the predicted amodal mask from each direction.
3.2 Test-time Adaptation for Vos
Sa Vos models spatiotemporal prior and as such should be robust against data distribution shift. However, certain components (especially the generative part) are sensitive to samples the training data. Type prior can be implicitly learned during training, which the model falls back and struggles
5

---

to "synthesize" novel masks. In this work we adopt one-shot rest-time adaptation as stated in [32] We don't expect new knowledge to be learned on single sample, but suppress unnecessary type prior and rely only on learned spatiotemopral prior. The advantage is that base model can be reused to tackle new data distribution which is not expected to be part of the long term sample repository.
Since training is on each single test sample independently, no change for the testing environment is required, only the inference time will increase. In (32). test-time self-supervised loss is used to help the model adapt to the test data distribution. Theoretically, if the gradient of the test-time loss is on the same direction of the main training-time loss, the model performance on the test domain will be improved. As our Sa Vos model is self-supervisedll learned, the test-time adaptation naturally apply. In practice, we optimize the loss on Equation 7 on test video without any change.
In experiments, test-time adaptation indeed helps on challenging distribution shifts test scenarios. We will have more detailed analysis on the learning dynamics and efficiency on the test-time adaptation for SoVos in the Experiment Section 4 comparing with the baseline.
3.3 Architecture
As depicted in Figure b). the overall architecture has three components: 1) the spatiaotemporal
embedding that summarizes object-level signals into a hidden representation Irf 2) the amodal mask completor that takes by and V to output the estimated amodal mask M,, and 3) the estimated amodal mask motion A,, The generator function Warp itself does not contain any parameters.
Spatiotemporal embedding module This module extracts features from video frames, aligns and aggregates the feature across frames. The encoder (Enc) takes the concatenation of raw image patches, optical flow patches and visible masks input. Then a recurrent architecture (Seq) aggregates information through temporal dimension.
fe = Enc (1, AVE, V²²)
(9)
h = Seq f!)
(10)
hy is the spatiotemporal embedding for object at frame t. Here, we implement the encoder Enc and Seq with CNNs and LSTMs, respectively. This module also learns reasonable deformation over time.
Amodal mask completor Amodal mask completor is an Encoder-Decoder architecture with an information bottleneck. The CNN encoder takes the visible mask V. and concatenate the output with h, then uses several de-convolutional layers DeConv to produce the full mask prediction:
Mr = DeConva ([hi, CNN. (V=*))
(II)
where a the parameters of the CNN and DeConv above.
Motion predictor Computing AM, uses the same general encoder-decoder architecture the amodal mask completor except it takes AV:: instead of V. as input. In addition, the computation takes the form of residual prediction, using the mean of visible mask motion signal AV.. as the base to correct. This inductive bias reflect the common-fate principle of Gestalt Psychology 39
AMA AV, + DeConv, ([h², CNN. (AV+))
(12)
where is the parameters of the CNN and DeConv above.
Experiments
We evaluate the proposed pipeline on both close-worke setting with no distribution shifts between training set and test set, as well as the setting has distribution shifts with new object types.
Chewing Gum Dataset Chewing Gum is a synthetic dataset consists of random generated polygons (that look like chewing gum). Each polygon has a random number of nodes ranging from 7 to 12 and the nodes randomly scattered on a circle. Object are occluding each other and have relative movement. The occluded object never shows its full shape. But each part is shown at least one frame. Because of the randomness in generation, the shape prior for certain type will not work.

---

Deocclusion
SaVos (Ours)
Figure 3: Qualitative comparison between the image-level baseline Self- Saff-Deccllusin 44] and our Vos model. Our model produces more consistent predictions across frames and the performance better when the occlusion rate high. This is expected considering the consistency loss and the spatiotemporal architecture Sa Vos has.
FISHBOWI. Dataset This dataset 33 consists of 10,000 training and 1,000 validation and test videos recorded from a publicly available WebGL demo of an aquarium [T] each with 128 frames with resolution at 480x320. It is positioned between simplistic toy scenarios and real world data.
KINS-Video Video-Car Dataset KINS an image-level amodal dataset labeled from the city driving dataset KITTI [9]. In order to have Sa work with KINS, we match images in KINS to its original video frame in KITTI. Since only training videos are available online, we re-split the original KITTI training set into three subsets for training, validation and test. We use PointTrack 42 to extract visible masks and object tracks to drive our video- based algorithm. We only run the algorithm for the Car category and mark this modified KINS dataset as KINS-Vidde Viieso-daa-..
Metrics and Settings. The metric to evaluate amodal segmentation mean-IoU as in Specifically, we compute mean-loU against the groundtruth full mask as well as only the occluded sub-area, in order to evaluate the overall and focused performance. Occluded mean-loU is usually a better indicator for amodal segmentation. We use groundtruth visible mask and tracking as inputs for FISHBOWL and Chewing Gum, and pre-compute visible mask and tracking from PointTrack model for CINS-Video-e completion. On FISHBOWL, we only compute mean-IoU for objects with the occlusion rate from 10% to 70%. On KINS- Video-Car, match the tracked objects and the annotated ones from KINS and only compute mean-IoU on the paired ones. All sulfvvupe models share the same input, while the supervised baseline is trained samples with labels. For test-time adaptation, video is given and adapted separately. We run repeating experiments on our own method but not all baselines since the training is costly. The performance difference between runs is around 0.02 on the occluded mean-loU, Paricularly, we further propose two new settings to evaluate the performance on distribution shifts. In the first setting, train a model four type of fishes in FISHBOWL and test on the rest type. In the second one, we train a model on FISHBOWL and evaluate on KINS Video-Car.
Note that self-supervision in this work is only for the amodal mask
test
Competitors. We use a simple heuristic method that just completes the object into a convex shape, a state-of-the
the-trr otate-of-the-art image-level self supervvee model Self deff-Deocclusion 44] and supervised oracles: recent wate-offtheeaa supervised method VRSP-Net 41 or U-Net 28 depending on the availability
of the codebook for VRSP-Net.
4.1 Experiment Results on Test Set with No Distribution Shifts
Performance on FISHBOWL and KINS- Video-Car Table T compares Sa Vos with baselines. Qualitatively, Figure 3 compares predictions between Sa and Self-Deoccclsion. Note that KINS- Vidco-Car poses several challenges: the model-inferred object visible mask and tracking are inevitably inaccurate; vidcos with camera motion bring complex temporal motion signals like zoom-in/out, lens distortion and change of view point. However, our model still works on this challenging scenario. Also, some pixels for target car in the video of Figure b) are never visible in any frames. This is representative case for the parked cars that line up along roadside with corners always invisible. Our model still produce complete amodal mask, which can not be achieved if only spatiotemporal prior is learnt. This is an indicator that Savos also learns type prior during training.
Performance on Chewing Gum dataset Chewing Gum is classless as every two objects are different. As shown in Table [ all image-level models fail to predict the occluded part. Since there is no type prior, image-level model, including the supervised one, can do much better than
7

---

Table 1: Mcan-IoU on FISHBOWL, SAN--oido- and Chewing Gum datasets. For the supervised oracle results, we use VRSP-Net(4) for KINS- Video-Car, and U-Net for FISHBOWL and Chewing Gum since the codebook for VRSP-Net is not available for these two datasets.
FISHBOWL
KINS-Video Car
Chewing Gum
Method
Supervised
Full
Occluded
Full
Occluded
Full
Occluded
Convex
X
0.7761
0.4638
0.7862
0.0829
0.9264
0.3182
Self-Deocclusion 44
0.8704
0.6502
0.8158
0.1790
0.9624
0.3307
SaVos (Ours)
0.8863
0.7155
0.8258
0.3132
0.9746
0.8046
Supervised Oracle
0.9162
0.7500
0.8551
0.4883
0.9613
0.3321
Saft
Usnel (Sperined) (Cum)
Ground
Frams 1+4
Selt
(uperiiee)) SaVes (Duns)
Figure 4: Amodal prediction on a Chewing Gum video on different frames. Sosos predicts consistent masks while for the image-based method the prediction varies even for the supervised method.
completing the object to its convex hull. SaVos outperforms the rest with a significant advantage. Note that the occluded part only occupies small portion of the entire full mask, thus mean-IoU on occluded part a better indicator on the model performance. Visualization is shown in Figure
4.2 Experiment Results on Test Set with Distribution Shifts
It's a major challenge for machine learning models to generalize under distribution shifts and test videos may contain objects from unseen types. Amodal methods depending only type prior suffers from these challenges, while a model that considers spatiotemporal prior can work better. When the model learns both type and spatiotemporal prior, test-time adaptation strategy can pick-up spatiotemporal prior to achieve good generalization performance. We verify this statement in the following experiments.
Test-time adaptation performance on FISHBOWL dataset with unseen fishes Trained on 4 types of fish, SoVos tries to fit new type of fish out of the ones it has already learned (Figure 5 This is an indicator SaVos also learns type prior. Savos with test-time adaptation produces competitive result compared with model trained with all types of fishes (Table 2). The image- level baseline model, although also test-time trained, doesn't show the same amount improvement. This demonstrates that Sa Vos has picked up spatiotemporal prior designed.
more challenging scenario: adapting the model trained on FISHBOWL to KINS- Video-Car dataset. To further check the ability of adaptation visually complex data, model trained on FISHBOWL dataset and adapt it to the MINS-Video-Car dataset at test-time. Under this setting,
SAA coilided IOU Salou Jiil, OU
akr. soodde
Teat www.ereeeerneggg te-atiro
Figure 5: a) Learning curve for test-time adaptation on one FISHBOWL video.b) Learning dynamics for Sa Vos test-time adaptation. The optimal stopping point that is closest to the groundtruth might not be the one has the lowest loss. e) d) Sa Vos first fits the unseen type of fish a type seen training time. After test-time adaptation, the incorrect type prior is gone. e) f) Amodal performance before and after test-time adaptation for SoVos trained on FISHBOWL and test on KINS- MONS-Vide--Car this case, though the model never see car during training, it still make reasonable predictions. Though it captures less detail and slightly invades the neighboring pixels compared model trained on ANNS-Vido-Caa in Figure

---

Table 2: Methods with (w.) or without (w./o.) test-time adaptation (TTA). We report mean-loU performance on unseen fishes. Sa shows the advantage after applying test-time adaptation. S-D: Seff-Deocclusion. Occ: occluded.
Unseen
SmallFishA
MediaFishA
MediaFishB
BigFishA
BigFishB
Overall
Full/Oce
Occ
Oce
Occ
Occ
Occ
Full
Occ
Self-D
0.4757
0.6207
05825
0.5965
0.5709
0.8459
0.5859
Self-D w. TTA
0.4629
0.6349
0.6268
0,5742
0,5944
0.8497
0.5971
SaVos w./o. TTA
0.5362
0.6678
0.6044
0.5931
0.4978
0.8428
0.5905
Sa w. TTA
0.5464
0.7147
0.7080
0.7151
0.6230
0.8663
0.6886
SaVos w. All
0.6299
0.7324
0.7256
0.7115
0.6894
0.8863
0.7155
Table 3: Adaptation performance from FISHBOWL training set to KINS Video- test set. With test-time adaptation, SoVos trained on FISHBOWL achieves better mean-loU compared with Self- Deocclusion trained MINS-Video-aar
Method
Training
Adaptation
Full
Occluded
FISHBOWL
0.7813
0.0658
Scif-Deacclusion
FISHBOWL
KINS-
Vidco-Car
0.7999
0.0969
KINS-Video Video- Car
X
0.8158
0.1790
FISHBOWL
X
0.8004
0.0994
Sa VourrOOus)
FISHBOWL
KINS- Video-Car
0.8235
0.2976
X
0.8258
0.3132
the model is challenged to adapt to new object type, camera motion, new image quality as well as visual details. In Table 3 with test-time adaptation, our model even outperforms the image-level baseline trained directly on KINS- Video-Car since our model leverages spatiotemporal prior, which leads to better generalization ability when distribution shift happens.
4.3 Analysis on The Effectiveness of Test-time Adaptation for Sa SaVos
Test-time adaptation optimization dynamic Figure 5 a) shows the learning curve of test-time adaptation on one FISHBOWL video. The model is trained on other four types of fishes and tuned to adapt to the unseen type BigFishB in that video. For our Vos model, we noticed the occluded IoU performance increases first and then slightly drops, while the mean-loU on the visible part keeps improving. similar observation has been made in Deep Image Prior (36] where on the image denoising task, the network first learns the denoised version of the input image and then reconstructs the noisy pixels later. We assume similar optimization dynamic on our case. As illustrated in Figure 5 b), the training loss in Equation 7 is a surrogate one that only optimizes the visible mask in the other frames. The ground-truth solution Mg: belongs to a manifold of points that have loss: However, there other points can get the same loss while not really achieving perfect amodal segmentation. Usually those points refer to the predictions that not only cover the full mask, but also intrude into the neighboring area that is never revealed in any frame. Type prior can help resolving this issue by telling the common shape of that type. Then for seen types we don't worry about that. When type prior is not available in test-time adaptation, We just stop when finishes recovering all the visible part in every frame, leaving less chance to make additional intrusive predictions. In the experiment in this paper, stop the optimization if the visible-part IoU improves less than 0.01 in the last 10 iterations. We also visualize the prediction before and after test-time adaptation in Figure 5 c)-f).
O.
try to tackle this using early stopping.
Why test-time adaptation is efficient on Sa Vos? Seems any self-supervised model is suitable in test-time adaptation since the same loss be used in the training phase and test-time adaptation phase. However, we also try test-time adaptation on Self-Deocclusion while we don't 't observe the same amount of test performance improvement Sa in Table
The assumptions explain this are sampling efficiency and lack of motion. Gelf-Deocclusion randomly selects another object as the occluder and overlay it on top of the visible mask of the target occludee to get supervision, again randomly. However, we would argue that such method useful given large training set but particularly not that efficient in test-time adaptation on single video. The area they produces training signal is on the visible part at the current frame, which no need to be completed in this frame anyway. Only if the same part is visible in another frame and

---

the occluder happens to be overlapped on the same position, that the training signal is contributing the current test time video. Considering the image resolution and occluder type, the chance to sample that training signal can be low. While for Sa Vos, we produce the training signal that is exactly on the occluded areas this video by building amodal motion across frames. The efficiency on test-time adaptation setting is an unique perspective to compare different loss design. Though both are self-supervised. SaVos is more efficient than Seff-Deccclusio after combining with test-time adaptation.
5 Conclusions
We propose SaVos, a self-supervised video amodal segmentation pipeline that simultaneously models the completed dense object motion and amodal mask. Beyond type prior, on which the existing image-leve models rely, SaVos also leverages spatiotemporal priors for amodal segmentation. Sa Vos not only outperforms image-level baseline on several synthetic and real- real-world datasets, but also generalize better with test-time adaptation.
6 Limitations and Future Works
We summarize several limitations and future works of the existing SaVos model:
Variational method can be introduced to handle uncertainty on the occluded area, especially for articulated non-rigid objects.
Inductive bias from 3D modeling can be introduced handle more complex ego and object motions.
Currently, we need to run visible mask segmentation and tracking beforehand to start Sa Sos. It would be valuable extend So to end-to-end pipeline, even all in self-supervised way. This leads to a future work of combining SaVos with video structured generative models like SCALOR([6). However, KINS-Video-F and found out object discovery on this dataset is still too challenging for existing methods. We'll also catch up with the progress of object discovery.
empirically tried to utilize the SCALOR code
All these future works can be built top of the idea of utilizing spatiotemporal information to mine amodal supervision signal and find evidence for mask completion from the existing SaVos.
7 Negative Social Impact
Sa Vos runs on object tracking result. Tracking on cars or pedestrians might have privacy concern.
References
[1] Webgl demo of an aquarium, 2022.
[2] Alex M Andrew. Multiple view geometry in computer vision. Kybernetes, 2001.
[3] Maxim Berman, Amal Rannen Triki, and Matthew B Blaschko. The lovász-softmax loss: tractable surrogate for the optimization of the intersection-over-uiion measure in neural networks. In Proceedings of the IEEE Conference on Computer Vision and Patler Recognition, pages 4413-4421, 2018.
[4] Jeff Donahue, Yangqing Jia, Oriol Vinyals, Judy Hoffmann, Ning Zhang, Eric Tzeng, and Trevor Darrell. Decaf: A deep convolutional activation feature for generic visual recognition. In International conference machine learning, pages 647-655. PMLR, 2014.
[5] Alexey Dosovitskiy, Philipp Fischer, Eddy Ilg, Philip Hausser, Caner Hazirbas, Vladimir Golkov, Patrick Van Der Smagt, Daniel Cremers, and Thomas Brox. Flownet: Learning optical flow with convolutional networks. Proceedings of the IEEE international conference on computer vision, pages 2758-2766 2015.
10

---

[6] John Duncan. Selective attention and the organization of visual information. Journal of experimental psychology: General, 113(4))501, 1984.
[7] Patrick Follmann, Rebecca König, Philipp Härtinger, Michael Klostermann, and Tobias Böuger. Learning to see the invisible: End-to-end trainable amodal instance segmentation. In 2019 IEEE Winter Conference on Applications of Computer Vision (WACV), pages 1328-1336. IEEE, 2019.
[8] David Forsyth and Jean Ponce. Computer vision: A modern approach. Prentice hall, 2011.
[9] Andreas Geiger, Phillip Lenz, and Raquel Urtasun. Are we ready for autonomous driving? the kitti vision benchmark suite. In 2012 IEEE conference on computer vision and pattern recognition, pages 3354-3361. IEEE, 2012.
[10] Miguel Granados, James Tompkin, K Kim, Oliver Grau, Jan Kautz, and Christian Theobalt. How be seen-object removal from videos of crowded scenes, In Computer Graphics Forum, volume 31, pages 219-228. Wiley Online Library, 2012.
[11] Yuan-Ting Hu, Hong-Shuo Chen, Kexin Hui, Jia-Bin Huang, and Alexander G Schwing. Sail- Vos: Semantic amodal instance level video object segmentation-a synthetic dataset and baselines. In Proceedings of the IEEE/CVF Conference on Compuler Vision and Puttern Recognition, pages 3105-3115, 2019.
[12] Jia-Bin Huang, Sing Bing Kang, Narendra Ahuja, and Johannes Kopf. Temporally coherent completion of dynamic video. ACM Transactions on Graphics (TOG), 35(6): 2016.
[13] David H Hubel and Torsten N Wiesel. Receptive fields and functional architecture in two nonstriate visual areas (18 and 19) of the cat. Journal of neurophysiology. 28(():229-295 1965.
[14] Eddy Ilg, Nikolaus Mayer, Tonmoy Saikia, Margret Keuper, Alexey Dosovitskiy, and Thomas Brox. Flownet 2.0: Evolution of optical flow estimation with deep networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2462-2470, 2017.
[15] Eddy Ilg, Tonmoy Saikia, Margret Keuper, and Thomas Brox. Occlusions, motion and depth boundaries with a generic network for disparity, optical flow or scene flow estimation. In Proceedings of the European Conference Computer Vision (ECCV), pages
2018.
[16] Jindong Jiang, Sepehr Janghorbani, Gerard De Melo, and Sungjin Ahn. Scalor: Generative world models with scalable object representations. arXiv preprint arXiv: 2019.
[17] Lei Ke, Yu-Wing Tai, and Chi-Keung Tang. Occlusion-aware video object inpainting, In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 14468- 1 14478, 2021.
[18] Adam Kosiorek, Hyunjik Kim, Yee Whye Teh, and Ingmar Posner. Sequential attend, infer, repeat: Generative modelling of moving objects. Advances in Neural Information Processing Systems, 31, 2018.
[19] Chenyang Lei, Yazhou Xing, and Qifeng Chen. Blind video temporal consistency via deep video prior. Advances in Neural Information Processing Systems, 33, 2020.
[20] Huan Ling, David Acuna, Karsten Kreis, Seung Wook Kim, and Sanja Fidler. Variational amodal object completion. Advances in Neural Information Processing Systems, 13:16246-16257, 2020.
[21] Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for se- mantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3431-3440, 2015.
[22] Wenhan Luo, Junliang Xing, Anton Milan, Xiaoqin Zhang, Wei Liu, and Kyun Kim. Multiple object tracking: A literature review. Artificial Intelligence, 293:103448, 2021.
[23] Benjamin Peters and Nikolaus Kriegeskorte. Capturing the objects of vision with neural networks. Nature human behaviour, 5(9):1127-1144, 2021.

---

[24] Lu Qi, Li Jiang, Shu Liu, Xiaoyong Shen, and Jiaya Jia. Amodal instance segmentation with kins dataset. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3014-3023, 2019.
[25] Joaquin Quiñonnro- Candella, Masashi Sugiyama, Anton Schwaighofer, and Neil D Lawrence. Dataset shift in machine learning. Mit Press, 2008
[26] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in neural information processing systems, 28:91 99, 2015.
[27] Maximilian Riesenhuber and Tomaso Poggio. Hierarchical models of object recognition in cortex. Nature neuroscience, 2011:1011-1025, 1999.
[28] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention, pages 234-241, Springer, 2015.
[29] Sara Sabour, Andrea Tagliasacchi, Soroosh Yazdani, Geoffrey Hinton, and David J Fleet. Unsupervised part representation by flow capsules. In International Conference on Machine Leurning, pages 9213-9223. PMLR, 2021.
[30] Yihong Sun, Adam Kortylewski, and Alan Yuille. Amodal segmentation through out-of-task and -iitt--iiittittti generalization with a bayesian model. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 1215-1224, 2022.
[31] Yu Sun, Xiaolong Wang, Zhuang Liu, John Miller, Alexei Efros, and Moritz Hardt. Test-time training with self-supervision for generalization under distribution shifts. International Conference Machine Learning, pages 9229-9248. PMLR, 2020.
[32] Yu Sun, XiaolongWang, Zhuang Liu, John Miller, Alexei A. Efros, and Moritz Hardt. Test-time training with self-supervision for generalization under distribution shifts. In ICMI., 2020.
[33] Matthias Tangemann, Steffen Schneider, Julius von Küigelgen, Francesco Locatello, Peter Gehler, Thomas Brox, Matthias Kümmerer, Matthias Bethge, and Bernhard Schölkopf. Unsupervised object learning via common fate. arXiv preprint
2021.
[34] Prune Truong, Martin Danelljan, and Radu Timofte. Glu-net: Global-local universal network for dense flow and correspondences. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6258-6268, 2020.
[35] ZHUOWEN TU, XIANGRONG CHEN, Alan Yuille, and Song-Chun Zhu. Image parsing: Unifying segmentation, detection, and recognition. In IJCV, 2005.
[36] Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Deep image prior. In Proceedings of the IEEE conference on computer vision and pallern recognition, pages 9446-9454, 2018.
[37] Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell, Tent: Fully test-time adaptation by entropy minimization. arXiv preprint 1rXiv:2006.10726, 2020.
[38] X. Wang, A. Jabri, and A. Efros. Learning correspondence from the cycle-consistency of time. In CVPR, 2019.
[39] Max Wertheimer. On perceived motion and figural organization. MIT Press, 2012.
[40] Yonatan Wexler, Eli Shechtman, and Michal Irani. Space-time completion of video. IEEE Transactions pattern analysis and machine intelligence, 463 2007.
[41] Yuting Xiao, Yanyu Xu, Ziming Zhong, Weixin Luo, Jiawei L., and Shenghua Gao. Amodal segmentation based on visible region segmentation and shape prior. arXiv preprint wwXiv:2012.05598 2020.
[42] Zhenbo Xu, Wei Zhang, Xiao Tan, Wei Yang. Huan Huang, Shilei Wen, Errui Ding, and Liusheng Huang. Segment as points for efficient online multi-object tracking and segmentation. In Proceedings of the European Conference Computer Vision (ECCV), 2020.
12

---

[43] Jianwei Yang. Zhile Ren, Mingze Xu, Xinlei Chen, David Crandall, Devi Parikh, and Dhruv Batra. Embodied amodal recognition: Learning to move perceive objects. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2040-2050, 2019.
[44] Xiaohang Zhan, Xingang Pan, Bo Dai, Ziwei Liu, Dahua Lin, and Chen Change Loy. Self- supervised seene de-occlusion. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3784-3792, 2020.
[45] Ziheng Zhang, Anpei Chen, Ling Xie, Jingyi Yu, and Shenghua Gao. Learning semantics- aware distance map with semantics layering network for amodal instance segmentation. In Proceedings of the 27th ACM International Conference on Multimedia, pages 2124-2132, 2019.
[46] Yan Zhu, Yuandong Tian, Dimitris Metaxas, and Piotr Dollár. Semantic amodal segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1464-1472, 2017.
[47] Daniel Zoran, Rishabh Kabra, Alexander Lerchner, and Danilo Rezende. Parts: Unsupervised segmentation with slots, attention and independence maximization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10439-10447, 2021.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either in the supplemental material or URL)? [Yes] We put the code as part of the supplemental material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they chosen)? [Yes]
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either in the supplemental material or as URL? [Yes]
We will put the code as part of the supplemental material.
(d) Did you discuss whether and how consent was obtained from people whose data you're 'rere
using/curating? [Yes]
(e) Did you discuss whether the data you using/curating contains personally identifiable information or offensive content? [Yes]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
13

---

(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
14


Reviews:
[{"review_id": "EbZvBJDdGJQ", "reviewer": "Reviewer_CKtM", "summary": "The paper proposes a method for amodal video object segmentation that does not rely on amodal segmentation annotations during training. Starting with image, optical flow and conventional segmentation as input, the method estimates an amodal mask and amodal motion for each instance and propagates this mask through time using the estimated flow. The learning signal comes from consistency terms with the input mask and flow through time.", "questions": "Questions\nThe questions follow the weaknesses above.\n1. How does the paper position itself compared to the related work above? This includes a potential discussion on unsupervised vs. weakly supervised methods.\n\n2. What is the influence of the accuracy of the input optical flow and segmentation on the  performance of the method?\n\n3. It is somewhat difficult to attribute the performance gains over [42] as the method uses a different architecture as well as a different formulation. What influence do the proposed architecture changes have on the final performance?\n\n4. In table 1, why is the supervised baseline called “supervised oracle”? It is useful to compare to a supervised baseline, but what makes it an oracle in addition?\n", "limitations": "Some limitations are discussed in the appendix. It could be helpful to expand them with actual examples from the results.\nSocietal impact is discussed only very briefly: one line in the appendix. This could be expanded considerably. It is important to discuss that these models often hallucinate segmentations of parts of objects that are never actually visible and thus cannot be trusted.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\nThe paper has a clear motivation for learning amodal segmentation without amodal supervision. The writing is clear and the explanations are comprehensive.\n\nThe method itself is intuitive, by using the assumption that cumulatively, over the course of a video most of the object will be visible.\n\nThe results show good performance as well as the importance of test-time fine-tuning and generalization to unseen objects and sequences.\n\n\nWeaknesses\nPositioning and comparisons\nThe paper is missing a discussion and positioning relative to many recent works in the area:\n-Amodal segmentation\nNguyen, Khoi, and Sinisa Todorovic. \"A weakly supervised amodal segmenter with boundary uncertainty estimation.\" ICCV 2021, performs image-based amodal segmentation.\nLing, Huan, et al. \"Variational amodal object completion.\" NeurIPS 2020, does amodal completion per object instance in single frames.\nZhou, Qiang, et al. \"Human De-occlusion: Invisible Perception and Recovery for Humans.\" CVPR 2021, learns occlusion for people.\nZheng, Chuanxia, et al. \"Visiting the Invisible: Layer-by-Layer Completed Scene Decomposition.\", IJCV 2021\n-Weakly supervised instance segmentation\nArun, Aditya, C. V. Jawahar, and M. Pawan Kumar. \"Weakly supervised instance segmentation by learning annotation consistent instances.\" ECCV 2020\nThe setup is quite similar to weakly supervised amodal segmentation - of course here applied to videos instead of images - so it would make sense to also classify this work as weakly supervised (uses segmentation and flow input) instead of unsupervised. \n\nInfluence of input accuracy\nAs the method uses optical flow and instance segmentation input, it would be interesting to understand the dependency of the method on the accuracy of that input. Instead of using the GT on synthetic datasets, one could utilize a sota (supervised) method to estimate flow and instance segmentation and compare the results to GT inputs. As the paper is making a case for _unsupervised_ amodal segmentation it could be achieved through the use of unsupervised instance segmentation and unsupervised optical flow methods, however likely at the cost of performance as these methods fall behind their supervised counterparts. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "7Y5pt4YZV-", "reviewer": "Reviewer_cqVs", "summary": "In this work, the authors mainly consider the amodal segmentation task for moving object in a video. hey propose a self- supervision strategy and a test-time adaptation pipeline for it. It achieves SOTA results on the FISHBOWL and KINS-Video-Car benchmark.", "questions": "My main concern is could the authors ease the strong assumption if would still continue in this direction. Maybe the authors could provide the total number of the cases under the strong assumption. Or could provide one option for the rest cases? ", "limitations": "The paper misses on some important baseline and comparisons in Structured generative model space, that do AMODAL RGB prediction and segmentation on videos, such as PARTS Kabra et al, Scalor (https://arxiv.org/abs/1910.02384), Flow Capsules (https://arxiv.org/abs/2011.13920) and SQAIR (https://arxiv.org/abs/1806.01794). ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This paper is well written and easy to follow. The method's novelty is based on the intuition that the occluded part of an object can be explained away if that part is visible in other frames. The core idea makes sense and is novel in some, even in most cases. But according to the Theoretical Analysis on The Loss Design of SaVos in the supplementary material, ‘… with the following assumptions, we claim that the region covered, … 3) each pixel of this object is visible in at least one frame. …’ the assumption might be too strong and it is easy to point out the opposite examples, such as Fig. 3 and the video in the supplementary material. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "5gJC6P5Srtw", "reviewer": "Reviewer_w3Qn", "summary": "The paper proposes a self-supervised method (SaVos) for amodal segmentation that tracks shapes of partially occluded objects in video frames to learn the complete (amodal) shape. Given visible modal segmentation masks, the method computes amodal masks and objects’ motion by relying on both spatiotemporal and type priors. The method generalizes well by applying one-shot test-time adaptation given that the focus is on the spatiotemporal prior in order to retrieve the output. The type prior is however useful in scenarios where some parts of the objects are never visible.", "questions": "-", "limitations": "The paper does not address the limitaitons of the method.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n+ The paper is very well written paper. The explanations are complete and make the paper is easy to follow.\n+ Amodal segmentation is an arduous task and the key ideas to solve it are very intuitive and relevant.\n+ Although the method is a combination of well known techniques, it is valuable in the the sense that there are improvements compared to the presented SotA even after a drastic distribution shift.\n+ The connection to human vision in the first paragraph of the introduction is inspiring and informative.\n\nWeaknesses\n\n- The related work is not adequately mentioned or compared with. It should at least include [A] Ling et al. (NeurIPS 2020) and [B] Sun et al. (CVPR 2022). In the experiments, the baseline used to compare the results should be Ling et al. in addition to Zhan et al. (CVPR 2020), since it is a newer and better performing model. Moreover, similarly to what has been done in Sun et al., it would be more informative to present results as a function of the different occlusion levels, which would enable studying the effect of increasing amount of partial occlusion.\n- In the experiments, the authors do not compare to any method that uses temporal information, which gives the proposed model an important advantage over related work. The authors state in lines 41-42 “traditional optical flow […] require[s] explicit visual correspondence across pixels, which would be impossible due to occlusions”. However, I actually think that aggregating optical flow across frames into a complete object shape would be the proper baseline for the proposed mode (also including the input provided, i.e. assuming rigid objects, slowly moving object, ground-truth bounding box and visible segmentation)?\n- The proposed method makes very strong assumptions regarding the supervision at training time. In particular, the model assumes that the object is perfectly discovered, including ground-truth bounding box, instance label and visible segmentation mask. Therefore, referencing this approach as self-supervised is somehow misleading. Moreover, these assumptions are rather unrealistic in real world scenarios, therefore the experiments should study how sensitive the proposed method when those inputs are predicted, e.g. using a Faster-RCNN model. \n- The authors state in lines 275-276 “occluded part only occupies a small portion of the entire full mask, thus mean-IoU on the occluded part is a better indicator on the model performance”: This does not mention that this indicator is also significantly more sensitive to noise. Results should include standard deviation estimates to ensure that the uncertainties are considered when comparing results.\n- It would be worthwhile to see how does this method generalizes to additional categories i.e. see how much the one-shot test-time adaptation to a new domain “unlearn” previous learned type priors, e.g. performing training on FISHBOWL, adaptation on KINS-Video-Car and re-evaluation on FISHBOWL would give enlightening clues about the generalization capacity of the method.\n- The authors state in the Supplementary material lines 140-141: “this makes it easy to squeeze out type information since type is a concise representation”: Were there any experiments performed to support this statement? It would be interesting to underline the possibility to apply classification or clustering to see if it is possible to extract type information as it was mentioned.\n- On what grounds are these statements based on? “reasonably learned” (l. 8-9), “as long as the deformation can be reasonably learned” (l. 56), “This module also learns reasonable deformation over time” (l. 216) \n- The paper mentions that the SAIL-VOS dataset is not used because it “has frequent camera switches” (l.79). It would still be interesting to study the performance of the model on that dataset. How long does a video sequence need to be for the proposed model to perform well compared to related work?\n\nMinor:\n\n- Concerning the alpha channel in the bi-direction prediction, does it help only at the beginning when the cold start problem is present or is it also useful at any point after? An analysis of the weights distribution of this channel to further detail its influence would be a great added value.\n- Reference to figures, equations and sections (especially in sec. 3.1) are inconsistent and sometimes missing (l. 175 and l.178). Please make sure to precise what you are referring to in order to ensure clarity.\n- The paper mentions that a limitation of related works is that they need to perform object discovery (l.  87.). At that location in the text it should be mentioned that this work assumes that the object is perfectly discovered, including ground-truth bounding box, instance label and visible segmentation mask.\n\n============================ POST REBUTTAL==========================\nAfter reading the other reviews and the rebuttal I increase my score to 6 and vote for accepting the paper. I very much appreciate the detailed comments of the authors which addressed my concerns and provide important clarifications regarding the points that I misunderstood. I think this is a solid work with moderate contribution. Remaining weaknesses to me are that it was mostly studied on rather simple video datasets and does not generalize to image collections such as COCO-A or OccludedPASCAL3D+ that are standard evaluation benchmarks in related works, due to the necessity for video data.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "3LWwU-6s96y", "reviewer": "Reviewer_k3zX", "summary": "**Problem**: This paper addresses the problem on predicting amodal segmentation masks of objects using full sequences of videos depicting the object in different states of occlusion. \n\n**Solution**: In this work, a novel model is proposed to address this problem which is trained in a self-supervised manner. For each frame, the proposed model predicts an amodal mask and an estimated motion of the object. These outputs are used jointly to estimate the mask in the subsequent frame. The paper proposes a novel objective that provides supervision using the estimated mask on the subsequent frame and the known modal mask on the subsequent frame. \n\n\n", "questions": "Some design choices of the proposed method are not entirely explained or intuitive:\n- As explained in Line 182-185, usage of the LSTM based model leads to the \"cold start problem\" *i.e.* amodal predictions on the first few frames are going to be poor due to lack of prior video data. While a workaround has been proposed, wouldn't a more intuitive choice be a transformer-like architecture or even a bidirectional LSTM?\n- Why is $\\delta V$ necessary? It seems like $V$ and $I$ for each frame should be sufficient to estimate the amodal masks. It would be interesting to see this ablation. \n- In addition to the previous question, is the image patch also really necessary to perform this inference? In theory, it should be possible to complete masks given long enough/diverse enough sequences of masks only. ", "limitations": "The limitations have not been thoroughly discussed. It would be useful to see at least a few examples of failure cases.\nThe societal impact also has not been discussed.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\n### Writing\nThe text of the paper is very well organize and clearly written. The motivation and inspiration for the work have been explained in great detail. To the best of my knowledge, the related work section thoroughly covers the diversity of relevant literature and discusses the relationship to the proposed work. The details of the proposed model and training methodology has been concisely explained. \n\n### Novelty \nThe architecture of the proposed model is very intuitive. A CNN is used to encode the modal mask, motion and image patch in each frame. An LSTM is used to capture the spatio-temporal relationships. And finally, an auto-encoder is used to predict the amodel mask using the modal mask and the LSTM features. To the best of my knowledge, this is a novel architecture designed specifically for the task of amodal object segmentation. \n\nOne of the key novel contributions of this work is the self-supervised objective that takes advantage of modal masks as supervision to estimate amodal masks. I believe this contribution might be of interest to researchers in this domain and possibly could be extended to other domains. \n\n### Results\n\nThe results demonstrate that the proposed model can outperform existing work and produce better amodal segmentations. The experimental evaluation is very thorough and covers multiple datasets for this task. Furthermore, the paper presents results using test-time adaptation to perform inference on samples outside of the training distribution. These results demonstrate that the proposed method is well-suited for test-time adaptation and demonstrates improved generalization compared to existing work. \n\n\n## Weaknesses\n\nOverall there are no major weaknesses of the proposed approach. However, some of the design choices are not entirely explained. I have listed these questions in the Questions section below.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept