Learning a Spatio-Temporal Embedding for Video Instance Segmentation
Authors: Anthony Hu
Alex Kendall
Roberto Cipolla
Venue:  (2020)
Keywords: computer
vision
video
instance
segmentation
metric
learning

Abstract: Understanding object motion is one of the core problems in computer vision. It requires segmenting and tracking objects over time. Significant progress has been made in instance segmentation, but such models cannot track objects, and more crucially, they are unable to reason in both 3D space and time.
We propose a new spatio-temporal embedding loss on videos that generates temporally consistent video instance segmentation. Our model includes a temporal network that learns to model temporal context and motion, which is essential to produce smooth embeddings over time. Further, our model also estimates monocular depth, with a self-supervised loss, as the relative distance to an object effectively constrains where it can be next, ensuring a time-consistent embedding. Finally, we show that our model can accurately track and segment instances, even with occlusions and missed detections, advancing the state-of-the-art on the KITTI Multi-Object and Tracking Dataset.

Under review as a conference paper at ICLR 2020
LEARNING A SPATIO-TEMPORAL EMBEDDING FOR VIDEO INSTANCE SEGMENTATION
Anonymous authors Paper under double-blind review
ABSTRACT
Understanding object motion is one of the core problems in computer vision. It requires segmenting and tracking objects over time. Significant progress has been made in instance segmentation, but such models cannot track objects, and more crucially, they are unable to reason in both 3D space and time. We propose a new spatio-temporal embedding loss on videos that generates temporally consistent video instance segmentation. Our model includes a temporal network that learns to model temporal context and motion, which is essential to produce smooth em- beddings over time. Further, a self-supervised loss, as the relative distance to an object effectively constrains where it can be next, ensuring time-consistent embedding. Finally, we show that our model can accurately track and segment instances, even with occlusions and missed detections, advancing the state-of-the-art the KITTI Multi-Object and Tracking Dataset.
model also estimates monocular depth, with
INTRODUCTION
Explicitly predicting the motion of actors a dynamic scene is a critical component of intelligent systems. Humans seamlessly track moving objects in their environment by using cues such as appearance, relative distance, and temporal consistency. The world is rarely experienced in a static way: motion (or its absence) provides essential information understand scene. Similarly, incor- porating past context through temporal model is essential to segment and track objects consistently over time and through occlusions.
From a computer vision perspective, understanding object motion involves segmenting instances, estimating depth, and tracking instances over time. Instance segmentation, which requires seg- menting individual objects at the pixel level, has gained traction with challenging datasets such as COCO (Lin et al. 2014), Cityscapes (Cordts et al., 2016) and Mapillary Vistas (Neuhold et al.) 2017]. Such datasets, which only contain single-frame annotations, do not allow the training of video models with temporally consistent instance segmentation, nor does it allow self-supervised monocular depth estimation, that necessitates consecutive frames. Yet, navigating in the real-world involves a three-dimensional understanding of the other agents with consistent instance segmen- tation and depth over time. More recently, new dataset containing video instance segmentation annotations was released, the KITTI Multi-Object and Tracking Dataset (Voigtlaender et al.) 2019). This dataset contains pixel-level instance segmentation on more than 8,000 video frames which effectively enables the training of video instance segmentation models.
In this work, we propose new spatio-temporal embedding loss that lears map video-pixels to a high- nigh-dimensional space") This space encourages video- pixels of the same instance to be close to- gether and distinct from other instances. We show that this spetoo-temporal embedding loss, jointly with a deep temporal convolutional neural network and self-supervised depth loss, produces consis- tent instance segmentations over time. The embedding accumulates temporal context thanks to the temporal model, as otherwise, the loss would only be based on appearance. The temporal model is a causal 3D convolutional network, which is only conditioned on past frames to predict the current embedding and is capable of real-time operation. Finally, we show that predicting depth improves the quality of the embedding as knowing the distance to an instance constrains its future location given that objects move smoothly in space.
See video demo of our model here: https//yoatuub///qRRXXXUII

---

Under review as a conference paper at ICLR 2020
Figure I: An illustration of our video instance segmentation model. Clockwise from top left: input image, predicted multi-object instance segmentation, visualisation of the high-dimensional embed- ding and predicted monocular depth.
To summarise our novel contributions, we:
introduce a new spatio-tempord embedding loss for video instance segmentation,
show that having temporal model improves embedding consistency time, improve how the embedding disambiguates objects with a self-supervised monocular depth loss,
handle occlusions, contrary to previous IoU based instance correspondence.
We demonstrate the efficacy of our method by advancing the state-of-the-art on the KITTI Multi- Object and Tracking Dataset (Voigtlaender al. 2019). An example of our model's output given by Section !
RELATED WORK
Two main approaches exist for single- single-image instance segmentation: region-proposal based (He al.
2017
Hu et al. 20188 Chen et al. 2018 Liu et al. (018)) and embedding based (Brabandere et al.
2017 Kong& & Fowlkes 2018] Fathi al. [217]] K al. 2018). The former method relies on
region of interest proposal network that first predicts bounding boxes then estimates the mask of
the object inside that bounding box. With such strategy, one pixel can belong to the overlap of many
bounding boxes, and is largely unclear how correspondence between pixels can be Iearned. We instead favour the embedding based method and extend to space and time.
Capturing the inter-relations of objects using multi-modal cues (appearance, motion, interaction) is difficult, as showcased by the Multi-Object Tracking (MOT) challenge (Xiang et al.) 2015). MOT's goal is infer the trajectories of objects and cover wide range of applications such as biology (birds (Luo et al. 2014), fish (Spampinato et al. 2008), robot navigation (Elfes 1989)) and au-
tonomous driving (Petrovskaya & Thrun 2009 Ess al. 2009)). [Sadeghian al. (2017) and Son et al. (2017) learned representation of objects that follows the ttracking-by-detettiee by-detection" paradigm where the goal
connect detections across video frames by finding the optimal assignment of graph-based tracking formulation (i.c. each detection is node, and an edge is the similarity score between two detections).
Collecting large-scale tracking datasets is necessary to train deep networks, but that process is expen- sive and consscnnu,, Vondrick et al. (2018) introduced video colourisation self-supervised method to learn visual tracking. They constrained the colourisation problem of grayscale image by learning to copy colors from reference frame, with the pointing mechanism of the model acting tracker once it is fully trained. The colourisation model is more robust than optical flow based models, especially in complex natural scenes with fast motion, occlusion and dynamic backgrounds.
Voigtlaender et al. (2019) extended the task of multi-object tracking to multi-object tracking and segmentation (MOTS), by considering instance segmentations as opposed to 2D bounding boxes. Motivated by the saturation of the bounding box level tracking evaluations [Pont-Tused al. (017), they introduced the KITTI MOTS dataset, which contains pixel-level instance segmentation on more than 8,000 video frames. They also trained model which extends Mask R-CNN (He al. 2017 by
2

---

Under review as a conference paper at ICLR 2020
incorporating 3D convolutions to integrate temporal information, and the addition of an association head that produces an association vector for each detection, inspired from person re-identification (Beyer et al. 2017). The temporal component of their model, however, is fairly shallow (one two layers), and is not causal, as future frames are used to segment past frames. More recently, Yang al. (2019) collected large-scale dataset from short YouTube videos (3-6 seconds) with video instance segmentation labels, and Hu et al. (2019) introduced a densely annotated synthetic dataset with complex occlusions to learn how estimate the spatial extent of objects beyond what is visible.
3 EMBEDDING-BASEE VIDEO INSTANCE SEGMENTATION LOss
Contrary methods relying on region proposals (He et al. 2017 Chen al. 2018), embedding- based instance segmentation methods map all pixels of a given instance to a high dimensional space with desirable properties. This overcomes several limitations of region-proposal methods. Firstly, two objects may share the same bounding box and in that situation, it is ambiguous which object mask the model should segment. Secondly, pixels can belong to two separate objects as each pre- diction is done independently. Finally, the number of detected objects is limited by the fixed number of proposals the network.
We propose spatio-temporal embedding loss that extends Brabandere et al. (2017)*s 's instance em- bedding loss video: each pixel belonging to a given instance space and time transformed into unique location in a high dimensional space, using cues such as appearance, context and motion.
More concretely, three terms are used in the loss: the attraction loss (Equation (T) to ensure pix-
els from the same instance are close to each other, the repulsion loss (Equation 2 to ensure two
separate instances are far from each other and a regularisation term (Equation so that instance
centers should not diverge too much from the origin.
Let us denote the number instances, K, and the subset of indices, S₁, corresponding to all the pixels belonging to instance k in the video. vi Sk. V/ is the embedding for pixel position and pa
the mean embedding of instance k: jı [8
Eies, Vi-
C.
max (0, I|ue 9ill2 - paa)
(1)
K
Jm
YESt
C.
=
max(0, 0,2pp- - ||kk, - Mkz ll2)
(2)
K(K 1)
K
Lug K
Σ
(3)
Where Pa denotes the attraction radius within a cluster: we want the embedding to be within Pa of the centroid. 2p, denotes the repulsion radius: we want the centroids of two different clusters be at least 2p, apart. Therefore, if we set > 2pa. given pixel embedding of a cluster will be closer all the pixel embeddings its cluster than any other pixel embedding.
The spatio-temporal embedding loss is the weighted sum of the attraction, repulsion and regularisa- tion losses:
Linstance = X1 + XzC, + Az
(4)
During inference, each pixel of the considered frame is assigned to an instance by randomly picking unassigned point and aggregating close-by pixels with the mean-shift algorithm (Comaniciu & Meer] 2002). In the ideal case, with a test loss zero, this will result in perfect clustering the repulsion radius, Pr, twice large as the attraction radius, Pa.
3

---

Under review as a conference paper at ICLR 2020
3.1 SELF-SUPERVISED DEPTH ESTIMATION
The relative distance of objects is a strong cue to segment instances in space and time, as the motion of objects is temporally smooth. Knowing the previous distance of object relative the camera assists tracking as the future position will be constrained by the object's 's current location.
Depth estimation with supervised methods requires a vast quantity of high quality annotated data, which is challenging to acquire in range of environments as laser measurements can be imprecise in natural scenes with motion and reflections. Because we have access to video in our instance segmentation dataset, we can leverage self supervisd depth losses from monocular videos, where the supervision comes from consecutive temporal frames. In addition to predicting the depth map, ego-motion also has to be inferred, but only during training to constrain the depth network.
Following Zhou al. (017)) and Godard al. (2019). we train a depth network with a separate pose estimation network with the hypothesis during training that scenes are mostly rigid, therefore assuming appearance change is mostly due to camera motion. The pixels that violate this assumption masked from the view synthesis loss, as they otherwise create infinite holes during inference for objects that are typically seen motion during training - more details in Appendix A. The training signal comes from novel view synthesis: generation of new images of the scene from different camera pose. Let us denote by (I, I2...,, Ir) sequence of images, with target view I, and source view I.. The view synthesis loss is given by:
Lue Ee(lesiem)) 8ft
(5)
with l-- the synthesised view I, from source image J, using the predicted depth Dj and the pre- dicted 4 x camera transformation Tr predicted from the separate pose network. The projection error function, e, is deseribed by Zhao al. (2017) weighted sum of L1, Structural Similarity Index (SSIM) and a smoothness regularisation term. Let us denote by p. the coordinate a pixel in the target image I, in homogeneous coordinates. Given the camera intrinsic matrix, K, and the mapping from image plane to camera coordinate, the corresponding pixel in the source image is provided by:
Ps Ki-ssp(K-))p (p.)))
The projected coordinates p. continuous values, the Spatial Transformer Network (Jader- berg et al., 2013) sampling mechanism to bilinearly interpolate the four neighbouring pixels to pop- ulate the reconstructed image /.--t-
Some pixels are visible in the target image, but are not in the source images, leading to a large projection error. As advocated by Godard al. (2019), instead of summing, taking the minimum projection error greatly reduces artifacts due to occlusion and results in sharper predictions. The resulting view synthesis loss
C.. = min m f.-1)
syt
The resulting video instance embedding loss is the weighted sum of the attraction, repulsion, regu- larisation and geometric view synthesis losses:
Linstance = X1 X2C, + AzGreg + Aahrs
(8)
MODEL ARCHITECTURE
Our model contains three components: an encoder, temporal model and the decoders. Each frame first encoded to more powerful and compact representation, then the temporal model learns the dynamics of the scene, and finally, the decoders output the instance embedding and depth prediction illustrated by Figure
4

---

Under review as a conference paper at ICLR 2020
Convolutional Encoder
Causal 3D Convolutions
Convolutional Decoders
Inputs
Instance Segmentation
Embedding
Ct
t
Depth
It 1
Tt
Our video instance segmentation and depth model architecture. The embedding, 2 is trained to explicitly encode appearance, motion and geometry cues in order to predict an instance embedding and monocular depth prediction.
Figure 2:
Encoder. We use ResNet-18 (He et al. 2016) with 14.6M parameters our encoder, which allows the network to run in real-time on sequences of images.
Temporal Model. The scene dynamics is learned with a causal 3D convolutional network com- posed of blocks of 3D residual convolutions (convolving in both space and time). For given time index, t, the network only convolves over images from indices s < to compute the temporal repre- sentation 2, It therefore does not use future frames and is completely causal. The temporal model does not decimate the spatial dimension of the encoding, but slowly accumulates information over time from the previous encodings 2, with S t. The temporal model is trained efficiently with convolutions as all input images are available during training, enabling parallel computations with GPUs. However, during inference, the model inherently has be sequential, but be made signif- icantly faster by caching the convolutional features over time and eliminating redundant operations, as proposed by Paine al. (2016) for WaveNet (Oord al.) (0166).
Decoders. The decoders then map the temporal encoding स to its instance embedding y of dimension p x height x width, with the embedding dimension, and depth de of dimension height width. The embedding values belonging to the same instance are pushed together in the high- diteemiiensii space IR, and pulled away from the other instances, over the whole video. Therefore, tracking instances simply requires comparing the mean embedding of newly segmented instance with previously segmented instances. A distance lower than Pr indicates match.
To segment the instances, we first run background mask network (trained separately) then we cluster the segmented embeddings using mean shift to discover dense regions of embeddings. Over time, the embeddings are accumulated up until the sequence length corresponding to the sequence length used during training to constrain instances spatio-temporally. This creates increasingly dense regions over time resulting in better clustering. To ensure that embeddings a particular instance smoothly vary over time, the embeddings have life span corresponding to the sequence length of the model.
Pose and Mask Model. For the pose network we use ResNet and for the mask network we use an encoder-decoder model, also based ResNet. Further details are in Appendix A.T A.I
EXPERIMENTS
Next describe experimental evidence which demonstrates the performance of our method by advancing the state-of-the-ait on the KITTI Multi-Object and Tracking Dataset (Voigtlaender et al, 2019).
5

---

Under review as a conference paper at ICLR 2020
5.1 DATASET
The KITTI Multi-Object Tracking and Segmentation (MOTS) dataset (Voigtlaender al. 2019) contains 8,008 frames with instance segmentation labels resulting in a total of 26,899 annotated cars. It is composed of 21 scenes with resolution of 375 x 1242 with consistent instance ID labels across time, allowing the training of video instance segmentation models. The frames are annotated 10 frames per second, which is suitable for self-supervised monocular depth prediction.
Scenes
Frames
Annotations 18,831
Avg. frames
Avg. annotations
Train
12
5,027
419
1,569
Validation
2,981
8,068
331
896
Table Details of the KITTI Multi-Object Tracking and Segmentation (MOTS) dataset.
The ApolloScape dataset (Huang al. 2018) also contains video instance segmentation labels for 49,287 frames, but the annotations are not consistent in time, rendering the training of a temporal model impossible. NuScenes (Caesar al. 2019) features 1,000 scenes of 20 seconds with annota- tions at 2Hz in a diverse range of environments (different weather, daytime, city) but only contains bounding box labels, failing to represent the fine-grained details of instance segmentation. Tempo- ral instance segmentation is also available on short snippets of the DAVIS dataset (Pont-Tuset et al., 2017), but each snippet is recorded by different camera and is short effectively learn depth model. For this reason, we focus on the KITTI MOTS dataset - it is the only dataset that contains consistent video instance segmentation sufficient quantity to train deep models.
5.2
PAREAR-RRAMESERS
We halve the input images our encoder use an input RGB resolution of 3 192 640. The resulting encoding is 128 x 24 x 80. The decoders then map the temporal encoding to its instance embedding Y/ of dimension p x 192 640, with the embedding dimension, and depth d, of dimension x 192 640. Except for the experiments in Table we train with a sequence length of which corresponds to 0.5 seconds temporal context since videos are 10Hz.
In the loss function, we set the attraction radius Pa 0.5 and repulsion radius Pr 1.5. We weight the losses with attraction and repulsion loss weight A = A22 1.0, regularisation loss X3 = 0.001 and depth loss x 1.0.
5.3 METRICS
Let us define multi-object tracking and segmentation metrics, which measures the quality of the segmentation as well as the consistency of the predictions over time. Contrary to bounding box detection, where ground truth box may overlap with several predicted boxes, in instance segmen- tation, since each pixel is assigned to at most one instance, only one predicted mask can have an Intersection Union (loU) larger than given threshold with given ground truth mask. Let us denote by H the set of predicted ids, M the set of ground truth ids and the mapping from hypothesis masks ground truth masks. g H M U is defined as:
(h)) aargmax, loU(h. if max, IoU(h, m) > threshold otherwise
0,
We thus define:
True positives TP {h E IHI (h)) + 0}. correctly assigned predicted masks.
False positives as: FP = (h IHI gh)) ()) = 0). predicted masks not assigned to any ground
truth mask.
False negatives as: FN = {m E Ml (m) 6). ground truth masks not covered by any hypothesis mask.
Soft number of true positives: TP Eberr loU(h,g(h))

---

Under review as a conference paper at ICLR 2020
Let the function pred: M MU U 0 map ground truth mask to its latest tracked predecessor (0 if the ground truth mask is seen for the first time). The set IDS of ID switches is defined as the set of ground truth masks whose predecessor
tracked by different ID.
Following Voigtlaender et al. (2019), we define the following MOTS metrics: multi-object tracking and segmentation precision (MOTSP), multi-object tracking and segmentation accuracy (MOTSA) and finally the soft multi-object tracking and segmentation accuracy (sMOTSA) that measures seg- mentation well detection and tracking quality.
MOTSP [TPl TP
(10)
MOTSA [FP + FN + |IDS
TP
|FP|
|IDS|
(11)
M
M
TTP| - FFP - IDS
sMOTSA
(12)
M
We also measure the average precision (AP), i.e. the normalised area under the precision/recall curve.
5.4 RESULTS
We compare our model to the following baselines for video instance segmentation and report the results in Table
Single-frame embedding loss (Brabandere et al. 2017), previous state-of tfetherrrt method where instance segmentations are propagated in time using intersection-over-nee associ- ation.
Without temporal model, spatio-temporal embedding loss, without the temporal model.
Without depth, temporal model and spatio-temporal embedding loss, without the depth loss.
We also report the results of Mask R-CNN (He et al. 2017 and Track R-CNN (Voigllenner al, 2019 even though direct comparison with the latter is not possible as their model was pretrained on Cityscapes and Mapillary Vistas, and is not causal they use future frames to predict the current frame instance segmentations.
KITTI only
Causal
MOTSA
SMOTSA
MOTSP
AP
recall
precision
Brabandere et al.
2017
0.575
0).423
0.803
0.612
0.770
0.841
Mask R-CNN
0.584
0.455
0.833
0.646
0.771
0.875
Voigtlaender et al. 2019
X
X
0.671
0.540
0.831
0.656
0.775
0,894
Without temporal model
0.582
0.426
0.799
0.607
0.777
0.835
Without depth
0.591
0.433
0.801
0.614
0.795
0.822
Ours
0.613
0.461
0.801
0,600
0.764
0.839
Table 2: KITTI MOTS validation set results comparing our model with baseline approaches.
The static detection metrics (average precision, recall, precision) are evaluated image by image without taking into account the temporal consistency of instance segmentations. As the compared models (Without temporal model, Without depth, Ours) are all using the same mask network, they show similar performance terms of detection. However, when evaluating performance on met- rics that measures temporal consistency (MOTSA and sMOTSA), best model shows significant improvement over the baselines.
The variant without the temporal model performs poorly as it does not have any temporal context to learn a spatio-temporal embedding and therefore only relies on spatial appearance. The temporal model on the other hand learns with the temporal context and local motion, which results in better embedding. Our model, which learss to predict both a spatio-temporal embedding and monocular depth, achieves the best performance. In addition to using from appearance and temporal

---

Under review as a conference paper at ICLR 2020
context, estimating depth allows the network to use information from the relative distance of objects to disambiguate them. Finally, we observe that our model outperforms Mask R-CNN (He et al. 2017] on the temporal metrics (MOTSA and sMOTSA) even though the latter exhibits higher detection accuracy, further demonstrating the temporal consistency quality our spatio-temporal embedding.
5.4.1 ANALYSIS OF CLUSTERING AND MASK SEGMENTATION.
Our model first relies on the mask segmentation to isolate which pixel locations to consider for instance clustering. We evaluate the impact of using the ground truth mask against our predicted mask in Table 3 The performance gain is significant, hinting that better instance segmentation would be possible by improving the mask network.
Next, we evaluate the effect of clustering. In the best scenario, the validation loss would be zero, and the clustering would perfect using the MeanShift algorithm. However, this scenario is unlikely and the clustering algorithm is affected by noisy embeddings. We evaluate the effect of this noise by clustering with the ground-truth mean for each instance, by thresholding with Pr around the ground truth instance embedding mean. This also results in a boost in the evaluation metrics, but most interestingly, a model that uses both ground truth instance embedding mean clustering and ground truth mask performs worse than model segmented with ground truth mask and our clustering algorithm. This is because our clustering algorithm accumulates embeddings from past frames and therefore creates attraction force for the mean shift algorithm that enables the instances to be matched more consistently.
GT Mean
GT Mask
MOTSA
sMOTSA
MOTSP
AP
Recall
Precision
X
0.613
0,461
0,801
0.600
0.764
0.839
X
0.700
0.616
0.897
0.691
0.816
0.886
X
/
0.804
0.751
0.936
0.786
0.832
0.971
/
0.714
0.644
0.915
0.710
0.824
0.892
Table 3: Comparing the effect noisy against ground-truth clustering and mask segmentation on the KITTI MOTS dataset.
5.4.2 EFFECT OF SEQUENCE LENGTH
Our model learns spatio-temporal embedding that clusters video-pixels from a given instance. Correspondence instances between frames is achieved by matching detected instances to previous instances if the embedding distance is below the repulsion radius, Pa- Instance tracking can occur for an arbitrarily long sequence of time, as long as the embedding changes smoothly over time, which is likely the case temporal context and depth must evolve gradually. However, when the pratio-tempora embedding is trained over sequences which are too long, the embedding learning collapses. This is because the attractive loss term is detrimental between distant frames, it pressures pixels from the same instance to be have corresponding embeddings when their appearance and depth no longer similar. It also suggests our model is able reason over lower order motion cues more effectively than longer term dynamics. This is seen experimentally in Table
Length MOTSA
3
5
7
10
15
0.575
0.590
0.613
0.555
0.538
0.402
sMOTSA
0.423
0.435
0.461
0.402
0.398
0.273
MOTSP 0.803
0.810
0.801
0.788
0.792
0.783
Table 4: Influence of the sequence length on model performance. This indicates that our model can learn short-term motion features effectively, but not long-term cues. We reason that this is because over longer sequences, the loss prevents the embedding smoothly shifting, which naturally occurs to changing pose, appearance, context and lighting in the scene. We find the optimum sequence length on this dataset to be five.
8

---

Under review as a conference paper at ICLR 2020
5.5 QUALITATIVE EXAMPLES
The instance segmentation of our model is consistent across frames as instances are clustered in both space and time. This provides demonstrate this with the following scenarios showing tracking through partial (Figure 3 and full occlusion (Figure 5 as well as continuous tracking through noisy detections (Figure 4 Additional examples and failure cases of our model are shown in Appendix A.2 and video demo can be viewed here: https:/yyoutubee/qqPXXUUOO
robust clustering compared a per-frame approaches. We
In each example, we show from left to right: RGB input image, ground truth instance segmentation, predicted instance segmentation, embedding visualised in 2D, embedding visualised RGB and predicted monocular depth. The embedding is visualised in 2D with the corresponding mean shift clustering. Each color represents a different instance, the inner circle is the attraction radius of instance mean embedding, and the outer circle is the repulsion radius of each instance. Additionally, we also visualise the embedding spatially in 3D, by projecting its three principal components to an RGB image.
We show in Appendix A.2 that incorporating depth context greatly improves the quality of the embedding, especially in complex scenarios such partial or total occlusion. We also observe that the embedding is much more structured when incorporating 3D geometry information.
CONCLUSIONS
We proposed: a new spatio-temports embedding loss that generates consistent instance segmentation time. The temporal network models the past temporal context and the depth network constrains the embedding to aid disambiguation between objects. We demonstrated that our model could ef- fectively track occluded instances instances with missed detections, by leveraging the temporal context. Our method advanced the state-of-the-art at video instance segmentation on the KITTI Multi-Object and Tracking Dataset.

---

Under review as a conference paper at ICLR 2020
Figure 3: Partial occlusion. The segmented brown is correctly segmented even when being partially occluded by the segmented red car, as the embedding contains past temporal context and is aware of the motion of brown car.
Figure 4: Continuous tracking. The segmented pink and purple cars are accurately tracked even with missing detections.
Figure 5: Total occlusion. The segmented green car correctly tracked, even though was com- pletely occluded by another car.
10

---

Under review as a conference paper at ICLR 2020
REFERENCES
Lucas Beyer, Stefan Breuers, Vitaly Kurin, and Bastian Leibe. Towards principled integration of multi-camera re-identification and tracking through optimal bayes filters. In arXiv: /705. 1755006705.046688 2017.
Bert De Brabandere, Davy Neven, and Luc Van Gool. Semantic instance segmentation with discriminative loss function. CVPRw, 0as//708.025511 2017.
Holger Caesar, Varun Bankiti, Alex H. Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: multimodal dataset for autonomous driving. arXiv preprint arXh::1903.11022 2012,
Liang-Chich Chich Chen, Alexander Hermans, George Papandreou, Florian Schroff. Peng Wang, and Hartwig Adam. Masklab: Instance segmentation by refining object detection with semantic and direction features. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018.
Dorin Comaniciu and Peter Meer. Mean shift: A robust approach toward feature space analysis. IEEE Transactions on Pattern Analysis & Machine Intelligence, pp. 603-619, 2002.
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proc. of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2016.
A. Elfes. Using occupancy grids for mobile robot perception and navigation. Computer, 22(6), June 1989.
Andreas Ess, Konrad Schindler, Bastian Leibe, and Luc Van Gool. Improved multi pellio tracking with active occlusion handling. In Proceedings IEEE ICRA 2009 workshop on people detection and tracking. 2009.
Alireza Fathi, Zbigniew Wojna, Vivek Rathod, Peng Wang, Hyun Oh Song, Sergio Guadarrama, and Kevin P. Murphy. Semantic instance segmentation via deep metric learning. ArXiv, Abs/1703.102777 2017.
Clément Godard, Oisin Mac Aodha, Michael Firman, and Gabriel J. Brostow. Digging into self-supervised monocular depth prediction. The International Conference Computer Vision (ICCV), October 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. CVPR, 2016.
Kaiming He, Georgia Gkioxari, Piotr Dollar, and Ross Girshick. Mask R-CNN. In Proceedings of the International Conference on Computer Vision (ICCV), 2017.
Yuan-Ting Hu, Jia-Bin Huang, and Alexander G. Schwing. Maskrmn: Instance level video ob- ject segmentation. NIPS, abs/1803. 11187, 2018. URL http::://bpp.mmatt.... html 4 abs 100e--8833-00 -100--11887
de/dee/eeb.e dbb
Yuan-Ting Hu, Hong-Shuo Chen, Kexin Hui, Jia-Bin Huang, and Alexander G. Schwing. Sail-vos: Semantic amodal instance level video object segmentation synthetic dataset and baselines. In CVPR, 2019.
Xinyu Huang, Xinjing Cheng, Qichuan Geng, Binbin Cao, Dingfu Zhou, Peng Wang. Yuanqing Lin, and Ruigang Yang. The apolloscape dataset for autonomous driving. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, June 2018.
Max Jaderberg, Karen Simonyan, Andrew Zisserman, and Koray Kavukcuoglu. Spatial transformer networks. In Proceedings of the 28th International Conference Neural Information Processing Systems Volume 2, NIPS' 15, pp. 2017-2023 Cambridge, MA, USA, 2015. MIT Press. URL
httt::/ddl acm. org ci tat on c ffm? dt 2969465

---

Under review as a conference paper at ICLR 2020
Alex Kendall, Yarin Gal, and Roberto Cipolla. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
Shu Kong and Charless Fowlkes. Recurrent pixel embedding for instance grouping. In 2018 Con- ference on Computer Vision and Pattern Recognition (CVPR), 2018.
Tsung-Y Yi Lin, Michael Maire, Serge J. Belongie, Lubomir D. Bourdev, Ross B. Girshick, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft coco:
Common objects in context. In ECCV,
Shu Liu, Lu Qi, Haifang Qin, Jianping Shi, and Jiaya Jia. Path aggregation network for instance segmentation. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 0759-8768, 2018.
Wenhan Luo, Tae-Kyun Kim, Bjorn Stenger, Xiaowei Zhao, and Roberto Cipolla. Bi-label propa- gation for generic multiple object tracking. In CVPR, 2014.
Gerhard Neuhold, Tobias Olimann, Samuel Rota Bulò, and Peter Kontschieder. The mapillary vistas dataset for semantic understanding of street scenes. In International Conference on Computer Vision (ICCV), 2017. (tttts:s: www.:..apllaaa....
com/ wataaattt/tteeeee setitvvitttee
Aaron den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Semior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio, 2016. /tt.http://arxxx wwwpa/iwwworg/b//////000 03499
Tom Le Paine, Pooya Khorrami, Shiyu Chang, Yang Zhang, Prajit Ramachandran, Mark A Hasegawa-Johnson, and Thomas S Huang. Fast wavenet generation algorithm. arXiv preprint 161.092611.09482.. 2016.
Anna Petrovskaya and Sebastian Thrun. Model based vehicle detection and tracking for autonomous urban driving. Auton. Robots, 26(2-3), April 2009.
Pont-Tuset, F. Perazzi, S. Caelles, P. Arbelaez, A. Workine-Hornung and L. Van Gool. The 2017 davis challenge on video object segmentation. arXiv:1704.00673 2017.
Amir Sadeghian, Alexandre Alahi, and Silvio Savarese. Tracking the untrackable: Learning to track multiple cues with long-term dependencies. In (CCV,2017
Jeany Son, Mooyeol Back, Minsu Cho, and Bohyung Han. Multi-object tracking with quadruplet convolutional neural networks. 2017 IEEE Conference on Computer Vision and Pattern Recogni- tion (CVPR), pp. 3786-3795, 2017.
C. Spampinato, Y.-H. Chen-Burger, G. Nadarajan, and R. Fisher. Detecting, tracking and counting fish in low quality unconstrained underwater videos. In Proc. 3rd Inl. Conf. on Computer Vision Theory and Applications (VISAPP). 2008
Paul Voigtlaender, Michael Krause, Aljosa Osep. Jonathon Luiten, Berin Balachandar Gnana Sekar, Andreas Geiger, and Bastian Leibe. Mots: Multi-object tracking and segmentation. In Proceed- ings IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2019.
Carl Martin Vondrick, Abhinav Shrivastava, Alireza Fathi, Sergio Guadarrama, and Kevin Murphy. Tracking emerges by colorizing videos. In ECCV, 2018.
Yu Xiang, Alexandre Alahi, and Silvio Savarese. Learning track: Online multi-object tracking by decision making. In International Conference on Computer Vision, 2015.
Linjie Yang, Yuchen Fan, and Ning Xu. Video instance segmentation. ICCV, 2019.
Hang Zhao, Orazio Gallo, luri Frosio, and Jan Kautz. Loss functions for image restoration with neural networks. IEEE Transactions on Computational Imaging, 3:47-57, 2017.
Tinghui Zhou, Matthew Brown, Noah Snavely, and David Lowe. Unsupervised learning of depth and regomootio from video. In Computer Vision and Pattern Recognition, 2017. URL hetps: v.rxxiv.oo wwwrxiv.org/abaae. 704. 07813
12

---

Under review as a conference paper at ICLR 2020
A APPENDIX
A.1 NETWORK ARCHITECTURE DETAILS
We report the details of each component of our model in this section. The number of parameters and layers of each module are in Table
Encoder
Temporal
Decoders
Pose
Mask
Parameters
14.6M
0.7M
0.4M
13.0M
14.8M
Layers
18
36
22
25
Table 5: Number of parameters and layers of each module.
Encoder. The encoder ResNet- 18 convolutional layer (He al. 2016), with 128 output chan- nels.
Temporal model. The temporal model contains residual 3D convolutional blocks, with only the first and last block convolving over time. Each residual block is the succession of: projection layer of kernel size 1 x 1x to halve the number of channels, 3D causal convolutional layer t x x3, projection layer X 1 x to double the number of channels.
We set the temporal kernel size to = 2, and the number of output channels to 128.
Decoders. The decoders for instance embedding and depth estimation identical and consist of 7 convolutional layers with channels [64, 64, 32, 32, 16, 16] and 3 upsampling layers. The final convolutional layer contains p channels for instance embedding and channel for depth.
Depth Masking. During training, we remove from the photometric reprojection loss the pixels that violate the rigid scene assumption, i.e. the pixels whose appearance do not change between only include pixels where the reprojection error is lower with the warped image Ia than the unwarped source image Ig:
adjacents frames. We set the mask
M = min c(le, 10+1 < min ((ee
Pose Network. The pose network is the succession of a ResNet-18 model followed by 4 convo- lutions with [256, 256, 256, 6] channels. The last feature map averaged to output a single 6-DoF transformation matrix.
Mask Network. The mask network is trained separately to mask the background and is the suc- cession of the Encoder and Decoder described above.
A.2 ADDITIONAL QUALITATIVE EXAMPLES
The following examples show qualitative results and failure examples of our video instance segmen- tation model on the KITTI Multi-Ohject and Tracking Dataset. From left to right: RGB input image, ground truth instance segmentation, predicted instance segmentation, embedding visualised in 2D, embedding visualised RGB and predicted monocular depth.
13

---

Under review as a conference paper at ICLR 2020
Figure 6: Video instance segmentation parked cars.
Figure 7: Video instance segmentation of other traffic.
14

---

Under review as a conference paper at ICLR 2020
Figure 8: Failure case: the vehicle segmented into two separate instances.
Figure 9; Failure case: two far-away cars segmented as instance.
15

---

Under review as a conference paper at ICLR 2020
A.3 QUALITATIVE IMPROVEMENTS WITH DEPTH
We show that our model greatly benefits from depth estimation, with the learned embedding being more structured, and correctly tracking objects difficult scenarios such as partial or total occlusion.
(a) Without depth estimation.
(b) With depth estimation
Figure 10: Without depth, the car circled in red is wrongly tracked frame and 9, while our model correctly tracks it as the network has learned consistent embedding based not only appearance, but also on 3D geometry. Also, the RGB projection of the embedding from our model is considerably better and much more structured.
16

---

Under review as a conference paper at ICLR 2020
(a) Without depth estimation.
(b) With depth estimation
Figure 11: Without depth, the cireled car merges into the red-segmented car, while our model does not as there is significant difference in depth between the two cars.
17

---

Under review as a conference paper at ICLR 2020
(a) Without depth estimation.
(b) With depth estimation
Figure 12: The model without depth is not able to handle complete occlusion, while ours can.
18


Reviews:
[{"review_id": "rkxg1rP5KH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\nThe paper presents a method to learn an embedding space for each pixel in a video that indicates the instance id of the objects. They also propose an auxiliary loss based on depth prediction to improve performance. This can be used to both segment and track objects in videos.\n\nStrengths\n1) The proposed approach is simple and general and handles the problem of occluding objects in videos. \n2) Their causal convolution architecture will be useful for other problems in videos.\n3) The authors perform a number of ablations to investigate how much each part of their solution contributes to the final performance.\n4) The paper is well-written and well-motivated.\n\nWeaknesses\n1) Comparisons to other video instance segmentation methods [1,2,3] are missing. The only comparison is done with a single-frame instance embedding method. \n2) The authors propose to predict depth as an auxiliary task. However, they do not use the predicted depth at test time. This is a missed opportunity. Difference in depth might help in identifying instances. Also, it might be worthwhile to investigate if instance segmentation is helping depth prediction.\n3) Experiments have been conducted on only one dataset. \n\n\nReferences\n[1] \"Video Instance Segmentation\" Linjie Yang and Yuchen Fan.\n[2] \"MaskRNN: Instance Level Video Object Segmentation\" Yuan-Ting Hu, Jia-Bin Huang, and Alexander G. Schwing.\n[3] \"SAIL-VOS: Semantic Amodal Instance Level Video Object Segmentation – A Synthetic Dataset and Baselines\" Yuan-Ting Hu, Hong-Shuo Chen, Kexin Hui, Jia-Bin Huang, Alexander Schwing.\n\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "rkl524DAKH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents learning a spatio-temporal embedding for video instance segmentation. With spatio-temporal embedding loss, it is claimed to generate temporally consistent video instance segmentation. The authors show that the proposed method performs nicely on tracking and segmentation task, even when there are occlusions.\n\nOverall, this paper is well-written. Section 3 clearly explains the loss functions. The main idea is not very complex, but generally makes sense. The authors mention that scenes are assumed to be mostly rigid, and appearance change is mostly due to the camera motion. I would like to see more argument about this, as there are cases if this is obviously not true; for instance, human changes pose significantly. If we limit the range of discussion to some narrow domain, such as self-driving, this might be more valid, but we may want to see some discussion about validity of this assumption.\n\nSome modules are not full explained in detail. For example, what is the background mask network? Which model was used, and how was it trained?\n\nIn experiment, the proposed method shows nice score on MOTSA and sMOTSA, but all other metrics, it is on the worse side. The authors are encouraged to discuss more about the metrics and experimental results with the other metrics as well. Other than these, the experiment was well-designed and conducted.", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "SJlSrPVAKB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper propose a video instance embedding loss for jointly tackling the instance tracking and depth estimation from self-supervised learning. \n\nPros:\n1: I think the method is moving towards the right direction that 3d geometry and 2d instance representation should be considered jointly under the scenario of video learning.\n2: The video instance embedding loss is also making sense as an extension of image instance embedding.\n\nCons: \n1: I think the major argument I have is this method is lack of technical novelty, since it is straight forward to adopt the loss of  Brabandere et.al 2017 to video cases for including pixels in the same group under ground truth tracking, and the self-supervised loss is exactly the same as previous methods. The fusion between depth and segments are relatively weak since it just ask the embedding to also decode depth, is there any further analysis of visual effect of explaining where the depth helps segments? \n\n2: In the experiments, the baseline for comparison over MOTS is fairly old, and I think it makes sense to include the number of MOTS paper, which is currently hard to align with that shown in the paper.  In Tab.2, the author only highlight the improved motion metric, while in per-frame AP the results are actually lower than the baselines. It also needs to be well explained. \n\n3: The paper claims `\"it generates temporal consistent segmentation \" (which is not guaranteed, maybe just statistically better but not exact). \n\nOverall, in my opinion I suggest it to be a workshop paper, but the contribution is somehow not significant for a major publication. \n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a spatio-temporal embedding loss for video instance segmentation. The proposed model (1) learns a per-pixel embedding such that the embeddings of pixels from the same instance are closer than embeddings of pixels from other instances, and (2) learns depth in a self-supervised way using a photometric reconstruction loss which operates under the assumption of a moving camera and a static scene. The resulting loss is a weighted sum of these attraction, repulsion, regularisation and geometric view synthesis losses.
The reviewers agree that the paper is well written and that the problem is well motivated. In particular, there is consensus that the 3D geometry and 2D instance representation should be considered jointly. However, due to the lack of technical novelty, the complexity of the final model, and the issues with the empirical validation of the proposed approach, we feel that the work is slightly below the acceptance bar.
Decision: Reject