Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper presents learning a spatio-temporal embedding for video instance segmentation. With spatio-temporal embedding loss, it is claimed to generate temporally consistent video instance segmentation. The authors show that the proposed method performs nicely on tracking and segmentation task, even when there are occlusions.

Overall, this paper is well-written. Section 3 clearly explains the loss functions. The main idea is not very complex, but generally makes sense. The authors mention that scenes are assumed to be mostly rigid, and appearance change is mostly due to the camera motion. I would like to see more argument about this, as there are cases if this is obviously not true; for instance, human changes pose significantly. If we limit the range of discussion to some narrow domain, such as self-driving, this might be more valid, but we may want to see some discussion about validity of this assumption.

Some modules are not full explained in detail. For example, what is the background mask network? Which model was used, and how was it trained?

In experiment, the proposed method shows nice score on MOTSA and sMOTSA, but all other metrics, it is on the worse side. The authors are encouraged to discuss more about the metrics and experimental results with the other metrics as well. Other than these, the experiment was well-designed and conducted.
review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: This paper propose a video instance embedding loss for jointly tackling the instance tracking and depth estimation from self-supervised learning. 

Pros:
1: I think the method is moving towards the right direction that 3d geometry and 2d instance representation should be considered jointly under the scenario of video learning.
2: The video instance embedding loss is also making sense as an extension of image instance embedding.

Cons: 
1: I think the major argument I have is this method is lack of technical novelty, since it is straight forward to adopt the loss of  Brabandere et.al 2017 to video cases for including pixels in the same group under ground truth tracking, and the self-supervised loss is exactly the same as previous methods. The fusion between depth and segments are relatively weak since it just ask the embedding to also decode depth, is there any further analysis of visual effect of explaining where the depth helps segments? 

2: In the experiments, the baseline for comparison over MOTS is fairly old, and I think it makes sense to include the number of MOTS paper, which is currently hard to align with that shown in the paper.  In Tab.2, the author only highlight the improved motion metric, while in per-frame AP the results are actually lower than the baselines. It also needs to be well explained. 

3: The paper claims `"it generates temporal consistent segmentation " (which is not guaranteed, maybe just statistically better but not exact). 

Overall, in my opinion I suggest it to be a workshop paper, but the contribution is somehow not significant for a major publication. 

review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: Summary
The paper presents a method to learn an embedding space for each pixel in a video that indicates the instance id of the objects. They also propose an auxiliary loss based on depth prediction to improve performance. This can be used to both segment and track objects in videos.

Strengths
1) The proposed approach is simple and general and handles the problem of occluding objects in videos. 
2) Their causal convolution architecture will be useful for other problems in videos.
3) The authors perform a number of ablations to investigate how much each part of their solution contributes to the final performance.
4) The paper is well-written and well-motivated.

Weaknesses
1) Comparisons to other video instance segmentation methods [1,2,3] are missing. The only comparison is done with a single-frame instance embedding method. 
2) The authors propose to predict depth as an auxiliary task. However, they do not use the predicted depth at test time. This is a missed opportunity. Difference in depth might help in identifying instances. Also, it might be worthwhile to investigate if instance segmentation is helping depth prediction.
3) Experiments have been conducted on only one dataset. 


References
[1] "Video Instance Segmentation" Linjie Yang and Yuchen Fan.
[2] "MaskRNN: Instance Level Video Object Segmentation" Yuan-Ting Hu, Jia-Bin Huang, and Alexander G. Schwing.
[3] "SAIL-VOS: Semantic Amodal Instance Level Video Object Segmentation – A Synthetic Dataset and Baselines" Yuan-Ting Hu, Hong-Shuo Chen, Kexin Hui, Jia-Bin Huang, Alexander Schwing.


review_title: Official Blind Review #2

Meta-review: This paper proposes a spatio-temporal embedding loss for video instance segmentation. The proposed model (1) learns a per-pixel embedding such that the embeddings of pixels from the same instance are closer than embeddings of pixels from other instances, and (2) learns depth in a self-supervised way using a photometric reconstruction loss which operates under the assumption of a moving camera and a static scene. The resulting loss is a weighted sum of these attraction, repulsion, regularisation and geometric view synthesis losses.
The reviewers agree that the paper is well written and that the problem is well motivated. In particular, there is consensus that the 3D geometry and 2D instance representation should be considered jointly. However, due to the lack of technical novelty, the complexity of the final model, and the issues with the empirical validation of the proposed approach, we feel that the work is slightly below the acceptance bar.
Decision: Reject