Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper provides a new self-supervised proposal-based approach for object detection and segmentation. The author introduces a Monte Carlo-based optimization to solve the inefficiency problem in the discrete proposal-based forward process defined in (Crawford and Pineau 2019). Also, the paper redefines the decoder part for self-supervised from minimizing reconstruction loss with background segmentation to maximize reconstruction error with learning a foreground segmentation.  The method is then verified with a suite of experiments for people-detection on video datasets.

The main benefit over many previous unsupervised object detection/segmentation approaches is that they did not make use of optical flow or other readily available cues during training. However, given that the framework directly came from (Crawford & Pineau 2019), and the only change is from variational inference to an importance-sampling (MC) approach. This would be fine if it is verified in experiments, however, the experiments did not show any comparison w.r.t. (Crawford & Pineau 2019) hence we have no way of understanding what is the relative performance w.r.t. that baseline approach.

Besides, in all the experiments a single object is in the view. How does the method perform in images where multiple objects are in the view?

A little bit of a philosophical question is whether this a problem worth pursuing as well. For self-supervised motion estimation (e.g. optical flow), it is clear why we want to do that. However, the current type of algorithm is dependent on the assumption that the background follows relatively consistent textures, this may not necessarily be true in practice, and hence the application could be quite limited. Many previous unsupervised video object segmentation methods make use of optical flow and boundary detection, which I thought are OK cues to be used, especially when both can be learned in a self-supervised manner. This is not entirely related to the assessment, but I would still like to hear what the authors think.

Minor:
In the paragraph after 'Training strategy'(Section 3.2, Page 5), is it 'the foreground objective O of Eq. (2)' or 'Eq.(4)'?


review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: The paper introduces a method to self-supervised train a model for object detection/segmentation. The idea is that the background is easy to reconstruct while the foreground/object is hard. Experiments demonstrate the effectiveness of the proposed methods.

Here are some high-level concerns.

1. As mentioned in the "Implementation details", 'naive end-to-end training is difficult... we use ImageNet-trained weights for initialization'. This is worrisome to justify the effectiveness. It may be possible the imagenet-trained model has already captured salient objects. To justify the claims and effectiveness of the method, it should include a comparison with [R1], which demonstrates the possibility of doing detection with a pretrained model. Other work along this line should be also good reference.

2. As a moving camera is available, it is also possible to segment background with frames through a 6DoF prediction on the camera, rotation and translation, e.g., [R2]. The supervision signal is from frame reconstruction through learning to predict both camera pose and pixel-level depth. This is also self-supervised learning. At least such a self-supervised trained model can act as an initialization.

Considering the above points, the paper does not appear compelling, due to lack of either careful claims or justification.


[R1] Learning deep features for discriminative localization
[R2] Unsupervised learning of depth and ego-motion from video
review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: This submission proposes a self-supervised segmentation method, that learns from single-object videos by finding the region where it can segment an object, remove the entire bounding box around it, inpaint it, then finally put the object back. The loss is a balance between a reconstruction error and a negative inpainting error (high error means object is probably present, due to the weak correlation with the background).

My decision is Weak Accept. I like the method very much and think it’s a clever and well-executed algorithm. The reason for being Weak is because the experimental evidence could be stronger, especially comparing with Croitoru et al. and Rhodin et al. The paper leaves some open problems, inviting future work to be built on top of it (i.e. leveraging time more and handling multiple objects better). I think it should be accepted to promote such future work.

Method:

The method is interesting and clever. Similar efforts have been made, such as Bielski & Favaro and Crawford & Pineau. However, key contributions relax some of the contrived requirements of these past methods (e.g. simple foreground translation over background; requirement of plain background). Thanks to the inpainter, the importance sampler, and avoidance of collapsing into trivial solutions, this paper is able to put together a method that works on moving cameras and fairly complex scene semantics (still limited to one object though). Actually getting this to converge with only two loss terms balanced against each other is impressive. Having certain heads of the network trained only on one of the two terms seems to be a key contribution. 

Experimental results:

The comparison with Rhodin et al. on H36M is characterized as “slightly” lower, but I would call 71% vs. 58% a significant difference. Of course, I understand that Rhodin et al. relies on the static background, so this is not a fair comparison.

Ski-PTZ-Dataset should then offer a better comparison, but here the method struggles to compete with Croitoru et al. 2019, which has a 11-point higher F-measure. On Handheld190k, the dataset proposed in this paper, is where the method finally shines, but still only offers a 1-point F-measure improvement over Croitoru et al. That being said, considering how different the methods are, and how Croitoru et al. requires two-stage training, there are many benefits to this method. Croitoru et al. also relies on video to extract the object features, and this requirement is not as explicit in this work. Actually, that brings me to one question I had. The paper states that “as long as videos or picture collections of a single object in front of the same scene are available.” I didn’t quite understand why this must be trained on multiple images of the same scene. If the inpainter is general to any background scenery, couldn’t it work on single-images as well? The conclusion even says you do not use temporal cues.

Other:

I don’t think it’s that meaningful to include precision/recall in the tables. It is also not that meaningful to point out that your method’s precision is higher than that of Croitoru et al., when the F-measure is shy 11 points. The reason is because many points on the precision/recall could be constructed simply by applying for instance a gamma curve on the segmentation predictions. The high precision is clearly at the cost of a low recall, and another point on this tradeoff curve could be presented. This is why F-measure and average precision are much better.
review_title: Official Blind Review #3

Meta-review: This work proposes a self-supervised segmentation method: building upon Crawford and Pineau 2019, this work adds a Monte-Carlo based training strategy to explore object proposals.
Reviewers found the method interesting and clever, but shared concerns about the lack of a better comparison to Crawford and Pineau, as well as generally a lack of care in comparisons to others, which were not satisfactorily addressed by authors response.
For these reasons, we recommend rejection.
Decision: Reject