Venue: CoRL 2021 Conference

Reviews:

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper presents a refinement method for instance masks in cluttered environments. The method uses the relationship between overlapping objects in a cluttered scene to improve the instance segmentation of each individual object. The presented algorithm combines a graph neural network and combinatorial optimization to find an ideal set of interacting instance masks.
strengths_and_weaknesses: + Well written
+ Good improvement over multiple baseline algorithms (general)
- Unlikely reproducible
- Somewhat arbitrary

The paper is generally well written and easy to follow. The presented algorithm is interesting, and yields a good improvement over multiple baselines.

While the paper does a good job explaining things that are contained in it, it lacks some details that would make both experiments and method reproducible. Are the authors open to releasing code and data upon a potential acceptance?
Some of the details that were unclear from the current submission: What data-augmentation is used when training the different SO-Nets? 
 * How are 'training' samples constructed? Do they take into account potentially wrong initial segments of baseline detectors?
 * How many of the baselines use the same input modalities (RGB-D) as the presented method? How many use just RGB or D? Might some of the gains come from these additional modalities?
 * Could RICE work from scratch? Why does it need a baseline detector?
Even with the additional details in the supplement, it would be hard to reproduce without code, which greatly diminishes the value of the submission.

Finally, the method is at times a bit arbitrary. There are many design choices that are simply states, and neither motivated or ablated. For example:
 * SplitNet could as well just predict a binary mask instead of its boundary that needs to be sampled and optimized. I understand that there is some ambiguity in the prediction (both the binary mask and its complement are valid answers), but this can easily be broken using either a single positive example, a hand-coded ordering (left is mask 1, right is mask 2), or even a Hungarian loss. It would be nice to know that the presented boundary prediction is in deed somewhat optimal. This is especially important since most of the presented method replies in a good split (and most other components either hurt or do not significantly improve the performance, see supplement).
* In a similar spirit, the AddNet relies on a proposal constructed from a connected component, why not use an RPN?

summary_of_recommendation: The paper starts with an interesting idea, but does not provide sufficient evidence to show the efficacy of the method.
issues: See issues above.

In addition, it would be nice to have a more detailed comparison of SplitNet and the entire pipeline, as the performance if split alone is almost the same as the complete combinatorial optimization.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: In this paper, the authors propose a neural-network-based method to refine segmentation masks provided by different models. In particular, their goal consists in improving unknown object segmentation masks in cluttered environments, where classical methods fail.The idea consists in (1) representing masks as graphs and use a GNN that produces variations of input masks, and (2) use an additional DNN to evaluate such variations and select them. The set of variations is generated iteratively through sampling and using CEM.
strengths_and_weaknesses: PROS:
* The paper is well structured and well written
* The idea is interesting and easy to apply, as it does not have particular requirements
* The proposed method can be applied (post-training) to several architectures to improve their results
* Although the approach is stochastic, the sampling statistic is is low and predictions are stable
* Experimental evaluation on 4 architectures, 2 of which are recent and task-optimized
* Experimental evaluation includes a use-case showing the utility of the method

CONS:
* std-dev is studied in the sampling procedure, but it is unclear on how many trial it is computed
* The paper does not report how many training runs have been performed: if results are average results, then std-dev is lacking
* The approach makes use of some hyper-parameters (e.g., branching factor, vicinity threshold, max node budget): there is no analysis on how those affect the results



summary_of_recommendation: The paper is well written and well structured. The proposed approach is interesting, easily applicable, and well supported by
experimental evaluations. This said, the document could be further improved by performing in-depth analysis of hyper-parameters
as well as better studying the average/std-dev of the training performance.
issues: * Section 3.2 is difficult to understand as it is currently placed. It would be better placed after 3.4
* Why not using IOU as a metric? For example, https://arxiv.org/pdf/2103.06796.pdf uses it.
* The paper does not report how many training runs have been performed: if results are average results, then std-dev is lacking
* How do the hyper-parameters affect the performance?
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: The authors propose a GNN-based refinement method for unknown instance segmentation based on RGB-D data. Therefore, they build a graph by connecting segments that are close in pixel space. Then they perform an informed perturbation of that graph by splitting, merging, adding and/or deleting segments and finally rank the resulting graphs using another GNN. It is shown that the method can successfully improve results of recent unknown instance segmentation methods. The approach is applied to object singulation where it helps a robot to only singulate the objects with uncertain contours. 
strengths_and_weaknesses: Strengths:

- A refinement of unknown segmentation masks by taking the whole scene into account had been missing in the field and the GNN-based approach is an expensive but convincing attempt. 
- The details how the network learns to predict splits, merges, adds and deletes are interesting
- The ablations (also in the detailed appendix) show the impact of individual components
- The method improves upon different baseline methods in the RGB-D domain on OCID and OSD
- Previously used performance metrics are corrected to be independent to object size. 
- The approach is demonstrated on a robotic system
- The paper is clearly written and seems relatively honest wrt. limitations

Weaknesses (of the approach, paper issues will be listed below):

- The large number of networks and forward passes due to the sampling operations make this method quite slow and memory-intensive to use in robotic applications (10-15s/frame). However, the authors and readers are aware of that and it is a good proof of concept. 
- The networks are completely agnostic to the approach that generated the initial masks. On the one hand, this is beneficial because the refinement can be easily used for different initial methods. On the other hand, it leads to a lot of redundancy, as you need a new feature extractor that learns what an object is. Also using the feature representation of UCN probably contains richer information than just a binary mask and the representation could be directly refined end-to-end by the GNN scores.
- In the robotic experiments it seems like the network is able to merge/split/delete existing masks but has more troubles in adding completely new segments or completing undersegmented masks by adding+merging. This limitation could be made more clear, especially by showing separate results for adding and deleting in Table 1 of the appendix. The problem is probably the generation of candidate masks. I would be curious if an oversegmentation through low thresholds in the initial methods and subsequently only merging/splitting/deleting wouldn't lead to better results.
- The approach is trained on non-photorealistic data (perhaps for better comparison) and then needs to be manually adapted to work better with real data. There might be a more elegant solution, maybe by directly training on more realistic data. 
- It would be interesting how the method deals with many similar objects
summary_of_recommendation: If my issues are resolved I recommend a strong accept. The paper presents an original idea on a relevant problem which has not been studied enough. The experiments are thorough and it is shown how the approach could benefit a robotic system.
issues: - "smart perturbations" are used throughout the paper. I would rather say "informed perturbations" or "learned perturbations".
- l275 "numerical relevance" should be defined as it is otherwise unclear and hard to find in the reference
- Table 2: Is the worst ranking really a good comparison?
- Figure 7: It is annoying to scroll up and down to get the description of the figure, which is actually in section 4.6. It would be great if you could move the descriptive part in the caption.

Since code release was not announced:
- l97: padding is not specified 
- l108: threshold is not specified
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper proposes an approach for image segmentation using graph-based representation and several deep neural networks trained to perturb and score different image segmentations. I like the framework architecture in general, and the results look very promising. But I'm a bit suspicious that this paper is more suited for a computer vision venue than for a robotics one. However, I'm going to give the authors the benefit of the doubt and vote for acceptance.
strengths_and_weaknesses: Strengths:
- The approach seems straightforward to re-produce.
- The paper is well-written and organized.

Weaknesses:
- As mentioned by the authors, the framework is pretty computation-heavy at the moment.
- The robotics experiment is very simplistic and it uses an out-of-the-box grasping method (probably developed by the authors in their previous work).
summary_of_recommendation: The robotic experiments seem very simplistic to me. But the segmentation approach is very interesting and the results look good, and since this can have lots of applications in robotics, I vote for acceptance.
issues: - In Line 98, h and w are not defined.
review_title: null

Meta-review: All authors agree that the proposed method is interesting, novel and applicable to a wide array of segmentation algorithms. All reviewers agree that the paper is well-written and criticism relates to its reproducibility and design choices (QM46) or choices in the evaluation (5yWX, cMBw). However, three of four reviewers recommend acceptance (1x strong accept, 2x weak accept, 1x weak reject). Although this paper may be a better fit for a computer vision conference, I am following this assessment. 
Decision: Accept (Poster)