RICE: Refining Instance Masks in Cluttered Environments with Graph Neural Networks
Authors: Chris Xie
Arsalan Mousavian
Yu Xiang
Dieter Fox
Venue: CoRL2021 Poster (2021)
Keywords: Unseen Object Instance Segmentation
Graph Neural Network

Abstract: Segmenting unseen object instances in cluttered environments is an important capability that robots need when functioning in unstructured environments. While previous methods have exhibited promising results, they still tend to provide incorrect results in highly cluttered scenes.  We postulate that a network architecture that encodes relations between objects at a high-level can be beneficial. Thus, in this work, we propose a novel framework that refines the output of such methods by utilizing a graph-based representation of instance masks.  We train deep networks capable of sampling smart perturbations to the segmentations, and a graph neural network, which can encode relations between objects, to evaluate the perturbed segmentations.  Our proposed method is orthogonal to previous works and achieves state-of-the-art performance when combined with them. We demonstrate an application that uses uncertainty estimates generated by our method to guide a manipulator, leading to efficient understanding of cluttered scenes. Code, models, and video can be found at https://github.com/chrisdxie/rice.

RICE: Refining Instance Masks in Cluttered Environments with Graph Neural Networks
Christopher Xie Arsalan Mousavian
Yu Xiang Dieter Fox
University of Washington
"NVIDIA
(chrisxie,fox)) fox) washington edu (emousaaviaan.yux)onvddaa..
Abstract: Segmenting unseen object instances in cluttered environments is an important capability that robots need when functioning in unstructured environ- ments. While previous methods have exhibited promising results, they still tend to provide incorrect results in highly cluttered scenes. We postulate that network architecture that encodes relations between objects at high-level can be beneficial. Thus, in this work, we propose a novel framework that refines the output of such methods by utilizing graph-based representation of instance masks. We train deep networks capable of sampling informed perturbations to the segmentations, and a graph neural network, which can encode relations between objects, to evaluate the perturbed segmentations. Our proposed method orthogonal to previous works and achieves state-of-thes of-the-art performance when combined with them. We demon- strate an application that uses uncertainty estimates generated by our method guide manipulator, leading to efficient understanding of cluttered scenes. Code, models, and video can be found at ht https: bpp::// //g/ /ggttub....... com/ thrisdxie// eetiiieeriie...
Keywords: Unseen Object Instance Segmentation, Graph Neural Networks, Robot Perception
Best Graph
Initial Segmentation
Encode Groph
Sample Perturbations
Uncertaintyy
Delete
Add
High-level overview of our proposed method. Given an initial segmentation, we encode as graph, sample perturbations, then score the resulting segmentation graphs. The highest scoring graph and/or contour uncertainty output. Best viewed in color and zoomed in.
Figure 1:
Introduction
Perception lies at the core of the ability of a robot to function in an unstructured environment. critical component of such perception system is its capability to solve Unseen Object Instance Segmentation (UOIS), as it infeasible to assume all possible objects have been seen in a training phase. Proper segmentation of these unseen instances can lead better understanding of the scene, which can then be exploited by algorithms such as manipulation [1, 2, 3] and re-an re-arrangement [4].
Many methods for UOIS directly predict segments from raw sensory input such as RGB and/or depth images. While recent methods have shown strong results for this problem [5, 6, 7, 8], they still tend to fail when dealing with highly cluttered scenes, which are common in manipulation scenarios. natural thought that an architecture with relational reasoning can benefit the predictions. For example, it can potentially learn to recognize common object configurations (e.g. realizing that one object stacked on top another). While relational inductive biases have shown to be useful for problems such as scene graph prediction [9, 10, 11], it remains to be seen whether it can be useful in identifying objects in dense clutter. In this work, we investigate the use of graph neural networks, which can encode relations between objects, for segmenting densely cluttered unseen objects.
5th Conference on Robot Learning (CoRI. 2021), London, UK.

---

In this paper, we propose a novel method for Refining Instance masks in Cluttered Environments, named RICE. Given an initial instance segmentation unseen objects, we encode into a segmen- tation graph, where individual masks are encoded as nodes and connected with edges when they are close in pixel space. Starting from this initial graph, we build a tree of sampled segmentation graphs by perturbing the leaves in a CEM-style (Cross Entropy Method) framework, where example perturbations include splitting and merging. We learn Sampling Operation Networks (SO-Nets) that sample efficient and informed perturbations that generally lead to better segmentations. The perturbed segmentation graphs are scored with a graph neural network, denoted Segmentation Graph Scoring Network (SGS-Nett Finally, can return the highest scoring segmentation or compute contour uncertainties, depending on the application. Figure provides a high-level illustration of our method.
RICE is able to improve the results of existing techniques to deliver state-of-the-an performance for UOIS. An investigatory analysis reveals that applying SGS-Net on top of the SO-Nets results in more accurate and consistent predictions. In particular, we find that SGS-Net learns to rank segmentation graphs better than SO-Nets alone. Additionally, we provide a proof-of-concepl efficient scene understanding application that utilizes uncertainties output by RICE to guide manipulator.
In summary, our main contributions are: 1) We propose a novel framework that utilizes a new graph-based representation of instance segmentation masks in cluttered scenes, where we learn deep networks capable of suggesting informed perturbations and scoring of the graphs. 2) Our method achieves state-of-the-art results for UOIS when combined with previous methods. 3) We demonstrate that uncertainty outputs from our method can be used to perform efficient scene understanding.
2 Related Work
Instance Segmentation Traditional methods for 2D instance segmentation include GraphCuts [12], Connected Components [13]. and LCCP [14]. Recently, seaminng-bssed approaches have provided more semantic solutions. For example, top-down solutions combine segmentation with object proposals in the form of bounding boxes [7, 15, 16, 17]. Mask R-CNN [7] predicts a foreground mask for each proposal produced by its region proposal network (RPN). However, when bounding boxes contain multiple objects (e-g. cluttered robot manipulation setups), the true instance mask is ambiguous and these methods struggle. Recently, a few methods have investigated bottom-up methods which assign pixels to object instances [18, 19, 20, 21, 22]. Some examples of this include contrastive losses [18] and unrolling mean shift clustering as a neural network to learn pixel embeddings [22].
Most of the montiomento algorithms provide instance masks with category-level semantic labels, which do not generalize to unseen objects novel categories. Class-agnostic methods [23, 24, 25, 26] and motion segmentation [27, 28, 29] methods have been investigated for this problem. In robotic perception, Xie et al. [30] proposed to separate the processing of depth and RGB in order to generalize their method from sim-to-real settings and provide sharp masks. Their follow-on work [5] proposed a 3D voting method to overcome the limitations of their earlier 2D method. Xiang et al. [6] showed that training network on RGB-D with simulated data and simple contrastive loss [18] can demonstrate strong results for this problem. While these methods show promise, they are not perfect and still admit mistakes in cluttered scenes, which can hamper downstream robot tasks that rely such perception. Our method is orthogonal to these works, and is designed to refine their outputs by sampling perturbations to result in better instance segmentations in the cluttered environments.
Graph Neural Networks Graph neural networks (GNN) in vision and roboties have recently become a useful tool for learning relational representations. They have found applications in many standard computer vision tasks such as image classification [31, 32), object detection [33], semantic segmentation [34], and question answering [35]. GNNs have also been used to perform "scene graph generation", which requires predicting not just object detections, but also the relations between the objects [9, 10, 11]. The resulting scene graphs have been used for applications such as image retrieval [36]. GNNs have also been used to learn object dynamics, properties, and relations for applications such differential physics engines [37, 38]. Our proposed work represents instance segmentation masks as graphs and utilizes this architecture in order to refine the predicted masks.
3 Method
Our method, RICE, is designed to Refine Instance masks of unseen objects in Cluttered Environ- ments. Given initial segmentation mask S E NHXW W of unseen objects, we first encode this

---

as a segmentation graph Gş, which is described in Section 3.1. Then, in Section 3.2, we build tree T of sampled segmentation graphs by perturbing the leaves CEM-style [39] framework. Section 3.3 details the sampling operations, which are parameterized by our Sampling Operation Networks (SO-Nets). Each candidate graph (tree node) is scored by a GNN named Segmentation Graph Scoring Network (SGS-Net), introduced in Section 3.4. Finally, the highest scoring graph in T and/or contour uncertainties are returned. Figure I provides a high-level illustration of RICE, and pseudocode can be found in the Supplement.
3.1 Node Encoder
Given single instance mask S; E (0, H for instance we crop the RGB image e R#xxxx organized point cloud D E xx (computed by backprojecting a depth image with camera intrinsics), and the mask S with some padding for context. We then resize the crops to h and feed these into multi- stream encoder network which we denote as the Node Encoder. This network applies separate convolutional neural network (CNN) to each input, and then fuses the flattened outputs to provide a feature vector v; for this node. See Figure for visual illustration of the network. Note that we also encode the background mask node in the graph. This gives the segmenta- tion graph Gş = (V, E), where each v, E V corresponds to an individual instance mask, and nodes are connected with undirected edges e (i.j) E if their set distance is less than threshold.
Figure 2: Given an initial instance segmentation mask (left), our segmentation graph representa- tion encodes each individual mask as graph node (red dots) with a corresponding feature vector v; (yellow bar) output by the Node Encoder (right). Edges (blue lines) connect nearby masks.
3.2 Building the Sample Tree
Our sample tree-building procedure operates in a CEM-style fashion. CEM [39] is an iterative sampling-based optimization algorithm that updates its sampling distribution based on an "clite set" of the top (or top percentile) samples. For more details, we refer the reader to [39]. Following this terminology, our clite set consists of the leaves of our sample tree I', each of which are guaranteed to be better with respect to our proxy objective function, SGS-Net. Then, the sampling distribution is implicitly defined by the SO-Nets; while cannot explicitly write out the distribution, we can certainly sample from with our sampling operations described in Section 3.3.
Our sample tree T starts off with the root Gg. We expand the tree from the leaves with K expansion iterations. For each expansion iteration, we iterate through the current leaves of T. For a leaf G, we randomly choose sample operation from Section 3.3 and apply it to G to obtain candidate graph G". We then compare the scores $G, SG output by SGS-Nct and add G' to T as a child sg' > SG. Thus, any leaf of T is guaranteed to be least good the root Gş w.r.t our proxy objective function SGS-Net. We apply this procedure B times for G. such that each tree node can have a maximum of B children. Thus, B is branching factor. Finally, due to constraints of limited GPU memory, we exit the process in an anytime fashion whenever we exceed a budget of maximum graph nodes and/or graph edges (not be confused with tree nodes/edges). See the Supplement for pseudocode and an example of the sample tree-building procedure.
G
It is important to note that while utilize learned SO-Nets and SGS-Net to build the sample tree T, they are applied in different manners (although they are trained on the same dataset). In order to add candidate graph to the tree, they must both agree in the sense that the perturbation must be suggested via an SO-Net and SGS-Net must approve of the candidate graph via its score. This redundancy offers level of robustness, Section 4.4 shows that the combination of these leads to more accurate performance with lower variance.
3.3 Sampling Operations
We consider four sampling operations: 1) splitting, 2) merging, 3) deleting, and 4) adding. However, randomly performing these operations leads inefficient samples which wastes computation time and memory. For example, it is not clear how to split or add an instance mask randomly such that it may result in better segmentation. Thus, we introduce two networks for these four operations, SplitNet and DeleteNet, which comprise our SO-Nets. They are learned to suggest informed perturbations to
3

---

(a) Split
(b) Merge
(c) Delete
(d) Add
Figure 3: We show real-world examples of the sampling operations and how they can refine the
original segmentation. Best viewed in color on computer screen and zoomed in.
bias the sampling towards better graphs, lowering the amount of samples needed in order to favorably refine the segmentation. Examples of each operation can be found Figure 3.
Split It is not clear how to randomly split mask such that it provides effective split. For example, a naive thing to do is to sample a straight line to split the mask, however in many cases this will not result in a reasonable split (see Figure 3a for an example). Thus, we propose learn deep network denoted SplitNet to handle this. SplitNet takes the output of the Node Encoder (before flattening), fuses them with concatenation followed by a convolution, then passes them through single decoder with skip connections. Essentially it is a mulli-stream architecture, much like Y-Net [27], except that has three streams for RGB, depth, and the mask. The output of SplitNet is pixel-dense probability map P; E [0. 1|ax0" of split-able object boundaries. To sample a split for instance mask S,, we first sample two end points on the contour of the original mask St, and calculate the highest prohability path from the end points that travels through pi, resulting in trajectory r = {(u,, 21) (n)) V/z of length L, We score the split with 8, = L. E, Ps (ug, vy] E R, which is the average probability along the sampled path. More details can be found in the Supplement.
incoder-decoder U-Net [40]
Merge We exploit the fact that merging is the opposite of splitting and adapt SplitNet for this operation. For each pair ji. of neighboring masks, take their union Sij and pass it through SplitNet to get Piy- Note that we do not consider merging disjoint masks that may belong to the same instance, which is a limitation of this work. To compute the merge score mij, we first compute the union of the boundaries of S; and Sj. denoted Bij E {0, 1) Then, we calculate the merge score as Maj = 1 - (Pij @ B1y/(1Tpij p.y1), 1)) where @ is element-wise multiplication, 1 is a vector of ones. This is essentially weighted average of Biy with weights Pty. This score indicates how likely SplitNet thinks S, and Sy correspond to different objects. Figure 3b shows an ideal merge operation.
Delete We design a network, DeleteNet, to provide delete scores d: E R for every instance (graph node) i. This network is also built top of the Node Encoder: it computes the difference v, Vog- where Vbg is the feature vector for the background node output by the Node Encoder. This difference is then provided as input to multi-layer perceptron (MLP) which outputs scalar d4. See Figure 3c for an example of how DeleteNet can help remove false positives from the segmentation.
Add Similarly to merging, we can exploit the fact that adding is the opposite of deleting. Given a candidate mask Sx+1 add to the graph, can use DeleteNet to compute its delete score dx+11 If d is below a threshold, we successfully add the mask to the graph. However, the question remains of how to generate such candidate masks. Given external foreground mask F E {0, 1) x (provided by UOIS-Net-3D [5]), we run connected components on F (UrS+), }. and use the discovered components as potential new masks. A successful addition operation can be seen in Figure 3d.
3.4 Segmentation Graph Scoring Network
While sample operations provide efficient samples that typically lead to better segmentation graphs, they can also suggest samples that worsen the segmentation. Thus, we Iearn SGS-Net which acts as proxy for the objective function in the CEM framework. Our proposed SGS-Net learns to score a segmentation graph by considering the fused feature vectors v, in context of their neighboring graph nodes (masks). We posit that this context will aid SGS-Net in predicting whether the perturbations improve the segmentation. For example, it can potentially learn to recognize common object configurations from the training set, and score such configurations higher.
high-level illustration of SGS-Net can be found in Figure 4. The initial node features v (0) are
(0) given by the Node Encoder, and we obtain initial edge features eij by running the Node Encoder on

---

Input Graph
Inibal Node Features
Residual GN Layer
Residual GN Layer
Output Layer
SG E R
Encoder
Figure 4: A high-level illustration of our Segmentation Graph Scoring Network (SGS-Net). It is composed of a Node Encoder (see Figure 2), multiple Residual GraphNet Layers, and an output layer. We borrowed elements from Figure of Battaglia al. [41].
all neighboring union masks Siy. Then, we pass them through multiple Residual GraphNet Layers (RGLs), which are essentially GraphNet Layers [41] with residual connection. We refer readers to Battaglia et al. [41] for details GraphNet Layers, Layers, and also provide a full mathematical specification of RGLS in the Supplement for completeness. The output of SGS-Net is scalar score in [0, 1].
3.5 Training Procedure
For SplitNet, we apply a weighted binary cross entropy (BCE) loss to the probability map pt Esplic Σn, wh, faoe (Pu,Pon where ranges over pixels, p E (0, 1x is ground truth boundary, and bro the binary cross entropy loss. The weight w. is inversely proportional to the number of pixels with labels equal to p... DeleteNet also trained with standard BCE loss. SGS-Net is trained with Lo to regress to .8F + FF @. 75, where F' is the Overlap F-measure [30] and F .75 is the Overlap F .75 measure [42]. The latter measures the percentage of correctly segmented instances. Thus, SGS-Net learns to predict score based on the number of correctly identified pixels and instances. Note that this regression problem very difficult to solve. However, the scores do not actually matter as long as the relative scoring is correct, since building the sample tree relies only on this (Section 3.2). In Section 4.5 we show that while SGS-Net may not solve the regression problem well, learns to rank graphs accurately. Further training and implementation details can be found in the Supplement.
4
Experiments
4.1 Encoding RGB and Modality Tuning
We use ResNet50 [43] with Feature Pyramid Networks [44] (FPN) to encode RGB images before passing them to the Node Encoder. However, since we are training with (a more cluttered version of) the non- pontpootorcalistic synthetic dataset from Xie et al. [30], we perform modality tuning [45]. where we fine-tune earlier convolutional layers of ResNet50 during training, and use the COCÓ [46] pretrained weights during inference. For all experiments, we modality tune the conv1 and conv2_1 blocks of ResNet. We provide an experiment in the Supplement that shows this setting optimal.
4.2
Datasets and Metrics
We evaluate our method on two real-world datasets of challenging cluttered tabletop scenes: OCID [47] and OSD [48], which have 2346 images of semi automatically constructed labels and 111 manually labeled images, respectively. Our SO-Nets and SGS-Net are trained on a more cluttered version of the synthetic Tabletop Object Dataset (TOD) [30], where each scene has anywhere between 20 and 30 ShapeNet [49] objects. We use 20k scenes in total, with 5 images per scene.
Xie et al. [30] introduced the Overlap P/R/F and Boundary P/R/F measures for the problem of UOIS. However, these metrics do not weight objects equally; they are dependent on the size and larger objects tend dominate the metrics. Thus, we introduce variation these metrics that equally weights the errors of individual objects regardless of their size. Given a Hungarian assignment A between the predicted instance masks {S;) and the ground truth instance masks {5,), compute our Object Size Normalized (OSN) P/R/F measures as follows:
Σ Ps ( N
Σ (IJ)A
Raj
Σ Fij (iJ)A
Σ 1(Fg >= 0.75) (AJ)
PR
R
F,
M, F,
M
max(M,N)
where Pijs Raj, Fe are the precision, recall, and F-measure of S,, Sj. Note that the F, @ .5 penalizes both false positive and false negative instances, as opposed to the normal F@. 75, which does not penalize false positives. Similarly to Xie al. [30], we can apply the OSN metries to the pixels and
5

---

Results OCID
Results on OS0
Meak- RCIM
NettGrosp
JOIC No-30
UCI
PuesCrouz
UDIS Not
UCN
70:700
21
22,378.
64)
FIONTS
X,06.75
Figure 5: Applying RICE to refine results from state-of-the-ar instance segmentation methods leads improved performance across the board. Note that standard deviation bars are shown, but are very tight and difficult to see.
Overlap Rg
Boundary Rn
SO-Nets
SCiS-Ne Net
F
P
F@0..7
F,000.5
x
X
(
83.0
77.8
84.6
76.5 ()
75.0
7
77.0()
(1.23)
89.4 (0.19)
82.3 (1
82.7 (1.37)
82.8 (0.19)
(m)
(0.26))
84.2 (1.26)
86.3 (0.03)
89.1 (0.01)
(0.05)
84.5 (0.04)
82.5 (0.04)
80.0 (0.04)
88.5 (0.02)
85.5 (0.05)
Table 1: Ablation to test the utility of SO-Nets and SGS-Net on OCID [47] starting from UOIS-
Net-3D [5] masks. Only using the sample operator networks (SO-Nets) in an iterative sampling scheme already provides an increase in performance, showing that the informed samples are generally improving the initial segmentations. However, the standard deviations (shown in parentheses) are relatively high. Adding in SGS-Net boosts performance while drastically lowering the variance, demonstrating the efficacy of SGS-Net in consistently filtering out bad suggestions by the SO-Nets.
boundaries, giving Overlap and Boundary P.,/ R., Fr measures. For comparison, we also show results with the normal Overlap and Boundary P/R/F measures in the appendix.
We run each experiment 5 times and show means and standard deviations for all metrics.
4.3 SOTA Improvements
We demonstrate how RICE can improve upon predicted instance segmentations from state-of-the-art methods. In particular, we apply it to the results of Mask R-CNN [7], PointGroup [8], UOIS-Net- 3D [5]. and UCN [6]. all of which consume RGB-D as input. We employ RICE by returning the best segmentation. For brevity, we only show Overlap F., Boundary F.. F@ .7,, and F, @.75 in Figure on both OCID and OSD. The light orange bars show the additional performance that RICE provides over the output of the methods. Standard deviations are shown as error bars, but are in general very narrow, showing that method provides consistent results despite its stochasticity. RICE provides substantial improvements to all methods. The largest gains occur Mask R-CNN and PointGroup, with 21.6% and 32.3% relative gain in F3 @.75 on OCID, respectively. Additionally, on the already strong results from UOIS-Net-3h and UCN, RICE achieves 11.0% and 4.0% relative gain in F, @ .75 on OCID, respectively. These results are similar on OSD, with the gains being slightly less pronounced, which we believe is due to OSD being smaller dataset with less clutter. Note that applying RICE increases both F and F, @ .7,, indicating that not only is capturing the object identities correctly, it is not simultaneously predicting more instances (false positives). In the appendix, we show full results for all metrics including P, R,, and normal P/R/F metries.
4.4 Ablation Study
We aim to answer two questions with this study: 1) how good are the samples suggested by our SO-Nets, and 2) to what degree does SGS-Net increase performance and robustness? We study these questions on the larger OCID.
Since the SO-Nets alone do not provide scores of the perturbed segmentation graphs, we structure our ablation such that this is not needed in order answer 1). Our SO-Nets are trained provide informed perturbations that are closer to the ground truth segmentation, so every sample is supposed to be better than the original graph. With this insight, we design an experiment where we run RICE with branch factor B = 1 and K = 5 iterations, always add the candidate graph to the tree without consulting SGS-Net, and return the final graph. Essentially, this can seen as an iterative segmentation graph refinement procedure where the sampled graph should be better than the previous in every iteration. Starting from initial masks provided by UOIS-Net-3D [5], we see in Table I that applying this iterative sampling scheme with SO-Nets only provides better results almost all

---

metrics than without. However, adding SGS-Net back into the procedure results in better Overlap Fn. Boundary F,, and F,, @.75, while significantly reducing the standard deviation of the results by two orders of magnitude. This demonstrates that having SGS-Net in RICE delivers not only more accurate performance, but also more robust performance with relatively small variance, which answers 2). Note that @ 75 is slightly lower with F,, @.75 higher, indicating that SO-Nets are suggesting more samples that better capture the objects, but are suggesting too many instance segments.
4.5 SGS-Net Ranking
Figure 6 shows an example of how difficult scoring the segmentations graphs is; the two slightly different segmen- tations have significant difference in their ground truth scores. In fact, SGS-Net does a poor job at scoring the graphs, with a mean absolute error (MAE) of 0.184 and even higher standard deviation shown in Table 2. These values are high given that the scores are in the range [o, 1]. Then, this begs the question, why does SGS-Net work well within our proposed RICE framework? Recall that the score magnitudes do not matter, only the relative scoring (Section 3.2). We claim that SGS-Net learns to rank the graphs accurately, and design an experiment to test this hypothesis.
0.820
0.848
Figure 6:
Can you spot the differences between the segmentations?
We leverage the normalized Discounted Cumula- tive Gain (nDCG) [50] which is a popular ranking metric in the information retrieval community. The 2 -1
MAE
nDCG
Minimum SO-Nets SGS-Net
0.844 196)
0.944 (0.098)
DCG is computed as E (og,((+1) where rel, is the numerical relevance (1 to p. higher is better) of the item at position i. This essentially computes weighted sum of the relevance with discount factor for further items, which places more emphasis on the high-ranking predictions. The normalized version divides DCG by the "ideal" version, i.e. the DCG of the correct ranking. This results in nDCG E [0, 1] with higher being better. We compute nDCG of the ranking of the iterative sampling experiment in Section 4.4, with relevance values in {o..... KJ. The ranking for SO-Nets is given .... by the order of the predicted graphs, and we use SGS-Ne scores to compute its ranking/relevance. We also compute the nDCG of the worst ranking, denoted "minimum". In Table 2. see that both SO-Nets and SGS-Net perform significantly better than the worst ranking. SGS-Net provides better ranking than SO-Nets with slightly lower variance, which helps to explain its effectiveness in RICE.
0.184 (0.212)
0.952 (0.095)
Table 2: Ranking study on OCID and OSD.
4.6 Visualizing Refinements
In the leftt side of Figure 7 (green box), we qualitatively demonstrate successful refinements from applying RICE to instance masks provided by state-of-the-ar methods. The first column shows an example where many nearby objects are under-segmented. Indeed, RICE manages to find all of the necessary splits except for one. In general, RICE is quite adept at splitting under-segmented instance masks. This is quantitatively confirmed in an additional ablation in the Supplement that studies the usefulness of each sampling operation. Column two shows an initial mask that is fixed with a merge operation. Column three shows false positive mask on the textured background, which is suppressed by RICE's deletion sampling operation. In the fourth column, the initial mask is missing quite few objects, and RICE is able to not only recover them but also correctly segment them, resulting in an almost perfect instance segmentation. In the last column, the bottom left segment is bleeding into neighboring segment, which is fixed through multiple perturbations (i.e. split, then merge).
4.7 Failures and Limitations
In the right side of Figure 7 (red box), we discuss some failure modes and limitations. The first column demonstrates failure mode where RICE tends to over-segment objects with lot of texture (e.g. cereal box). We believe that this is due to TOD lacking texture on many of its objects [5]. The second column shows a limitation: since RICE only considers merging neighboring masks, it cannot merge non-neighboring masks that belong to the same object. RICE does nothing and the book is still incorrectly segmented in two pieces. We leave this as an interesting avenue for future work.
4.8 Guiding a Manipulator with Contour Uncertainties for Efficient Scene Understanding
Fully segmenting and understanding a scene of cluttered objects necessary for various manipulation tasks, such as counting objects or ce-arranging and sorting them. One way for doing this is to actively
7

---

Ground After
Figure 7: We demonstrate successful refinements (left, green box) for each of the sampling operations. Failure modes (right, red box) include textured objects and non noighbiggg masks that belong to the same object. Best viewed in color and zoomed in computer screen.
UCN Masks
Contour
Uncertainty
Interactions
t 0
t=1
t=
Figure 8: UCN masks [6] (top row) and contour uncertainties from RICE (bottom row, uncertainties shown in red with average contours in green) in trial of our understanding experiment. After grasping the milk carton and red cup, the scene is segmented with full certainty, indicating that the scene is fully understood. Thus, the algorithm terminates without having to singulate each object.
singulate each object [51]. However, such an approach be extremely inefficient. Here we show how contour uncertainties extracted from RICE can help to solve this problem with potentially far less interactions. Specifically, we extract contour uncertainties by computing the standard deviation of the mask contours of each Icaf graph. These uncertainties let us distinguish between objects that already confidently segmented and those that require physical interaction resolve segmentation uncertainty. We grasp [52] any object that has uncertain contours in order to determine its correct segmentation, and repeat this until no more uncertainty persists. Thus, interactions are only required to resolve the uncertain portions of the scene, which can potentially be much less than the number of objects, leading more efficient scene understanding method. For example, Figure 8, only two grasps are required to fully understand the scene. See the Supplemental video for more results.
Conclusion and Future Work
We have proposed a novel framework that utilizes a graph-based representation of instance segmenta- tion masks. incorporates deep networks capable of sampling smart perturbations, and a graph neural network that exploits relational inductive biases. Our experimental analysis revealed insight into why our method achieves state-of-the-ar performance when combined with previous methods. We further demonstrated that uncertainty outputs can be utilized to perform efficient scene understanding.
main limitation our work is the computational burden; algorithm runs at 10-15 seconds per frame, depending the expansion of the sample tree. Additionally, it is GPU-memory intensive as the sample tree must stored GPU memory. Future work will explore how to make the method more computationally efficient, along with solving the inherent limitations mentioned in Scction 4.7.

---

Acknowledgements
This work was funded in part by NSF NRI grant IIS-2024057.
References
[1] A. Mousavian, C. Eppner, and D. Fox. 6-dof graspnet: Variational grasp generation for object manipulation. In IEEE International Conference on Computer Vision (ICCV), 2019.
[2] A. Murali, A. Mousavian, C. Eppner, C. Paxton, and D. Fox. 6-dof grasping for target-driven object manipulation in clutter. In IEEE Conference Robotics and Automation (ICRA), 2020.
[3] C. Mitash, R. Shome, B. Wen, A. Boularias, and K. Bekris. Task-driven perception and manipulation for constrained placement of unknown objects. IEEE Robotics and Automation Letters, 2020.
[4] M. Danielczuk, A. Mousavian, C. Eppner, and D. Fox. Object rearrangement using learned implicit collision functions. arXiv preprint arXiv:2011.10722 2020.
[5] C. Xie, Y. Xiang, A. Mousavian, and D. Fox. Unseen object instance segmentation for robotic environments. IEEE Transactions on Robotics (T-RO), 2021.
[6] Y. Xiang, C. Xie, A. Mousavian, and D. Fox. Learning rgb-d feature embeddings for unseen object instance segmentation. In Conference on Robot Learning (CoRL), 2020.
[7] K. He, G. Gkioxari, P. Dollár, and R. Girshick. Mask r-cnn. In IEEE International Conference on Computer Vision (ICCV), 2017.
[8] L. Jiang, H. Zhao, S. Shi, S. Liu, C.-W. Fu, and J. Jia. Pointgroup: Dual-set point grouping for 3d instance segmentation. In IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2020.
[9] D. Xu, Y. Zhu, C. B. Choy, and L. Fei-Fei. Scene graph generation by iterative message passing. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017.
[10] V. Chen, P. Varma, R. Krishna, M. Bernstein, C. Re, and L.. Fei-Fei. Scene graph prediction with limited labels. Proceedings of the IEEE International Conference on Computer Vision, 2019.
[11] R. Zellers, M. Yatskar, Thomssn, and Choi. Neural motifs: Scene graph parsing with global context. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018.
[12] P. F. Felzenszwalb and D. P. Huttenlocher, Efficient graph-based image segmentation. Interna- tional Journal of Computer Vision (IJCV), 2004.
[13] A. Trevor, S. Gedikli, R. Rusu, and H. Christensen. Efficient organized point cloud segmentation with connected components. In 3rd Workshop on Semantic Perception Mapping and Exploration (SPME), 2013.
[14] S. Christoph Stein, M. Schoeler, J. Papon, and F. Worgotter. Object partitioning using local convexity. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2014.
[15] Y. Li, H. Qi. J. Dai, X. Ji, and Y. Wei. Fully convolutional instance-aware semantic segmentation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
[16] L.-C. Chen, A. Hermans, G. Papandreou. F. Schroff, P. Wang, and H. Adam. Masklab: Instance segmentation by refining object detection with semantic and direction features. In IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2018.
[17] A. Kirillov, Wu, He, and R. Girshick. Pointrend: Image segmentation as rendering. In IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2020.

---

[18] B. De Brabandere, D. Neven, and L. Van Gool. Semantic instance segmentation with discriminative loss function. arXiv preprint 2Xiv::708.02551, 2017.
[19] D. Neven, B. De Brabandere, M. Proesmans, and L. Van Gool. Instance segmentation by jointly optimizing spatial embeddings and clustering bandwidth. In IEEE Conference on Computer Vision and Puttern Recognition (CVPR), 2019.
[20] D. Novotny, S. Albanic, D. Larlus, and A. Vedaldi. Semi-convolutional operators for instance segmentation. In European Conference Computer Vision (ECCV), 2018.
[21] I.. Shao, Y. Tian, and J. Bohg. Clusternet: 3d instance segmentation rgb-d images. arXiv preprint arXiv: 1807 08894, 2018.
[22] S. Kong and C. Fowlkes. Recurrent pixel embedding for instance grouping. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
[23] M. Danielczuk, M. Matl, S. Gupta, A. Li, A. Lee. J. Mahler, and K. Goldberg. Segmenting unknown 3d objects from real depth images using mask r-cnn trained on synthetic data. In IEEE Conference on Robotics and Automation (ICRA), 2019.
[24] O. Pinheiro, R. Collobert, and Dollár. Learning to segment object candidates. In Advances in Neural Information Processing Systems (NeurIPS), 2015.
[25] O. Pinheiro, T.-Y. Lin, R. Collobert, and P. Dollár. Learning to refine object segments. In European Conference Computer Vision (ECCV), 2016.
[26] W. Kuo, A. Angelova, Malik, and T.-Y. Lin. Shapemask: Learning to segment novel objects by refining shape priors. In IEEE International Conference Computer Vision (ICCV), 2019.
[27] C. Xie, Y. Xiang, Z. Harchaoui, and D. Fox. Object discovery in videos as foreground motion clustering. In IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2019.
[28] A. Dave, P. Tokmakov, and D. Ramanan. Towards segmenting everything that moves. arXiv preprint 1rXiv:1902.000 00010.0......... 2019.
[29] L. Shao, P. Shah, V. Dwaracheria, and J. Bohg. Motion-based object segmentation based on dense rgb-d scene flow. IEEE Robotics and Automation Letters, 3:3797-3888 2018.
[30] C. Xie, Y. Xiang, A. Mousavian, and D. Fox. The best of both modes: Separately leveraging rgb and depth for unseen object instance segmentation. Conference on Robot Learning (CoRL), 2019.
[31] V. Garcia and J. Bruna. Few-shot learning with graph neural networks. Conference Learning Representations (ICLR), 2018.
International
[32] X. Wang, Y. Ye, and A. Gupta. Zero-shot recognition via semantic embeddings and knowledge graphs. In Proceedings of the IEEE conference computer vision and pattern recognition, 2018.
[33] H. Hu, J. Gu, Z. Zhang, J. Dai, and Y. Wei. Relation networks for object detection. In Proceedings of the IEEE conference on computer vision and patlern recognition, 2018.
[34] X. Liang, L. Lin, X. Shen, J. Feng, S. Yan, and E. P. Xing. Interpretable structure-evolvvng Istm. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2017.
[35] A. Santoro, D. Raposo, D. G. Barrett, M. Malinowski, R. Pascanu, Battaglia, and T. Lillicrap. A simple neural network module for relational reasoning. In Advances in Neural Information Processing Systems (NeurIPS), 2017.
[36] J. Johnson, R. Krishna, M. Stark, Li,J, retrieval using scene graphs. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2015.
Li, D. Shamma, M. Bernstein, and L. Fei-Fei. Image
10

---

[37] P. Battaglia, R. Pascanu, M. Lai, D. Jimenez Rezende, et al. Interaction networks for learning about objects, relations and physics. Advances neural information processing systems, 29: 4502-4510, 2016.
[38] M. B. Chang. T. Ullman, A. Torralba, and J. B. Tenenbaum. A compositional object-based ap- proach to learning physical dynamics. In International Conference on Learning Representations, (ICLR), 2017.
[39] P.-T. De Boer, D. P. Kroese, S. Mannor, and R. Y. Rubinstein. A tutorial the cross-entropy method. Annals of operations research, 134(1):19-67, 2005.
[40] O. Ronneberger, P. Fischer, and T. Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI), 2015.
[41] P. W. Battaglia, J. B. Hamrick, V. Bapst, A. Sanzhez- V. Zambaldi, M. Malinowski, A. Tacchetti, D. Raposo, A. Santoro, R. Faulkner, et al. Relational inductive biases, deep learning, and graph networks. arXiv preprint arXiv: 1r0Xiv1806.000 082661.66, 2018.
[42] P. Ochs, J. Malik, and T. Brox. Segmentation of moving objects by long term video analysis. IEEE transactions on pattern analysis and machine intelligence, 2014.
[43] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
[44] T.-Y. Lin, Dollár, R. Girshick, K. He, B. Hariharan, and S. Belongie. Feature pyramid networks for object detection. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
[45] Y. Aytar, L. Castrejon, C. Vondrick, H. Pirsiavash, and A. Torralba. Cross-modal scene networks. IEEE transactions pattern analysis and machine intelligence, 40(10):2303-23144 2017.
[46] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick. Microsoft coco: Common objects in context. In European Conference Computer Vision (ECCV), 2014.
[47] M. Suchi, T. Patten, and M. Vineze. Fasylabel: A semi-automatic pixel-wise object annotation tool for creating robotic rgb-d datasets. In IEEE Conference on Robotics and Automation (ICRA), 2019.
[48] A. Richtsfeld,
T. Mörwald, J. Prankl, M. Zillich, and M. Vineze. Segmentation of unknown objects in indoor environments. In IEEF/RSJ International Conference on Intelligent Robots and Systems (IROS), 2012.
[49] A. X. Chang, T. Funkhouser, L. Guibas, P. Hamrahan, Q. Huang, Z. Li, S. Savarese, M. Savva, S. Song, H. Su, J. Xiao, L. Yi, and F. Yu. ShapeNet: An Information-Rae Rich 3D Model Repository. Technical Report
2015.
[50] K. Järvelin and J. Kekäläinen. Cumulated gain-based evaluation of ir techniques. ACM Transactions on Information Systems (TOIS), 20(4):422-446, 2002.
[51] L. Chang, J. R. Smith, and D. Fox. Interactive singulation of objects from pile. In 2012 IEEE International Conference on Robotics and Automation, 2012.
[52] M. Sundermeyer, A. Mousavian, R. Triebel, and F. Dieter. Contact-graspne Efficient 6- dof grasp generation in clutteredscenes. IEEE International Conference on Robotics and Automation (ICRA), 2021.
11


Reviews:
[{"review_id": "ny-62AO_8J2", "reviewer": "Reviewer_QM46", "summary": "The paper presents a refinement method for instance masks in cluttered environments. The method uses the relationship between overlapping objects in a cluttered scene to improve the instance segmentation of each individual object. The presented algorithm combines a graph neural network and combinatorial optimization to find an ideal set of interacting instance masks.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "+ Well written\n+ Good improvement over multiple baseline algorithms (general)\n- Unlikely reproducible\n- Somewhat arbitrary\n\nThe paper is generally well written and easy to follow. The presented algorithm is interesting, and yields a good improvement over multiple baselines.\n\nWhile the paper does a good job explaining things that are contained in it, it lacks some details that would make both experiments and method reproducible. Are the authors open to releasing code and data upon a potential acceptance?\nSome of the details that were unclear from the current submission: What data-augmentation is used when training the different SO-Nets? \n * How are 'training' samples constructed? Do they take into account potentially wrong initial segments of baseline detectors?\n * How many of the baselines use the same input modalities (RGB-D) as the presented method? How many use just RGB or D? Might some of the gains come from these additional modalities?\n * Could RICE work from scratch? Why does it need a baseline detector?\nEven with the additional details in the supplement, it would be hard to reproduce without code, which greatly diminishes the value of the submission.\n\nFinally, the method is at times a bit arbitrary. There are many design choices that are simply states, and neither motivated or ablated. For example:\n * SplitNet could as well just predict a binary mask instead of its boundary that needs to be sampled and optimized. I understand that there is some ambiguity in the prediction (both the binary mask and its complement are valid answers), but this can easily be broken using either a single positive example, a hand-coded ordering (left is mask 1, right is mask 2), or even a Hungarian loss. It would be nice to know that the presented boundary prediction is in deed somewhat optimal. This is especially important since most of the presented method replies in a good split (and most other components either hurt or do not significantly improve the performance, see supplement).\n* In a similar spirit, the AddNet relies on a proposal constructed from a connected component, why not use an RPN?\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper starts with an interesting idea, but does not provide sufficient evidence to show the efficacy of the method.", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {"issues": "See issues above.\n\nIn addition, it would be nice to have a more detailed comparison of SplitNet and the entire pipeline, as the performance if split alone is almost the same as the complete combinatorial optimization."}}, {"review_id": "fj631LfLvuf", "reviewer": "Reviewer_5yWX", "summary": "In this paper, the authors propose a neural-network-based method to refine segmentation masks provided by different models. In particular, their goal consists in improving unknown object segmentation masks in cluttered environments, where classical methods fail.The idea consists in (1) representing masks as graphs and use a GNN that produces variations of input masks, and (2) use an additional DNN to evaluate such variations and select them. The set of variations is generated iteratively through sampling and using CEM.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "PROS:\n* The paper is well structured and well written\n* The idea is interesting and easy to apply, as it does not have particular requirements\n* The proposed method can be applied (post-training) to several architectures to improve their results\n* Although the approach is stochastic, the sampling statistic is is low and predictions are stable\n* Experimental evaluation on 4 architectures, 2 of which are recent and task-optimized\n* Experimental evaluation includes a use-case showing the utility of the method\n\nCONS:\n* std-dev is studied in the sampling procedure, but it is unclear on how many trial it is computed\n* The paper does not report how many training runs have been performed: if results are average results, then std-dev is lacking\n* The approach makes use of some hyper-parameters (e.g., branching factor, vicinity threshold, max node budget): there is no analysis on how those affect the results\n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper is well written and well structured. The proposed approach is interesting, easily applicable, and well supported by\nexperimental evaluations. This said, the document could be further improved by performing in-depth analysis of hyper-parameters\nas well as better studying the average/std-dev of the training performance.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {"issues": "* Section 3.2 is difficult to understand as it is currently placed. It would be better placed after 3.4\n* Why not using IOU as a metric? For example, https://arxiv.org/pdf/2103.06796.pdf uses it.\n* The paper does not report how many training runs have been performed: if results are average results, then std-dev is lacking\n* How do the hyper-parameters affect the performance?"}}, {"review_id": "b0_46MmOFM", "reviewer": "Reviewer_cMBw", "summary": "This paper proposes an approach for image segmentation using graph-based representation and several deep neural networks trained to perturb and score different image segmentations. I like the framework architecture in general, and the results look very promising. But I'm a bit suspicious that this paper is more suited for a computer vision venue than for a robotics one. However, I'm going to give the authors the benefit of the doubt and vote for acceptance.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The approach seems straightforward to re-produce.\n- The paper is well-written and organized.\n\nWeaknesses:\n- As mentioned by the authors, the framework is pretty computation-heavy at the moment.\n- The robotics experiment is very simplistic and it uses an out-of-the-box grasping method (probably developed by the authors in their previous work).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The robotic experiments seem very simplistic to me. But the segmentation approach is very interesting and the results look good, and since this can have lots of applications in robotics, I vote for acceptance.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "1: Not very relevant to the robot learning community.", "issues": "- In Line 98, h and w are not defined.", "reviewer_expertise": "Fair: Some knowledge of the area"}, "extra_text": {}}, {"review_id": "0l8Ub0P0v3", "reviewer": "Reviewer_1fhJ", "summary": "The authors propose a GNN-based refinement method for unknown instance segmentation based on RGB-D data. Therefore, they build a graph by connecting segments that are close in pixel space. Then they perform an informed perturbation of that graph by splitting, merging, adding and/or deleting segments and finally rank the resulting graphs using another GNN. It is shown that the method can successfully improve results of recent unknown instance segmentation methods. The approach is applied to object singulation where it helps a robot to only singulate the objects with uncertain contours. ", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- A refinement of unknown segmentation masks by taking the whole scene into account had been missing in the field and the GNN-based approach is an expensive but convincing attempt. \n- The details how the network learns to predict splits, merges, adds and deletes are interesting\n- The ablations (also in the detailed appendix) show the impact of individual components\n- The method improves upon different baseline methods in the RGB-D domain on OCID and OSD\n- Previously used performance metrics are corrected to be independent to object size. \n- The approach is demonstrated on a robotic system\n- The paper is clearly written and seems relatively honest wrt. limitations\n\nWeaknesses (of the approach, paper issues will be listed below):\n\n- The large number of networks and forward passes due to the sampling operations make this method quite slow and memory-intensive to use in robotic applications (10-15s/frame). However, the authors and readers are aware of that and it is a good proof of concept. \n- The networks are completely agnostic to the approach that generated the initial masks. On the one hand, this is beneficial because the refinement can be easily used for different initial methods. On the other hand, it leads to a lot of redundancy, as you need a new feature extractor that learns what an object is. Also using the feature representation of UCN probably contains richer information than just a binary mask and the representation could be directly refined end-to-end by the GNN scores.\n- In the robotic experiments it seems like the network is able to merge/split/delete existing masks but has more troubles in adding completely new segments or completing undersegmented masks by adding+merging. This limitation could be made more clear, especially by showing separate results for adding and deleting in Table 1 of the appendix. The problem is probably the generation of candidate masks. I would be curious if an oversegmentation through low thresholds in the initial methods and subsequently only merging/splitting/deleting wouldn't lead to better results.\n- The approach is trained on non-photorealistic data (perhaps for better comparison) and then needs to be manually adapted to work better with real data. There might be a more elegant solution, maybe by directly training on more realistic data. \n- It would be interesting how the method deals with many similar objects", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "If my issues are resolved I recommend a strong accept. The paper presents an original idea on a relevant problem which has not been studied enough. The experiments are thorough and it is shown how the approach could benefit a robotic system.", "technical_quality": "Excellent: Very convincing and thorough.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "Excellent: Expert knowledge on the topic of the paper"}, "extra_text": {"issues": "- \"smart perturbations\" are used throughout the paper. I would rather say \"informed perturbations\" or \"learned perturbations\".\n- l275 \"numerical relevance\" should be defined as it is otherwise unclear and hard to find in the reference\n- Table 2: Is the worst ranking really a good comparison?\n- Figure 7: It is annoying to scroll up and down to get the description of the figure, which is actually in section 4.6. It would be great if you could move the descriptive part in the caption.\n\nSince code release was not announced:\n- l97: padding is not specified \n- l108: threshold is not specified\n"}}]

Author rebuttal:

Meta-review: All authors agree that the proposed method is interesting, novel and applicable to a wide array of segmentation algorithms. All reviewers agree that the paper is well-written and criticism relates to its reproducibility and design choices (QM46) or choices in the evaluation (5yWX, cMBw). However, three of four reviewers recommend acceptance (1x strong accept, 2x weak accept, 1x weak reject). Although this paper may be a better fit for a computer vision conference, I am following this assessment. 
Decision: Accept (Poster)