Connectivity-constrained interactive annotations for panoptic segmentation
Authors: Ruobing Shen
Bo Tang
Ismail Ben Ayed
Andrea Lodi
Thomas Guthier
Venue:  (2020)
Keywords: Panoptic Segmentation
Semantic Segmentation
Interactive Segmentation
Integer Programming

Abstract: Large-scale ground truth data sets are of crucial importance for deep learning
based segmentation models, but annotating per-pixel
masks is prohibitively time consuming. In this paper, we investigate interactive graph-based segmentation algorithms that enforce connectivity. To be more precise, we introduce an instance-aware heuristic of a discrete Potts model, and a class-aware Integer Linear Programming (ILP) formulation that ensures global optimum. Both algorithms can take RGB, or utilize the feature maps from any DCNN, whether trained on the target dataset or not, as input. We present competitive semantic (and panoptic) segmentation results on the PASCAL VOC 2012 and Cityscapes dataset given initial scribbles. We also demonstrate that our interactive approach can reach $90.6\%$ mIoU on VOC validation set with an overhead of just $3$ correction scribbles. They are thus suitable for  interactive annotation on new or existing datasets, or can be used inside any weakly supervised learning framework on new datasets.

Under review as a conference paper at ICLR 2020
CONNECTIVIY--COSTTRANDD INTERACTIVE ANNO- TATIONS FOR PANOPTIC SEGMENTATION
Anonymous authors
Paper under double-blind review
ABSTRACT
Large-scale ground truth data sets are of crucial importance for deep learning based segmentation models, but annotating per-pixel masks prohibitively time consuming. In this paper, we investigate interactive graph-based segmentation al- gorithms that enforce connectivity. aware heuristic of discrete Potts model, and a class-aware Integer Linear Pro- gramming (ILP) formulation that ensures global optimum. Both algorithms can take RGB, utilize the feature maps from any DCNN, whether trained on the target dataset or not, as input. We present competitive semantic (and panoptic) segmentation results the PASCAL VOC 2012 and Cityscapes dataset given ini- tial scribbles. We also demonstrate that our interactive approach can reach 90.6% mloU on VOC validation set with overhead of just correction scribbles. They are thus suitable for interactive annotation on new or existing datasets, or can be used inside any weakly supervised learning framework on new datasets.
more precise, we introduce an instance-
INTRODUCTION
Deep Convolutional Neural Networks (DCNNs) excel at wide range of image recognition tasks (He) ct al. 2016; Ren et al. 2017; Shelhamer et al. 2017), such as semantic segmentation (Chen et al. 201 8b Shelhamer et al. 2017 Zhao al. 201 7) and panoptic segmentation (Kirillov al. 2019b
Yang et al. 2019 Xiong et al. 2019 Kirillov lov et al. 2019a). Semantic segmentation studies the tasks of assigning class label to each pixel of an mmgee while instance segmentation (Chen et al. 2018a) detects and segment each object instance. Panoptic segmentation unifies both tasks that investigate to segment both things (such as person, cars) and stuff (such as road, sky) classes.
While DCNNs show outstanding results for semantic and panoptic segmentation, they have two con- ceptional problems. First, they require huge amounts of annotated data. Annotating image segmen- tation masks is a very time consuming and labor extensive task. For example, annotating semantic image mask took "more than 1.5h on average" on the Cityscapes dataset (Cordis et al. 2016). In addition, DCNNs rely on their implicitly learned generalization probability and most of the state- of-the-art architectures do not make use of any domain specific knowledge, such as neighborhood relations and connectivity priors for segments. On the contrary, classical graph based segmenta- tion models (Boykov et al. 2001] Yedidia et al., (203) do not require any learning data and can incorporate specific domain knowledge. Their major drawback is that they rely on -emanndesigned similarity features and require complex optimization algorithms or solvers, which are mainly CPU based and non-suitable for real time applications.
In this work, we explore the combination of DCNNs and graph-based algorithms for annotating ground truth segmentation. We propose two interactive algorithms that both enforce the connectiviry prior (to be more precise in Sec. 2.T Specifically, we design a heuristic region growing method based on the Potts model (Potts) 1952). as well as an integer linear programming (II.P) formulation of the markov random field (MRF) (Boykov et 1998). We utilize scribble based annotations from human annotators as initialized or iterative hard constraints for our algorithms, which is typical in (HITL) annotation process. We explore two different scenarios. In the first scenario, we assume that for the segmentation task, there is already a pre-trained DCNN with the same class mapping. Thus, input to our algorithms. In the second scenario, we assume that no pre-trained DCNN for the same objective is available, This is true for lot of existing datasets, e.g., Cityscapes does not contain any
could make full use of DCNN's probability map and scribbles as

---

Under review as a conference paper at ICLR 2020
Left: image with scribbles from Paseal VOC dataset. Mid-left: semantic segmentation result of our heuristic with layer 3 of ResNet 101. Mid-right: result of our ILP with probability map of DeepLab V2. Right: ILP without connectivity prior (with Deeplab V2 prob. map).
Figure 1:
class labels for lane marking, which is crucial information for an autonomous vehicle. In this case, we cannot use the class specific probability map, but the more generic low level features of a DCNN can be utilized as feature description for the algorithm.
To investigate the general purpose of low level DCNN features, we compare DCNN trained on ImageNet (Russakovsky et al. 2015) and COCO (Lin et al. 2014) with and without fine tuning on the target dataset. Our experiments show that incorporating the connectivity prior well as the DCNN features greatly improves the algorithm performance. We present competitive semantic (and panoptic) segmentation results the PASCAL 2012 (Everingham al. 2015) and Cityscapes dataset. To the best of our knowledge, our method is the first non-DCNN" panoptic segmentation algorithm Cityscapes with competing results, which shows the potential improvement gained by a combined approach. See Fig. for a visualization that uses initialized scribbles as input.
We also prototype algorithms for the task of interactive annotation. Based initial scribbles and a baseline DCNN's probability map, our algorithms acheive 90.9% mloU on VOC validation set, with just rounds (1 scribbles per image each round) of correction scribbles.
Summarized, our key contributions are
novel heuristic algorithm and class-aware ILP formulation with connectivity prior,
in depth analysis of combination of DCNN and graph-based segmentation algorithms,
extensive evaluation of the scribble based interactive algorithms for semantic and panoptic segmentations on two challenging datasets.
Our proposed algorithms have multiple use cases in annotating datasets for segmentation. First, they can be used inside any HITL annotation tool, as the annotator interacts with the image in forms of scribbles until satisfaction. Second, given initial scribbles, they can used inside any weakly or semi-supervised learning framework for semantic (and panoptic) segmentation.
Related Work. The procedure of annotating per pixel segmentation masks is similar interac- tive image segmentation, which is widely studied in the past decade. The method using bounding boxes (Rother al. (2004) is suitable for instance segmentation, which requires the user draw the box as tight as possible. Recently, 4 extreme points (Maninis et al. [2018 Papadopoulos et al. (017)) clicking is used methods (Russell et al. 2008) require users to carefully click the extreme points of things and stuff, and the accuracy heavily depends on the number of clicks, Among others, scribbles are recognized
alternative which shows superior result. On the other hand, polygon based
as more user-friendly way (Boykov & Jolly. 2001 Levin et al. 2008). Moreover, it is also natural to annotate stuff classes using scribbles.
Modern annotation tools often adopt interactive deep learning based methods, including Polygon- RNN++ (Acuna et al.) 2018) and Curve-GCN (Ling et al. 2019), which allows the annotator to click on the boundaries. Notably, recent approach (Agustsson et al. 2019) enables both extreme points clicking and scribbles correction. Besides, they can also take advantage ensemble learn- ing (Zhoul 2009), combine several inference results to produce a better segmentation. However, these methods requires an existing ground truth dataset for DCNNs to learn on the first hand, which may not be available when unknown domains new classes introduced.
2

---

Under review as a conference paper at ICLR 2020
As cheaper alternative, weakly supervised learning has drawn a lot of attention recently. (Dai et al. 2013 Khoreva et al. 2017 Li et al. 2018) claim that weakly iteratively trained by just bounding boxes and image tags, the DCNN can achieve 95% segmentation score compared to fully supervised on VOC. Since this method emphasizes on thing classes, it has worse score (or even none) on stuff' classes. Instead, (Lin et al.) 2016, Can al., 2018) claim that iteratively training a DCNN by scribble annotations suffers only small degradation in performance on both thing and stuff classes.
widely used for denoising and segmentation. The authors of (Shen et al. 2017) formulate the problem as an ILP and try to solve the global optimum, but only to reduced image size, while (Nguyen & Brown 2015) proposed an effi- cient region-fusion fusion-based heuristic algorithm. The MAP-MRF (maximizing posterior in an MRF) has been well studied for image segmentation. Previous methods focus on local priors (Boykov & Veksler, 2006 Krähenbühl & Koltun| 2011). and efficient approximate algorithms exist, e.g., graph cut (Boykov et al. 2001) and belief propagation (Yedidia al. (003). MRF with connectivity prior has been studied in (Remplier et al. 2016; Nowozin & Lampert 2009) and formulated as ILP, which can be solved by any ILP solver (Anand et al., (017)). Since solving an ILP exactly is in general -ar-hard (Land & Doig 1960), they either solve relaxation of the ILP, or focus the binary MRF case.
For graph-based methods, the (discrete) Potts model (Potts) 1952)
Our proposed interactive (in forms of scribble) multi-label annotation algorithms are graph based, with glohal connectivity prior, i.e., they enforce pixels of the same label to be connected, which allows the annotator better control over the final segmentation.
PROPOSED APPROACH
2.1 PREREQUISITE
Given an image, we build an undirected graph G = (V, where the node set V represents a of pixels (or superpixels) and E' set of edges consisting of un-ordered pairs of nodes. Image segmentation can be transformed into a graph labeling problem, where the label set C' is pre-defined.
When talking about segmentation, we need to first distinguish between class, instance, and region ID of a node. In semantic segmentation, the task is to assign a class label to each node in a graph. In panoptic segmentation, one has further assign an instance ID to the node that belong the "thing" class. In this paper, our algorithms require an additional region ID, which is linked to scribble and we assume nodes with the same region ID must be connected (to be explained See. .3.2. This is to deal with the case where an object of the "thing" or "stuff" class is separated into several connected regions, e.g., the car in Fig. 2 is separated into two regions by tree. Afterwards, the class and region labels can be used to generate panoptic segmentation.
2.2 CONNECTIVITY
OPTIMIZATION ALGORITIMMS
We first discuss the formal definition of connectivity, and the two proposed algorithms in details, ie., the class-agnostic heuristic algorithm of the discrete Potts model and the class-awan ILP of MRF with connectivity constraints.
2.2.1 THE CONNECTIVITY PRIOR
Two nodes u, graph G are connected if there a (u, v)- in G. called connected every pair of nodes connected in G, otherwise it is disconnected. Let Ge C G be connected subgraph where every node is labeled & e C. Then, the image segmentation with connectivity constrains corresponds to find partition of G into (k- ICT) connected (and disjoint) subgraphs. Enforcing connectivity constraints itself is proven to be P-P-hard in (Nowozin & Lampert 2009).
2.2.2 THE to REGION FUSION BASED HEURISTIC
Given graph G(V, E), let y, be the information (either RGB or features from any DCNN) of node i, and w; be its estimated value, the discrete Potts model (Potts 1952) has the following form:
min Σ kev I|w; -vill2 + Σ (iee || wyllp
(1)
3

---

Under review as a conference paper at ICLR 2020
where À is the regularization parameter. Here, the first term is the data fitting and the second is the regularization term. We recall that the to norm of a vector gives its number of nonzero entries.
In this paper, introduce an iterative scribble based region fusion heuristic algorithm (which call f with the "class" and "region" ID for each node. In the beginning. the nodes covered by the same scribble grouped together and labeled with the same IDs, while all other nodes unlabeled and in their individual group. Note that different regions can share the same class ID. Then, the algorithm iterates over each group (outer loop) and its neighbors (inner loop) and decide whether to merge the neighbors or not, by checking their region IDs. If both groups have region IDs and are different, they cannot be merged. In all other cases, i.e., if both groups have no region ID or only one has it, the following merging criteria (Nguyen & Brown 2015) are checked:
o, 0 || - Y,l12 S -79 (a; + o.).
(2)
where o, denotes the number of pixels in group i. Y; the mean of image information (e.g., RGB color) of group i, and "ij denotes the number of neighboring pixels between groups and j. Here, B the regularization parameter, and increases over the outer loop number.
If (2) is satisfied, two groups are merged, and their labels are updated according to the following rule. If both groups have no "region" ID, the merged group still have none, hence unlabeled. If only one group has region ID, the merged group inherits the label, hence labeled.
After one outer loop over all groups, will increase and follows the exponential growing strategy where "iter" is the current number of outer loop and 1j is the growing parameter. The procedure goes on until all groups are labeled, and the complexity of to H is On),, where n. is the number of nodes.
of (Nguyen & Brown 2015). i.e., 3 = ( iter )2
50
Note that the above algorithm approximate to problem [ and connectivity of each region is enforced at every step. Given desired scribbles, &o is able to generate panoptic segmentations (and also semantic segmentations). Also note that the class does not play any role the algorithm, inherits from the scribble and propagates with the region ID. Hence, this algorithm is class-agnostic.
2.2.3
THE ILP FORMULATION WITH CONNECTIVITY CONSTRAINTS
The MRF with pairwise data term can be formulated as the following ILP:
min, Evec Eiev dat E
dislaf - 251
(3)
E EiEo xf = f ViEV, LEC,
ViE V,
3
where of denotes the unary data term for assigning class label l to node i (hence class- dy the simplified pairwise term for assigning i. different labels, and is the regularization parameter. Constraint (3 enforces that to each node is assigned exactly one label, i.e., xf if and only if node i is labeled f. Note that the absolute term can be easily transformed into linear terms by intro- ducing additional continuous variables. The model above does, however, not guarantee connectivity, which is instead enforced as follows.
Connectivity constraints with root node Let r (the first node of scribble ? denote the root node of subgraph G.. Then, the following constraints of (Rempfler et al. 2016) suffice to characterize the set of all connected subgraphs of label that contain
SES 2. Vi V\r (i,r) & E, VS S(i,r),
(4)
where S' is the vertex-separate set of (i,r), i.c., if the removal of S from G disconnects i and And S(i, r) is the collection of all seeta--eeparator sets of (i,
The number of constraints 4 is exponential with respect to the number of nodes in G. hence it is not possible to include into problem (3) all of them at in once. In practice, they are added iteratively when needed (called the cutting planes method (Kelley) 1960)). In this paper, adopt the K- nearest cut generation strategy (Rempfler et al., 2016) to add a few of them at each iteration. See appendix for more details.
4

---

Under review as a conference paper at ICLR 2020
Sky
Sky
Road
/
Car
Road
Figure 2: Left: our scribble policy to draw as many scribbles as there are separated regions. Right: ILP-P that introduces pseudo edges which connect multiple regions the same class.
For the simple where the region IDD coincides with the class ID, i.e., the number of regions equals that of classes, problem (3) with connectivity prior is solved as follows. We first solve (3) and check if all subgraphs G; are connected. If not, we identify disconnected components, adds constraints (4). and solve the resulting ILP again. This procedure continues until all subgraphs are connected. This method ensures global optimumality if no time limit is restricted.
For the case where k regions share the same class ID, we introduce an improved formulation (we call ILP-P) where we add k pseudo edges that "connect" all separated regions of the same class (illustrated in Fig. 2 In particular, this does not increase the number of variables and is class-aware. Then, the solving process follows the aforementioned framework. But ILPP- only designed for semantic segmentations, i.e., it is region-aware. Post-processing methods are needed to separate instances within the same class for panoptic segmentation.
3 OVERALL WORKFLOW
In this section, describe the pre-processing steps well the scribble policy for panoptic and interactive segmentation.
3.1 SUPERPIXELS DIMENSION REDUCTION
Superpixels have long been used for image segmentation (Stutz et al. 2017 Achanta et al. 2012). as they can greatly reduce the problem size while not sacrificing much of the accuracy. In this paper, we the PASCAL VOC 2012 dataset,
adopt SEEDS (den Bergh et al. 2012) generate superpixels while using a deep learning based method (Tu et al. 2018) on the more challenging Cityscapes dataset. We then build region adjacency graph (RAG) G(V, E) of the superpixels, where each superpixel forms a node (vertex) and edges connect two adjacent superpixels.
3.2 EXTRACTING IMAGE FEATURES USING SCRIBBLES
Our segmentation algorithms scribble supervised, which two folded. On the one hand, the node labels, such as class, instance and region IDs are fixed the nodes are covered by the corre- sponding scribbles. On the other hand, for the ILP algorithm, if no high level image information (i.e., probability map) exists, the scribbled superpixels will be used to extract information for the class, i.e., one can use the average of the scribbled superpixels information represent each class.
Although the input to the algorithms can be as simple as image RGB information, one can also take advantage of the modern DCNN extract deeper features. We distinguish scenarios:
No previous training data is available starts annotating images in dataset.

---

Under review as a conference paper at ICLR 2020
Training data available one continues annotating more images of an existing dataset.
In the former case, other than RGB, one can also adopt any base network (i.e.. ResNet 101 (He et al, 2016)) mre-trained other datasets (i.e., ImageNet) and use the output of the low level features that extracts image edges, textures, etc. In the later case, one can fully utilize any modern DCNN trained
on the existing dataset, and use the output of the final layer (i.e., probability map).
Scribble generation policy For panoptic segmentation, first of all, the scribble itself must be connected. Second, one has draw as many scribbles as there are connected regions (both "thing" and "stuff" class) presented in the image. For example, if an object is cut into separated regions, one has to draw a scribble on each region. One sample image with scribbles shown in Fig.
3.3 INTERACTIVE SEGMENTATION USING SCRIBBLES
In a HITL annotation framework, it is typical that the annotator makes corrections to the current segmentation until satisfaction. In this paper, we simulate the human correction scribbles (described in Appendix A.3) We then report analysis of the segmentation results up to 3 rounds of scribbles correction in Sec.
4 SEMANTIC AND PANOPTIC SEGMENTATION EXPERIMENTS
4.1 EXPERIMENTAL. SETUP
In this session, conduct extensive experiments the Pascal VOC 2012 and Cityscapes validation set. In all our experiments, when we mention base network, we refer to the publicly released ResNet 101 that is pre-trained on ImageNet and COCO dataset. We adopt DeepLab V2 (Chen al. 2018b) (without CRF as post-processing) and DRN (Yu et al., (017) as our baseline DCNN to get the probability maps, trained on their corresponding training sets. We adopt IBM Cplex (Bliek et al., 2014) version 12.8 to solve the ILP. All computational experiments are performed on a Intel(R) Xeon(R) CPU E5-2620 v4 machine, with 64 GB memory.
We report the semantic and panoptic segmentation scores, where the pixel mean intersection over union (mloU) commonly used for semantic segmentation, and the panoptic quality (PQ) metric is newly introduce in (Kirillov et al.] (019b) and is a combination of segmentation quality (SQ) and recognition quality (RQ). The parameters for both heuristic and ILP is described in Appendix A.4
4.2 RESULTS ON PASCAL VOC 2012
Pascal VOC 2012 has 20 "thing" classes and single "background" class for all other classes. We evaluate our algorithms on the 1449 validation images. We first apply (den Bergh et al.] (0122) to produce around 700 superpixels, and use the public scribbles set provided by (Lin et al. (0066) as ini- tial scribbles. Since some of these scribbles do not meet our policy (deseribed in Sec. 3.2). only the semantic segmentation is reported, in terms of mloU averaged across the 21 classes. Visualization of 2 experiments are presented in Appendix A.5
No training data In this case, one can either use RGB or the output of lower level features of base network as input (y,) to our algorithms. We compare our class-agnostic heuristic (fg/d) using RGB or different low level features from ResNet 101 as input, against ILP--. We do not set any time or step limit for the heuristic, but time limit of 10 seconds for ILP-P (denoted ILP-P-10). We report Table [ the detailed comparison, where we use the RGB, first and third layer of ResNet 101 as input to H, and "Dim" is the dimension of the input feature map.
We can see in Table the advantage by incorporating lower level features maps of ResNet 101, that improves H of RGB by 1.8%, even though it is pre-trained on completely different dataset. ILP-P adopts llayer as initial solution, and further increase the mloU by 0.3%.
With training data If training data is available, use the probability map of DeepLab V2 (with baseline mloU 70.5%) trained on VOC training set as input to our algorithms, foHl get another huge boost of 10.1% to 81.7% compared using layer 3. We compare II.P (3) without connectivity

---

Under review as a conference paper at ICLR 2020
Table Comparison on VOC 2012 val set when Table 2: Comparison when training data avail- no training data is available.
able, bascline DeepLab V2 (Chen et al. 2018b).
Model
Dim
Time
mloU
Model
Time (s)
mloU (%)
4gH-RGB
2.2
69.8
DeepLab V2
70.5
&o Hlayer 1
64
2.9
70.8
MRF-pprbb
0.2
80.8
H-
256
3.9
71.6
tg H-rob
0.8
81.7
ILP-P-10
9.7
71.9
ILP-P-10
7.9
84.6
Figure 3: Top: Cityscapes original image and image with superpixels and artificial scribbles, Bot-
tom: semantic and panoptic segmentation of
with DCNN's probability map as input.
that is solved to optimum (MRF-prob) with H and found out it is 0.9% worse, which shows the importance of the connectivity prior. We then compare ILP-P with H as initial solutions (81.7% mloU) and a time limit of 10 seconds, Table 2 suggests that ILP-P-10 improves the baseline of Dee V2 by 14.1% and -prob by 2.9%. An example of visual comparison is illustrated in Fig. I Since ILP (3) is class-aware and encodes pairwise term daj. further boost on the performance can be expected, given better baseline DCNN and an edge detector.
4.3 RESULTS ON CITYSCAPES
Cityscapes has "thing" classes, and 11 "stuff" classes. While there is public scribbles
simulate our own that meet the policy in Sec. 3.2 The procedure is described in Appendix A.2, and one can see Fig. for an example. We evaluate our algorithms the 500 validation images with the scribbles, where we first apply (Tu et al. 2018) to produce around 2000 superpixels.
No training data We use RGB and output of layer 3 of Resnet 101 as input to H, and report in Table 3 both mloU and PQ. We also compare H to recent weakly supervised learning method (Li et al. 2018). uses a more powerful PSPNet (Zhao et al., 2017) supervised by bounding boxes and image tags, but requires end end training. / shows superior results on both semantic and panoptic segmentation, both around 10% boost compared to (Li et al. 2018). Note that it is not fair comparison since ours requires scribbles while (Li et al. 2018) does
inference time.
With training data We use the public full-supervised (trained Cityscapes) DRN (Yu et al.) 2017 as our baseline (71.4% mloU), and run H and ILP-P using its probability map. Table shows HoH-pro -prob improves the baseline by 4.4%, and by 1.5% compared to HoHellyer 3 Besides, PQ also increases from 49.6% to 51.2%. Because the rich class and instance information (more than 15 classes per image) of Cityscapes and also that around 2000 superpixels (around 30k

---

Under review as a conference paper at ICLR 2020
Table Comparison on Cityscapes val set with and without its training data set. The later is based on DRN (Yu et al. 2017))ss probability map.
Model
Time
mloU
PQ
SQ
RQ
loH-RGB
6.5
74.2
49.6
74.3
63.8
lo Hollllyerr
7.2
74.3
49.6
74.5
63.7
Weakly (Li et al. 2018
63.6
40.5
DRN (baseline)
71.4
HgH-proo
7.2
75.8
51.2
75.6
64.8
binary variables), ILP struggles to find any better solution within the time limit of 10 seconds. The score remains unchanged compared to &o f, and hence not shown the table. More visualization examples can be seen in Appendix A.6 A.6
4.4 RESULTS INTERACTIVE SEGMENTATION
As described in Appendix A.3 we simulate 3 rounds of scribbles (1 scribble per image per round) to correct the largest error previous segmentation. We report their mloU, compared to the base- line DCNN (DeepLab V2) and current reported state-of-the-art (SOTA) deep learning approach (DeepLab V3+ with 84.6%) (Chen et al., 2018c) on VOC validation set. Results show that our al- gorithms benefit from addition scribbles and the performance increases significantly. The best result is reported by ILP-P-10 at iteration 3 (90.6%), a 6.0% mloU gain with just 3 correction scribbles.
VOC 2012 val set
95.0%
90.6%
90.0%
87.9%
50.2%
89.0%
85.0%
86.6%
mlou
84.6%
91.7%
61.1%
Deeplab v2
30.0%
ljt+layera
70.4%
(ith prob
ILP-P-10
71.6%
70.0%
SOTA
iter
Rerl
iter2
iterr
Figure 4: Interactive segmentation on VOC 2012 val set. The mloU increases significantly with just one correction scribble per iteration. Blue dot is our baseline and red dot the current SOTA DCNN.
5 CONCLUSIONS AND FUTURE WORK
In this paper, we proposed two interactive multi-label graph based algorithms with connectivity prior. With DCNN's features and initial scribbles input, both achieve competitive semantic (and panoptic) segmentation results on VOC and Cityscapes validation set. The interactive approach further boost the performance with little overhead of correction scribbles. It would be interesting to investigate our approach in weakly supervised setting, where connectivity constraints are imposed on the outputs of a deep network to leverage unlabeled data. Training with such discrete high-order constraints can be explored via ADM (Alternatigg Direction Method) schemes, for instance, in way similar to Marin et al. (2019). which showed promising performances of imposing discrete MRF on the outputs of convolutional networks. Finally, note that all the reported performance upper bounded by the accuracy of superpixels. Hence, using better superpixel algorithms or increasing its number may further influence the performance.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
R. Achanta, A. Shaji, K. Smith, A. Lucchi, P. Fua, and S. Susstrunk. Slic superpixels compared to state-of-the-ar superpixel methods. The IEEE Transactions on Puttern Analysis and Machine Intelligence (TPAMI), (4(11)22774-22288 2012.
D. Acuna, H. Ling, A. Kar, and S. Fidler. Efficient interactive annotation of segmentation datasets with polygon-rnn++. The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
E. Agustsson, J. R. R. Uijlings, and V. Ferrari. Interactive full image segmentation by considering all regions jointly. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 11622-11631, June 2019.
R. Anand, D. Aggarwal, and Kumar. comparative analysis of optimization solvers. Journal of Statistics and Management Systems, 20(4):623-63) 635, 2017.
C. Blick, P. Bonami, and A. Lodi. Solving mixed-integer quadratic programming problems with ibm-cplex: a progress report. Proceedings of the RAMP Symposium, 2014.
Y. Boykov and O. Veksler. Graph cuts in vision and graphics: Theories and applications. Handbook of Mathematical Models in Computer Vision, pp. 79-96, 2006.
Y. Boykov, O. Veksler, and R. Zabih. Markov random fields with efficient approximations. In The IEÉE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 648 655,,,, June 1998.
Y. Boykov, O. Veksler, and R. Zabih. Fast approximate energy minimization via graph cuts. The IEÉE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 23(11):1222-12333 Nov 2001.
Y. Boykov and M. Jolly. Interactive graph cuts for optimal boundary amp; region segmentation of objects in n-d images. In The Eighth IEEE International Conference Computer Vision (ICCV), pp. 105-112 vol.1,Jlly 2001.
Y. B. Can, K. Chaitanya, B. Mustafa, L. M. Koch, E. Konukoglu, and C. Baumgartner. Learning to segment medical images with scribble-supervision alone. In Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support, volume 11045, pp. 236-244, 2018.
I. Chen, A. Hermans, G. Papandreou, F. Schroff, P. Wang, and H. Adam. Masklab: Instance seg- mentation by refining object detection with semantic and direction features. The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4013-4022, 2018a.
L. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crís. The IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 40(4):834 848, April 2018b.
L. Chen, Y. Zhu, G. Papandreou, F. Schroff, and H. Adam. Encoder-decoder with atrous separable convolution for semantic image segmentation. arXiv e-prints, Feb 2018c.
M. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler, R. Benenson, U. Franke, S. Roth, and B. Schiele. The cityscapes dataset for semantic urban scene understanding. In The IEEE Conference Computer Vision and Puttern Recognition (CVPR), 2016.
Dai, K. He, and Sun. Boxsup: Exploiting bounding boxes to supervise convolutional networks for semantic segmentation. The IEEE International Conference Computer Vision (ICCV), pp- 1635-1643, 2015.
M. V. den Bergh, X. Boix, G. Roig. B. de Capitani, and L. Van Gool. Seeds: Superpixels extracted via energy-driven sampling. The European Conference on Computer Vision (ECCV), pp. 13-26, 2012.

---

Under review as a conference paper at ICLR 2020
M. Everingham, S. M. A. Eslami, L. Van Gool, C. K. 1. Williams, J. Winn, and A. Zisserman. The pascal visual object classes challenge: A retrospective. International Journal of Computer Vision, 111(1):98-136, Jan 2015.
K. He, Zhang, Ren, and Sun. Deep residual learning for image recognition. The IEEE Conference on Computer Vision and Puttern Recognition (CVPR), pp. 770-778, June 2016.
J. E. Kelley. The outting-plann method for solving convex programs. Journal of the Society for Industrial and Applied Mathematics, 8(4):703-712 1960.
A. Khoreva, Benenson, J. Hosang, M. Hein, and B. Schiele. Simple does it: Weakly supervised instance and semantic segmentation. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
A. Kirillov, R. Girshick, K. He, and P. Dollár. Panoptic feature pyramid networks. arXiv e-prints, Jan 2019a.
A. Kirillov, K. He, R. Girshick, C. Rother, and Dollár. Panoptic Segmentation. In The IEEE Conference Computer Vision and Pattern Recognition (CVPR), June 2019b.
Krähenbühl and Koltun. Efficient inference in fully connected erfs with gaussian edge poten- tials. Advances in Neural Information Processing Systems, pp. (09-117,4 2011.
A. H. Land and A. G. Doig. An automatic method for solving discrete programming problems. ECONOMETRICA, 28(3)49975520, 1960.
A. Levin, D. Lischinski, and Y. Weiss. A closed-form solution natural image matting. The IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 30(2):228-24 -228, Feb 2008.
Q Li, A. Arnab, and P. H.S. Torr. Weakly- and semi-supervised panoptic segmentation. In The European Conference on Computer Vision (ECCV), Sept 2018.
D. Lin, J. Dai, J. Jia, K. He, and J. Sun. Scribblesup: Seribble-supervised convolutional networks for semantic segmentation. The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3159-3167, 2016.
T. Lin, M. Maire, S. Belongie, Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick. Mi- crosoft coco: Common objects in context. In The European Conference Computer Vision (ECCV), pp. 740-755, 2014.
H. Ling, J. Gao, A. Kar, W. Chen, and S. Fidler. Fast Interactive Object Annotation with Curve- GCN. arXiv e-prints, Mar 2019.
K.-K. Maninis, S. Caelles, J. Pont-Tuset, and L. Van Gool. Deep extreme cut: From extreme points to object segmentation. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
D. Marin. M. Tang, R. I. Ben Ayed, and Y. Boykov. Beyond gradient descent for regularized seg- mentation losses. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
R. M. H. Nguyen and M. Brown. Fast and effective gradient minimization by region fusion. In The IEEE International Conference on Computer Vision (ICCV), pp. 208-216, Dec 2015.
Nowozin and C. H. Lampert. Global connectivity potentials for random field models. In The IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp. 818-825 Jun 2009.
D. Papadopoulos, J. Uijlings, F. Keller, and V. Ferrari. Extreme clicking for efficient object anno- tation. In The IEEE International Conference on Computer Vision (ICCV), pp. 4940-4949, Dec. 2017.
R. B. Potts. Some generalized order-disorder transformations. Mathematical Proceedings of the Cambridge Philosophical Society, 48(1):
1952.
10

---

Under review as a conference paper at ICLR 2020
M. Rempfler, B. Andres, and B. H. Menze. The minimum cost connected subgraph problem in medical image analysis. The Interaational Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI), 2016.
S. Ren, K. He, Girshick, and J. Sun. Faster r-cnm: Towards real-time object detection with region proposal networks. The IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI). 39(06):
Jun 2017.
C. Rother, Kolmogorov, and A. Blake. "grabcut": Interactive foreground extraction using iterated graph cuts. ACM Trans. Graph., pp. 309-314, Aug 2004.
O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and F. Li. Imagenet large scale visual recognition challenge. Interna- fional Journal of Computer Vision, 115(3):211-2555 2015.
B. C. Russell, Torralba, K. Murphy, and W. T. Freeman. Labelme: A database and web-based tool for image annotation. International Journal of Computer Vision, 77(1):157-173, May 2008,
E. Shelhamer, J. Long, and T. Darrell. Fully convolutional networks for semantic segmentation. The IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 39(4):640-651, Apr 2017.
R. Shen, G. Reinelt, and S. Canu. A first derivative potts model for segmentation and denoising using milp. In Operations Research Proceedings, 2017.
D. Stutz, A. Hermans, and B. Leibe. Superpixels: an evaluation of the state-of-the-art. Computer Vision and Image Understanding, pp. 1-32, 2017.
W. Tu, M. Liu, V. Jampani, D. Sun, S. Chien, M. Yang, and J. Kautz. Learning superpixels with segmentation-aware affinity loss. In The IEEE Conference Computer Vision and Pattern Recognition (CVPR), June 2018.
Y. Xiong, R. Liao, H. Zhao, R. Hu, M. Bai, E. Yumer, and R. Urtasun. Upsnet: unified panoptic segmentation network. arXiv e-prints, Jan 2019.
Yang, M. Collins, Y. Zhu, Hwang, T. Liu, X. Zhang, V. Sze, G. Papandreou, and Chen. Deeperlab: Single-shot image parser. arXiv e-prints, Feb 2019.
J. S. Yedidia, W. T. Freeman, and Y. Weiss. Exploring artificial intelligence in the new millennium. pp. 239-2699, 2003.
Yu, V. Koltun, and T. Funkhouser. Dilated residual networks. In The IEEE Conference on Com- puter Vision and Pattern Recognition (CVPR), 2017.
H. Zhao, Shi, X. Qi, X. Wang, and Jia. Pyramid scene parsing network. The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
7. Zhou. Ensemble learning. Encyclopedia of Biometrics, pp. 270-273, 2009.

---

Under review as a conference paper at ICLR 2020
A APPENDIX
A.1 A-NEAREST CUT GENERATION ALGORITHM
We restrict ourselves label l only, and the approach described below will be repeated for the other as active nodes, and we look for all
labels. Upon solving problem (3). we denote those with connected components in Gg. We then iterate over all active components pair rH, and use the following strategy to select K constraints (4) each time.
Precisely, we run breath-first search algorithm starting from H collect K disjoint sets Sm (m 1..... KK where S. contains all the inactive nodes with distance m to H. The search terminates if K equals the number of nodes in or another active node is reached. The idea is illustrated in Fig- 5 where active nodes are shown in black and denotes the root node. The two separator sets are marked in red and blue. Here K 2, because reaches the number of nodes in H. The K-Nearest strategy is reported in (Rempfler al., 2016) to be one of the most successful (among five) in terms of solved instances and computational efficiency.
H S
S2
Figure 5: K-Nearest cut generation strategy. Active nodes are shown in black, and the two vertex
separator are marked in red and blue.
A.2 SIMULATING SCRIBBLE GENERATIONS
Since there is no public scribble set for Cityscapes, hack the ground truth instance segment and apply erosion and skeleton algorithms to simulate scribbles that meet our policy in Sec. 3.2
We generated artificial scribbles for both training and validation sets of Cityscapes. We adopt kernel size adapted erosion and skeleton method and apply them on the instance-level ground truth masks of Cityscapes. Therefore, the format of the scribbles are also the same as Cityscapes, which is 16 bit grayscale PNG images with both class and instance labels. The code that generates the scribbles will be open-source All scribbles can be downloaded online!
A.3 SIMULATING CORRECTION SCRIBBLES
For VOC 2012, simulate only one scribble per image per iteration. Given the VOC ground truth and current segmentation mask, we first identify the largest error region which is connected and mislabelled. To simulate scribble from the operation to smooth the region. Then, we use skeletonization to shrink the region to 1 pixel wide without breaking the connectivity. We can then get the scribble by cutting branches in the skeleton tree. One can see Fig. for an illustration of generating correction scribbles. The code will also be open-source soon.
region, first apply opening morphological
A.4 PARAMETERS SETTING FOR THE EXPERIMENTS
Parameter y: is the information contained in pixel i, which could come from vector of RGB channels, layer or layer feature of ResNet 101, or from the probability map of a trained DCNN.
12

---

where ILP-P--rob adopts the result of Under review as a conference paper at ICLR 2020
HIM
Lajp
Figure 6: Top: Images from VOC with initial scribbles and 2 rounds of scribbles correction. Bottom:
Their corresponding panoptic segmentation result.
For VOC 2012, the probability map comes from trained DeepLab V2. For Cityscapes, we use DRN to get the probability map inference.
For H, after several trials, the growing parameter 7 is set to 0.1, 20, 100 and 0.3 for RGB, layer 1, layer and probability map both VOC 2012 and cityscapes.
For ILP, when training data is available, we can use the probability map p; and c = || Pa where e is k (k being the number of classes) dimensional vector with || position equals 1. When there is no training data, we compute the average of the nodes information (y:) covered by scribbles of the same class (i.e., class (), and use this to represent class (denote as Yo). Then of = |lys Yell, The regularization parameter À for all the ILP is set to 100, and the pairwise term dej = e llv. 1lve-,,l1,
and the l's
A.5 VISUALIZATION EXPERIMENTS PASCAL VOC 2012
In this section, we present 2 experiments Pascal VOC using optimization algorithms, shown Fig. 7 The figures are displayed in order of Table HyH-proh its initial solution.
Table 4: Layout of the Pascal experimental images
Left
Middle
Right
Top Bottom
Original image
HGGB MRF-prob
Hayerr
to Hort-rrbb
ILP-PP-rob-5
In both experiments, could see that using probability map is better than using low level feature maps. ILP-P-prob improves the result in the first one, while verifies the optimally in the second experiment, i.e., the initial solution of & Hprob is already the global optimization for ILP-P-prob.
We observe that -.P-P--rob-5 has the same solution as MRF-prob in the first experiment, while improves MRF-prob in the second one.
A.6 VISUALIZATION EXPERIMENTS CITYSCAPES
In this section, present experiments Cityscapes using our optimization algorithms. The figures are displayed order Table [5] where "SEEG" and "Panoptic" denotes semantic and panoptic segmentation, respectively.
13

---

Under review as a conference paper at ICLR 2020
Figure 7: Visualization of 2 experiments on Pascal VOC validation set.
Supervised by scribbles, both semantic and panoptic segmentation have already good shape and sharp boundaries, even using just low level features of DCNN (pre-trained on other completely dif- ferent datasets). The probability map from Cityscapes trained DCNN further improves the perfor- mance. But they are still limited by the quality of superpixels, especially for small and thin objects.
Table 5: Layout of the Cityscapes experimental images
Left
Right
Top Middle
Original image
Our artificial Scribbles
llyer SSEG
l, Hayeer Panoptic
Bottom
H-pobb SSEG
H Panoptic
14

---

Under review as a conference paper at ICLR 2020


Reviews:
[{"review_id": "r1e_52FhcH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces post-processing methods for panoptic (a combination of semantic and instance) segmentation, which are capable of using scribble annotations provided interactively by users. The proposed methods rely on (i) a discrete Potts model making use of RGB or DCNN features of the image, as well as the edge connectivity in a superpixel graph, and (ii) an integer linear program corresponding to a MRF with a pairwise data term. The proposed methods are evaluated on the Pascal VOC 2012 and Cityscapes datasets.\n\nThe paper is generally well written and easy to follow. The problem of panoptic segmentation is fundamental to computer vision, and as such, of relevance to the ICLR community. The proposed methods appear novel and worth pursuing.\n\nA first reservation about the paper is that the method is primarily one of post-processing (after, e.g., extracting primary features from a DCNN), but the most common means of post-processing, namely conditional random fields, are not even mentioned, let alone compared against.\n\nThe other main reservation about the paper is that there are very few comparisons to the abundant literatures on either semantic or instance segmentation, and as such it is difficult to appreciate the paper’s contributions to these areas. Of note:\n\n1. Evaluate on the COCO dataset, which is the current standard for segmentation ;\n2. The scribble supervision method of Lin et al (2016) is mentioned, but not compared against.\n\nSeparately, the paper should compare the proposed method for semantic and instance segmentation with other methods that use weak-labels such as:\n* Laradji, I. H., Vazquez, D., & Schmidt, M. (2019). Where are the Masks: Instance Segmentation with Image-level Supervision. arXiv preprint arXiv:1907.01430.\n* Laradji, I. H., Rostamzadeh, N., Pinheiro, P. O., Vazquez, D., & Schmidt, M. (2019). Instance Segmentation with Point Supervision. arXiv preprint arXiv:1906.06392.\n* Cholakkal, H., Sun, G., Khan, F. S., & Shao, L. (2019). Object counting and instance segmentation with image-level supervision. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 12397-12405).\n* Zhou, Y., Zhu, Y., Ye, Q., Qiu, Q., & Jiao, J. (2018). Weakly supervised instance segmentation using class peak response. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 3791-3800).\n* Zhu, Y., Zhou, Y., Ye, Q., Qiu, Q., & Jiao, J. (2017). Soft proposal networks for weakly supervised object localization. In Proceedings of the IEEE International Conference on Computer Vision (pp. 1841-1850).\n* Ahn, J., & Kwak, S. (2018). Learning pixel-level semantic affinity with image-level supervision for weakly supervised semantic segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 4981-4990).\n\nAs the paper currently stands, given the gaps in the experimental evaluation, it is difficult to appreciate the contributions and complementarities of the proposed methods to the panoptic segmentation problem. As such, the paper would require more work before recommending acceptance at ICLR.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper."}, "extra_text": {}}, {"review_id": "SkxvMZ5i9B", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes two graph-based deep network feature fusion methods with connection constraints for semantic and panoptic segmentation. By incorporating additional scribble information to DCNN features, the methods yield improved results over the original network predictions on two popular semantic and panoptic segmentation datasets. Through interactively correcting error regions with the scribbles, further performance increase can be obtained. \n\nI am not completely convinced by the novelty and experiments.\n(1) First, the idea of smart annotation can be formalized as a weekly supervised segmentation problem where only part of the annotation is available. Can the authors justify how your work differs from those works solving the weekly supervised problem and what's your advantages. (Seed Expand and Constrain ... Alexander Kolesnikov; STC: A simple to Complex Framework for Weakly... Yunchao Wei; FickleNet Jungbeom Lee; etc..) Or, if possible, could you make a fair comparison with some existed weekly supervised approach on the final (semantic) result. Second, Potts model, MRF, K-nearest cut are known approaches. Thus I would like to know the deeper contribution of this work other than set constraints and solve ILP.\n(2) The authors did not justify the use of less powerful models (DeepLabV2 and DRN) as both the inputs for l0H and ILP-P and the baseline comparison. The authors mentioned the current SOTA model (DeepLabV3+), which has achieved 79.55% mIoU on the CityScapes val set. However, they did not perform experiments using its probability map. It would be more convincing if the same performance gain can be achieved by using the SOTA model as inputs to the algorithms.\n(3) The argument of achieving competitive results for panoptic segmentation is rather weak. To approach the panoptic segmentation problem, the authors essentially used scribbles to separate semantic region predictions into individual instances. Since the proposed algorithm requires as many scribbles to be drawn as there are regions, the baseline network only needs to predict semantic classes, and the algorithms uses the provided region IDs from the scribbles to segment individual instances. While this still has numerous applications in data annotation, it is somewhat unjust to claim that this method achieves competitive results in panoptic segmentation.\n(4) The artificial scribbles for CityScapes experiments do not resemble human-drawn scribbles. Compared to the scribbles data for VOC12, the artificially generated scribbles for CityScapes experiments are visually idealistic. Rather than a stroke through the object, the generated is more similar to an outline of the object, which conveys a lot more information than a single line. Particularly when applied on super-pixels, it seems that super-pixels can easily be merged together by grouping any super-pixels within a scribble outline.\nThere are some other minor suggestions. For example, it might be clearer and easier to read if section 2.2.2 is presented in an algorithm format. Some minor typos and grammatical mistakes should also be corrected.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "BygLwR_cYH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n- key problem: efficiently leveraging scribbles as interactive supervision (at test time) for panoptic segmentation;\n- contributions: 1) two algorithms leveraging scribbles via a superpixel connectivity constraint (one class-agnostic local diffusion heuristic, one class-aware with a MRF formulation), 2) experiments on PASCAL VOC 2012 and Cityscapes showing that both methods i) can achieve good performance without training data (using RGB values or pretrained features), ii) can improve the performance of a fully supervised baseline when using its probability maps as representation, and iii) can significantly improve performance beyond the state of the art on PASCAL when used interactively (90% mIoU with 3 rounds of corrective scribbles).\n\nRecommendation: weak reject\n\nKey reason 1: unclear novelty and relevance to ICLR.\n- The paper proposes to apply two existing algorithms (Nguyen & Brown 2015, Rempfler et 2016) to a new task (interactive panoptic segmentation): what is the claimed novelty? What is specific to panoptic segmentation vs semantic or instance segmentation? Could the difference with related work in Section 2 be discussed more precisely?\n- Furthermore, there seems to be no learning (representation or otherwise) involved in this submission. The paper mentions potential applications to weakly-supervised learning in Section 5, but it does not provide clear insights into what would be the benefits in terms of representation learning (vs. RGB, pre-trained features, or probability maps).\n- Overall, this paper might be more tailored for a Computer Vision venue like CVPR.\n\nKey reason 2: lack of sensitivity / robustness analysis.\n- The scribbles are \"simulated\" using morphological operations on the ground truth (A.2, A.3): does this lead to realistic scribbles? Figure 3 (which is unclear) shows that the \"scribbles\" might be precise outlines or contours, which are very different than the expected scribbles illustrated in Figure 2. Contours provide much stronger information for segmentation, and are much more likely to effectively leverage the connectivity prior (esp. with the diffusion heuristic), but are they really scribbles / cheap supervision?\n- What is the importance of the superpixel coverage by scribbles or missing scribbles or the location of scribbles relative to segment boundaries? What are the impact of realistic deviations from the expected scribble policy that are likely to happen in practice? Measuring sensitivity to different types of noise (by perturbing / dropping out scribbles) seems important to assess the practical usefulness and robustness of the method.\n- PASCAL VOC and Cityscapes are small datasets. Experiments on bigger more recent ones like Mapillary Vistas and COCO are becoming the standard protocol in the instance/semantic/panoptic segmentation community. How would this method fare on those much more challenging datasets? What are the benefits of the proposed interactive methods in terms of scalability?\n\nAdditional Feedback:\n- Fig. 4 is too low resolution / blurry;\n- typos: \"tarining set\", \"weekly supervised\".\n\n## Update following the rebuttal\n\nThanks to the authors for their replies. Sadly, my concerns are only answered at a high-level, and the consensus among reviewers is clear. Hence I confirm my rating to reject. I hope the feedback provided above will assist the authors in improving the work or finding a more suitable venue.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BJlbD83nYr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper investigates scribble-based interactive semantic and panoptic segmentation.  The algorithms described build a graph on superpixels and do not require deep features or labels but rely on “scribbles” for supervision.  Given that semantic (and panoptic) annotation is very labor-intensive, advances in scribble-based annotation could significantly improve annotation time for new datasets; in applications where real-time performance is not required, scribble-based refinement of predictions could also be advantageous.\n\nThe experiments compare the proposed algorithms to deep baselines for VOC2012 and Cityscapes panoptic segmentation, and show impressive performance even without deep features. However they do not compare results to other scribble supervision methods to highlight the advantages of their approach over prior work.  I’d like for the experiments section to have a proper comparison to prior scribble algorithms (e.g. in section 4.4, comparing to other algorithms with the SOTA approach as baseline) to clearly show the advantage of their approach.\n\nThe results are impressive compared to the deep learning baseline, but I think further experimental validation should exist for properly comparing to prior work.\n\nPost-rebuttal: I maintain my recommendation.", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes two methods for interactive panoptic segmentation (a combination of semantic and instance segmentation) that leverages scribbles as supervision during inference. Reviewers had concerns about the novelty of the paper as it applies existing algorithms for this task and limited empirical comparison with other methods. Reviewers also suggested that ICLR may not be a good fit for the paper and I encourage the authors to consider submitting to a vision oriented conference. 
Decision: Reject