Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper introduces post-processing methods for panoptic (a combination of semantic and instance) segmentation, which are capable of using scribble annotations provided interactively by users. The proposed methods rely on (i) a discrete Potts model making use of RGB or DCNN features of the image, as well as the edge connectivity in a superpixel graph, and (ii) an integer linear program corresponding to a MRF with a pairwise data term. The proposed methods are evaluated on the Pascal VOC 2012 and Cityscapes datasets.

The paper is generally well written and easy to follow. The problem of panoptic segmentation is fundamental to computer vision, and as such, of relevance to the ICLR community. The proposed methods appear novel and worth pursuing.

A first reservation about the paper is that the method is primarily one of post-processing (after, e.g., extracting primary features from a DCNN), but the most common means of post-processing, namely conditional random fields, are not even mentioned, let alone compared against.

The other main reservation about the paper is that there are very few comparisons to the abundant literatures on either semantic or instance segmentation, and as such it is difficult to appreciate the paper’s contributions to these areas. Of note:

1. Evaluate on the COCO dataset, which is the current standard for segmentation ;
2. The scribble supervision method of Lin et al (2016) is mentioned, but not compared against.

Separately, the paper should compare the proposed method for semantic and instance segmentation with other methods that use weak-labels such as:
* Laradji, I. H., Vazquez, D., & Schmidt, M. (2019). Where are the Masks: Instance Segmentation with Image-level Supervision. arXiv preprint arXiv:1907.01430.
* Laradji, I. H., Rostamzadeh, N., Pinheiro, P. O., Vazquez, D., & Schmidt, M. (2019). Instance Segmentation with Point Supervision. arXiv preprint arXiv:1906.06392.
* Cholakkal, H., Sun, G., Khan, F. S., & Shao, L. (2019). Object counting and instance segmentation with image-level supervision. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 12397-12405).
* Zhou, Y., Zhu, Y., Ye, Q., Qiu, Q., & Jiao, J. (2018). Weakly supervised instance segmentation using class peak response. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 3791-3800).
* Zhu, Y., Zhou, Y., Ye, Q., Qiu, Q., & Jiao, J. (2017). Soft proposal networks for weakly supervised object localization. In Proceedings of the IEEE International Conference on Computer Vision (pp. 1841-1850).
* Ahn, J., & Kwak, S. (2018). Learning pixel-level semantic affinity with image-level supervision for weakly supervised semantic segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 4981-4990).

As the paper currently stands, given the gaps in the experimental evaluation, it is difficult to appreciate the contributions and complementarities of the proposed methods to the panoptic segmentation problem. As such, the paper would require more work before recommending acceptance at ICLR.
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: This paper proposes two graph-based deep network feature fusion methods with connection constraints for semantic and panoptic segmentation. By incorporating additional scribble information to DCNN features, the methods yield improved results over the original network predictions on two popular semantic and panoptic segmentation datasets. Through interactively correcting error regions with the scribbles, further performance increase can be obtained. 

I am not completely convinced by the novelty and experiments.
(1) First, the idea of smart annotation can be formalized as a weekly supervised segmentation problem where only part of the annotation is available. Can the authors justify how your work differs from those works solving the weekly supervised problem and what's your advantages. (Seed Expand and Constrain ... Alexander Kolesnikov; STC: A simple to Complex Framework for Weakly... Yunchao Wei; FickleNet Jungbeom Lee; etc..) Or, if possible, could you make a fair comparison with some existed weekly supervised approach on the final (semantic) result. Second, Potts model, MRF, K-nearest cut are known approaches. Thus I would like to know the deeper contribution of this work other than set constraints and solve ILP.
(2) The authors did not justify the use of less powerful models (DeepLabV2 and DRN) as both the inputs for l0H and ILP-P and the baseline comparison. The authors mentioned the current SOTA model (DeepLabV3+), which has achieved 79.55% mIoU on the CityScapes val set. However, they did not perform experiments using its probability map. It would be more convincing if the same performance gain can be achieved by using the SOTA model as inputs to the algorithms.
(3) The argument of achieving competitive results for panoptic segmentation is rather weak. To approach the panoptic segmentation problem, the authors essentially used scribbles to separate semantic region predictions into individual instances. Since the proposed algorithm requires as many scribbles to be drawn as there are regions, the baseline network only needs to predict semantic classes, and the algorithms uses the provided region IDs from the scribbles to segment individual instances. While this still has numerous applications in data annotation, it is somewhat unjust to claim that this method achieves competitive results in panoptic segmentation.
(4) The artificial scribbles for CityScapes experiments do not resemble human-drawn scribbles. Compared to the scribbles data for VOC12, the artificially generated scribbles for CityScapes experiments are visually idealistic. Rather than a stroke through the object, the generated is more similar to an outline of the object, which conveys a lot more information than a single line. Particularly when applied on super-pixels, it seems that super-pixels can easily be merged together by grouping any super-pixels within a scribble outline.
There are some other minor suggestions. For example, it might be clearer and easier to read if section 2.2.2 is presented in an algorithm format. Some minor typos and grammatical mistakes should also be corrected.

review_title: Official Blind Review #4

final_score: rating: 3: Weak Reject
review_content: This paper investigates scribble-based interactive semantic and panoptic segmentation.  The algorithms described build a graph on superpixels and do not require deep features or labels but rely on “scribbles” for supervision.  Given that semantic (and panoptic) annotation is very labor-intensive, advances in scribble-based annotation could significantly improve annotation time for new datasets; in applications where real-time performance is not required, scribble-based refinement of predictions could also be advantageous.

The experiments compare the proposed algorithms to deep baselines for VOC2012 and Cityscapes panoptic segmentation, and show impressive performance even without deep features. However they do not compare results to other scribble supervision methods to highlight the advantages of their approach over prior work.  I’d like for the experiments section to have a proper comparison to prior scribble algorithms (e.g. in section 4.4, comparing to other algorithms with the SOTA approach as baseline) to clearly show the advantage of their approach.

The results are impressive compared to the deep learning baseline, but I think further experimental validation should exist for properly comparing to prior work.

Post-rebuttal: I maintain my recommendation.
review_title: Official Blind Review #1

final_score: rating: 1: Reject
review_content: Summary:
- key problem: efficiently leveraging scribbles as interactive supervision (at test time) for panoptic segmentation;
- contributions: 1) two algorithms leveraging scribbles via a superpixel connectivity constraint (one class-agnostic local diffusion heuristic, one class-aware with a MRF formulation), 2) experiments on PASCAL VOC 2012 and Cityscapes showing that both methods i) can achieve good performance without training data (using RGB values or pretrained features), ii) can improve the performance of a fully supervised baseline when using its probability maps as representation, and iii) can significantly improve performance beyond the state of the art on PASCAL when used interactively (90% mIoU with 3 rounds of corrective scribbles).

Recommendation: weak reject

Key reason 1: unclear novelty and relevance to ICLR.
- The paper proposes to apply two existing algorithms (Nguyen & Brown 2015, Rempfler et 2016) to a new task (interactive panoptic segmentation): what is the claimed novelty? What is specific to panoptic segmentation vs semantic or instance segmentation? Could the difference with related work in Section 2 be discussed more precisely?
- Furthermore, there seems to be no learning (representation or otherwise) involved in this submission. The paper mentions potential applications to weakly-supervised learning in Section 5, but it does not provide clear insights into what would be the benefits in terms of representation learning (vs. RGB, pre-trained features, or probability maps).
- Overall, this paper might be more tailored for a Computer Vision venue like CVPR.

Key reason 2: lack of sensitivity / robustness analysis.
- The scribbles are "simulated" using morphological operations on the ground truth (A.2, A.3): does this lead to realistic scribbles? Figure 3 (which is unclear) shows that the "scribbles" might be precise outlines or contours, which are very different than the expected scribbles illustrated in Figure 2. Contours provide much stronger information for segmentation, and are much more likely to effectively leverage the connectivity prior (esp. with the diffusion heuristic), but are they really scribbles / cheap supervision?
- What is the importance of the superpixel coverage by scribbles or missing scribbles or the location of scribbles relative to segment boundaries? What are the impact of realistic deviations from the expected scribble policy that are likely to happen in practice? Measuring sensitivity to different types of noise (by perturbing / dropping out scribbles) seems important to assess the practical usefulness and robustness of the method.
- PASCAL VOC and Cityscapes are small datasets. Experiments on bigger more recent ones like Mapillary Vistas and COCO are becoming the standard protocol in the instance/semantic/panoptic segmentation community. How would this method fare on those much more challenging datasets? What are the benefits of the proposed interactive methods in terms of scalability?

Additional Feedback:
- Fig. 4 is too low resolution / blurry;
- typos: "tarining set", "weekly supervised".

## Update following the rebuttal

Thanks to the authors for their replies. Sadly, my concerns are only answered at a high-level, and the consensus among reviewers is clear. Hence I confirm my rating to reject. I hope the feedback provided above will assist the authors in improving the work or finding a more suitable venue.
review_title: Official Blind Review #3

Meta-review: The paper proposes two methods for interactive panoptic segmentation (a combination of semantic and instance segmentation) that leverages scribbles as supervision during inference. Reviewers had concerns about the novelty of the paper as it applies existing algorithms for this task and limited empirical comparison with other methods. Reviewers also suggested that ICLR may not be a good fit for the paper and I encourage the authors to consider submitting to a vision oriented conference. 
Decision: Reject