Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: The submission proposes a module for discrete labeling within panoptic segmentation models that can still be optimized end-to-end. The panoptic labels are discrete labels giving the class and instance of each pixel in the image. This paper considers labels produced by the asymmetric multiway cut problem (AMWC) [36] and uses a perturbation-based method for determining the gradients of this module. They can then be back-propagated into a neural network model that produces the potentials in the AMWC problem.
main_review: **Originality**

Combining graphical models, including discrete optimization, with deep learning has a long history with deep learning for vision & segmentation:

  * Chen et. al. "Learning Deep Structured Models" ICML 2015
  * Kirillov et. al. "Efficient Likelihood Learning of a Generic CNN-CRF Model for Semantic Segmentation" ICLR 2016

And the submission includes good discussion of the most closely related literature, citing [2, 62, 71] among others. Directly considering the discrete optimization problem as in the submission, rather than a continous model or a continuous relaxation, is a less common choice.

It's easy to anticipate this being applied to panoptic segmentation too. I am not aware of previous work taking this kind of approach for this particular segmentation subproblem.

How the models are combined, and the gradients calculated, is nontrivial, especially when the discrete problem is hard as in this case. Some complex and interesting design decisions made are here.

**Quality**

[36] is a very good choice of which discrete problem to use. It can explicitly handle instances for a subset of classes in a way that directly maps to the panoptic segmentation problem.

A multi-way cut optimization is a very principled way to handle the discreteness of the panoptic labels. The original work proposing the panoptic segmentation problem [32] used discrete labels and a quality measure (PQ) that more or less assumes this (using their definition of IoU).

The core novel part of the method, the robust perturbation technique and how it's applied, all looks correct to me.

The experiments are simple but straightforward. The submission does not introduce much in the way of additional confounds or hyperparameters that need to be tuned, so the comparisons seem good. Experiments are done with multiple trials, and the source of the values of the hyperparameters that are present is specified.

One set of experiments, that I did not find but would ideally be present, are any showing the quality of the AMWC solutions or the gradients. Section A.1 in the supplementary material describes a heuristic greedy solver to the AMWC problem. How does this perform in practice in terms of the how close to optimal its solutions are, and how consistently? How much more accurate (by whatever measure) are the gradients for the robust perturbation than [52]? This or other questions, that directly measure how the discrete optimization step is performing, would provide some insight beyond the end-to-end PQ results.

**Clarity**

The submission is very clear. The presentation style is simple and direct. Pseudocode and mathematical definitions are use well for more precise description of the optimization method.

**Significance**

The overall method gets a lower score on panoptic quality (PQ) than the state-of-the art, but it does convincingly improved upon a bottom-up baseline. So it seems very likely that this kind of approach, at least generally considering discrete optimization within the training and inference, is a significant step forward in panoptic segmentation and image segmentation in general.

**Minor Comments**

* Double-checking: does the loss function L, specified in 193, take real-valued or binary labels as input?
* Typo "Trail" in Table 5 in the supplement.
limitations_and_societal_impact: The limitation of the core novel component of this method are probably primarily that the solutions to the asymmetric cut problem, and the gradients, not being computed exactly. The submission is clear that it is using approximations here, and discusses how they dealt with the drawbacks of their choice of approximation.

Societal impact is going to depend on application, as the authors state.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper focuses on the problem of panoptic segmentation. The author proposes an end-to-end trainable approach with a combinatorial optimization layer for post-processing. AMWC formulation and surrogate loss are utilized for panoptic segmentation. The experimental evaluations are conducted on both Cityscapes and COCO datasets, and the results seem promising.
main_review: Strengths:
1. Use of an AMWC solver to address the problem of panoptic segmentation seems interesting.

Weaknesses:
1. Some of the experimental comparisons are missing. For example, what would the results be when given different backbones, as shown in Table 1?
2. The results on the Cityscapes dataset seem good when compared to other methods. While on the COCO dataset, the results are much worse, e.g., when compared to the UPSNet. What would be the possible reason for that? Would the method not work well for complex scenes like COCO?
3. The introduced computations, parameters, and the running time of the proposed module are not discussed in the manuscript.
4. It is suggested to rerank the order of different approaches in Table 1 by their performance.
5. Typos: 'CityScapes' should be 'Cityscapes'; two dots at line 97, etc. The paper needs more proofreading.
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes an end-to-end trainable approach for panoptic segmentation incorporating a combinatorial optimization layer for post-processing and directly learning a panoptic surrogate loss. It solves the panoptic segmentation based on the bottom-up manner and views the instance separation as a multiway cut problem. Experiments prove the effectiveness.
main_review: This paper proposes an end-to-end trainable approach for panoptic segmentation incorporating a combinatorial optimization layer for post-processing and directly learning a panoptic surrogate loss. It solves the panoptic segmentation based on the bottom-up manner and views the instance separation as a multiway cut problem. Overall, the method is elegant and experiments prove its effectiveness. Some detailed concerns are given below:

Some concerns:
1. This paper uses a pre-trained model as baseline and optimizes it with the proposed panoptic quality loss. I'm wondering if it is possible to optimize them simultaneously in an iterative manner? Because the pretrained manner actually brings more cost to the whole process and may lead to a suboptimal solution.
2. The universality of the proposed method. Could the proposed loss function be viewed as a plug-and-play module in other methods for panoptic or instance segmentation? It could bring more technical value.
3. The performance on the COCO dataset is not satisfactory. The suboptimal performance on the COCO dataset could be attributed to the complex scenarios compared with the Cityscapes dataset, especially in Thing categories. Is this brought by the drop in SQ or RQ? A detailed analysis is recommended to be given in the main paper or appendix.
limitations_and_societal_impact: The limitations and potential negative societal impact are discussed and addressed.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper designs an end-to-end panoptic segmentation pipeline with combinatorial optimization. The authors propose to use asymmetric multiway-cut to make the panoptic segmentation prediction, with a Panoptic Quality surrogate loss as a direct supervision signal and a perturbation-based gradient estimation technique to enable training.
main_review: ---

[+] The paper is well-written and easy to follow.

[+] The idea in the paper sounds novel. The idea is followed by a simple and elegant panoptic segmentation pipeline. The proposed key algorithm, AMWC, greatly simplifies the panoptic prediction.

[+] The paper contributes several techniques to implement the AMWC algorithm for large-scale training, including a surrogate loss and a multi-scale back-propagation technique. Particularly, the surrogate loss mitigates the existing gap between supervision signals and the evaluation metric.

[+] The proposed solution outperforms SOTA, e.g., Panotptic-Deeplab and SSAP, in segmentation quality.

--- 

[-] The perturbation-based gradient estimation might limit the scalability of the method. The gradient estimation might get inaccurate when we have more parameters to optimize, for example, when we optimize the whole model instead of only the classification layer as in the paper.

[-] The solver (2 seconds per Cityscape image) is too slow for real-time applications.

[-] Compared to the baseline, the PQ_stuff even dropped on COCO after fine-tuning. It would be better if the drop could be explained.

[-] The experiment part is a little weak. It would be better if more ablation studies and analysis, e.g., on different backbones and training time memory consumption, can be provided.

---

In summary, the main idea of the paper about using a combinatorial optimizer to simplify the notoriously complicated panoptic prediction task is interesting and novel. However, I have concerns about the scalability and the real-world impact of the method since the solver is currently too slow and a sophisticated optimization method has to be employed in training.

---

Typos:

L97: "architectures.." -> "architectures."

Fig 2. caption: "to a affinity branch" -> "to an affinity branch"

L169: "The AMWC can be though of as a" -> "The AMWC can be thought of as a"
limitations_and_societal_impact: The authors adequately addressed the limitations and potential negative societal impact of their work.
review_title: null

Meta-review: The authors propose a method for simultaneous semantic and instance segmentation which can be optimized end-to-end. In particular, the authors make use of predictions obtained by solving the asymmetric multiway-cut problem, coupled with a surrogate loss whose gradient is estimated using a perturbation-based approach. Promising empirical results were presented on COCO and CityScapes datasets.
Reviewers appreciated the clarity of exposition and felt that the proposed approach was a rather elegant solution, notwithstanding the fact that the empirical results were not state-of-the-art. The rebuttals did a great job addressing most concerns, but scalability and practicality remain one of the key drawbacks. Nevertheless, the reviewers felt that the contribution is significant enough to merit acceptance. I agree and I would like to see the authors include a detailed limitations section to highlight the remaining issues.
Decision: Accept (Poster)