Combinatorial Optimization for Panoptic Segmentation: A Fully Differentiable Approach

Abstract: We propose a fully differentiable architecture for simultaneous semantic and instance segmentation (a.k.a. panoptic segmentation) consisting of a convolutional neural network and an asymmetric multiway cut problem solver. The latter solves a combinatorial optimization problem that elegantly incorporates semantic and boundary predictions to produce a panoptic labeling. Our formulation allows to directly maximize a smooth surrogate of the panoptic quality metric by backpropagating the gradient through the optimization problem. Experimental evaluation shows improvement by backpropagating through the optimization problem w.r.t. comparable approaches on Cityscapes and COCO datasets. Overall, our approach of combinatorial optimization for panoptic segmentation (COPS) shows the utility of using optimization in tandem with deep learning in a challenging large scale real-world problem and showcases benefits and insights into training such an architecture.


Reviews and author responses:

 Sorry for the last-minute response.

Thank you to the authors for clarifying some points of the paper I thought were missing or found confusing. Generally, the further details in the rebuttal make sense to me. I continue to recommend acceptance.

This paper focuses on the problem of panoptic segmentation. The author proposes an end-to-end trainable approach with a combinatorial optimization layer for post-processing. AMWC formulation and surrogate loss are utilized for panoptic segmentation. The experimental evaluations are conducted on both Cityscapes and COCO datasets, and the results seem promising.  Strengths:
1. Use of an AMWC solver to address the problem of panoptic segmentation seems interesting.

Weaknesses:
1. Some of the experimental comparisons are missing. For example, what would the results be when given different backbones, as shown in Table 1?
2. The results on the Cityscapes dataset seem good when compared to other methods. While on the COCO dataset, the results are much worse, e.g., when compared to the UPSNet. What would be the possible reason for that? Would the method not work well for complex scenes like COCO?
3. The introduced computations, parameters, and the running time of the proposed module are not discussed in the manuscript.
4. It is suggested to rerank the order of different approaches in Table 1 by their performance.
5. Typos: 'CityScapes' should be 'Cityscapes'; two dots at line 97, etc. The paper needs more proofreading. Yes

 Some of my major concerns are well addressed in the rebuttal, and the author promises to make the presentation in the revised manuscript more clear. Thus I would like to increase my rating.

 Thanks for your explanation. Most of my concerns have been addressed from the rebuttal. I'm looking forward to see the discussion of the universality and the experimental performance in the revision. And I'd like to keep my recommendation rating of this paper.

 Thanks for the detailed explanation, most of my concerns have been addressed. However, since the authors agree that scaling up the gradient estimation and improving running time could be open challenges, my concern about the work's real-world impact at the moment remains, and I will not further increase my score.

 Thanks for the elaborate review, positive appraisal and pointing out of typos.

* **Perturbation based gradient limits scalability:**  
    1. We argue that our approach can be used to train all parameteres end-to-end: The perturbation based methods have been used for training all parameters of a model in [49, 52]. Our robust backpropagation technique improves upon [52] showing better convergence and thus enables usage of such methods in more applications requiring backpropagation through optimization layers. For example [52] showed training of a complete ResNet18 for a variety of discrete optimization problems with optimal solvers. 
    2. We currently avoid training the whole model with end-to-end training for the following reason: IoU based metrics including the PQ quality (and our surrogate) are not separable under expectations  w.r.t. different images (see eq. 15 in [Ma18]). To get good estimates of the loss we therefore require many images, i.e. larger batch sizes for end-to-end training than for pre-end training. This restriction makes it infeasible to train all layers due to GPU memory limitations.
    
     We agree that end-to-end training of all parameters with suboptimal solvers in a challenging scenario would be interesting. We argue however that another application with a simpler, more separable loss would be a better candidate for this.

* **The solver is too slow for real-time:** We agree that this is a current limitation of our work. We hope that faster parallel AMWC solver will improve running times, see also our conclusion. Developing such a solver is, however, beyond the scope of our paper. We further hope that our work will stimulate research into efficient algorithms that are suitable for scenarios such as ours.

* **PQ_stuff performance drops on COCO:** COCO dataset has 80 _thing_ classes and _53_ stuff classes, hence performance of _thing_ classes is more important for the overall PQ metric than performance on _stuff_ classes. Since we optimize for overall panoptic quality the optimization process can trade performance on _stuff_ classes with _thing_ classes due to this imbalance.

* **Experimental part lacks ablation w.r.t. different backbones:** We will add statistics w.r.t. runtime, memory consumption etc. in the final version of the paper. For our response regarding ablations w.r.t. different backbones see the corresponding point in the general comments section.
    
[Ma18] - Berman et. al. The lovász-softmax loss: A tractable surrogate for the optimization of the intersection-over-union measure in neural networks. CVPR 2018. 

 Thank you for the in-depth review, positive appraisal and interesting ideas related to the universality of our method.

* **Simultaneous iterative optimization (Combined loss):** Using the panoptic quality surrogate loss during pre-training is a good idea. However, we follow our two-step approach due to computational efficiency, since the combinatorial part takes a significant amount of time. Therefore, pre-training  with the panoptic surrogate loss is infeasible due to the computational burden incurred by the AMWC solver. We hope that with better and faster AMWC solvers training can be converted to a single stage in the future. 

* **Universality of the loss function:** We discuss universality of our approach w.r.t. 1. AMWC + PQ surrogate loss and 2. PQ surrogate loss alone.
    1. Our loss along with our optimization layer can be plugged into any network which produces semantic and affinity scores. This can be used to create an instance or panoptic segmentation pipeline with a different neural network. For example the networks from [23, 63] can directly make use of our loss and optimization layer in a plug-n-play fashion.
    2. Additionally, if a different but differentiable layer computes panoptic or instance labels our loss alone can also be directly used.

* **Performance on COCO dataset:** Actually according to our oracle study the performance on COCO dataset is primarily tied to semantic segmentation and not on instance separation. Such conclusion is also made on Panoptic DeepLab [13] in the unpublished technical [report](https://github.com/bowenc0221/panoptic-deeplab/blob/master/docs/tech_report.pdf). We also would like to point out that we outperform the comparable approach [13] on COCO.
Detailed results breaking down the performance on COCO into recognition quality (RQ), semantic quality (SQ) are as follows:

  | Method | PQ | SQ | RQ  | PQ_th | SQ_th | RQ_th | PQ_st | SQ_st | RQ_st
  | --- | --- | ---  | --- | --- | --- | --- | --- | --- | --- |
  | Our baseline | 34.2 | 78.1 | 42.6 | 34.9 | 77.8 | 43.5 | **33.2** | **78.6** | **41.1**
  | [13] |  35.5 | 77.2 | 44.6 | 37.8 | 77.8 | 47.5 | 32.0 | 76.4 | 40.3
  | Our end-to-end | **36.3** | **78.1** | **45.2** | **39.3** | **80** | **49** | 31.7 | 78.3 | 39.5 |
  
  The table shows that we mainly improve the overall recognition quality by end-to-end training whereas the overall semantic quality of our baseline is already better than the comparable method [13] but we conjecture that it is limited by the backbone (see our oracle study in appendix section 8). We point out that the _thing_ category, where we improve, is the more complicated one, since there might be multiple instances of each _thing_ class. Here, our AMWC solver and its end-to-end training helps. Our panoptic surrogate loss is taking into account SQ and RQ and automatically finds the improving direction w.r.t. the overall metric. Since there are more _things_ than _stuff_ classes in COCO, our overall PQ loss finds it profitable to improve _thing_ classes even at the expense of _stuff_ classes.
In summary, our approach may lead to lopsided improvement if this helps the overall PQ score. We will also add the above table and a corresponding discussion to the final paper. Please also see the corresponding point in the overall comments section for more details on COCO performance.

 Thank you for the knowledgeable review. We will correct the mentioned typos and check for more. 

* **Different backbones:** Please see the corresponding reply in the overall comment section. 

* **Performance on COCO dataset:** Our performance is better than the comparable work [13]. Please refer to the corresponding point in the overall comments section.
  
* **Computations, parameters ... of the module:** Our model is very similar to Panoptic Deeplab [13]. We will add exact statistics in the final manuscript.

* **Rerank Table 1:** We ordered the methods by comparability (top-down, bottom-up, complexity of backbone). We agree that this makes comparison w.r.t. performance only more difficult, but on the other hand allows for easier comparison with directly comparable methods. We will improve categorization to make overall comparison easier.

 Thank you for the very detailed review, positive feedback, catching of typos and additional references. We will correct typos and add suggested references in the final paper. We will condense our discussion w.r.t. the quality of the solver and the gradients and add it as well to the final paper.

* **Quality of optimization solutions:** It is correct that we cannot currently estimate the quality of our AMWC heuristic. The original AMWC paper [36] introduced a cutting plane ILP solver for its solution. Due to the limited scalability of their approach AMWC problems could only be solved up to moderate sizes. This precludes at the current time numeric estimation of bounds on the solution quality for our very large scale instances. It is an interesting research direction to come up with scalable solvers that also output approximative lower bounds against which we could test our heuristic, similar as was done for the closely related multicut problem in [La18]. However, it was shown that a similar primal heuristic, i.e. greedy additive edge contraction on which our heuristic is based, gives solution close to optimum on typical computer vision problems, see [SW17]. Better and faster solvers with lower bounds are interesting future research directions that will benefit our approach.

* **Quality of gradients:** This is a good question with multiple facets. 
  1. One source of error is that we obviously do not solve the problem to optimality, hence the gradient will be different in those variables where a non-optimal solution has been computed. However, we argue that this is not necessarily that problematic: The supervisory signal computed by the suboptimal solver will tell the network about the costs it needs to reach the right solution (even if those weights are different than those needed by an optimal solver). In other words the network to some degree learns the suboptimality of the solver and can adapt to it.
  2. Another factor is the quality of the gradient produced by the perturbation scheme. If we directly use the approach of [52] the gradient can only take values in $\\{-1 / \lambda, 0, 1/\lambda \\} $. Thus the gradient is giving a direction of loss decrease/increase but without any differences in the absolute value. By our robust backpropagation approach the gradients get averaged and thus they become more informative, including different absolute values.
We have created a gif [animation](https://imgur.com/a/zKzFLAr) showing the original image and gradients w.r.t. the semantic branch with different numbers $N$ of $\lambda$-values. What we see is that for single pertubations ($N=1$) the magnitude of the gradient is either zero or $|1/\lambda|$, while our robust backpropagation with multiple $\lambda$ results in a gradient with multiple magnitude levels (shown by varyings shades of grey). This, we argue, helps in driving the neural network weights by larger updates whenever the loss indicates that severe errors are made and by smaller updates when loss is less sensitive w.r.t. the errors made with the current weights.

* **Does the loss function L, specified in 193, take real-valued or binary labels as input?** The loss function in our case takes binary labels as input, we will correct this.

[La18] Lange, Jan-Hendrik, Andreas Karrenbauer, and Bjoern Andres. "Partial optimality and fast lower bounds for weighted correlation clustering." in ICML 2018.

[SW17] Swoboda, Paul, and Bjoern Andres. "A message passing algorithm for the minimum cost multicut problem." in CVPR 2017.

 We thank all the reviewers for their knowledgeable and constructive criticism. Points raised by multiple reviewers are discussed here. We will incorporate minor criticism without further comment.

* **Performance on the COCO dataset (5rdz, vDMJ):** For the COCO dataset we perform an oracle study in Section 8 of the appendix. The study shows that the main challenge for good panoptic segmentation performance on COCO is to get good semantic scores. The reason for this is that COCO contains more diverse scenarios thus requiring a more sophisticated neural network (e.g. UPSNet uses deformable convolutions and a MaskRCNN for top-down information, MaxDeepLab uses transformers). Nonetheless, in comparison to other bottom-up approaches [13, 23, 64] our method is still performing best when compared on identical backbones. 

* **Different backbones(5rdz, XpFH):** The goal of our study was to tackle the challenging task of combining an optimization module with a neural network by enabling end-to-end training. Since the bottom-up approaches [13, 23, 64] most relevant for our comparison used ResNet-50, we believe that to be the common denominator that would best showcase our contribution in comparison to competing methods. One can exchange neural network blocks to improve the performance, yet retain end-to-end training and its benefits. Such larger experimental study was however beyond the scope of our work.


The submission proposes a module for discrete labeling within panoptic segmentation models that can still be optimized end-to-end. The panoptic labels are discrete labels giving the class and instance of each pixel in the image. This paper considers labels produced by the asymmetric multiway cut problem (AMWC) [36] and uses a perturbation-based method for determining the gradients of this module. They can then be back-propagated into a neural network model that produces the potentials in the AMWC problem.  **Originality**

Combining graphical models, including discrete optimization, with deep learning has a long history with deep learning for vision & segmentation:

  * Chen et. al. "Learning Deep Structured Models" ICML 2015
  * Kirillov et. al. "Efficient Likelihood Learning of a Generic CNN-CRF Model for Semantic Segmentation" ICLR 2016

And the submission includes good discussion of the most closely related literature, citing [2, 62, 71] among others. Directly considering the discrete optimization problem as in the submission, rather than a continous model or a continuous relaxation, is a less common choice.

It's easy to anticipate this being applied to panoptic segmentation too. I am not aware of previous work taking this kind of approach for this particular segmentation subproblem.

How the models are combined, and the gradients calculated, is nontrivial, especially when the discrete problem is hard as in this case. Some complex and interesting design decisions made are here.

**Quality**

[36] is a very good choice of which discrete problem to use. It can explicitly handle instances for a subset of classes in a way that directly maps to the panoptic segmentation problem.

A multi-way cut optimization is a very principled way to handle the discreteness of the panoptic labels. The original work proposing the panoptic segmentation problem [32] used discrete labels and a quality measure (PQ) that more or less assumes this (using their definition of IoU).

The core novel part of the method, the robust perturbation technique and how it's applied, all looks correct to me.

The experiments are simple but straightforward. The submission does not introduce much in the way of additional confounds or hyperparameters that need to be tuned, so the comparisons seem good. Experiments are done with multiple trials, and the source of the values of the hyperparameters that are present is specified.

One set of experiments, that I did not find but would ideally be present, are any showing the quality of the AMWC solutions or the gradients. Section A.1 in the supplementary material describes a heuristic greedy solver to the AMWC problem. How does this perform in practice in terms of the how close to optimal its solutions are, and how consistently? How much more accurate (by whatever measure) are the gradients for the robust perturbation than [52]? This or other questions, that directly measure how the discrete optimization step is performing, would provide some insight beyond the end-to-end PQ results.

**Clarity**

The submission is very clear. The presentation style is simple and direct. Pseudocode and mathematical definitions are use well for more precise description of the optimization method.

**Significance**

The overall method gets a lower score on panoptic quality (PQ) than the state-of-the art, but it does convincingly improved upon a bottom-up baseline. So it seems very likely that this kind of approach, at least generally considering discrete optimization within the training and inference, is a significant step forward in panoptic segmentation and image segmentation in general.

**Minor Comments**

* Double-checking: does the loss function L, specified in 193, take real-valued or binary labels as input?
* Typo "Trail" in Table 5 in the supplement. The limitation of the core novel component of this method are probably primarily that the solutions to the asymmetric cut problem, and the gradients, not being computed exactly. The submission is clear that it is using approximations here, and discusses how they dealt with the drawbacks of their choice of approximation.

Societal impact is going to depend on application, as the authors state.

This paper proposes an end-to-end trainable approach for panoptic segmentation incorporating a combinatorial optimization layer for post-processing and directly learning a panoptic surrogate loss. It solves the panoptic segmentation based on the bottom-up manner and views the instance separation as a multiway cut problem. Experiments prove the effectiveness.  This paper proposes an end-to-end trainable approach for panoptic segmentation incorporating a combinatorial optimization layer for post-processing and directly learning a panoptic surrogate loss. It solves the panoptic segmentation based on the bottom-up manner and views the instance separation as a multiway cut problem. Overall, the method is elegant and experiments prove its effectiveness. Some detailed concerns are given below:

Some concerns:
1. This paper uses a pre-trained model as baseline and optimizes it with the proposed panoptic quality loss. I'm wondering if it is possible to optimize them simultaneously in an iterative manner? Because the pretrained manner actually brings more cost to the whole process and may lead to a suboptimal solution.
2. The universality of the proposed method. Could the proposed loss function be viewed as a plug-and-play module in other methods for panoptic or instance segmentation? It could bring more technical value.
3. The performance on the COCO dataset is not satisfactory. The suboptimal performance on the COCO dataset could be attributed to the complex scenarios compared with the Cityscapes dataset, especially in Thing categories. Is this brought by the drop in SQ or RQ? A detailed analysis is recommended to be given in the main paper or appendix. The limitations and potential negative societal impact are discussed and addressed.

The paper designs an end-to-end panoptic segmentation pipeline with combinatorial optimization. The authors propose to use asymmetric multiway-cut to make the panoptic segmentation prediction, with a Panoptic Quality surrogate loss as a direct supervision signal and a perturbation-based gradient estimation technique to enable training.  ---

[+] The paper is well-written and easy to follow.

[+] The idea in the paper sounds novel. The idea is followed by a simple and elegant panoptic segmentation pipeline. The proposed key algorithm, AMWC, greatly simplifies the panoptic prediction.

[+] The paper contributes several techniques to implement the AMWC algorithm for large-scale training, including a surrogate loss and a multi-scale back-propagation technique. Particularly, the surrogate loss mitigates the existing gap between supervision signals and the evaluation metric.

[+] The proposed solution outperforms SOTA, e.g., Panotptic-Deeplab and SSAP, in segmentation quality.

--- 

[-] The perturbation-based gradient estimation might limit the scalability of the method. The gradient estimation might get inaccurate when we have more parameters to optimize, for example, when we optimize the whole model instead of only the classification layer as in the paper.

[-] The solver (2 seconds per Cityscape image) is too slow for real-time applications.

[-] Compared to the baseline, the PQ_stuff even dropped on COCO after fine-tuning. It would be better if the drop could be explained.

[-] The experiment part is a little weak. It would be better if more ablation studies and analysis, e.g., on different backbones and training time memory consumption, can be provided.

---

In summary, the main idea of the paper about using a combinatorial optimizer to simplify the notoriously complicated panoptic prediction task is interesting and novel. However, I have concerns about the scalability and the real-world impact of the method since the solver is currently too slow and a sophisticated optimization method has to be employed in training.

---

Typos:

L97: "architectures.." -> "architectures."

Fig 2. caption: "to a affinity branch" -> "to an affinity branch"

L169: "The AMWC can be though of as a" -> "The AMWC can be thought of as a" The authors adequately addressed the limitations and potential negative societal impact of their work.

Meta-review: The authors propose a method for simultaneous semantic and instance segmentation which can be optimized end-to-end. In particular, the authors make use of predictions obtained by solving the asymmetric multiway-cut problem, coupled with a surrogate loss whose gradient is estimated using a perturbation-based approach. Promising empirical results were presented on COCO and CityScapes datasets.
Reviewers appreciated the clarity of exposition and felt that the proposed approach was a rather elegant solution, notwithstanding the fact that the empirical results were not state-of-the-art. The rebuttals did a great job addressing most concerns, but scalability and practicality remain one of the key drawbacks. Nevertheless, the reviewers felt that the contribution is significant enough to merit acceptance. I agree and I would like to see the authors include a detailed limitations section to highlight the remaining issues.

Decision: accept