Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper presents a unified approach for Semantic, instance, and panoptic segmentations tasks.  Proposed method segments both instances and semantic categories consistently by a group of learnable kernels, where each kernel is responsible for generating a mask for either a potential instance or a stuff class.
The model is training using bipartite matching, and the training and inference are naturally NMS-free and box-free. 

main_review: The panoptic segmentation approaches usually use a separate method for prediction things and stuff. It’s nice that this paper proposes a unified approach for them.

The proposed method is evaluated on three tasks of coco panoptic segmentation, coco instance segmentation and ADE20k semantic segmentation.
They show that by adding k-net on top of different backbone and model architectures they can consistently improve the performance of the model.
On the coco panoptic segmentation task they could achieve a new state-of-the-art result.
However, the best result on coco instance segmentation is much lower than state-of-the-art. Also the improvement on instance segmentation task seems to be less significant. I’m wondering why is that? Does k-net give more improvement on stuff compared to things?
It would be nice to provide per category improvements by adding k-net to one of the models for the panoptic segmentation task..

limitations_and_societal_impact: The discussion about broader impact look good to me.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper presents a convolutional neural network method for panoptic (multi-class, multi-instance) image segmentation. Semantic segmentation is addressed in the standard fashion with pre-trained model weights (convolutional kernels). Novelty is introduced in the instance segmentation component where the parameters of the convolutional kernels responsible for producing activation maps, one for each instance, are adapted at test-time to fit the given image. Overall this is a neat idea and the experimental results are compelling. However, I found the paper difficult to understand on first readings and have several suggestions for improvement of the presentation. 
main_review: Overall I found the idea of adaptive model components for instance segmentation appealing but the presentation of the paper lacked clarity. It was not immediately clear (although obvious after reading Section 3) that the kernels referred to in the paper are filter kernels for convolutional neural networks. The term "kernel" has multiple meanings in the machine learning literature and this should be clarified much earlier in the paper.

My biggest concern relates to details of the initialization and updating of the adaptive instance kernels, which is the core contribution of the paper. First, it is not clear how the number, N, of kernels is chosen and what happens (at test time) if this number is over- or under-estimated. Second, it seems at least when initialized that the instance kernels should be more sensitive to spatial location than appearance, and then adapt to appearance. Can the authors comment? Also, is there a need to treat "thing" and "stuff" instances differently?

Third, it may be helpful to summarise the iterative updating of the kernels  in a formal algorithm. Eqn (2) gives the high-level overview although it is not clear why function $f$ is indexed by $i$. The spatial indices $(i,j)$ in Eqn (3) clash with the index $i$ for the iteration number and its not clear how the batches are indexed (presumably the operations are applied independently for each batch). Eqn (5) introduces $\tilde{K}$ without providing an explicit relationship to $K_i$ (from the following paragraph appear to be a transformer followed by multi-layer perceptron?). All of this can be presented more rigorously.

Last, it is not clear what effect the number of iterations applied to adapting the kernels has on the results. Table 3(c) suggests that accuracy saturates at four iterations for ADE20K and doesn't decrease thereafter but further data points should be given. Does the same finding hold for COCO? How many iterations (rounds) were used during training and does this affect the finding?

Some other minor things to address:
1. The image on page 1 is poorly placed and distracts the flow of the paper. I suggest moving to the top of the second page.
2. The idea of joint semantic and instance segmentation appears in the literature well before the term panoptic segmentation was proposed (line 20). For example, Tu et al., IJCV 2005 propose the idea of image parsing, Gould et al., NeurIPS 2009 propose the idea of image decomposition, and Yao et al., CVPR 2012 develop the idea of holistic scene understanding.
limitations_and_societal_impact: I could not identify any societal impacts.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper tries to unify semantic and instance segmentation with a group of learnable kernels that is conditioned on image features. Although the idea of using dynamic kernels (aka dynamic convolutions) has been already explored in instance segmentation [44, 47] and panoptic segmentation [30], this paper proposes to use sparse dynamic kernels instead of using kernels that are generated on dense grid in [30, 44, 37]. This paper further borrows the idea of bipartite matching loss in DETR [4] to make their model NMS free. The experiments show competitive results compared to state-of-the-art methods on multiple segmentation tasks.
main_review: First of all, I have concerns to the writing:

(1) In my optional, Table 4 (a) is a very important and also interesting experiments, however, there is not much description to this experiment. First, what is the baseline model without A.K.U. and K.I.? This is very important, because the improvements of these two components seem to be very significant. I want to make sure the author is not making comparison to some unreasonable baseline. Especially for without A.K.U., is kernel feature simply updated by $\tilde K = F^K + K_{i-1}$? If this is the case, why gating makes so much improvement? Another question is, without A.K.U., why K.I. makes performance worse? This does not make much sense, adding self-attention is possible to not improve the performance, but I don't expect it to decrease the performance.

(2) L295, positional encoding is mentioned here for the first time. It is unclear where positional encoding is added.

(3) L42, the authors claim they make two "novel" designs here. I would suggest the authors to not make a such strong claim. Especially for the second design: bipartite matching loss is proposed by DETR [4] and the authors should not claim it as their novel design.

Second, I have some questions/concerns on Adaptive Kernel Update (A. K. U.): According to the ablation study in Table 4 (a), the proposed A.K.U. is the most important component in K-Net. However, the motivation of A.K.U. is not very clear in my opinion. In the draft, the authors claim since $M_i$ is not precise, $F^K$ may contain noises and the gating mechanism can reduce the adverse effect of the noise. My questions are:
(1) Since there is no spatial resolution anymore, the only way to reduce the effect of noise is to reduce the amount of information from $F^K$ to $\tilde K$. But why the multiplication between $F^K$ and $K_{i-1}$ can provide such information?

(2) Why there is also a gate needed for $K_{i-1}$ as well?

(3) The proposed gating seems very simple according to the text description, so I have tried such gating myself with a simplified K-Net architecture (without semantic features on instance segmentation). However, I found the gating mechanism does not work as well as that is reported in Table 4 (a). In fact, it always decreases performance. I'm wondering if the authors provide the exact description on how they implement the A.K.U. and is there any special design that is not mentioned in the paper?

Some minor suggestions:

(1) L182: $g_i$ is undefined?

(2) L217: does it mean there is still conflict between instance and semantic segmentation so you need to paste instances first?

In general, this paper proposes an interesting architecture and performs enough ablation studies to validate the architecture. But the lack of motivation to the most important component (A.K.U.) and the difficulty to reproduce this important component lead to the weak accept score. If the authors can well address my concerns (especially on the A.K.U. part), I am happy to raise the score.




----
\#### Post rebuttal comment ####

The authors have addressed my concern on the reproduction issue. The new ablation Table is much more reasonable than the original one in the submission. On the other hand, the new ablation Table shows one of the proposed components is less effective and may not even be necessary.

As I said in my original review, this paper proposes an interesting architecture and it is of value to the community. Since the authors address my concerns, I will raise my score to something between 6 and 7 (more towards 7).

As pointed out by multiple reviewers, the presentation of this paper is not satisfactory: it misses many important details. Thus, I would suggest the authors to check their writing more carefully, and provide all the necessary context and details.
limitations_and_societal_impact: yes
review_title: null

final_score: rating: 9: Top 15% of accepted NeurIPS papers, strong accept
review_content: summary: This paper propose a unified framework named K-Net which segments both instances and semantic categories consistently by a group of learnable kernels, where each kernel is responsible for generating a mask for either a potential instance or a stuff class. The authors propose a refined kernel update strategy to group categories efficiently and effectively. Extensive experiments on three different segmentation tasks prove that such simple design can achieve STOA results while running faster than many previous hand-crafted segmentation methods. 

main_review: Overall, from the motivation aspect, this is the most interesting paper in the topic of segmentation I have reviewed in this year. It brings a new way of modeling three segmentation tasks in one framework.  The new concept is to unify all the segmentation tasks which is also the goal of panoptic segmentation. This will be a fulture direction of segmentation task.

Although MaxDeeplab[1] and DETR[2] solve the panoptic segmentation in a NMS free manner, the former requires specific backbone and attention layers and the latter needs much longer time to converge also along with the heavy mask head. 
K-Net can fairly compare with previous work using ResNet backbone (PanopticFCN, Panoptic FPN) and generalize well on large backbone (Swin-Transformer[3]). 


Here are the more details:

1, Writing: The overall writing is good and easy to follow. 

2, Method: The proposed framework is simple yet effective which has novel components such as kernel update for interative refine. The kernel fusion is updated by the  learnable gate which is reasonable.  
Moreover such design is also computationally efficient which can be a plug-in module for other tasks such as human couting and pose estimation.
 
3, Experiment: The proposed K-Net surpasses all previous state-of-the-art single-model results of panoptic segmentation on MS COCO and semantic segmentation on ADE20K.  Most importantly, the comparison is fair and convincing. 
The ablation studies are enough and good to prove the effectiveness of each part in method.  

4, Potential: The framework is easy to extend into related tasks such as video segmentation tasks.

Thus I give my init rating as strong accept. 
Hope the code can be available for the community soon. 

[1] MaX-DeepLab: End-to-End Panoptic Segmentation with Mask Transformers CVPR-2021 

[2] End-to-End Object Detection with Transformers ECCV-2020

[3] swin transformer: hierarchical vision transformer using shifted windows.
limitations_and_societal_impact: No major weakness on both method and experiment parts.

Here are several detailed suggestions and questions. 

Suggestions: 

1. Better cite and compare Max-Deeplab [1]. 
2. It seems that using kernel is a little confusing. Better to discuss the difference with object query in DETR [2] or Max-Deeplab [1].
3. Better give the detailed network architectures. (You can put it into supplementary just like Max-Deeplab).
4. What is g  in Line 182 (need the defination)?  I guess it is a fc layer. 


[1] MaX-DeepLab: End-to-End Panoptic Segmentation with Mask Transformers CVPR-2021 
[2] End-to-End Object Detection with Transformers ECCV-2020
review_title: null

Meta-review: The submission discusses how to unify methods for semantic, instance and panoptic segmentation. For this the authors propose to use a group of learnable kernels and a corresponding update strategy. All reviewers appreciated the contributions of this paper and recommend acceptance/leaning to accept after having read each others reviews and the rebuttal, which addressed some concerns about missing/incomplete ablations. However, all reviewers also pointed out that presentation and writing are not satisfactory and lack clarity. AC read the paper and concurs. E.g., "With the constructed M_0, K_0, and F, the kernel update head can start produce group-aware kernels K_s, s = {1, 2, ..., S} iteratively by S times and obtain the refined mask prediction M_S." (randomly picked sentence). Sentences like this are hard to parse and may prevent others from replicating the results. This being said, the authors promised to release code and models.
Decision: Accept (Poster)