Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This paper focuses on the problem of semantic segmentation. Traditional semantic segmentation methods mainly adopt the FCN kind of structure and treat the task as per-pixel classification. While in this paper, the author reformulates the task into binary mask prediction and extra-label classification. The proposed method named MaskFormer could be utilized for both semantic segmentation and panoptic segmentation tasks, and the experimental evaluations on different datasets look promising.
main_review: Strengths:
1. The way of conducting semantic segmentation as the task of mask prediction and classification is interesting. It is different from previous lots of DL-based semantic segmentation solutions and provides some new insights in this field.

2. The experimental evaluations on different datasets are promising and convincing. Even though some of the results are not SOTA, while some others are nice, e.g., the performance on ADE20K validation set and full set.

3. The writing is clear and easy to follow.

Weaknesses:
1. The way of utilizing mask classification for panoptic segmentation is previously adopted in MaX-DeepLab, using conditional convolutions. In this paper, the author also uses mask classification while with transformer and bipartite matching. It is not clear which solution is better regarding the accuracy, efficiency, etc.

2. It seems that most of the reported results are in validation sets. It would be nice to include some results on the test sets, like ADE20K, Cityscapes, and COCO.
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: In this paper the authors present a technique to directly predict the mask and class label for semantic and panoptic segmentation. This is different from current state of the aret where per pixel labels are produced which are then process to obtain region masks and labels. The method consists of a backbone encoder which produces an image embedding, whic is fed into the backbon decoder to get pixel level embedding and to the transform decoder, the out of the transformer decode processed by an MLP provides the mask embedding. The mask embeddings and the pixel embeddings are used to get mask prdication. Whereas the class predictions are obtained throught the MLP output.

The method is tested on semantic segmentation datsets and produce comparable or better results on four datasets.
main_review: The paper tackles the problem of semantic and panoptic segmentation by proposing MaskFormer which simulatenously predict the binary mask and it's corresponding class label. The method can work on any semantic semgentation and panoptic segmentaiton backbone. Following are the main strenghts of the paper.

1) The paper is well-written, the main ideas are effectively presented and it is easy to read.

2) The method is of use to the community, the approach is sound and make intuitive sense.

3) Despite the fact that all the modules used in the setup are form the literature, the setup for the problem is novel.

4) The experiment section shows the effectiveness of the proposed method. 
limitations_and_societal_impact: The paper presents a novel technique for simualtaneous mask segmentaiton and class labelling in segmentation problems. I found the few design choices to be a bit arbitrary and not well motivated.


1) Line 149-150: "Note, we empirically find it is beneficial to not enforce mask predictions to be mutually exclusive to each other by using a softmax activation.". Line 167: "We empirically find that marginalization over probability-mask pairs...". Line 172-173: "we empirically observe directly maximizing per-pixel class  likelihood for MaskFormer leads to poor performance."
Claims like the obove should be motivate a bit more, probably even just showing the experiments which led to these design choices would suffice.


2) I could not find in the paper if the masks output by the algorithm are always contiguous. If yes, why, if not how do we tackle the panoptic segmentation when lets say 2 instance of the same class are labelled in one mask form the algorithm. Are we controlling for this with large N? as this will decouple different instance of same class.


3) "We observe the model that predicts N = 100 masks consistently performs the best across datasets with different numbers of classes, suggesting that N is a stable hyper-parameter". N, the number of mask seems to be an important hyperparameter for the algortihm and should not be fixed across datasets. It might work for these datasets but in general I dont see why it should work in all cases.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper discusses per-pixel classification and mask classification for semantic segmentation. It shows that mask classification is sufficiently general to solve semantic and instance segmentation in a unified manner. Thorough experiments are conducted on several benchmarks for both semantic and panoptic segmentation. 
main_review: Strengthens:
1. This paper is well-written and easy to follow.
2. The most valuable point in this paper is its highlight for pointing out the differences between per-pixel classification and mask classification. This point is not explicitly discussed before, while the authors show that mask classification is better than per-pixel classification with thorough experiments. 
3. The experiments are thorough on many benchmarks. 

Weaknesses:
1. The key insight of this paper is interesting. However, this operation is not first proposed in this paper. The paper Max-DeepLab [39] similarly tackled panoptic segmentation. The authors discussed the differences in Line 81-84. However, the differences in the losses are not significant enough. 

Overall, this paper is well-contained with great presentation and thorough ideas, but the methodology part is incremental vs. existing work. Nevertheless, I indeed agree that the key insight here is meaningful for discussion. Hence, my initial rating is borderline acceptance. 

I also have several questions. 
1. The method segments images with full resolution. How about the performance with the other resolutions? A trade-off table between performance and efficiency is appreciated. 
2. The method merges semantic masks from different queries. However, during the training process, the masks are trained with one-to-one matching. A visualization or statistical analysis for how the queries are merged during the inference is beneficial for understanding the optimized models. 
3. In Line 261-262, there are several tricks for the inference process. The confidence threshold is 0.7, not 0.5. Ablation studies about the inference process are appreciated. 

Typos:
Line 280: “testbed”. 
limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes a transformer-based model, called MaskFormer for semantic segmentation task. MaskFormer treats semantic segmentation as a mask classification problem, and predicts a set of binary masks with the corresponding class labels.  The proposed approach can be easily extended for panoptic segmentaion task. Experimental evaluations for semantic segmentation and panoptic segmentation tasks demonstrate the effectiveness of proposed method.
main_review: Strengths:

1. The proposed idea of  treating semantic segmentation as a mask classification problem is interesting. 

2. Semantic segmentation and panoptic segmentation results are impressive. 

Weaknesses:
1.	The proposed framework can be seen as a box-free simplification of DETR.  Hence, the advantages and novel contributions of the proposed approach compared to DETR needs to be further clarified. 
2.	For the panoptic segmentation task, DETR predicts boxes around both stuff and thing classes, while the proposed approach directly predicts masks for both stuff and things classes. 
Lines 290-293: Please clarify why predicting masks based on boxes is inferior  to predicting masks directly. In many scenarios, the box prediction may provide some prior information indicating possible image regions containing the desired category, which may be helpful even for the stuff classes.   Hence, it will be interesting to investigate if the superior PQ^stuff performance over DETR in Table 3 is really due to this direct prediction of mask, or due to the encoder-decoder structure? 
3.	Why the MaskFormer  operation (matrix multiplication operation) is superior to the panoptic head in DETR for the stuff segmentation task. It will be interesting to perform an ablation study that replaces the matrix operation in MaskFormer with the  DETR panoptic head module. 


limitations_and_societal_impact: There is no major concerns regarding the negative societal impact of this work.
review_title: null

Meta-review: Reviewers are unanimously positive about this paper and recommend to accept it. I see no reason to overturn their decision.
The paper is well written, with a clear message and good results. The authors are encouraged to include some of the discussions related to Max-DeepLab and DETR in the main paper (if they can find space for it).
Decision: Accept (Spotlight)