Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper focuses on the topic of long tail in semantic segmentation. It designs a model-agnostic multi-expert decoder and output framework, making certain improvements for some classical segmentation models. A diverse data distribution-aware loss function is proposed for preventing over-confidence of minority categories. Besides, it advocates mAcc as a more important metric to evaluate the performance for body and tail categories in long-tailed semantic segmentation. The paper demonstrates some analyses to prove such opinion.
strength_and_weaknesses: Strength；
1.	This paper makes a survey on the topic of long-tail problem in semantic segmentation.
2.	It proposes a model-agnostic multi-expert decoder to resolve such a problem and makes certain improvements for some classical segmentation models.
3.	A diverse data distribution-aware loss function is proposed for preventing over-confidence of minority categories.
4.	The paper conducts some analyses to give the ideal results for the proposed method, showing a promising direction of future research.

Weakness;
1.	This paper claims that it is the first to explicitly focus on the long-tailed semantic segmentation. Yet, There are already some published papers which concentrate on such topic (eg, Region Rebalance for Long-Tailed Semantic Segmentation ,cvpr2022)
2.	The paper duplicate the decoder head twice, bringing a large quantity of extra params. I am not pretty sure that whether the improvements are resulted from such a change. 
3.  The overall pipeline does not differ much from some multi-expert methods in long-tail classification/detection. I would recommend to compare with some of them, as they also conduct segmentation experiments such as "Distribution Alignment: A Unified Framework for Long-tail Visual Recognition"
4. The paper propose that the mAcc metric is a more important metric for long-tail segmentation, giving degenerated mIoU results and better mAcc results in some experiments. Yet, this problem is also mentioned by paper "Region Rebalance for Long-Tailed Semantic Segmentation in cvpr2022" and has been well addressed.


clarity,_quality,_novelty_and_reproducibility: This paper is of good clarity and quality in writing, and is well-organized. Yet, it should have compared with some existing methods in long-tailed classification, which also provide semantic segmentation results. Besides, the novelty is rather limited. Such a method is widely used in multi-expert long-tail classification/detection task. As for the results, I believe that such results can be easily reproduced.
summary_of_the_review: This paper presents the long-tail problem in semantic segmentation and claims that they are the first to explicitly focus on the long-tailed semantic segmentation. Yet, there are already some published papers focusing on such topic (eg, Region Rebalance for Long-Tailed Semantic Segmentation, cvpr2022). Besides, the main idea of this paper is widely adapted by multi-expert long-tailed methods. The paper also advocates that the mAcc metric is more important for long-tail segmentation, and gives degenerated mIoU results. Yet, the paper, Region Rebalance for Long-Tailed Semantic Segmentation, also encounters such a problem and well address this problem.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper considers the semantic segmentation problem. The motivation of this paper is based on the relatively poor performance for the tailed classes and in order to overcome this issue, the author propose to use multiple experts to process pixels with different category frequency. In the experimental section, the paper illustrates some improvements especially for the rare classes on Cityscapes and ADE20k datasets.
strength_and_weaknesses: Strength
The paper is well-written, and the experiments are well built to cover two most popular datasets for semantic segmentation. The ablation is comprehensive to cover the main contribution of this paper

Weakness

1. What is the contextual module? It needs to be illustrated with more details.

2. The auxiliary loss is unclear to me. What is the reality distribution 'q'. I do not quite understand this comment 'It is inevitable that we need to classify these pixels, even if they fall into majority categories'. From my understanding, for experts that process body or tail categories, these majority categories can be ignored without causing any gradient. In equation 4, the auxiliary loss tends to minimize the logits for these classes, which needs more explanation. In summary, the role played by this loss is quite confusing.

3. I have a big concern about the effectiveness of the proposed method. In table 2, the performance of the proposed method is even worse than the baseline. Despite the oracle results are promising, it is not applicable since we cannot priorly know which expert should process which pixel. For table 4, I can see some improvement for the rare classes on cityscapes, but the improvement is also quite minor. I admit the improvement on ADE20k is interesting, but only obtaining good results on one dataset is not enough to validate the effectiveness of the proposed method. 

4. The paper divides the categories into three sets. But during training, the experts process some overlapped categories. Why does the author device such an architecture instead of letting each expert work on a separate class set.

5. The author only compare their method to the straight baseline. It is okay but it would be better if it can comprehensively compare the method to the SOTA long-tail techniques used in semantic segmentation.
clarity,_quality,_novelty_and_reproducibility: The clarity, quality and novelty is good. The reproducibility cannot be told since the code is not contained.
summary_of_the_review: Overall speaking, the paper is interesting, but it has a big space to be further improved. I would like to see the response of the author to my questions.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper tackles the problem of improving the accuracy of semantic segmentation on long-tail classes. To this end, it introduces separate heads to the network output that focus only on specific segments of the class distribution, which are grouped heuristically into “head”, “body” and “tail” categories, based on their frequency in the dataset. A few decoder layers are trained to aggregate the predictions from these heads into the final model prediction. This approach improves the mean pixel accuracy across classes, but mIoU stays largely the same.
strength_and_weaknesses: Strengths:
- I concur with the authors that the long-tail problem in recognition deserves more research.
- The proposed approach is intuitive in its intention and improves the mean accuracy.
- Experiments with multiple model architectures and datasets.
- The reference numbers for the oracle scenario are interesting.

Weaknesses:

I understand that the approach does not pan out in terms of the IoU, hence the need to push mAcc as the more important metric. The work provides some formal argument, which I do not find convincing, however. The interpretation of the results — mAcc improves, mIoU does not — has a simple explanation to me: the method reduces the number of false negatives at the cost of increasing the number of false positives. I do not consider this is an overall improvement.

Some technical steps are not explained to a sufficient extent:
- The discussion leading up to Eq. 4 is unclear. What norm is imposed on z and why this need? What is the “reality” data distribution?

The notation has many loose ends, which impedes understanding of the actual implementation:
- In Eq. 4 the sum scope is unclear;
- The second line above Eq. 6: the condition for y membership is unclear, because the sets S are disjoint, as defined above.

Some of the claims are not backed up:
- Second line after Eq. 5: “K experts are good and distinctive from each other” — how are the goodness and distinctiveness measured?
clarity,_quality,_novelty_and_reproducibility: - Some arguments need fine-tuning to improve the clarity and quality in the paper, especially when it comes to formalizing them. Another example to the ones above is the last paragraph on page 4: X and Y should be apparently defined as a tuple (X, Y), but the definition of Y as a sample-agnostic label set is confusing. 
- There is limited novelty in this work. The approach to process groups of categories with a standalone classifier is a common theme in the literature on long-tail recognition.
- The implementation details seem sufficient for reproducibility.
summary_of_the_review: The empirical results do not show a significant improvement in terms of mIoU, which accounts for false positives and false negatives. The argument that mAcc is a more important measure is not convincing, since one would need to consider a very specific scenario of a downstream task for semantic segmentation. Since we commonly study semantic segmentation in an application-agnostic way, mIoU remains an important metric even if the focus is on the long-tail categories.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper is about semantic image segmentation. The authors notices that there are some categories have less pixels than other categories, and propose to divide all the categories into different groups according to number of pixels and proposes to use mixture models to different group of categories. Experiments on Cityscapes and ADE20k prove the effectiveness.
strength_and_weaknesses: The paper introduces a hierarchical structure, improves the quality of long-tailed distribution in semantic
segmentation. The experiments show the competitive performance of the proposed method. The paper provides detailed information for reproduction, opens a new direction for future research.

There are some questions. 

1.Equation (4) looks strange.

2.In Table 2, the results on Cityscapes is consistently lower than baseline accross different method, I think something should be done to improve the experimental results
clarity,_quality,_novelty_and_reproducibility: Writing is good.
summary_of_the_review: This paper proposes to use a two-stage model for semantic image segmentation. The idea is straight-forward and the main question is experiment, which is not good enough.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: All reviewers for this paper concurred that while the paper addresses a useful problem, there are a number of weaknesses that keep the paper below the acceptance threshold for ICLR. Multiple reviewers expressed similar concerns about the experimental results and the effectiveness of the proposed method, and in particular also questioned the paper’s focus on the mAcc metric. There was also some question about the methodical approach and novelty compared to previous works. After reviewing the paper and author and reviewer comments, I agree with the shared assessment of the reviewers and do not recommend acceptance of this paper at this time.
justification_for_why_not_higher_score: I consider the weaknesses of the paper with respect to experimental results and metric, as well as methodological approach and novelty, too significant to justify acceptance of the paper.
Decision: Reject