MEDOE: A Multi-Expert Decoder and Output Ensemble Framework for Long-tailed Semantic Segmentation
Authors: Junao Shen
Long Chen
Tian Feng
Zhang Wei
Fei Wu
Kun Kuang
Venue: Submitted to ICLR 2023 (2023)
Keywords: semantic segmentation
long-tailed distribution

Abstract: Long-tailed distribution of semantic categories, which has been often ignored in conventional methods, causes unsatisfactory performance in semantic segmentation on tail categories. In this paper, we focus on the problem of long-tailed semantic segmentation. Although some long-tailed recognition methods (e.g., re-sampling/re-weighting) have been proposed in other problems, they are likely to compromise crucial contextual information in semantic segmentation. Therefore, these methods are hardly adaptable to the problem of long-tailed semantic segmentation. To address this problem, we propose a novel method, named MEDOE, by ensembling and grouping contextual information. Specifically, our MEDOE is a two-sage framework comprising a multi-expert decoder (MED) and a multi-expert output ensemble (MOE). The MED includes several ``experts", each of which takes as input the dataset masked according to the specific categories based on frequency distribution and generates contextual information self-adaptively for classification. The MOE then ensembles the experts' outputs with learnable decision weights. As a model-agnostic framework, MEDOE can be flexibly and efficiently coupled with various popular deep neural networks (e.g., Deeplabv3+, OCRNet, and PSPNet) to improve the performance in long-tailed semantic segmentation. Experimental results show that the proposed framework outperforms the current methods on both Cityscapes and ADE20K datasets by up to 2\% in mIoU and 6\% in mAcc.

Under review as a conference paper at ICLR 2023
MEDOE: A MULTI-EXPERT DECODER AND OUTPUT ENSEMBLE FRAMEWORK FOR LONG-TAILED SEMAN- TIC SEGMENTATION
Anonymous authors Paper under double-bling blind review
ABSTRACT
Long-tailed distribution of semantic categories, which has been often ignored in conventional methods, causes unsatisfactory performance semantic segmen- tation on tail categories. In this paper, focus on the problem of long-tailed semantic segmentation. Although some long-tailes recognition methods (e.g., re- sampling/re-weightinnn have been proposed in other problems, they are likely compromise crucial contextual information in semantic segmentation. Therefore, these methods hardly adaptable to the problem of long-tailed semantic seg- mentation. To address this problem, we propose a novel method, named MEDOE, by ensembling and grouping contextual information. Specifically, our MEDOE is a two-sage framework comprising a multi-expert decoder (MED) and multi- expert output ensemble (MOE). The MED includes several "experts", each of which takes as input the dataset masked according to the specific categories based on frequeney distribution and generates contextual information self-adaptively for classification. The MOE then ensembles the experts" outputs with learnable de- cision weights. As model-agnostic framework, MEDOE can flexibly and efficiently coupled with various popular deep neural networks (e.g., Deeplabv3+, OCRNet, and PSPNet) to improve the performance in long-tailed semantic seg- mentation. Experimental results show that the proposed framework outperforms the current methods on both Cityscapes and ADE20K datasets by up to 2% in mloU and 6% in mAcc.
INTRODUCTION
Semantic segmentation is defined the task to predict the semantic category for each pixel in an image. As fundamental computer vision task, semantic segmentation is of great importance in various real-world applications (e.g., clinical analysis and automatic driving). Conventional methods follow the backbone-conter" module architecture, where the context module contributes significantly and can be modified to enhance the extraction and aggregation of surrounding pixels and global information through large-scale field convolution model or attention mechanisms (Chen al., 2018: 88; He et al., 2019; Huang et al., 2019; Yuan et al., 2019; Zhao et al., 2017). Such modifications have enabled the recent state-of-the-arr methods to improve the performance in semantic segmentation on various benchmark datasets (Caesar et al., 2018; Ding et al., 2019; Zhu et al., 2019).
Despite the overall impressive perfor- mance, the above-mentioned semantic segmentation methods still face challenges from the perspective of data distribution. For example, Table shows the results for head, body, and tail categories (i.e,, cate- gories ranked from top to bottom by pixel frequency) benchmark datasets using DeepLabv3+ (He et al., 2016), where pos- itive correlations can be found between re- sults and data distribution. In other words.
Table 1 Comparison of results in terms of mloU (%) and mAce (%) for head, body, and tail categories on Cityscapes (Cordts et al., 2016) and ADE20K (Zhou al., 2017) for semantic segmantation using Deeplabv3+ and ResNet-50c.
Head mloU
Body mAce mloU 73.74
Tail
mAce mloU
mAec
Cityscapes ADE20K
95.06
97.74
83.92 91.49
81.43
65.50
78.71 44.75 59.75
33.64 42.97
the performance declines for tail categories.
This suggests the problem of long- tailed distribution in

---

Under review as a conference paper at ICLR 2023
Figure 1:
Example of semantic segmentation on Cityscapes, where tail categories (e.g., "pole" and "wall"" are not well segmented. From left to right: street scene image, ground truth, and segmenta- tion map using DeepLabv3+.
semantic segmentation: a few head categories dominate the majority of pixels, whereas many tail categories correspond to significantly less pixels. Specifically, if all categories are processed in the same pattern while long-tailed distribution exists, head categories may excessively influence the training and negatively impact on learning the contextual information about tail ones, leading to un- satisfactory pixel-level classification. Solving the problem of long-tailed semantic segmentation is critical to real-world scenarios. As shown in Figure 1, distinguishing poles from street scene images may prevent potential accidents.
To the best of our knowledge, this paper is the first to explicitly focus on the long-tailed semantic segmentation, and aims to provide a straightforward solution by extending the existent recognition methods, which adopt various strategies to solve the problem of long-tailed distribution in image classification (Buda et al., 2018; Huang et al., 2016; Gupta al., 2019). These methods can be gen- erally categorized into re-weighting, re-sampling, and ensembling and grouping. The re- weighting methods increase the weights of tail categories but decrease those of head ones (Cao et al., 2019b; Liu et al., 2019), following the assumption that images nearly independent and identically (i.e., ii.d.) distributed to address the imbalance of training set. This assumption enables the classification accuracy for each category to depend on the frequency of the corresponding images (Cao et al., 2019b; Cui et al., 2022). The re-sampling methods conduct under-sampling for the head categories and over-sampling
even data augmentation for the tail ones (He & Garcia, 2009; Kim al., 2020; Chu et al., 2020), usually following a random sampling strategy to ensure fairness. The ensembling and grouping methods start with training feature extractor on the whole of imbalanced dataset as representation learning, then adjust the margins of classifiers using euulii-expert frameworks (Xiang et al., 2020; Zhang et al., 2021; Zhou et al., 2020) for rebbaannn (Kang et al., 2019).
Nevertheless, the mentionen recognition methods for long-tailed image classification can hardly be adapted for long-tailed semantic segmentation. Specifically, the re-weighting methods are not able to serve pixel-level classification pixel is usually highly correlated to the surround- ing ones (i.e., not i..d.) given the contextual information in the image (Lin et al., 2017). They may also cause see-saw phenomenon: the accuracy for head categories is compromised whereas tail categories are on purpose emphasized. Based on random sampling, the re-sampling methods lead to large number of independent pixels that undermine the contextual information of image and can be detrimental to semantic segmentation. The ensembling and grouping methods rely heavily on re-balancing, where classifier re-adjusting is not well adapted for semantic segmentation due to ignoring the difference among head, body, and tail categories in contextual information.
Motivated by the observation about the ensembling and grouping methods, we propose MEDOE, two-stage multi-expert decoder and output ensemble framework for long-tailed semantic segmenta- tion. At Stage the feature map extracted by a backbone trained on the whole of an imbalanced dataset, which represents the elementary knowledge, is first passed to a multi-expert decoder (MED). Each expert (i.e., a pair of context module and classifier head) in the MED works on the dataset where pixels and their corresponding labels of dominant categories (i.e.. body and tail categories) in each image are masked. Together with other constraints, this expert-specific pixel-masking strategy enables the experts to reduce the impact of head categories and irrelevant pixels and acquire the con- textual information body or tail categories. The following Stage 2 deploys a multi-expert output ensemble (MOE) to ensemble the outputs of all experts from Stage I using decision weights. Instead of being user -ser-spe,,,,,d, these weights are learned by a decision- of the constraints. The proposed framework is model-agnostic and can be integrated with any pop- ular semantic segmentation method, such as DeepLabv3+ (Chen et al., 2018), PSPNet (Zhao et al., 2017), OCRNet (Yuan et al., 2019) and so on.
to avoid the negative impact
2

---

Under review as a conference paper at ICLR 2023
To evaluate the MEDOE's performance in long-tailed semantic segmentation, especially for tail categories, we conduct theoretical and experimental analyses using the metrics mean Intersection- over-Union (mloU) and mean pixel accuracy (mAcc). They demonstrate the effecttiveness of the proposed framework in boosting mAcc while keeping mloU from decrease. In addition, we examine the potentials of MED and MOE despite that the information needed by the expert-specific pixel- masking strategy is unavailable in the inference phase of long-tailed semantic segmentation.
The key contributions of this paper can be summarized as follows:
We conduct an empirical study to discover the problem of long-tailed distribution in semantic segmentation and reveal its significance.
We advocate mAce as more important metric to evaluate the performance for body and tail categories in long-tailed semantic segmentation.
We propose a model-agnostic multi-expert decoder and output ensemble framework that outper- forms the several popular methods up to 2% in mloU and 6% in mAcc on Cityscapes (Cordts et al., 2016) and ADE20K (Zhou al., 2017) datasets.
We demonstrate the ideal information referred to Oracle improves substantially the results. with an average gain of 5% in mIoU and 6% in mAce on Cityscapes and 12% mloU and 19% in mAcc on ADE20K datasets. The results open very promising direction for future research.
RELATED WORK
Semantic Segmentation. FCN (Long et al., 2015) is regarded as the pioneer the field because of introducing the full convolution the whole image and formulating the semantic segmentation task as per-pixel classification with the basic framework of backbone-conten module. Subsequently, various advanced methods have been introduced based on FCN, they be roughly divided into two directions. One is to design novel backhone for more robust feature representation (Wang al., 2020a; Yu & Koltun, 2017). HRNet introduced parallel backbone network to generate high- resolution representations. The other is to enrich contextual information for each pixel (Fu et al., 2019; He et al., 2019; Huang et al., 2019; Yuan al., 2019). For instance, combining high-level fea- ture and low-level feature to extract global information (Amirul Islam et al., 2017; Badrinarayanan al., 2017), introducing large receptive field, such as dilated or atrous convolutions (Chen al., 2018; Zhao et al., 2017) to gather multi-scale contextual cues and building the feature pyramids (Kir- illov et al., 2019). However, these approaches ignore the impact of data distribution, and our work is the first to explicitly focus on the long-tailed data distribution in semantic segmentation.
Long-tailed visual recognition. Re-balancing/R--eighhiin forward solution for long-tailed distribution is re-balancing the contribution of each category in the training phase. Re-weighting approaches (Cao et al., 2019b; Liu et al., 2019; Lin et al., 2017) adjust the loss function or boost larger weights tail categories. Re-balancing approaches (He & Garcia, 2009; Kim et al., 2020; Chu et al., 2020) achieve data balance based on over-sampling the low- frequency categories, under-sampling the high-frequency categories even data augmentation, by generating additional samples to complement tail categories. However, whether sample-wise or loss- wise re-balancing approaches increase the performance of minority categories while the accuracy of majority ones is compromised. Ensembling and Grouping: Recent studies show a new trend of overcoming the long-tailed problem by multi-expert or branch strategy. BBN (Zhou al., 2020) adopted two-brande cro-branches to focus on normal and reversed sampling. LFME (Xiang et al., 2020), RIDE (Wang et al., 2020b) and ACE (Cai et al., 2021) trained on relatively balanced sub- uub-groups, which separated training data by category frequency, and ensembled together into a multi-expert ar- chitecture. Those methods learn diverse classifiers parallel with knowledge distillation, distribution- aware expert selection complementary experts. Despite being suffered from re- balance tricks, the success gained on image recognition has shown the great potential of these methods.
The most widely-used and straight-
METHODS
Following the case described in Table 1, each category in semantic segmentation is tagged head, body, or tail, according to its rank based the corresponding pixel frequency. In particular, body
3

---

Under review as a conference paper at ICLR 2023
Stage 1
Lice
Laux
Head Datasel S
Vo,
Hi.
Head Expert &q
Lev - Laux
Body Detasel Sك
Vo=
H
Backbone Ju Fearure Map
Body Expert E2
Lice Axrox
S3
Context: Module V4,
Classifier Head Jf
Vo
Hoz
Tail Expert
Expert
Multi-Expert Decoder
Priari Distribution
Stage
Axelect
Loss Back-propagation
Learnable Desision
Use
Selected Label
Dsteset Guidence
Multi-Expper Outout Ensemble
The overview of the proposed framework. Stage 1, Followed by shared backbone for representation training, the priori distribution allocate a specific masked dataset for each expert with the priori knowledge, then a MED module combined with context module and classifier self- adaptive generate contextual information on own dominating categories and classify pixels with Loe and Laux- Stage 2, we generate the select label Yee by priori distribution and MOE ensemble the outputs of all experts by a learnable decision- maker with the guidance of Yee.
Figure 2:
categories refer to those ranked lower than the head but higher than the tail. Figure provides an overview about the two-stage MEDOE framework. At Stage 1, given low-level feature map tracted by a backbone from the input image, a multi-expert decoder (MED) is employed to discover contextual information, which has considerable importance in semantic segmentation. The MED includes several experts, each of which is composed of a context module and a classifier head, cor- responding to head, body, and tail categories (respectively referred as head expert, body expert, and tail expert). Following an expert-specific dataset with the image pixels of certain categories masked, which ensures the expert focus on its corresponding categories and predict with higher confidence. Minimizing both Cross-Entrony and auxiliary loss functions, the head expert updates the backbone's parameters on the entire dataset, whereas the others refine their context modules, in order to constrain the effect of interfering pixels and the distribution of masked data. At Stage 2, a multi-expert output ensemble (MOE) is designed to aggregate the outputs from different experts for predicting the label of each pixel labels. The weights used by MOE are from a learnable decision maker, instead of those user-specified, so as to avoid the negative impact of the mentioned constraints.
strategy, each expert processes unique
(CE)
3.1 MED: MULTI-EXPERT DECODER
The MED is designed to draw out different contextual information from the feature map extracted by the backbone, order to adjust the boundaries of classifiers, based on the observation that model's performance in semantic segmentation tend to be superior for head categories. Specifi- cally, the parameters an expert are determined by its corresponding pixel-masking dataset, while the local optima of head categories can be avoided. The conventional context modules with large- scale receptive field convolutions, such as ASPP (Chen et al., 2018), PPM (Zhao et al., 2017) and self-adaptively generate different contextual information from domi-
OCR (Yuan et al.,, 2019), nant categories the expert-specific pixel-maskin datasets. This process can be defined as follows,
Given training set DD {(X, C), X denotes the data and Y denotes ground truth labels, with categories in total. Then we realign C according to the categories frequency in pixel level, C' 21,2..... c), where (F(i) F(J), if (i A)), F(i) denotes the frequency of categories
4

---

Under review as a conference paper at ICLR 2023
i. For K experts E {&i, E....., Ek), ) we assign K expert-specific masked datasets for different experts S = {S1, ,,...,,,K J. In this paper, we set K = by default, based on the intuition of head, body and tail categories, cb and ot denotes the first body and tail categories.
S = (1. S2 {co, Cb+1,...,, ot,, S3 = (er, Cttl......
(1)
In this case, the body and tail categories can be exposed to their dominant experts, and we also adopted some overlapped categories to expose fewer categories to more experts. For hard pixel seg- mentation, the overlapping strategy allows the experts who dominate the category to integrate the outputs of others. Followed the shared backbone fe. experts have chance to extract contextual infor- mation focused on "experts" own categories by MED. To reduce overfitting in minority categories, we only adopt a small number of convolutions in context module V {we, 1822...., ,,0x J. and classifier head HH ho,
høg): J. Finally, the expert & outputs prediction:
24 h9, (vo, (fo())))
Diverse data distribution-aware loss function. We require each expert perform well in their expert-specific masked dataset. Therefore, instead of constraining the final prediction as prior ap- proaches, we apply the diverse data distribution-aware loss function each separately. For each expert & we use the CE loss function to guide the pixel classification on the dataset S;
Lee (Zi, y: 0.) -. yi log (softmax (2:)).
(3)
However, semantic segmentation trains each image end end, and some categories were defined as interfering categories (IC). IC set S.C refers to the pixels' categories not in S,, which means S!C US, = C.Since we expect each expert to focus on the dominating categories, inevitably, S!Cis the main source of confusion & We devise novel suppression compensation measure by defining an auxiliary loss function to suppress the IC with an L2 regularization term. In addition to avoiding the over-confidence of confusing categories, for each expert &,, we minimize KL-divergence between the expert classification probability p with the reality label distribution q in ground truth y over S; categories.
Traux (2t, y: 0.) = -E 5 || -
E Peş
log
Pry
Pi = softmax(2))
(4)
Overall, the diverse data distribution-awarrc loss function for expert & is
L ( Zi y5 0.) Ieee (zi, yr 0s) al-aux (21, y; 0c),
(5)
where a is the syper-parameter to balance the loss of Lee and Leuas We empirically set a 0.2 by default. Since these dominant categories are complementary to each other, the K experts learned at Stage are good and distinctive from each other.
Oracle case with known decision information. As mentioned, the MED, loss terms and expert- specific pixel-masking strategy enable each expert focus better on their dominant categories. For the intuitive assumption of the Stage 1, our model will achieve the theoretically optimal result when each pixel gets the prediction through the expert who dominates its category. We call this situation the "Oracle" case, and take such a case as the theoretical upper bound of our method. Specifically, due to the overlap strategy, when the ground truth label y of pixel æ satisfied: y E S,&y & S,41. we choose expert &,, to output the final classification probability Po of instance x::
Po softmax(he, (Va, (fa(x))))
3.2 MOE: MULTI-EXPERT OUTPUT ENSEMBLE
The MOE with learnable decision-maker is devised upon unavailable information on decisions, and aims to ensemble the probabilities for classification. The MOE takes as input the outputs from all experts at Stage 1, and makes decision on label selection. For the n-th expert, if the ground truth label y of pixel x belongs to the expert's dominant categories (i.e., E S,,). ideally the expert should be selected, yee We construct a feature embedding layer and a classifier to learn this preision-maker Specifically, we concatenate the classification probabilities p for x through the K experts from first to k-th expert and project them to a scale by 2D convolutional layer Wi,
5

---

Under review as a conference paper at ICLR 2023
followed by ReLU, and finally, apply k-th classifier W with Softmax function to get selection probabilities s between each expert in [0,1]:
s(x) = softmax (ReLU(Ws(pp @ p2 @ @ Ptl))).
In the training phase, the decision-maker is optimized with CE loss:
Lectect (s(z), Use) - EE log (s(x):)
(8)
In the inference phase, instead of simply hard decision strategy, for instance a we adopt the select probability as soft decision weight W for all experts, the final classification probabilities:
Prinal = h: E w.P.
[w 2 W2 + 2 wwl== W
3.3 METRIC THEORETICAL ANALYSIS
Theoretical analysis between mIoU and mAce in Semantic Segmentation. Due to the exper- imental results Cityscapes, in Sec 4, we were surprised to find that the mAcc made a great improvement while the mloU remained stable. This misalignment inspires us to explore the corre- lation between mloU and mAce in long-tailed semantic segmentation because primary evaluation metric in segmentation, but traditional long-tailed recognition adopts mAce as standard metric. We refer TP, FN and FP to true positives, false negatives and false positives. Ace; and ToU, refer to pixel accuracy and Intersection-over-unni
mloU the
category i can be present as,
Acc; TP:/ FN.), IoU; = TP::/TPP PN: + FP.).
(10)
The correlation between FP and FN refers to: FP; FN; Meanwhile, Σi=1 FN, more affected by the incorrect instances in dominant head categories than the tail categories (See Appendix for details). Therefore, when evaluating the IoU on tail categories, the FN N-related FP will be suppressed by head categories, and make IoU remain stable. We believe that mloU is metric to focus on the segmentation results of whole images, which will be affected by head categories. Different from mloU, the terms in mAce formula are correlated their own categories, which is more fair metric of the improvement of body and tail categories. Thus, regarded mAce relatively important metrie our experiments. Meanwhile, consider situation where, in pursuit of accuracy growth, resulting in a substantial increase about FP,, which is harmful for segmentation. According to Remark 1, only satisfies the precondition of no significant drop in mloU, the mAcc increase does not cause the harm mentioned previously and tends to better performance in tail categories.
classify all the pixels nearby the tail category instances into this category,
EXPERIMENTS
We conducted the experiments on two challenging benchmark datasets, by integrating our MEDOE framework into DeepLabv3+, PSPNet, OCRNet and Segformer evaluate the effecttveness. Due to the limited space, benchmarks & implementation details are left in the Appendix.
4.1 COMPARISON RESULTS
Cityscapes. To validate the flexibility and efficiency of our MEDOE framework, we experiment with our method Cityscapes. As shown in Table 2 and 4, compared with the current meth- ods, MEDOE significantly improved mAce without reducing mloU and too many extra parameters. Specifically, for ResNet-50c, ResNet-101c, HRNet-W48 and MIT-B3 (Transformer-baseel are 3.36%, 3.46%, 1.03% and 2.88% gains in mAee DeepLabv3+, PSPNet, OCRNet and Seg- method achieved more impressive results than the ordinary re-weighting
there
Former. Especially, method (Focal Loss). This is because Focal Loss through the simple re-weighting method slightly improves the accuracy of the tail categories with significant decrease in the performance of the head categories, which will cause an unacceptable decline in mloU.
ADE20K. ADE20K is challenging benchmark due to its various image scales and plenty of mantic categories in reality, which cause the long-tailed distribution. In this scenario, our method shows better performance than the current methods. The experimental results are summarized in

---

Under review as a conference paper at ICLR 2023
Table Comparison of performance and memory size (Params) on the validation set of Cityscapes with current methods. i: Oracle results for the ideal scenario.
Methods
Backbone
mloU (%)
mAce (%)
Params (GBs)
DecpLaby3++ (Chen et al,, 2018)
ResNet-50c
80.37
86.68
24.16(1.0x)
DeeDLabx3+MMEDEE
ResNet-50c
80.20(-0..1)
90.04(-3.36)
34.31(1.4x)
D
ResNet-50c
84.14(43..77
92.38( (2.38(-5.700 (5770)
34.31
PSPNet (Zhao al., 2017)
ResNet-50c
78.32
85.52
18.33(1.0x)
PSPNet-MEDOE
ResNet-50e
78.02(-0.30)
88.98(
88.98(-3.405 3.46)
20.37(1.1x)
PSPNet-MEDOEf
ResNet-50c
84.32( 4-1.66)
91.90( 5.34)
20.37(1.1x)
DeepLaby3+
ResNet-10lc
80.67
87.58
25.02(1.0x)
DeeLLab33--MEDE
ResNet-101c
80.23(- 1233-00.4 0.44)
91.12(
3.52)
35.03(1.4x)
DeepLab33+
ResNet-
84.51(+3.84)
92.63(
(55055)
35.03(1.4x)
PSPNet
ResNet-101c
79.76
86.56
20.11(1.0x)
PSPNet-MEDOE
ResNet-101c
79.79(+0.03)
90.88( 1.32)
23.81
PSPNet-MEDOE
ResNet-101c
(4.22)+4.66)
91.90( 5.34)
23.81(1.1x)
OCRNet (Yuan et al., 2019)
HRNet-W48
80.70
88.11
23.41(1.0x
OCRNet-MEDOR
HRNet-Way W448
80.21(-0.49)
89. 89.14( (.00)))0))
30.27(1.3x)
OCRNet-MEBL MEDOE
HRNet-
-4 W48
85, (9(++4559)
93.38( 93.38(-5.277 -5.27)
30.27(1.3x)
SegFormer (Xie et 2021)
MIT-B3
81.94
88.28
30.02(1.0x)
SegFomee-MEDDF
MTT-B3
81.67
91.16( 2.88)
38.65(1.3x)
SegFomerr-MEDOF!
MTT-B3
85.67( 3.7.3)
93.49 5.21)
38. 38.65(1.32
Table 3: Comparison of performance and memory size (Params) on the validation set of ADE20K with with current methods. t: Oracle results for the ideal scenario.
Methods
Backbone
mioU (%)
MAcc (%)
Params (GBs)
DeepLabv3+ (Chen ct al., 2018)
ResNet-50c
42.11
54.13
25.82(1.0x)
DeepLaby3+-MEDOR
ResNet-50c
43.82(++1.71)
(+..2(
35.49(1.4x)
DeepLaby3+-MEDDE
ResNet-50c
53.34(++11.23))
73.97( (5.977(199841)
35.49(1.4x)
PSPNet (Zhao et al., 2017)
ResNet-50c
40.46
51.42
19.81(1.0x)
PSPNet-MEDOE
ResNet-50c
41.76(++1.0))
54.27((+2.85)
23.49(1.1x)
PSPNet-MEDOE*
ResNet-50c
52.00(+11.54)
71.61(-20.11
23.19(1.1x)
DeepLabv3+
Res
44.60
56.28
26.31(1.0x)
ResNet-101c
16.13(+1.42)
61.12(+44.8)
36.27(1.4x)
DepLabv3+-MEDOE
ResNet-101c
55.18(+10.58)
(6.82(-20
36.27(1.4x)
PSPNet
Rescet-101c
43.33
54.51
22.94(1.0x)
PSPNEt-MEDOE
ResNet-
44.31(+00.88
59.85(+5.19)
25.69(1.1x)
PSPNet-MEDOF
ResNet-101c
53.47(+10.14)
72.86(-18.45)
25.69(1.1x)
OCRNet (Yuan et et., 2019)
HRNet- W48
42.53
54.91
33.03(1.0x)
OCRNeI-MEDOE
HRNet- W448
43.31(+0.78)
58.96(+44.05)
37.67(1.1x)
OCRNet-MEDOE
HRNet-W445
51.99( (99.6)
74.68( (4.68(-19.77)
37.67(1.1x)
SegFormer (Xie al., 2021)
MIT-B3
47.13
60.84
30.02(1.0x)
SegFormer-MEDEE SegFomer-MHDDFF
MIT-B3
48.22(+1.00)
(4..664+3,220
33.96(1.3x)
MTT-B3
56.49( 49936)
79.41(-18.55
33.96(1.3x)
Table and 4, method achieved 1.71%, 1.30%, 0.78% and 1.09% gains in mloU and 5.89%, 2.85%, 4.05% and 3.22% gains in mAce for DeepLabv3+, PSPNet, OCRNet and SegFormer. Fur- thermore, we illustrated the qualitative results in Figure the validation set of ADE20K. We can see our method achieved better segmentation results than baseline on these tail categories in different scenarios, such as pillow and lamp, which shows the consistent effectiveness of our method.
Results in diverse category data subsets. As illustrated in Table 4, our method achieved impressive mAce gains for body and tail data subsets on Cityscapes and ADE20K. Specifically, we achieved overall improvement on ADE20K. For further, Category-wiss performance improved comparisons between MEDOE and baseline shown in Figure 3. MEDOR achieved both loU and Ace im- provements on Cityscapes and ADE20K benchmark datasets in almost all categories. Especially, our method has significant advantages for body and tail categorics data subsets, which verifies our motivation that MED can extract effective contextual information to help pixel classify.
Oracle results. We investigated the ideal scenario to provide the exact experts' selecting decisions in the inference phase substitute the MOE, such that W W Reported results in this ideal scenario, referred to as Oracle in Table 2 and 5, shown impressive improvements in mloU and mAce over both current methods, with consistent gain across all datasets. Specifically, method gains 3%.4%

---

Under review as a conference paper at ICLR 2023
improvemeen
0.3
0.
009
002
604
002
1/4
603
-002
Category HHodd 1e 2
Catepray Senad to
(a) Performance improvement on Cityscapes
(b) Performance improvement on ADE20K
Figure 3: Comparisons of each category performance on Cityscapes and ADE20K with MEDOE and baseline. Our MEDOE gains both IoU and Acc improvements in minority categories on Cityscapes and ADE2KK, as well as avoid decreasing the majority performance.
(a) Image
(b) Groud Truth
(c) DeepLabv3+
(d) Ours
Figure 4: Qualitative Visualization the validation set of ADE20K with ResNer-50c as backbone. All the models here are trained under the same setting.
in mloU and 5%-7% in mAce on Cityscapes and particularly the improvements raise to 11%-113 in mloU and 19%-22% in mAcc on ADE20K. We believe these impressive improvements convey several important messages. 1) proves that our components Stage have trained series of robust and effective experts which focused on their dominant categories. 2) Compared to the results of our method, they indicate that there exists a strategy building on top of MED that can largely outperform kfate-of-the-at methods. This suggests that MED has significant prospects, more efforts could be directed towards to constrain the expert decision and opening a door to promising avenues.
4.2 ABLATION STUDY
Ablation of MED. To illustrate that our MED self-adaptively extracts different contextual informa- tion guided by diverse masked datasets, indeed, rather than attributing entirely the modifying clas- sifiers' boundaries operation or simply increasing extra parameters (i.e. f.e.model-ensemlle We designed Multi-Classifie network (MC) with a shared backbone and context module and model- ensemble method for comparison experiment in the whole datasets on Cityscapes, the results are shown in Table 5. Just modifying the classifier boundaries not effective in the tail categories, and reduces the overall performance. Since the task is really difficult, the model-ensemble method simply enhancing extra parameters amulti-backbonn our training strategies, it improves head performance but ignores the tail categories. Thus MED helps the network better classify pixels by different contextual information.
method).
can not make overall improvements. Without
Ablation of MOE. As mentioned, introduced various user-specified output ensemble meth- ods. For each pixel instance x, (2) Softmax method get the output 2; of expert E, from to k-th and pixel category G j if (softmax(z:)) > B, is set 0.3, (3) Argmax method determine
8

---

Under review as a conference paper at ICLR 2023
Table Comparison of performance (%) on each data subsets from Cityscapes and ADE20K with DeepLabv3+ and ResNet-50c. t: Baseline with Focal Loss, $: Oracle results for the ideal scenario.
Methods
All
Head
Body
Tail
mloU
mAce
mloU
mAcc
mloU
mAcc
mloU
mAcc
Cityscupes
80.37
86.68
95.26
97.74
83.42
91.49
73.01
81.43
Cityscapes
76.23
85.00
91.18
90.97
80.55
90.33
68.79
84.56
Cityscapes- MEDOO
80.20
90.04
95.03
96.52
83.39
94.25
73.65
86.15
Sryssapes-MEDDOR
84.14
92.38
96.43
97.72
87.12
96.70
78.59
88.88
ADE20K
42.11
54.13
65.50
78.71
44.75
59.75
33.64
42.97
ADE20K1
37.61
55.29
60.01
74.89
41.96
61.28
30.28
46.95
ADE20K-MEDOL
43.82
60.02
66.99
77.90
46.32
63.52
35.38
52.33
ADE20K-MEDOR 53.34
73.97
67.38
75.49
55.65
74.84
46.75
74.09
the outputs according to the confidence of the maximum softmax on each expert, (4) Group aver- age method ensemble 2, of & by grouping average weight. Comparisons between MOE with the above-mentioned methods in Table 6. Our MOE achieved improvements overall.
Table 5: Ablation of MC network, model-ensemble method and MED (ours) with DeepLabv3+ and ResNet-5Oc on Cityscapes to verify the effectiveness of multi-context decoder module. Results are reported in mloU (%), mAce (%) and Params (GBs)
All
Many
Medium
Few
Methods
Params
mIoU
mAce
mIoU
mAcc
mIoU
mAcc
mloU
mAcc
Baseline
42.11
54.13
65.50
78.71
44.75
59.75
33.64
42.97
25.82
+MC
10.15
53.79
62.01
75.51
12.53
56.20
32.15
45.18
28.39(1.1x)
+Ensemble
12.50
51.46
65.59
78.62
45.00
60.31
33.87
42.95
77.98 (3.0x)
+MEDOE
43.82
60.02
66.99
77.90
46.32
63.52
35.38
52.33
35.49(1.4x)
Aggregation
mloU
mAcc
550.
MOE
80.20
90.04
Baseline Outs
Softmax(2)
72.31
82.32
475-
Argmax(3)
70.45
83. 14
Ous
Group Avg(4)
78.87
86.49
Figure 5: Ablation w/ or w/o Auxiliary Loss and the hyper-pan parameter on ADE20K with DeepLabv3+ and ResNet-50c
Table 6: Ablation study various outputs ensemble methods with DeepLaby3+ and ResNet-50c Cityscapes.
Effectiveness the auxiliary loss function. ablation with or without an auxiliary loss function and the syper-parameter on ADE20K was conducted to prove the motivation that we introduce auxiliary loss function. The results in Figure 5 shown that the case of using CE loss function alone (Ours with a=0.0), our method can achieve the improvement compared to baseline, but the improvement limited. Meanwhile, when the hyper-parameter a==0.2 we gain the best performance. Training without the auxiliary loss function constraint more likely to be confused by the unavoid- able IP and decrease the performance overall. The introduction of auxiliary loss function obtains a more consistent and stable improvement.
5 CONCLUSION
In this paper, we investigate and identify the long-tailed distribution in semantic segmentation, and motivated by the fact that existing methods ignore this issue and the performance declines for tail categories. We proposed a MEDÒE framework to overcome this challenging problem. The MED, MOE module and series of strategies (c.g., auxiliary loss function and priori distribution) are introduced to help each expert extract different contextual information and promote the performance in dominant categories. Our method achieved great improvements across different current methods up to 2% gains in mloU and 6% gains in mAcc on Cityscapes and ADE20K. Furthermore, training a robust output ensemble decision-maie inference, demonstrated with the significantly improved results obtained by the oracle.
appears to be a very promising way of constraining the

---

Under review as a conference paper at ICLR 2023
REFERENCES
Md Amirul Islam, Mrigank Rochan, Neil DB Bruce, and Yang Wang. Gated feedback refinement network for dense image labeling. In Proceedings of the IEEE conference on computer vision and patter recognition, pp. 3751-3759, 2017.
Vijay Badrinarayanan, Alex Kendall, and Roberto Cipolla. Segnet: A deep convolutional encoder- decoder architecture for image segmentation. IEEE transactions on pattern analysis and machine intelligence, (9))2444881 1-295, 2017.
Mateusz Buda, Atsuto Maki, and Maciej A Mazurowski, A systematic study of the class imbalance problem in convolutional neural networks. Neural Networks, 106:249-2550 259, 2018.
Holger Caesar, Jasper Uijlings, and Vittorio Ferrari. Coco-stuff: Thing and stuff classes in context. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1209- 1218, 2018.
Jiarui Cai, Yizhou Wang, and Jenq-Neng Hwang. Ace: Ally complementary experts for solving long-tailes recogmition one-shot. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 112-121, 2021.
Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning imbalanced datasets with label-distribution- aware margin loss. Advances in neural information processing systems, 32, 2019a.
Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning imbalanced datasets with label-distribution-awart margin loss. Advances neural information processing systems, 32, 2019b.
Liang- Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder- decoder with atrous separable convolution for semantic image segmentation. In Proceedings of the European conference computer vision (ECCV), pp. 801-818, 2018.
Peng Chu, Xiao Bian, Shaopeng Liu, and Haibin Ling. Feature space augmentation for long-tailed data. In European Conference on Computer Vision, pp. 694-710. Springer, 2020.
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3213-3223, 2016.
Jiequan Cui, Yuhui Yuan, Zhisheng Zhong, Zhuotao Tian, Han Hu, Stephen Lin, and Jiaya Jia. Region rebalance for long-tain semantic segmentation. arXiv preprint arXiv:2204.01966 2022.
Henghui Ding, Xudong Jiang, Ai Qun Liu, Nadia Magnenat Thalmann, and Gang Wang. Boundary- aware feature propagation for scene segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 6819-68122 6622, 2019.
Jun Fu, Jing Liu, Haijie Tian, Yong Li, and Yongjun Bao. Zhiwei, fang, and hanqing lu. Dual attention network for scene, segmentation. In, CVPR, 2, 2019.
Agrim Gupta, Piotr Dollar, and Ross Girshick. Lvis: A dataset for large vocabulary instance segmen- tation. In Proceedings of the IEEE/CVF conference on computer vision and paltern recognition, pp. 5356-5364, 2019.
Haibo He and Edwardo A Garcia. Learning from imbalanced data. JEEE Transactions on knowledge and data engineering, 21(9): 2009.
Junjun He, Zhongying Deng, and Yu Qiao. Dynamic multi-scale filters for semantic segmentation. In Proceedings of the IEEF/CVE International Conference on Computer Vision, pp. 3562-3572,
2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
10

---

Under review as a conference paper at ICLR 2023
Chen Huang, Yining Li, Chen Change Loy, and Xiaoou Tang. Learning deep representation for imbalanced classification. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.
2016.
Zilong Huang, Xinggang Wang. Lichao Huang. Chang Huang, Yunchao Wei, and Wenyu Liu. Cenet: Criss-cross attention for semantic segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 603-612, 2019.
Bingyi Kang, Saining Xie, Marcus Rohrbach, Zhicheng Yan, Albert Gordo, Jiashi Feng, and Yannis Kalantidis. Decoupling representation and classifier for long-tailed recognition. arXiv preprint arXiv:1910.0922 2019.
Jachyung Kim, Jongheon Jeong, and Jinwoo Shin. M2m: Imbalanced classification via major-to- minor translation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13896-139999 2020.
Alexander Kirillov, Ross Girshick, Kaiming He, and Piotr Dollár. Panoptic feature pyramid net- works. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6399-64035 2019.
Tsung-Yi Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. Proceedings of the IEEE international conference on computer vision. pp. 2980-2988, 2017.
Ziwei Liu, Zhongqi Miao, Xiaohang Zhan, Jiayun Wang, Boqing Gong, and Stella Yu. Large- scale long-tailed recognition in open world. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 2537-2546, 2019.
Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1431-3440. 2015.
Jiaqi Wang, Wenwei Zhang, Yuhang Zang, Yuhang Cao, Jiangmiao Pang, Tao Gong, Kai Chen, Ziwei Liu, Chen Change Loy, and Dahua Lin. Seesaw loss for long-tailed instance segmentation. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp-
9695-9704 2021.
Jingdong Wang, Ke Sun, Tianheng Cheng, Borui Jiang, Chaorui Deng, Yang Zhao, Dong Liu, Yadong Mu, Mingkui Tan, Xinggang Wang, et al. Deep high-resolution representation learn- ing for visual recognition. IEEE transactions on pattern analysis and machine intelligence, 43 (10):3349-334) 2020a.
Xiaolong Wang, Ross B. Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. CoRR, abs/1711.07971, 2017. URL http:/ /ttp///axvvvv/vv///////e////.viv orgoo///////o.. 1111117117777
Xudong Wang, Long Lian, Zhongqi Miao, Ziwei Liu, and Stella X Yu. Long-tailed recognition by routing diverse distribution-awen experts. arXiv preprint arXiv:2010.01809 2020b.
Liuyu Xiang, Guiguang Ding, and Jungong Han. Learning from multiple experts: Self-paced knowl- edge distillation for long-tailed classification. In European Conference on Computer Vision, pp. 247-263. Springer, 2020.
Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo. Seg- former: Simple and efficient design for semantic segmentation with transformers. Advances in Newral Information Processing Systems, 34:12077-12099, 2021.
Fisher Yu and Vladlen Koltun. omas funkhouser, "dilated residual networks,". In Proceedings of the IEEE conference computer vision and patlern recognition, 2017.
Yuhui Yuan, Xiaokang Chen, Xilin Chen, and Jingdong Wang. Segmentation transformer: Object- contextual representations for semantic segmentation. arXiv preprint arXiv: /065, 2019.
11

---

Under review as a conference paper at ICLR 2023
Yongshun Zhang, Xiu-Shen Wei, Boyan Zhou, and Jianxin Wu. Bag of tricks for long-tale long-tailed visual recognition with deep convolutional neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 3447-3455, 2021.
Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. In Proceedings of the IEEE conference on computer vision and pultern recognition, pp. 2881-2890, 2017.
Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In Proceedings of the IEEE conference computer vision and paltern recognition, pp. 633-641, 2017.
Boyan Zhou, Quan Cui, Xiu-Shen Wei, and Zhao-Min Chen. Bbn: bilateral-brande network with cumulative learning for long-tailed visual recognition. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 9719-9728, 2020.
Yi Zhu, Karan Sapra, Fitsum A Reda, Kevin J Shih, Shawn Newsam, Andrew Tao, and Bryan Catan- zaro. Improving semantic segmentation via video propagation and label relaxation. Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1885-88665 2019.
12

---

Under review as a conference paper at ICLR 2023
A
APPENDIX
A.1 BENCHMARKS & IMPLEMENTATION DETAILS
Benchmarks: Cityscapes (Cordts al., 2016) is dataset that focuses semantic understanding of urban street scenes and contains 19 semantic classes. It contains 5K annotated images with pixel- level fine annotations and 20K coarsely annotated images. The finely annotated 5K images are split into sets with numbers 2975, 500 and 1525 for training, validation and testing.
ADE20K (Zhou et al., 2017) is challenging scene parsing dataset. It contains 150 categories and diverse scenes with 1038 image-level labels. The training and validation sets consist of 20K and 2K images, respectively.
Implementation Details. We initialize backbones with the weights pre-trained on ImageNet. For DeepLabv3+, we using CNN-based ResNet-50c and ResNet-10lc backbones, which switch the first 7x7 convolution layer to three 3 convolutions. HRNet- W448 (Wang et al., 2020a) is adopted for OCRNet. MIT-B3 a Transformer based backbone is adopted for SegFormer (Xie al., 2021), both of which are popular in semantic segmentation. For CNN-based network, we use SDG and 0.9
poly learning rate schedule (Zhao et al., 2017) with factor (1-rolafre) totaliter
The initial learning
rate is set as 0.01 and weight decay 0.0005. We adopt Adam with 6 x 10 learning rate and 0.01 weight decay. We set the image crop size to 512x 1024, batch size as 8 and training iterations as 80K on Cityscapes by default. For ADE20K, the crop size of images set as 512x512, the batch size is set 16 and training iterations are set as 80K if not stated otherwise. In the training phase, we augment data samples with the standard random scale the range of [0.5, 2.0], random horizontal flipping. random cropping, as well random color jittering. For inference, the input image size of ADE20K is the same as the size during training, but for Cityscapes the input image is sealed to x024x2048, no ticks(e.g.mullisschtt sullt- with flipping) will be adopted during testing. All experiments are implemented on the Nvidia A6000.
A.2 ADDITIONAL PROOF OF THEORETICAL ANALYSIS BETWEEN MIOU AND MACC
Before we proceed with the additional proof, we introduce some formulas for mAce and mloU,
mAcc- 1
1
TP:
Acc,
C
C
FN,
(11)
C
C
TP.
mloU =
LoU
=
0
TP. 17++FN FN, +FP
1=1
i=1
C
Σ FP; Σ FN,
(12)
1=1
i=1 TP: + FP;
num;
(13)
A.2.1 ADDITIONAL PROOF OF MACC IS THE BETTER
METRIC
According to the prior information from experimental statisties, the proportions of pixel instances of the head, body, and tail categories in the semantic segmentation are: 80%, 15% and 5%. We analyzed the items in the mloU and mAce formulas and found that the difference between them was the F P.. According to Eq. 12 We derive
C
Cn = Σ FN, = Σ FN + Σ
C FN, +
C. Σ
FP.
FP,
FN:
1=1
1=1
C₁ Acc;) x numm Σ (1-Ac)) num; + Σ (1 Acc;)) x nemmg
Co.
(14)
= =(1 mmAcs)) Σ num;;
numn; + (1 mmAcc))) num; mmccs) Σ
= (1 - m.Acs)) x 0.8NUM + (1 - 1-mAccs) x 0.15NUM M + +(1 (1--Acc)) x 0.05NUM
13

---

Under review as a conference paper at ICLR 2023
where, h, b and refer to head, body, tail categoris and NUM refers to the pixel instance number of overall datasets.
From Eq. 14, obtain the conclusion that due the large instance base of the head categories, FN, and FN-related FP; items are dominated by head categories. Therefore, the IoU; of each category and mloU will be dominated by head categories because of the item FP:.
Instead of mloU, for each category of mAcc, the items of Ace; are only related to its own category i, and mAce will not be dominated by the head categories. Thus mAcC is a fair and tail-sensitive metric.
A.2.2 ADDITIONAL. PROOF OF REMARK
To better understand the correlation between mean IoU and mean Ace in long-tailed semantic seg- mentation and Remark 1. According the equation Eq. 10 and the precondition from the experi- ments. The category Acc and IoU in our method become
LoU, = IoUn,
(15)
Acc; = (1 ACCC
And it clear from Eq. We obtain the result:
Acc; (1+1)ACC.
TP.
TP,
TP.
Acc:
= +(++))
TP, FN. (1 ()+D)ACC, (16)
TP; PN;
TP; + FN;
TP, TPP + FN, + FP, TP. FN, + FP, FN,+ +N.
TP:
TP,
IoU, = IOU, = IOU,
= IoU;
= (1+ p)TP; x (TP; FN, =P:::=PPP (TP, FN,+
+ (1 +DTP, x (num, + -PP.)=PPP x (num; + FFP)
AFP, = p Noum++PPP
(17)
TP: TP: FN,
TP: & IOU, TP: FN, FP: = PP,
Acc: ToU,
Acc;
nurn; numn
(18)
where AFP, is the increased false positive from baseline our method.
To guarantee the classifier effective and segment more tail categories, should satisfied:
APP: = x (num; + FPt) < num;
(19)
We observe that the results in Cityscapes benchmark has achieved a high value, Log./oU; 0.8 and Acc; 0.85. According to this precondition and Eq. 18, FP. = 0.0625 xn m, and p are both small value, which means F,P; P; is minimum value and satisfied Eq. 19. This is just the reason that Acc improved while IoU not decrease is significant for tail categories segmentation.
A.3 ADDITIONAL PROOF OF THE CORRELATION BETWEEN PERFORMANCE AND DATA DISTRIBUTION
As shown of Figure 6, the baseline method on Cityscapes and ADE20K perform not well on certain categories, and clearly see that these categories mainly fall in the tail and body data subsets. This further demonstrates that the baseline method by constraining the accuracy of certain categories.
data distribution limits the overall performance of the
14

---

Under review as a conference paper at ICLR 2023
Rocs Nxr Bookine
Data Driabiicuutee
Sooeines
08
03
Gdagery thoud
Calagery Proud
(a) Data distribution and performance with bascline (b) Data distribution and performance with bascline ADE20K.
on Cityscapes.
Figure 6: The existence of long-tailed distribution in semantic segmentation, set baseline as DeepLabv34 and ResNet-50c by default. (a) and (b) shown the data distributions on Cityscapes and ADE20K long-tailed, which cause the better performance the majority categories yet suppress the minority categories. It should be noted that we reordered the categories in Cityscapes according to Pixel Frequency.
A.4 ABLATION WITH LONG-TAILED METHODS
Table 7: Pearson coefficients between categories frequency and accuracy, the lower value means much weaker correlation.
Dataset PX,Y
CIFAR-I00
75.9
ADE20K-pixel 36.8
Cityscapes-pixer 58.9
In this section, we first calculate the Pearson correlation coefficient to show the correlation be- tween category accuracy and category frequency (image category frequency on CIFAR-100) in Ta- ble 7. The weak correlation between category accuracy and pixel level frequency on Cityscapes and ADE20K causes challenge to re-weighting in semantic segmentation, which can demonstrated in Table 8. We compared our work with traditional long-tailed classification methods to further explore the contribution of our work. We adopt the re-weighting method to modify the loss function and put larger weights tail categories. Despite improving mAcc, the re-weighting method caused lot of decrease in mloU. For re-sampling, the contextual information is corrupted resulting in segmen- tation metrics very low level. In general, the current Pixel level re-balance approaches can not work well with the long tail distribution on semantic segmentation.
Table 8: Ablation with method and pixel level re-weighting and ra-sampling, all networks adopt ResNet-50c backbone.
Benchmarks
Methods
mloU
mAcc
Baseline
80.37
86.68
Beweighting-Focalll FocalLoss (Lin et al., 2017)
76.23(-4.14)
85.00(-1.68)
Cityscapes
Reweighting-LDAMLL (Cao et al., 2019a)
77.52(-2.88
85.15(-1.43)
SeweeightingSSesaal.... (Wang et al., 2021)
67.58(-12.79 -12.79)
(..553-12.3)
Re-sampling
66.79( 66.79(-13.55 13358)
75.21 -1147)
Basclinc+MED
80.20 -0.17)
90.04
+3.36)
Bascline
42.11
54.13
Reweighting-Focal.. (Lin et al., 2017)
37.61(-4.50)
55.29 (1.116)
ADE20K
Reweighting -DAMMLsss (Cao et al., 2019 2019a)
40.39 -1.72)
48.78 -5.35)
Branghhting-Scasawn.n (Wang et al., 2021)
33.87 -8.24)
40.19( 13.994
Resampling
Baseline+MED
43.82 +1.71)
60.02 (+5.8)
15

---

Under review as a conference paper at ICLR 2023
(a) Image
(b) Groud Truth
(c) DecpLabv3+
(d) Ours
Figure 7: Qualitative Visualization results on the validation set Cityscapes with ResNet-50c as backbone. All the models here are trained under the same setting.
A.5 ANALYSTS OF THE PERFORMANCE GAP BETWEEN CITYSCAPES AND ADE20K
According to the experiments results of Sec4, our method has achieved impressive performance in both mIoU and mAcc on ADE20K dataset. However, there seems to gap between the perfor- mance on Cityscapes and ADE20K. We analyze the causes: 1) Compared to ADE20K, there are fewer categories in Cityscapes and more pronounced long-tail distribution (higher proportion of head category instances), so it is easier to fall into local optimum when training body and tail experts, resulting in the overfitting of these categories. Finally, it caused that the increase FNR and FPnr on the overall datasets, FN;; and FPsj denote to the pixel instance which the ground truth belongs to category but the prediction is j. 2) The performance level. According the above two reasons, method improves the overall TP on Cityscapes, but will increase FN of the head categories and FP ht of the tail categories, respectively, resulting in mloU being relatively stable value.
baseline methods on Cityscapes at a high
From the macro level of image visualization, as shown Figure 7, our method segments the sur- rounding part of the head categories into tail categories while segmenting the tail categories. We believe that such head accuracy decreases are acceptable and meaningful in real-world scenarios.
A.6 QUANTITATIVE VISUALIZATION COMPARISONS CITYSCAPES AND ADE20K
In this section, we demonstrate the better performance of MEDOE framework with quantitative vi- sualization on Cityscapes and ADE20K shown in Figure 7 and 8. We adopt ResNet-50C -0cc as backbone and all models trained under the same setting. In most semantic scenarios, our MEDOE method can achieve better performance segmenting tail categories.
A.7 ADDITIONAL EXPLANATIONS
A.7.1 CONTEXTUAL MODULE
Contextual module, important module in semantic segmentation, refers to the extraction and aggregation of contextual information for pixels through series of operations (i.e., feature pyramids, atrous convolution, large-scale convolutional, attention mechanisms, and global pooling). Existing contextual modules include: ASPP (Chen et al., 2018), PSP (Zhao et al., 2017), or non- local (Wang et al., 2017).
Contextual information means the relationship between this pixel and the surrounding pixels and global information is regarded contextual information. The reasons why segmentation needs contextual module. When solving semantic segmentation tasks if each pixel considers only its deep features, such texture and color, will be difficult to classify into the correct category (i.e.the
16

---

Under review as a conference paper at ICLR 2023
Image
(b) Groud Truth
(c) DeepLabv3+
(d) Ours
Figure 8: Qualitative Visualization the validation set of ADE20K with ResNet-50c as backbone. All the models here are trained under the same setting.
deep features of leaves and green grass will be very similar). At the same time, as stated in the work, semantic segmentation believes that each pixel not I.I.D., but is related to the surrounding pixels. The correlation information can better help the segmentation task (f.e. the leaves are surrounded by branches, and the grass is likely to be surrounded by roads).
A.8 ABLATION OF DIFFERENT EXPERTS
To measure the goodness and distinctiveness, We adopt the mAcc and per-category bias in Table 9 as the metrics to describe good and distinctive from each other, rather than simply boosting confidence through multi-expert training.
each expert. seems to demonstrate that the K experts learned at Stage I are
Furthermore, according the well-known bias- decomposition (Wang al., 2020b), per- category bias denotes:
Error(æ; E [(h(::; D) y) Bias(a; + Variance(z;; h) irreducible error (æ), (20)
17

---

Under review as a conference paper at ICLR 2023
Table 9: Ablation of each expert with the mean accuracy and per-category bias on Cityscapes with
Deeplabv3+ and ResNet-50c. The higher mean accuracy and lower bias are better.
Many
Medium
Few
mAcc
bias
mAcc
bias
mAce
bias
Overall
0.96
0.10
0.88
0.22
0.81
0.30
Expert
0.97
0.09
0.86
0.24
0.75
0.36
Expert
0.95
0.09
0.86
0.19
Expert 3
0.91
0.13
A.9 LONG-TAILED SEMANTIC SEGMENTATION
To the best of our knowledge, there is a contemporaneous (Cui al., 2022) work with paper. Although, we almost simultaneously focus on the long-tailed distribution as an important reason for constraining semantic segmentation performance, there are some differences between our work and Region rebalance (Cui et al., 2022):
Region rebalance was concerned about solving the problem of category re- balance, while our work was more focused on improving the recognition of tail and body categories and placed special emphasis on the significance of segmenting the body, and tail categories.
Perspective difference:
Methods difference:
Region rebalance relieved the categories imbalance with an auxiliary region classification branch by adjusting segmentation boundaries however motivated by the ensembling and grouping methods, we proposed MEDOR framework to encourage different experts to learn more balanced distribution in the feature space, and finally adjust classification boundaries. Com- pared to Region Rebalance, the motivation of our work was completely different and provided dif- ferent research directions.
Interpretations of mloU and mAce difference: Region rebalance only explain the cause of mloU with previous long-tailed methods. Different from Region rebalance, we take a further step and explore the significance of mAce in segmenting the body and tail categories.
Datasets difference: RR used COCO164 (164 categories) and ADE20K (150 categories) datasets, both of which are massive categories datasets. We used Cityscapes (19 categories) and ADE20K to verify both a small number of categories and massive categories scenarios. The decline in mloU may be partly due the impact of the dataset, we also demonstrated the gap between Cityscapes and ADE20K in Appendix A.5. Our method demonstrated excellent results on a large number of categories of datasets such as ADE20K.
A.10 SIGNIFICANCE OF BODY AND TAIL CATEGORIES IN REAL- WORLD SCENARIOS
We believe in a large number of real-world scenarios, it is more important to be able to identify body and tail categories than accurately segment the edge pixels of head categories. (i.e.In the automatic driving scenario, we need to segment some tail categories objects that appear on the driving path, such as "poles" or "fire hydrants", to avoid traffic accidents. Segmenting small lesions on medical images can help doctors detect underlying diseases). Generally, the benefits of this segmentation are far greater than the decrease of certain head categories edges.
A.11 COMPARISON WITH TRADITIONAL MULTI-EXPERTS METHODS
We compared the differences from existing multi-expert methods, there is three main difference:
Model architecture: The pipeline of traditional multi-expert methods contain a backbone and multi-classifiers to adjust classifier boundaries and finally ensemble the outputs. 1)We pioneered the combination of contextual modules and classifiers to become experts and learn more balanced distribution in the feature space, and finally adjust classification boundaries. 2yThen we provided each expert with soft weight based on the final contextual information and classification results through a learning mechanism ensemble the outputs.
Training strategies: Traditional methods often focus on constraining dominating categorics and ignoring the confusing categories. Differing from them, proposed the expert-specific pixel-
18

---

Under review as a conference paper at ICLR 2023
masking strategy and diverse data distribution loss function to ensure model architecture focuses on the confusing categories and has better performance.
Training step: Advanced multi-expert methods in long-tailed classification, such BBN (Zhou ct al., 2020), RIDE (Wang et al., 2020b), LFME (Xiang et al., 2020). They all take multiple steps to train, mainly including 1. training backbone, 2. training classifiers, and 3. Distillation learning (optional). However, our method can update the backbone parameters while training the head expert, so it is one-step training.
19


Reviews:
[{"review_id": "hllL6Aj3wt", "reviewer": "Reviewer_FVTB", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper considers the semantic segmentation problem. The motivation of this paper is based on the relatively poor performance for the tailed classes and in order to overcome this issue, the author propose to use multiple experts to process pixels with different category frequency. In the experimental section, the paper illustrates some improvements especially for the rare classes on Cityscapes and ADE20k datasets.", "summary_of_the_review": "Overall speaking, the paper is interesting, but it has a big space to be further improved. I would like to see the response of the author to my questions.", "strength_and_weaknesses": "Strength\nThe paper is well-written, and the experiments are well built to cover two most popular datasets for semantic segmentation. The ablation is comprehensive to cover the main contribution of this paper\n\nWeakness\n\n1. What is the contextual module? It needs to be illustrated with more details.\n\n2. The auxiliary loss is unclear to me. What is the reality distribution 'q'. I do not quite understand this comment 'It is inevitable that we need to classify these pixels, even if they fall into majority categories'. From my understanding, for experts that process body or tail categories, these majority categories can be ignored without causing any gradient. In equation 4, the auxiliary loss tends to minimize the logits for these classes, which needs more explanation. In summary, the role played by this loss is quite confusing.\n\n3. I have a big concern about the effectiveness of the proposed method. In table 2, the performance of the proposed method is even worse than the baseline. Despite the oracle results are promising, it is not applicable since we cannot priorly know which expert should process which pixel. For table 4, I can see some improvement for the rare classes on cityscapes, but the improvement is also quite minor. I admit the improvement on ADE20k is interesting, but only obtaining good results on one dataset is not enough to validate the effectiveness of the proposed method. \n\n4. The paper divides the categories into three sets. But during training, the experts process some overlapped categories. Why does the author device such an architecture instead of letting each expert work on a separate class set.\n\n5. The author only compare their method to the straight baseline. It is okay but it would be better if it can comprehensively compare the method to the SOTA long-tail techniques used in semantic segmentation.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "cSytYlh-ai", "reviewer": "Reviewer_9fjo", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper focuses on the topic of long tail in semantic segmentation. It designs a model-agnostic multi-expert decoder and output framework, making certain improvements for some classical segmentation models. A diverse data distribution-aware loss function is proposed for preventing over-confidence of minority categories. Besides, it advocates mAcc as a more important metric to evaluate the performance for body and tail categories in long-tailed semantic segmentation. The paper demonstrates some analyses to prove such opinion.", "summary_of_the_review": "This paper presents the long-tail problem in semantic segmentation and claims that they are the first to explicitly focus on the long-tailed semantic segmentation. Yet, there are already some published papers focusing on such topic (eg, Region Rebalance for Long-Tailed Semantic Segmentation, cvpr2022). Besides, the main idea of this paper is widely adapted by multi-expert long-tailed methods. The paper also advocates that the mAcc metric is more important for long-tail segmentation, and gives degenerated mIoU results. Yet, the paper, Region Rebalance for Long-Tailed Semantic Segmentation, also encounters such a problem and well address this problem.", "strength_and_weaknesses": "Strength；\n1.\tThis paper makes a survey on the topic of long-tail problem in semantic segmentation.\n2.\tIt proposes a model-agnostic multi-expert decoder to resolve such a problem and makes certain improvements for some classical segmentation models.\n3.\tA diverse data distribution-aware loss function is proposed for preventing over-confidence of minority categories.\n4.\tThe paper conducts some analyses to give the ideal results for the proposed method, showing a promising direction of future research.\n\nWeakness;\n1.\tThis paper claims that it is the first to explicitly focus on the long-tailed semantic segmentation. Yet, There are already some published papers which concentrate on such topic (eg, Region Rebalance for Long-Tailed Semantic Segmentation ,cvpr2022)\n2.\tThe paper duplicate the decoder head twice, bringing a large quantity of extra params. I am not pretty sure that whether the improvements are resulted from such a change. \n3.  The overall pipeline does not differ much from some multi-expert methods in long-tail classification/detection. I would recommend to compare with some of them, as they also conduct segmentation experiments such as \"Distribution Alignment: A Unified Framework for Long-tail Visual Recognition\"\n4. The paper propose that the mAcc metric is a more important metric for long-tail segmentation, giving degenerated mIoU results and better mAcc results in some experiments. Yet, this problem is also mentioned by paper \"Region Rebalance for Long-Tailed Semantic Segmentation in cvpr2022\" and has been well addressed.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "V-RyeXNqyv", "reviewer": "Reviewer_Yf4C", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper is about semantic image segmentation. The authors notices that there are some categories have less pixels than other categories, and propose to divide all the categories into different groups according to number of pixels and proposes to use mixture models to different group of categories. Experiments on Cityscapes and ADE20k prove the effectiveness.", "summary_of_the_review": "This paper proposes to use a two-stage model for semantic image segmentation. The idea is straight-forward and the main question is experiment, which is not good enough.", "strength_and_weaknesses": "The paper introduces a hierarchical structure, improves the quality of long-tailed distribution in semantic\nsegmentation. The experiments show the competitive performance of the proposed method. The paper provides detailed information for reproduction, opens a new direction for future research.\n\nThere are some questions. \n\n1.Equation (4) looks strange.\n\n2.In Table 2, the results on Cityscapes is consistently lower than baseline accross different method, I think something should be done to improve the experimental results", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "0Qe3YIv8xcj", "reviewer": "Reviewer_Vw5N", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper tackles the problem of improving the accuracy of semantic segmentation on long-tail classes. To this end, it introduces separate heads to the network output that focus only on specific segments of the class distribution, which are grouped heuristically into “head”, “body” and “tail” categories, based on their frequency in the dataset. A few decoder layers are trained to aggregate the predictions from these heads into the final model prediction. This approach improves the mean pixel accuracy across classes, but mIoU stays largely the same.", "summary_of_the_review": "The empirical results do not show a significant improvement in terms of mIoU, which accounts for false positives and false negatives. The argument that mAcc is a more important measure is not convincing, since one would need to consider a very specific scenario of a downstream task for semantic segmentation. Since we commonly study semantic segmentation in an application-agnostic way, mIoU remains an important metric even if the focus is on the long-tail categories.", "strength_and_weaknesses": "Strengths:\n- I concur with the authors that the long-tail problem in recognition deserves more research.\n- The proposed approach is intuitive in its intention and improves the mean accuracy.\n- Experiments with multiple model architectures and datasets.\n- The reference numbers for the oracle scenario are interesting.\n\nWeaknesses:\n\nI understand that the approach does not pan out in terms of the IoU, hence the need to push mAcc as the more important metric. The work provides some formal argument, which I do not find convincing, however. The interpretation of the results — mAcc improves, mIoU does not — has a simple explanation to me: the method reduces the number of false negatives at the cost of increasing the number of false positives. I do not consider this is an overall improvement.\n\nSome technical steps are not explained to a sufficient extent:\n- The discussion leading up to Eq. 4 is unclear. What norm is imposed on z and why this need? What is the “reality” data distribution?\n\nThe notation has many loose ends, which impedes understanding of the actual implementation:\n- In Eq. 4 the sum scope is unclear;\n- The second line above Eq. 6: the condition for y membership is unclear, because the sets S are disjoint, as defined above.\n\nSome of the claims are not backed up:\n- Second line after Eq. 5: “K experts are good and distinctive from each other” — how are the goodness and distinctiveness measured?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject