Title: AUCSeg: AUC-oriented Pixel-level Long-tail Semantic Segmentation
Venue: NeurIPS 2024
Accepted: yes
Keywords: Semantic Segmentation
Abstract: The Area Under the ROC Curve (AUC) is a well-known metric for evaluating instance-level long-tail learning problems. In the past two decades, many AUC optimization methods have been proposed to improve model performance under long-tail distributions. In this paper, we explore AUC optimization methods in the context of pixel-level long-tail semantic segmentation, a much more complicated scenario. This task introduces two major challenges for AUC optimization techniques. On one hand, AUC optimization in a pixel-level task involves complex coupling across loss terms, with structured inner-image and pairwise inter-image dependencies, complicating theoretical analysis. On the other hand, we find that mini-batch estimation of AUC loss in this case requires a larger batch size, resulting in an unaffordable space complexity. To address these issues, we develop a pixel-level AUC loss function and conduct a dependency-graph-based theoretical analysis of the algorithm's generalization ability. Additionally, we design a Tail-Classes Memory Bank (T-Memory Bank) to manage the significant memory demand. Finally, comprehensive experiments across various benchmarks confirm the effectiveness of our proposed AUCSeg method. The code is available at https://github.com/boyuh/AUCSeg.

1

Abstract

The Area Under the ROC Curve (AUC) is a well-known metric for evaluating instance-level long-tail learning problems. In the past two decades, many AUC optimization methods have been proposed to improve model performance under long-tail distributions. In this paper, we explore AUC optimization methods in the context of pixel-level long-tail semantic segmentation, a much more complicated scenario. This task introduces two major challenges for AUC optimization techniques. On one hand, AUC optimization in a pixel-level task involves complex coupling across loss terms, with structured inner-image and pairwise inter-image dependencies, complicating theoretical analysis. On the other hand, we find that mini-batch estimation of AUC loss in this case requires a larger batch size, resulting in an unaffordable space complexity. To address these issues, we develop a pixel-level AUC loss function and conduct a dependency-graph-based theoretical analysis of the algorithm's generalization ability. Additionally, we design a Tail-Classes Memory Bank (T-Memory Bank) to manage the significant memory demand. Finally, comprehensive experiments across various benchmarks confirm the effectiveness of our proposed AUCSeg method. The code is available at https://github.com/boyuh/AUCSeg.

1 Introduction

Semantic segmentation aims to categorize each pixel within an image into a specific class, which is a fundamental task in image processing and computer vision [33, 51, 75]. Over the past decades, substantial efforts [58, 23, 43, 96] have advanced the field of semantic segmentation. The mainstream paradigm is to develop innovative network architectures that encode more discriminative features for dense pixel-level classifications. Typical backbones include CNN-based [58, 14, 80] and newly emerging Transformer-based methods [108, 83, 65, 15, 32], which have achieved the state-of-the-art (SOTA) performance. Beyond this direction, researchers [24, 66, 48, 41] have recently realized the Pixel-level Long-tail issue in Semantic Segmentation (PLSS) , as shown at the top of Figure 1. Similar to the flaws of traditional long-tail problems, the major classes will dominate the model learning process, causing the model to overlook the segmentation of minority classes in an image. Several remedies have been proposed to alleviate this [49, 61, 10, 56, 104, 92, 78]. For example, [77] introduces a category-wise variation technique inversely proportional to distribution to achieve balanced segmentation; [66] introduces a sequence-based generative adversarial network for imbalanced medical image segmentation, and [41] develops a re-weighting scheme for semi-supervised segmentation.

Currently, mainstream studies fall into two camps. One is to develop carefully designed backbones for long-tail distributions but leave the effect of loss functions unconsidered. The other is to conduct empirical studies on the loss functions without exploring their theoretical impact on the generalization performance. A question then arises naturally:

Can we find a theoretically grounded loss function for PLSS on top of SOTA backbones?

This paper provides an affirmative answer from the AUC perspective and proposes a novel framework called AUC-oriented Pixellevel Long-tail Semantic Segmentation (AUCSeg) . Specifically, AUC indicates the likelihood that a positive sample scores higher than a negative one, which has been proven to be insensitive to data distribution [86, 101]. Applying AUC to instance-level long-tail classifications has shown promising progress in the machine learning community [86, 100, 88, 68]. Motivated by its success, this paper starts an early trial to study AUC optimization for PLSS. The primary concern is to study its effectiveness for PLSS from a theoretical perspective. The key challenge is that the standard techniques for generalization analysis [62, 8, 21] require the loss function to be expressed as a sum of independent terms. Unfortunately, the proposed loss function does not satisfy this assumption due to the dual effect of structured inner-image dependency and pairwise inter-image dependency. This complicated structure poses a big challenge to understanding its generalization behavior. To address this, we decompose the loss function into inner-image and inter-image terms. On top of this reformulation, we deploy the dependency

10

log10(Pixel Number)

HeadClassesMiddleClasses

9

8

9

100

mloU (%)

80

Performance Gap

(Head classes)

60

STDC

HRNet

40

DeepLabV3+

AUCSeg (Ours)

Performance Gap

(Middle classes)

PerformanceGap

(Tail classes)

Ba

jilding

Vege

Caf

SKY

Label

pole

Wall

Traffic

Sign

Bicycle

Bus

Train

cLight

Motorcycle

graph [103] to decouple the interdependency. Finally, we reach a bound of ˜ O ( τ √ log ( τNk ) /N ) , where τ behaves like an indicator for imbalance degree, and k denotes the number of pixels in each image. This suggests optimizing AUC loss could ensure a promising performance under PLSS.

Back to the practical AUC learning process, we realize that the stochastic gradient optimization (SGD) for structured pixel-level tasks imposes a greater computational burden compared to instancelevel long-tail problems. Specifically, the SGD algorithm of AUC requires at least one sample from each class in each mini-batch [99, 85]. In light of this, the primary choice is to adopt the so-called stratified sampling on all images [70, 60, 86] for mini-batch generation (See Equation (7)). Unfortunately, as shown in Figure 3(a) and (b), this is hard to implement under PLSS because pixel-level labels are densely coupled in each image . Meanwhile, as shown in Proposition 1, we also argue that directly using random sampling to include all classes would require an extremely large batch size. This leads to unaffordable GPU memory demands for optimization, as described in the experiments in Appendix G.6.

To alleviate this, a novel Tail-class Memory Bank (T-Memory Bank) is carefully designed. The main idea is to identify those missing pixel-level classes in each randomly generated mini-batch and then complete these absences using stored historical class information from the T-Memory Bank. This enables efficient optimization of AUCSeg with a light memory usage, enhancing the scalability of our proposed method, as shown in Figure 1. Finally, comprehensive empirical studies consistently speak to the efficacy of our proposed AUCSeg.

Our main contributions are summarized as follows:

This paper starts the first attempt to explore the potential of AUC optimization in pixel-level long-tail problems.

We theoretically demonstrate the generalization performance of AUCSeg in semantic segmentation. To our knowledge, this area remains underexplored in the machine-learning community.

We introduce a Tail-class Memory Bank to reduce the optimization burden for pixel-level AUC learning.

2 Related Work

2.1 Semantic Segmentation

Semantic segmentation is a subtask of computer vision, which has seen significant development since the inception of FCN [58]. The most common framework for semantic segmentation networks is the encoder-decoder. For the encoder, researchers typically use general models such as ResNet [37] and ResNeXt [84]. As the segmentation tasks become more challenging, some specialized networks have emerged, such as HRNet [75], ICNet [105], and multimodal networks [76, 42]. For the decoder, a series of studies focus on strengthening edge features [23, 107], capturing global context [43, 31, 45], and enhancing the receptive field [96, 67, 12, 13]. Recently, the transformer has shown immense potential, surpassing previous methods. A series of methods related to Vision Transformer [108, 83, 65, 15, 74] are proposed. SegNeXt [32], which is the current state-of-the-art (SOTA) method, possesses the same powerful feature extraction capabilities as the Vision Transformer and the same low computational requirements as CNN. Apart from improving the network, some research [48, 56, 24, 11, 10, 66, 39] is directed toward addressing the issue of class imbalance in semantic segmentation. However, the effectiveness of these methods is not significant. In this paper, we aim to improve the performance of long-tailed semantic segmentation from an AUC optimization perspective.

2.2 AUC Optimization

The development of AUC Optimization can be divided into two periods: the machine learning era and the deep learning era. As a pioneering study, [20] ushers in the era of AUC in machine learning. It studies the necessity of AUC research, which points out that AUC maximization and error rate minimization are inconsistent. After that, AUC gains significant attention in linear fields such as Logistic Regression [38] and SVM [46, 47]. Then researchers begin to explore the online [106, 28] and stochastic [94, 63] optimization extensions of the AUC maximization problem. Research from the perspectives of generalization analysis [2, 73, 17] and consistency analysis [1, 29] provides theoretical support for AUC optimization algorithms. [57] is the first to extend AUC optimization to deep neural networks, ushering in the era of AUC in deep learning. Meanwhile, a series of AUC variants [87, 86, 68, 88, 69, 90] emerge, gradually enriching AUC optimization algorithms. Furthermore, in practice, AUC optimization demonstrates its effectiveness in various class-imbalanced tasks, such as recommendation systems [5, 6, 4, 7], disease prediction [79, 30], domain adaptation [89], and adversarial training [40, 91].

Despite significant progress, existing studies of AUC optimization mainly pay attention to the instancelevel imbalanced classification tasks. This paper starts an early trial to introduce AUC optimization to semantic segmentation. However, due to the high complexity of pixel-level multi-class AUC optimization, such a goal cannot be attained by simply using the current techniques in the AUC community.

3 Preliminaries

In this section, we briefly introduce the semantic segmentation task and the AUC optimization problem.

3.1 Semantic Segmentation Training Framework

Let D = { ( X i , Y i ) n i =1 | X i ∈ R H × W × 3 , Y i ∈ R H × W × K } be the training dataset, where H and W represent the height and width of the images, and K denotes the total number of classes. Let f θ be a semantic segmentation model ( θ is the model parameters), which commonly follows an encoder-decoder backbone [75, 105, 108, 83, 32]. Let ˆ Y i = f θ ( X i ) ∈ R H × W × K be the dense

pixel-level prediction, i.e. ,

where the encoder f e θ extracts features from the image X i , and then the decoder f d θ predicts each pixel based on extracted features and outputs a dense segmentation map with the same size as Y i .

Furthermore, let Y i u,v and ˆ Y i u,v represent the ground truth and prediction of the ( u, v ) -th pixel of the i -th image, respectively. To train the model f θ , most current studies [14, 80, 83, 65] usually adopt the cross-entropy (CE) loss:

where Y ic u,v and ˆ Y ic u,v are the one-hot encoding of ground truth and the prediction of pixel ( X i u,v , Y i u,v ) in class c , respectively.

3.2 AUC Optimization

Area under the Receiver Operating Characteristic Curve (AUC) is a well-known ranking performance metric for binary classification task, which measures the probability that a positive instance has a higher score than a negative one [35]:

where ( X + , y + ) and ( X -, y -) represent positive and negative samples, respectively. When AUC → 1 , it indicates that the classifier can perfectly separate positive and negative samples.

According to [87, 86, 88], given finite datasets, maximizing AUC ( h θ ) is usually realized by maximizing its unbiased empirical estimation:

where ℓ is a differentiable surrogate loss [86] measuring the ranking error between two samples, n + and n -denote the number of positive and negative samples, respectively.

Moreover, we can directly optimize the following problem for AUC maximization:

Note that, AUC has achieved significant progress in long-tailed classification [100, 88, 68]. Due to the limitations of space, we refer interested readers to the literature [86, 99] for more introductions to AUC. However, most existing studies merely focus on the instance-level or image-level problems. Inspired by its distribution-insensitive property [26], this paper starts an early trial to introduce AUC to PLSS.

4 AUC-Oriented Semantic Segmentation

In this section, we introduce our proposed AUCSeg method for semantic segmentation. A brief overview is provided in Figure 2. AUCSeg is a generic optimization method that can be directly applied to any SOTA backbone for semantic segmentation. Specifically, AUCSeg includes two crucial components: (1) AUC optimization where a theoretically grounded loss function is explored for PLSS and (2) Tail-class Memory Bank , an effective augmentation scheme to ensure efficient optimization of the proposed AUC loss. In what follows, we will go into more detail about them. For clarity, we include a table of symbol definitions in Appendix A.

一

一

Store

Ground Truth

Image

road

vegetation

rider

Sample

Tail

Branch

T-Memory Bank

Input

Memory

AugmentedBatch

traffic light bus

rider

bicycle

Branch

Tail Classes

Image*

Sample

Output*

Resize

Branch

Ratio

Encoder

AUCoro

AUC Optimization

Decoder

0+---0

Prediction

>fθ

+入

Maximize AUC

+++

Prediction

fauc

[D(f)-ED[D(f)]≤0(T√log(TNk)/N

Theoretical Result:

4.1 Pixel-level AUC Optimization

Semantic segmentation is a multi-class classification task. Therefore, to apply AUC, we follow a popular multi-class AUC manner, i.e. , the One vs. One (ovo) strategy [64, 34, 86], which is an average of binary AUC score introduced in Section 3.2. Specifically, on top of the notation of Section 3.1, we further denote D p = { ( X i u,v , Y i u,v ) | i ∈ [1 , n ] , u ∈ [0 , H -1] , v ∈ [0 , W -1] } as the set of all pixels; the j -th element ( j ∈ [1 , n × ( H -1) × ( W -1)] ) in D p is abbreviated as ( X p j , Y p j ) for convenience. Given the model prediction f θ = ( f (1) θ , . . . , f ( K ) θ ) , ∀ c ∈ [ K ] , f ( c ) θ ∈ [0 , 1] , where f ( c ) θ serves as a continuous score function supporting class c , AUC ovo seg calculates the average of binary AUC scores for every class pair:

̸

To this end, as introduced in Section 3.2, the goal is to minimize the following unbiased empirical risk:

̸

where we adopt the widely used square loss ℓ sq ( x ) = (1 -x ) 2 as the surrogate loss [29]; ℓ c,c ′ ,m,n sq := ℓ sq ( f ( c ) θ ( X p m ) -f ( c ) θ ( X p n )) ; N c = { X p k | Y p k = c } represents the set of pixels with label c in the set D p , and |N c | denotes the size of the set.

4.2 Generalization Bound

In this section, we explore the theoretical guarantees of the AUC loss function in semantic segmentation tasks and demonstrate that AUCSeg can generalize well to unseen data.

A key challenge is that standard techniques for generalization analysis [62, 8, 21] require the loss function to be expressed as a sum of independent terms. Unfortunately, the proposed loss function does

not satisfy this assumption because there are two layers of interdependency among the loss terms . On one hand, semantic segmentation can be considered a structured prediction problem [16], where couplings between output substructures within a given image create the first layer of interdependency. On the other hand, the AUC loss creates a pairwise coupling between positive and negative pixels, so any pixel pairs sharing the same positive/negative instance are interdependent, resulting in the second layer of interdependency.

We present our main result in the following theorem and the proof is deferred to Appendix B.

Theorem 1 (Generalization Bound for AUCSeg) . Let E D [ ˆ L D ( f ) ] be the population risk of ˆ L D ( f ) . Assume F ⊆ { f : X → R H × W × K } , where H and W represent the height and width of the image, and K represents the number of categories, ˆ L ( i ) is the risk over i -th sample, and is µ -Lipschitz with respect to the l ∞ norm, (i.e. ∥ ˆ L ( x ) -ˆ L ( y ) ∥ ∞ ≤ µ · ∥ x -y ∥ ∞ ). There exists three constants A > 0 , B > 0 and C > 0 , the following generalization bound holds with probability at least 1 -δ over a random draw of i.i.d training data (at the image-level):

where

n ( c ) max = max X n ( X ( c ) ) , n ( c ) mean = ∑ N i =1 n ( X ( c ) i ) , N = |D| , k = H × W and X ( c ) represents the pixel of class c in image X .

Remark 1. We achieve a bound of ˜ O ( τ √ log ( τNk ) /N ) , indicating reliable generalization with a large training set. Here, τ represents the degree of pixel-level imbalance. More interestingly, even though we have k classifiers for every single image, the generalization bound only has an algorithm dependent on k , suggesting that pixel-level prediction doesn't hurt generalization too much.

4.3 Tail-class Memory Bank

Motivation. Although we have examined the effectiveness of AUC for PLSS from the theoretical point of view, there is a practical challenge when conducting AUC optimization for semantic segmentation, as discussed in Section 1. Specifically, the stochastic AUC optimization, as defined in Equation (7), requires at least one sample from each class in a mini-batch. In instance-level AUC optimization , recent studies [87, 86] often use a stratified sampling technique that generates batches consistent with the original class distribution, as shown in Figure 3(a). Such a strategy will work well when each image belongs to a unique category (say, 'Banana', 'Apple', or 'Lemon') in traditional classifications. Yet it cannot apply to pixel-level cases because each sample involves multiple and coupled labels, making it hard to split them for stratified sampling, as illustrated in Figure 3(b). Meanwhile, we also provide a bound (Proposition 1) to show that simply adopting random sampling will suffer from an overlarge batch size B , making an unaffordable GPU memory burden.

Proposition 1. Consider a dataset D that includes images with K different pixel categories. Let p i represent the probability of observing a pixel with label i in a given image. Randomly select B images from D as training data, where

Then with probability at least 1 -δ , for any c ∈ [ K ] , there exists X in the training data that contains pixels of label c .

Remark 2. The proof is deferred to Appendix C. Proposition 1 suggests that the value of B is inversely proportional to min i p i . Note that p i will be smaller as the long-tail degree becomes more

Instance-Level (Classification)

Pixel-Level (Semantic Segmentation)

Head Class

Tail Class

Tail Class

Dataset

Dataset

Tail-class

Memory

Bank

Apple

Lemon

Lemon

Stratified

Stratified

Random

Sampling

Sampling

Sampling

Training

Batch

1

31

Training

Batch

Augmented

Batch

MissingPixels

Indivisible!

in Tail Class

Random

Pixels

Coverage

(a)

(b)

?

severe, leading to a larger B . For example, in terms of the Cityscapes dataset with K = 19 classes, assuming δ = 0 . 01 and min i p i = 1% , B should be at least 759 to guarantee that each class of pixels appears at least once with a high probability. This results in a significant strain on GPU memory.

To address this, considering that the tail-class samples generally have less opportunity to be included in a mini-batch and are often more crucial for final performance, we thus develop a novel Tailclass Memory Bank (T-Memory Bank) to efficiently optimize Equation (7) and manage GPU usage effectively. As depicted in Figure 3(c), the high-level ideas of the T-Memory Bank are as follows: 1) identify missing tail classes of all images involved in a mini-batch and 2) randomly replace some pixels in the image with missing classes based on stored historical class information in T-Memory Bank. In this sense, we can obtain an approximated batch-version of Equation (7), i.e. ,

̸

̸

where N c and T c represent the set of pixels with label c in the original image and those pixels stored in the T-Memory Bank, respectively; ˜ ℓ c,c ′ ,m,n sq := ℓ sq ( f ( c ) θ ( ˜ X p m ) -f ( c ) θ ( ˜ X p n )) ; ˜ X p represents the sample after replacing some pixels with tail classes pixels from the T-Memory Bank.

Detailed Components. As shown in Figure 2, T-Memory Bank comprises three main parts: (1) Memory Branch stores a set with S M (the Memory Size) images for each tail class. We define the set as M = {M c 1 , . . . , M c nt } , where C t = { c i } n t i =1 denotes the labels of tail classes, and n t is the total number of selected tail classes; (2) Retrieve Branch selects pixels from the Memory Branch to supplement the missing tail classes and (3) Store Branch updates the Memory Branch whenever a new image arrives. Algorithm 1 summarizes a short version of AUCSeg equipped with T-Memory Bank. Please refer to the detailed version in Appendix D. Note that we introduce CE loss as a regularization term for our proposed AUCSeg, which is widely used in the AUC community [99] to pursue robust feature learning. Experiments demonstrate that the performance is insensitive to the regularization weight λ , as shown in Figure 5d.

At the start of training, the Memory Branch is empty. In this case, we only calculate the loss function ℓ for the classes present in the mini-batch, while the Retrieve Branch will not take any action. Meanwhile, the Store Branch will continuously append pixel data of tail classes to the Memory Branch. As the Memory Branch reaches its maximum capacity S M , we adopt a random replacement strategy to update the Store Branch (Lines 5 to 6 in Algorithm 1 or Lines 6 to 11 in Algorithm 2).

As the training process progresses, if the Memory Branch is not empty, the Retrieve Branch kicks in to count the missing classes in each image of the mini-batch, denoted as C miss . It then calculates the number of classes needed to be added for optimization, n sample = ⌈|C miss | × R S ⌉ . Here, we introduce a tunable sample ratio R S to strike a trade-off between the original and missing tail-class semantic information. Finally, it uniformly retrieves the corresponding pixels of n sample missing classes from the Memory Branch, resizes them by the resize ratio R R , and randomly selects positions to overwrite (Lines 7 to 8 in Algorithm 1 or Lines 13 to 18 in Algorithm 2).

Algorithm 1: AUCSeg Algorithm (Short Version)

Discussions. We recognize that Memory Bank [82, 36] has achieved great success in deep learning. However, our T-Memory Bank behaves differently compared to earlier studies. The key difference is that the Memory Bank and T-Memory Bank are designed for different tasks. The goal of the previous Memory Bank is to facilitate the traditional classifications by storing instance-level or image-level features, while our T-Memory Bank specifically stores the original pixels for each object. This strategy is particularly beneficial for our PLSS task. Additionally, the T-Memory Bank enables AUCSeg without substantially increasing GPU workload as well as the number of samples per mini-batch by selectively replacing non-essential pixels. Our experiments, detailed in Appendix G.6, include a comparison of GPU overhead. For more discussion on the T-Memory Bank, please refer to Appendix E.

5 Experiments

In this section, we describe some details of the experiments and present our results. Due to space limitations, please refer to Appendix F, Appendix G and Appendix H for an extended version.

5.1 Experimental Setups

The experiment includes three benchmark datasets: Cityscapes [19], ADE20K [109], and COCOStuff 164K [9]. We use SegNeXt [32] as the backbone for our model and the mean of Intersection over Union (mIoU) as the evaluation metric. We compare our method with 13 recent advancements and 6 long-tail approaches in semantic segmentation. All the long-tail methods also use SegNeXt as the backbone. To ensure fairness, we re-implement the listed methods using their publicly shared code and test them on the same hardware. Detailed introductions are deferred to Appendix F.

5.2 Overall Performance

Table 1 shows the quantitative performance comparisons. We draw the following conclusions: First, most current algorithms perform poorly in long-tail scenarios. Specifically, performance drops sharply from head to tail classes. For instance, the performance gap for PointRend and OCRNet on the Cityscapes reaches up to 40% . Second, models using long-tail approaches generally achieve better results than those that do not. However, they still fail to produce satisfactory outcomes. One possible reason is that these long-tail approaches focus on reweighting, giving too much attention to the tail classes and leading to overfitting. Additionally, our proposed AUCSeg method surpasses all competitors in most metrics. This success is due to the appealing properties of AUC. Our method consistently outperforms the runner-up by +1 . 21% , +0 . 75% , and +0 . 38% in tail classes mIoU across the datasets. Overall mIoU also improves by +1 . 05% , +0 . 27% , and +0 . 31% . In some Head/Middle

metrics, AUCSeg does not achieve the best performance. Even in these cases, AUCSeg still secures the runner-up status. We analyze the performance trade-off between head and tail classes in Appendix H. These experimental results underscore the effectiveness of our proposed method. We further present the results for each tail class in Appendix G.1, and analyze the reasons for the varying performance improvements of tail classes across the three datasets in Appendix G.2.

Figure 4 displays the qualitative results on the Cityscapes validation set. Benefiting from our proposed AUC and T-Memory Bank techniques, AUCSeg segments objects in tail classes more accurately. It correctly distinguishes between bicycles and motorcycles and successfully identifies distant traffic lights, which other methods overlook. More qualitative results can be found in Appendix G.3.

5.3 Backbone Extension

In Section 5.1, we select the current SOTA SegNeXt as the backbone. Nevertheless, AUCSeg can also adapt to other backbones, consistently delivering effective results. Table 2 presents the experimental results of AUCSeg when using DeepLabV3+, EMANet, OCRNet, and ISANet as backbones. These results reveal significant improvements in both overall mIoU and tail classes mIoU with AUCSeg. Notably, on ISANet, the increases are 5 . 54% and 6 . 49% . Moreover, AUCSeg enhances performance across various model sizes and different pixel-level long-tail problems, as detailed in Appendix G.4 and Appendix G.4. This demonstrates the superiority of our proposed AUCSeg for long-tailed semantic segmentation.

5.4 Ablation studies

We perform several ablation studies to test the effectiveness of different modules and hyperparameters. All experiments are conducted on the ADE20K validation set.

(a) Input

(c) DeepLabV3+

(d) SegNeXt

(e) AUCSeg (Ours)

Overall

Tail

Best

40

42

44

46

48

50

1

3

5

8

10

20

S M

mIoU (%)

(a) Memory Size (

S

M

).

40

42

44

46

48

50

0.01

0.04

0.05

0.07

0.1

0.15

R S

mIoU (%)

R

S

).

40

42

44

46

48

50

0.3

0.4

0.5

0.6

0.7

1

R R

mIoU (%)

R

R

).

40

42

44

46

48

50

1/6

1/5

1/4

1/3

1/2

1

l

mIoU (%)

(d) The weight

λ

.

The Effectiveness of AUC Optimization and T-Memory Bank. Table 3 details our step-bystep ablation study on the AUC and T-Memory Bank components of AUCSeg. Compared to the baseline SegNeXt, AUC enhances performance by +1 . 01% and +1 . 42% in overall and tail classes. The T-Memory Bank further addresses the imbalance issue, yielding improvements of +1 . 75% overall and +2 . 24% in tail classes. Our results also show that employing T-Memory Bank without AUC yields no signif-

icant improvements, underscoring the necessity of AUC Loss. More ablation experiments on the effectiveness of AUC optimization and TMB are deferred to Appendix G.6, G.7, G.8, and G.9.

Ablation Study on Hyper-Parameters. Figure 5a ablates the maximum number of images stored per class in the Memory Branch, referred to as Memory Size ( S M ). For the ADE20K dataset, optimal performance occurs when S M = 5 , and performance shows little sensitivity to changes in S M . Figure 5b ablates the Sample Ratio ( R S ), the fraction of classes sampled from the Memory Branch relative to the total number of missing tail classes. Figure 5c ablates the Resize Ratio ( R R ), the scaling factor for the sampled pixels. For ADE20K, the best result is obtained when R S = 0 . 05 and R R = 0 . 4 . A potential reason for their small value is that the original image is overwritten when R S and R R are too large, resulting in poor training performance. Figure 5d ablates the weight λ for ℓ ce and ℓ auc , with λ = 1 4 providing slightly better results. The influence of λ on performance is minimal. Detailed results from this hyper-parameter ablation study are available in Appendix G.10 and G.11.

6 Conclusion

This paper explores AUC optimization in the context of PLSS tasks. To begin with, we theoretically study the generalization performance of AUC-oriented PLSS by overcoming the two-layer coupling issue across the loss terms of AUCSeg therein. The corresponding results show that applying AUC optimization to PLSS could also enjoy a promising performance. Subsequently, we propose a novel TMemory Bank to reduce the significant memory demand for the mini-batch optimization of AUCSeg. Finally, comprehensive experiments suggest the effectiveness of our proposed AUCSeg.

Acknowledgments

This work was supported in part by the National Key R&D Program of China under Grant 2018AAA0102000, in part by National Natural Science Foundation of China: 62236008, U21B2038, U23B2051, 61931008, 62122075, 92370102, 62406305, 62471013 and 62476068, in part by Youth Innovation Promotion Association CAS, in part by the Strategic Priority Research Program of the Chinese Academy of Sciences, Grant No. XDB0680000, in part by the Innovation Funding of ICT, CAS under Grant No.E000000, in part by the Postdoctoral Fellowship Program of CPSF under Grant GZB20240729 and GZB20230732, and in part by the China Postdoctoral Science Foundation under Grant No.2023M743441.

References

Shivani Agarwal. Surrogate regret bounds for bipartite ranking via strongly proper losses. The Journal of Machine Learning Research , 15(1):1653-1674, 2014.

Shivani Agarwal, Thore Graepel, Ralf Herbrich, Sariel Har-Peled, Dan Roth, and Michael I Jordan. Generalization bounds for the area under the roc curve. Journal of Machine Learning Research , 6(4), 2005.

Massih-Reza Amini and Nicolas Usunier. Learning with partially labeled and interdependent data . Springer, 2015.

Shilong Bao, Qianqian Xu, Ke Ma, Zhiyong Yang, Xiaochun Cao, and Qingming Huang. Collaborative preference embedding against sparse labels. In ACMMM , pages 2079-2087, 2019.

Shilong Bao, Qianqian Xu, Zhiyong Yang, Xiaochun Cao, and Qingming Huang. Rethinking collaborative metric learning: Toward an efficient alternative without negative sampling. IEEE TPAMI , 45(1):10171035, 2022.

Shilong Bao, Qianqian Xu, Zhiyong Yang, Yuan He, Xiaochun Cao, and Qingming Huang. The minority matters: A diversity-promoting collaborative metric learning algorithm. In NeurIPS , pages 2451-2464, 2022.

Shilong Bao, Qianqian Xu, Zhiyong Yang, Yuan He, Xiaochun Cao, and Qingming Huang. Improved diversity-promoting collaborative metric learning for recommendation. IEEE TPAMI , 2024.

Peter L Bartlett and Shahar Mendelson. Rademacher and gaussian complexities: Risk bounds and structural results. JMLR , 3(Nov):463-482, 2002.

Holger Caesar, Jasper Uijlings, and Vittorio Ferrari. Coco-stuff: Thing and stuff classes in context. In CVPR , pages 1209-1218, 2018.

Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning imbalanced datasets with label-distribution-aware margin loss. In NeurIPS , 2019.

Robin Chan, Matthias Rottmann, Fabian Hüger, Peter Schlicht, and Hanno Gottschalk. Application of decision rules for handling class imbalance in semantic segmentation. arXiv preprint arXiv:1901.08394 , 2019.

Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE TPAMI , 40(4):834-848, 2017.

Liang-Chieh Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. arXiv preprint arXiv:1706.05587 , 2017.

Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoderdecoder with atrous separable convolution for semantic image segmentation. In ECCV , pages 801-818, 2018.

Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexander Kirillov, and Rohit Girdhar. Maskedattention mask transformer for universal image segmentation. In CVPR , pages 1290-1299, 2022.

Carlo Ciliberto, Lorenzo Rosasco, and Alessandro Rudi. A general framework for consistent structured prediction with implicit loss embeddings. JMLR , 21(98):1-67, 2020.

Stéphan Clémençon, Gábor Lugosi, and Nicolas Vayatis. Ranking and empirical minimization of u-statistics. The Annals of Statistics , 2008.

MMSegmentation Contributors. MMSegmentation: Openmmlab semantic segmentation toolbox and benchmark. https://github.com/open-mmlab/mmsegmentation , 2020.

Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In CVPR , pages 3213-3223, 2016.

Corinna Cortes and Mehryar Mohri. Auc optimization vs. error rate minimization. In NeurIPS , 2003.

Felipe Cucker and Steve Smale. On the mathematical foundations of learning. Bulletin of the American mathematical society , 39(1):1-49, 2002.

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR , pages 248-255, 2009.

Henghui Ding, Xudong Jiang, Ai Qun Liu, Nadia Magnenat Thalmann, and Gang Wang. Boundary-aware feature propagation for scene segmentation. In ICCV , pages 6819-6829, 2019.

Rongsheng Dong, Xiaoquan Pan, and Fengying Li. Denseu-net-based semantic segmentation of small objects in urban remote sensing images. IEEE Access , 7:65347-65356, 2019.

Mingyuan Fan, Shenqi Lai, Junshi Huang, Xiaoming Wei, Zhenhua Chai, Junfeng Luo, and Xiaolin Wei. Rethinking bisenet for real-time semantic segmentation. In CVPR , pages 9716-9725, 2021.

Tom Fawcett. An introduction to roc analysis. Pattern recognition letters , 27(8):861-874, 2006.

Jun Fu, Jing Liu, Haijie Tian, Yong Li, Yongjun Bao, Zhiwei Fang, and Hanqing Lu. Dual attention network for scene segmentation. In CVPR , pages 3146-3154, 2019.

Wei Gao, Rong Jin, Shenghuo Zhu, and Zhi-Hua Zhou. One-pass auc optimization. In ICML , pages 906-914, 2013.

Wei Gao and Zhi-Hua Zhou. On the consistency of auc pairwise optimization. arXiv preprint arXiv:1208.0645 , 2012.

Damian Gola, Jeannette Erdmann, Bertram Müller-Myhsok, Heribert Schunkert, and Inke R König. Polygenic risk scores outperform machine learning methods in predicting coronary artery disease status. Genetic epidemiology , 44(2):125-138, 2020.

Meng-Hao Guo, Zheng-Ning Liu, Tai-Jiang Mu, and Shi-Min Hu. Beyond self-attention: External attention using two linear layers for visual tasks. IEEE TPAMI , 45(5):5436-5447, 2022.

Meng-Hao Guo, Cheng-Ze Lu, Qibin Hou, Zhengning Liu, Ming-Ming Cheng, and Shi-Min Hu. Segnext: Rethinking convolutional attention design for semantic segmentation. In NeurIPS , pages 1140-1156, 2022.

Meng-Hao Guo, Tian-Xing Xu, Jiang-Jiang Liu, Zheng-Ning Liu, Peng-Tao Jiang, Tai-Jiang Mu, SongHai Zhang, Ralph R Martin, Ming-Ming Cheng, and Shi-Min Hu. Attention mechanisms in computer vision: A survey. Computational visual media , 8(3):331-368, 2022.

David J Hand and Robert J Till. A simple generalisation of the area under the roc curve for multiple class classification problems. Machine learning , 45:171-186, 2001.

James A Hanley and Barbara J McNeil. The meaning and use of the area under a receiver operating characteristic (roc) curve. Radiology , 143(1):29-36, 1982.

Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR , pages 9729-9738, 2020.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR , pages 770-778, 2016.

Alan Herschtal and Bhavani Raskutti. Optimising area under the roc curve using gradient descent. In ICML , pages 385-392, 2004.

Md Sazzad Hossain, John M Betts, and Andrew P Paplinski. Dual focal loss to address class imbalance in semantic segmentation. Neurocomputing , 462:69-87, 2021.

Wenzheng Hou, Qianqian Xu, Zhiyong Yang, Shilong Bao, Yuan He, and Qingming Huang. Adauc: End-to-end adversarial auc optimization against long-tail problems. In ICML , pages 8903-8925, 2022.

Hanzhe Hu, Fangyun Wei, Han Hu, Qiwei Ye, Jinshi Cui, and Liwei Wang. Semi-supervised semantic segmentation via adaptive equalization learning. In NeurIPS , pages 22106-22118, 2021.

Cong Hua, Qianqian Xu, Shilong Bao, Zhiyong Yang, and Qingming Huang. Reconboost: Boosting can achieve modality reconcilement. In ICML , pages 19573-19597, 2024.

Zilong Huang, Xinggang Wang, Lichao Huang, Chang Huang, Yunchao Wei, and Wenyu Liu. Ccnet: Criss-cross attention for semantic segmentation. In ICCV , pages 603-612, 2019.

Svante Janson. Large deviations for sums of partly dependent random variables. Random Structures & Algorithms , 24:234-248, 2004.

Zhenchao Jin, Xiaowei Hu, Lingting Zhu, Luchuan Song, Li Yuan, and Lequan Yu. Idrnet: Interventiondriven relation network for semantic segmentation. In NeurIPS , 2024.

Thorsten Joachims. A support vector method for multivariate performance measures. In ICML , pages 377-384, 2005.

Thorsten Joachims. Training linear svms in linear time. In SIGKDD , pages 217-226, 2006.

Michael Kampffmeyer, Arnt-Borre Salberg, and Robert Jenssen. Semantic segmentation of small objects and modeling of uncertainty in urban remote sensing images using deep convolutional neural networks. In CVPR workshops , pages 1-9, 2016.

Ganesh Ramachandra Kini, Orestis Paraskevas, Samet Oymak, and Christos Thrampoulidis. Labelimbalanced and group-sensitive classification under overparameterization. NeurIPS , pages 18970-18983, 2021.

Alexander Kirillov, Yuxin Wu, Kaiming He, and Ross Girshick. Pointrend: Image segmentation as rendering. In CVPR , pages 9799-9808, 2020.

Fahad Lateef and Yassine Ruichek. Survey on semantic segmentation using deep learning techniques. Neurocomputing , 338:321-348, 2019.

Antoine Ledent, Yunwen Lei, and Marius Kloft. Improved generalisation bounds for deep learning through l ∞ covering numbers. arXiv preprint arXiv:1905.12430 , 2019.

Feiran Li, Qianqian Xu, Shilong Bao, Zhiyong Yang, Runmin Cong, Xiaochun Cao, and Qingming Huang. Size-invariance matters: Rethinking metrics and losses for imbalanced multi-object salient object detection. In ICML , pages 28989-29021, 2024.

Shaojie Li and Yong Liu. Towards sharper generalization bounds for structured prediction. In NeurIPS , pages 26844-26857, 2021.

Xia Li, Zhisheng Zhong, Jianlong Wu, Yibo Yang, Zhouchen Lin, and Hong Liu. Expectationmaximization attention networks for semantic segmentation. In ICCV , pages 9167-9176, 2019.

Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In ICCV , pages 2980-2988, 2017.

Mingrui Liu, Zhuoning Yuan, Yiming Ying, and Tianbao Yang. Stochastic auc maximization with deep neural networks. In ICLR , 2020.

Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In CVPR , pages 3431-3440, 2015.

Ilya Loshchilov and Frank Hutter. Fixing weight decay regularization in adam. In ICLR , 2018.

Xiangrui Meng. Scalable simple random sampling and stratified sampling. In ICML , pages 531-539, 2013.

Aditya Krishna Menon, Sadeep Jayasumana, Ankit Singh Rawat, Himanshu Jain, Andreas Veit, and Sanjiv Kumar. Long-tail learning via logit adjustment. In ICLR , 2020.

Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar. Foundations of machine learning . MIT press, 2018.

Michael Natole, Yiming Ying, and Siwei Lyu. Stochastic proximal algorithms for auc maximization. In ICML , pages 3710-3719, 2018.

Foster Provost and Pedro Domingos. Tree induction for probability-based ranking. Machine learning , 52:199-215, 2003.

René Ranftl, Alexey Bochkovskiy, and Vladlen Koltun. Vision transformers for dense prediction. In ICCV , pages 12179-12188, 2021.

Mina Rezaei, Haojin Yang, and Christoph Meinel. Recurrent generative adversarial network for learning imbalanced medical image semantic segmentation. Multimedia Tools and Applications , 79(21-22):1532915348, 2020.

Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention , pages 234-241, 2015.

Huiyang Shao, Qianqian Xu, Zhiyong Yang, Shilong Bao, and Qingming Huang. Asymptotically unbiased instance-wise regularized partial auc optimization: Theory and algorithm. In NeurIPS , pages 38667-38679, 2022.

Huiyang Shao, Qianqian Xu, Zhiyong Yang, Peisong Wen, Gao Peifeng, and Qingming Huang. Weighted roc curve in cost space: Extending auc to cost-sensitive learning. In NeurIPS , pages 17357-17368, 2024.

Ravindra Singh, Naurang Singh Mangat, Ravindra Singh, and Naurang Singh Mangat. Stratified sampling. Elements of survey sampling , pages 102-144, 1996.

Ke Sun, Bin Xiao, Dong Liu, and Jingdong Wang. Deep high-resolution representation learning for human pose estimation. In CVPR , pages 5693-5703, 2019.

Nicolas Usunier, Massih R Amini, and Patrick Gallinari. Generalization error bounds for classifiers trained with interdependent data. In NeurIPS , 2005.

Nicolas Usunier, Massih-Reza Amini, and Patrick Gallinari. A data-dependent generalisation error bound for the auc. In ICML Workshop , 2005.

Haonan Wang, Qixiang Zhang, Yi Li, and Xiaomeng Li. Allspark: Reborn labeled features from unlabeled in transformer for semi-supervised semantic segmentation. In CVPR , 2024.

Jingdong Wang, Ke Sun, Tianheng Cheng, Borui Jiang, Chaorui Deng, Yang Zhao, Dong Liu, Yadong Mu, Mingkui Tan, Xinggang Wang, et al. Deep high-resolution representation learning for visual recognition. IEEE TPAMI , 43(10):3349-3364, 2020.

Yikai Wang, Xinghao Chen, Lele Cao, Wenbing Huang, Fuchun Sun, and Yunhe Wang. Multimodal token fusion for vision transformers. In CVPR , pages 12186-12195, 2022.

Yuchao Wang, Jingjing Fei, Haochen Wang, Wei Li, Tianpeng Bao, Liwei Wu, Rui Zhao, and Yujun Shen. Balancing logit variation for long-tailed semantic segmentation. In CVPR , pages 19561-19573, 2023.

Zitai Wang, Qianqian Xu, Zhiyong Yang, Yuan He, Xiaochun Cao, and Qingming Huang. A unified generalization analysis of re-weighting and logit-adjustment for imbalanced learning. In NeurIPS , pages 48417-48430, 2023.

Andrew Westcott, Dante PI Capaldi, David G McCormack, Aaron D Ward, Aaron Fenster, and Grace Parraga. Chronic obstructive pulmonary disease: thoracic ct texture analysis and machine learning to predict pulmonary ventilation. Radiology , 293(3):676-684, 2019.

Huikai Wu, Junge Zhang, Kaiqi Huang, Kongming Liang, and Yizhou Yu. Fastfcn: Rethinking dilated convolution in the backbone for semantic segmentation. arXiv preprint arXiv:1903.11816 , 2019.

Yu-Huan Wu, Yun Liu, Le Zhang, Ming-Ming Cheng, and Bo Ren. Edn: Salient object detection via extremely-downsampled network. IEEE Tip , 31:3125-3136, 2022.

Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via nonparametric instance discrimination. In CVPR , pages 3733-3742, 2018.

Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo. Segformer: Simple and efficient design for semantic segmentation with transformers. In NeurIPS , pages 12077-12090, 2021.

Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In CVPR , pages 1492-1500, 2017.

Tianbao Yang and Yiming Ying. Auc maximization in the era of big data and ai: A survey. ACM computing surveys , 55(8):1-37, 2022.

Zhiyong Yang, Qianqian Xu, Shilong Bao, Xiaochun Cao, and Qingming Huang. Learning with multiclass auc: Theory and algorithms. IEEE TPAMI , 44(11):7747-7763, 2021.

Zhiyong Yang, Qianqian Xu, Shilong Bao, Yuan He, Xiaochun Cao, and Qingming Huang. When all we need is a piece of the pie: A generic framework for optimizing two-way partial auc. In ICML , pages 11820-11829, 2021.

Zhiyong Yang, Qianqian Xu, Shilong Bao, Yuan He, Xiaochun Cao, and Qingming Huang. Optimizing two-way partial auc with an end-to-end framework. IEEE TPAMI , 2022.

Zhiyong Yang, Qianqian Xu, Shilong Bao, Peisong Wen, Yuan He, Xiaochun Cao, and Qingming Huang. Auc-oriented domain adaptation: From theory to algorithm. IEEE TPAMI , 2023.

Zhiyong Yang, Qianqian Xu, Xiaochun Cao, and Qingming Huang. Task-feature collaborative learning with application to personalized attribute prediction. IEEE TPAMI , 43(11):4094-4110, 2020.

Zhiyong Yang, Qianqian Xu, Wenzheng Hou, Shilong Bao, Yuan He, Xiaochun Cao, and Qingming Huang. Revisiting auc-oriented adversarial training with loss-agnostic perturbations. IEEE TPAMI , 2023.

Zhiyong Yang, Qianqian Xu, Zitai Wang, Sicong Li, Boyu Han, Shilong Bao, Xiaochun Cao, and Qingming Huang. Harnessing hierarchical label distribution variations in test agnostic long-tail recognition. In ICML , pages 56624-56664, 2024.

Minghao Yin, Zhuliang Yao, Yue Cao, Xiu Li, Zheng Zhang, Stephen Lin, and Han Hu. Disentangled non-local neural networks. In ECCV , pages 191-207, 2020.

Yiming Ying, Longyin Wen, and Siwei Lyu. Stochastic online auc maximization. In NeurIPS , 2016.

Changqian Yu, Changxin Gao, Jingbo Wang, Gang Yu, Chunhua Shen, and Nong Sang. Bisenet v2: Bilateral network with guided aggregation for real-time semantic segmentation. IJCV , 129:3051-3068, 2021.

Fisher Yu and Vladlen Koltun. Multi-scale context aggregation by dilated convolutions. arXiv preprint arXiv:1511.07122 , 2015.

Yuhui Yuan, Xilin Chen, and Jingdong Wang. Object-contextual representations for semantic segmentation. In ECCV , pages 173-190, 2020.

Yuhui Yuan, Lang Huang, Jianyuan Guo, Chao Zhang, Xilin Chen, and Jingdong Wang. Ocnet: Object context for semantic segmentation. IJCV , 129(8):1-24, 2021.

Zhuoning Yuan, Zhishuai Guo, Nitesh Chawla, and Tianbao Yang. Compositional training for end-to-end deep auc maximization. In ICLR , 2021.

Zhuoning Yuan, Yan Yan, Milan Sonka, and Tianbao Yang. Large-scale robust deep auc maximization: A new surrogate loss and empirical studies on medical image classification. In ICCV , pages 3040-3049, 2021.

Zhuoning Yuan, Dixian Zhu, Zi-Hao Qiu, Gang Li, Xuanhui Wang, and Tianbao Yang. Libauc: A deep learning library for x-risk optimization. In KDD , pages 5487-5499, 2023.

Hang Zhang, Kristin Dana, Jianping Shi, Zhongyue Zhang, Xiaogang Wang, Ambrish Tyagi, and Amit Agrawal. Context encoding for semantic segmentation. In CVPR , pages 7151-7160, 2018.

Rui-Ray Zhang and Massih-Reza Amini. Generalization bounds for learning under graph-dependence: A survey. Machine Learning , pages 1-31, 2024.

Songyang Zhang, Zeming Li, Shipeng Yan, Xuming He, and Jian Sun. Distribution alignment: A unified framework for long-tail visual recognition. In CVPR , pages 2361-2370, 2021.

Hengshuang Zhao, Xiaojuan Qi, Xiaoyong Shen, Jianping Shi, and Jiaya Jia. Icnet for real-time semantic segmentation on high-resolution images. In ECCV , pages 405-420, 2018.

Peilin ZHAO, Steven CH HOI, Rong JIN, and Tianbo YANG. Online auc maximization. In ICML , pages 233-240, 2011.

Mingmin Zhen, Jinglu Wang, Lei Zhou, Shiwei Li, Tianwei Shen, Jiaxiang Shang, Tian Fang, and Long Quan. Joint semantic segmentation and boundary detection using iterative pyramid contexts. In CVPR , pages 13666-13675, 2020.

Sixiao Zheng, Jiachen Lu, Hengshuang Zhao, Xiatian Zhu, Zekun Luo, Yabiao Wang, Yanwei Fu, Jianfeng Feng, Tao Xiang, Philip HS Torr, et al. Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers. In CVPR , pages 6881-6890, 2021.

Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In CVPR , pages 633-641, 2017.

Ding-Xuan Zhou. The covering number in learning theory. Journal of Complexity , 18:739-767, 2002.

Contents

A Symbol Definitions

In this section, Table 4 includes a summary of key notations and descriptions in this work.

B Generalization Bounds and Its Proofs

B.1 Preliminary Lemmas

Lemma 1 (Jensen's Inequality) . If X is a random variable and φ is a convex function, then

Assumption 1. Assume that ℓ is µ -Lipschitz continuous, that is

Assumption 1 is a pretty mild assumption. The square loss ℓ sq ( x ) = (1 -x ) 2 satisfies Assumption 1.

Lemma 2. The empirical Rademacher complexity of function g with respect to the predictor f is defined as:

where F ⊆ { f : X → R K } is a family of predictors, and N refers to the size of the dataset, and σ i s are independent uniform random variables taking values in {-1 , +1 } . The random variables σ i are called Rademacher variables.

Lemma 3. Let E [ g ] and ˆ E [ g ] represent the expected risk and empirical risk, and F ⊆ { f : X → R K } . Then with probability at least 1 -δ over the draw of an i.i.d. sample S of size n , the generalization bound holds:

Definition 1 (Fractional Independent Vertex Cover, and Fractional Chromatic Number [103]) . Let a graph be G = ( V, E ) .

A fractional vertex cover of G is a family { ( F j , ω j ) } j of pairs ( F j , ω j ) , where F j ⊆ V ( G ) , ω j ∈ (0 , 1] , and ∑ j : v ∈ F j ω j = 1 , ∀ v ∈ V ( G ) .

An independent set of G is a set of vertices in G with no two adjacent. Let I ( G ) denote the set of independent sets of G .

A fractional vertex cover is a fractional independent vertex cover { ( I j , ω j ) } j of G if ∀ j , I j ∈ I ( G ) .

A fractional coloring of a graph G is a mapping g : I ( G ) → (0 , 1] such that ∑ I ∈I ( G ): v ∈ I g ( I ) ≥ 1 , ∀ v ∈ V ( G ) . The fractional chromatic number χ f ( G ) is the minimum of the value ∑ I ∈I ( G ) g ( I ) over fractional colorings of G .

Notably, the minimum of ∑ j ω j over all fractional independent vertex covers { ( I j , ω j ) } j of G is the fractional chromatic number χ f ( G ) .

Definition 2 (Dependency Graph [44]) . An (undirected) graph G = ( V, E ) is called a dependency graph associated with a random vector (or random variables) x = ( x 1 , . . . , x m ) if

For all disjoint vertex sets I, J ⊆ [ m ] , if I, J are not adjacent in G , then random variables { x i } i ∈ I and { x j } j ∈ J are independent.

A useful result is Janson's decomposition property [44], which combines the concept of dependency graphs with fractional independent vertex covers. The property states that if interdependent random variables ( x i ) i ∈ [ m ] is associated with a dependency graph G with a fractional independent vertex cover ( I j , ω j ) j ∈ [ J ] , then, the sum of the interdependent variables, can be equivalently decomposed into a weighted sum of sums of independent variables:

B.2 Key Lemmas

Lemma 4. Let D represents the training set, and + / -respectively denote the categories c/c ′ . The function f (+) i,j represents the score function for the ( i, j ) -th pixel in category c . For a set A , |A| denotes the number of elements in the set. We have:

where,

and n ( X + ) /n ( X -) represents the number of positive/negative samples in image X .

Definition 3 (Covering Number [110, 54]) . Let F be class of real-valued fucntions, defined over a space Z and S := { ( X 1 , Y 1 ) , . . . , ( X n , Y n ) } ∈ Z n of cardinality n . For any ϵ > 0 , the empirical ℓ ∞ -norm covering number N ( F , || · || ∞ , S, ϵ ) w.r.t S is defined as the minimal number m of a collection of vectors v 1 , . . . , v m ∈ R n such that ( v j i is the i -th component of the vector v j )

In this case, we call { v 1 , . . . , v m } an ( ϵ, ℓ ∞ ) -cover of F w.r.t S .

Lemma 5. Let P ( X ) represents the pixels in image X , F = {{ f (+) i,j } : ( i, j ) ∈ P ( X ) , ( X , Y ) ∈ S } and ℓ ◦ F = { ℓ ◦ { f (+) i,j } : { f (+) i,j } ∈ F } . According to Definition 3, we have:

where ρ max = max X |D| n ( X + ) n ( X -) |N + ||N -| .

Proof. For any g = ℓ ◦ f , we can find ˜ g = ℓ ◦ ˜ f that satisfies the following conditions:

where

According to Assumption 1, we have:

where ρ max ≜ max X ρ ( X ) .

Define a ϵ 2 µρ max -covering of the class F with || · || ∞ norm:

with

∞ max There exists a f C k = {{ f C k i,j } : ( i, j ) ∈ P ( X ) } , such that for any f = {{ f (+) i,j } : ( i, j ) ∈ P ( X ) } ∈ C k ∩ F :

which implies that

where g f = ℓ ◦ f and g f C k = ℓ ◦ f C k .

Denote

then {C g, 1 , . . . , C g,N } realizes an ϵ -covering of ℓ ◦ f . Hence, the minimum size of the ϵ -covering is at most N . Mathematically, we then have:

This completed the proof.

Lemma 6 ([52]) . Let F be a real-valued function class taking values in [0 , 1] , and assume that 0 ∈ F . Let S be a finite sample of size n . For any 2 ≤ p ≤ ∞ , we have the following relationship between the Rademacher complexity ˆ R ( F ) and the covering number N ( F , || · || p , S, ϵ ) .

Lemma 7 ([72, 3]) . Given a sample ˜ D = { (˜ x i , ˜ y i ) } i m =1 where ˜ x i ∈ ˜ X , ˜ y i ∈ ˜ Y and ˜ D is associated with a dependency graph G , where χ f ( G ) is its fractional chromatic number, and a loss function L : ˜ X × ˜ Y × ˜ F → [0 , M ] , where ˜ F = { ˜ f : ˜ X → R } . Then, for any δ ∈ (0 , 1) , the following generalization bound holds with probability at least 1 -δ :

where ̂ R ∗ ˜ D ( L ◦ ˜ F ) is the empirical fractional Rademacher complexity of the loss space.

Lemma 8. Given a sample ˜ D = { (˜ x i , ˜ y i ) } 2 i m =1 where ˜ x i ∈ ˜ X , ˜ y i ∈ ˜ Y and ˜ D is associated with a dependency graph G , where χ f ( G ) is its fractional chromatic number, and each fractional independent vertex cover contains two independent samples, one positive and one negative. Under these conditions, χ f ( G ) satisfies:

Proof. The calculation of the fractional chromatic number can be transformed into finding how many groups can be formed where each group contains m ordered pairs of positive and negative samples. In each group, each sample appears only once, and there are no duplicate ordered pairs of positive and negative samples across all groups.

For example, in a dataset where 2 m = 4 , there exist 6 groups:

Therefore, χ f ( G ) = 6 .

For 2 m samples, we can extract A 2 2 m ordered pairs of positive and negative samples. Since these samples have an equal status in the dataset, they appear the same number of times among these A 2 2 m pairs.

After selecting the first group from these A 2 2 m pairs, A 2 2 m -m pairs of positive and negative samples remain. The frequency of each sample appearing in these remaining pairs remains equal.

We continue to select the second group, and so on, until the last group. The frequency of each sample in the remaining pairs still remains equal.

Thus, we can find A 2 2 m groups, i.e. , χ f ( G ) = 2(2 m -1)

B.3 Proof of the Main Result

Restate of Theorem 1 (Generalization Bound for AUCSeg) . Let E D [ ˆ L D ( f ) ] be the population risk of ˆ L D ( f ) . Assume F ⊆ { f : X → R H × W × K } , where H and W represent the height and width of the image, and K represents the number of categories, ˆ L ( i ) is the risk over i -th sample, and is µ -Lipschitz with respect to the l ∞ norm, ( i.e. ∥ ˆ L ( x ) -ˆ L ( y ) ∥ ∞ ≤ µ · ∥ x -y ∥ ∞ ). There exists three constants A > 0 , B > 0 and C > 0 , the following generalization bound holds with probability at least 1 -δ over a random draw of i.i.d training data (at the image-level):

where

n ( c ) max = max X n ( X ( c ) ) , n ( c ) mean = ∑ N i =1 n ( X ( c ) i ) , N = |D| , k = H × W and X ( c ) represents the pixel of class c in image X .

Proof. First, we find that the calculation of pair-wise AUC requires both positive and negative samples. These two samples can come from the same image or from two different images. Therefore,

we transform the original problem into two sub-problems:

where + / -respectively denote the categories c/c ′ .

For Part 1, we use the complexity measure technique of the covering number.

Assuming log N ( F , || · || ∞ , S, ϵ ) ≤ A ϵ 2 log [ B ϵ · |D| · k + C ] , where k = H × W is the pixel count in an image. A , B and C represent constants. Based on Lemma 5, we have:

where:

Denoted by a := 4 Aµ 2 ρ 2 max , b := 2 Bµρ max |D| k and c := C . Based on Lemma 6 and Lemma A.3 in [54], we have:

Substituting this result into Lemma 3, with probability at least 1 -δ , we have

For Part 2, we use the complexity measure technique of the fractional chromatic number and covering number.

According to Lemma 7, calculating the generalization bound only requires knowing the chromatic complexity. Based on Equation (13), we have

where ρ ( x ) = |D| ( |D|1) n ( X + ) n ( X -) |N + ||N -| , and ω j denotes the weight assigned to the subset in the fractional vertex cover.

Similar to Part 1, using the covering number we can get

where we define c j as ln 2 |D| · A · µ 2 · ρ 2 max · log ( 2 B · µ · ρ max · |D| 2 · k + C ) as c j , and define m j as | I j | .

Assume that the number of images in the dataset is even, i.e. , | D | ≡ 0 (mod 2) . We have:

where ( ∗ ) follows the Lemma 8.

The second-order inequality is based on the fact that:

define ρ j max as the largest ρ in the j -th cluster, and that:

where:

Combining Equation (34) and Equation (36), we obtain:

Based on Lemma 7 and Lemma 8, it comes:

Therefore, by combining Equation (28),Equation (31) and Equation (38), we can obtain:

where η inner = 48 µτ ln N N , η inter = 2 2 τ , N = |D| and k = H × W . This completed the proof.

C Proof for Propositions of Tail-class Memory Bank

Restate of Proposition 1. Consider a dataset D that includes images with K different pixel categories. Let p i represent the probability of observing a pixel with label i in a given image. Randomly select B images from D as training data, where

Then with probability at least 1 -δ , for any c ∈ [ K ] , there exists X in the training data that contains pixels of label c .

Proof. Define event A j as the extraction of N images where the pixels of class j appear at least once. Let p m = min { p 1 , p 2 , . . . , p K } , where m ∈ [ K ] .

The probability that each category appears at least once when randomly selecting N images:

Therefore,

This completed the proof.

D Details of T-Memory Bank Algorithm

In this section, Algorithm 2 provides a full version of Algorithm 1.

̸

E More Discussions about T-Memory Bank

E.1 Discussion on the Improved Version of Stratified Sampling

In this section, we introduce the definition of the improved version of stratified sampling and explain why it is not applicable to the PLSS task.

The definition of the improved version of stratified sampling. The improved version of stratified sampling starts by grouping images according to their categories. Due to the multi-label nature, different categories may contain the same images. After that, stratified sampling is applied to each category, making sure that even if an image is sampled more than once (as both head and tail), each mini-batch still includes at least one sample from every category.

Reasons for inapplicability to PLSS task. Conventional stratified sampling can hardly cover all the involved classes with a small batch size. To ensure coverage, one has to employ a much larger batch size, which results in a significantly higher computational burden. Although the improved version of stratified sampling can cover all classes, images from tail classes may appear repeatedly, leading to overfitting and consequently a degradation in performance. In contrast, our Tail-class Memory Bank only involves pasting a portion of one image onto another, effectively functioning as an implicit data augmentation. This approach mitigates the sampling problem without compromising generalization ability. The following empirical results support our assertion.

First, we counted the number of images containing pixels from each class in the Cityscapes, ADE20K, and COCO-Stuff 164K datasets. The results are shown in Table 5, Table 6, and Table 7.

The results indicate that images containing tail class pixels are very limited. Specifically, in the ADE20K dataset, there are only 41 images in the training set of 20210 images that contain pixels from the tail class with ID 97 . As a result, tail class images are repeatedly sampled when using stratified sampling, leading to overfitting on such repeated images.

Next, we trained on the ADE20K dataset using the improved version of the stratified sampling method. The results show that, compared to using the Tail-class Memory Bank, the performance on tail classes dropped by over 3% due to heavy sample repetition in the batch. However, our T-Memory Bank, with its random pasting technique, diversifies the backgrounds of the tail classes, enabling the model to better learn the features of these tail classes.

E.2 Discussion on Why the T-Memory Bank Works

In this section, we discuss why the primary function of the T-Memory Bank is not to enhance the diversity of tail samples.

As shown in Table 3, using a memory bank does indeed increase the diversity of tail classes as an implicit form of augmentation (comparing the rows for SegNeXt and SegNeXt+TMB in the table). However, we cannot rely solely on the bank to fully address the long-tail issue, as the bank's capacity

is always limited. This is why we also need to consider the problem from the perspective of the loss function. We find that the AUC loss focuses only on the ranking loss between positive and negative samples and is not sensitive to data distribution, fundamentally avoiding the risk of underfitting caused by insufficient training samples. We believe that the use of the T-Memory Bank is intended to both facilitate the effectiveness of the AUC loss and enhance the diversity of tail samples.

E.3 Discussion on AUC and Contrastive Learning from the Perspective of the Loss Function

In this section, we compare AUC and contrastive learning from the perspective of loss functions.

Theorem 2 (Comparison between AUC Loss and Contrastive Loss) . Minimizing the weighted contrastive loss approximately corresponds to minimizing an upper bound of the logistic AUC loss:

̸

̸

̸

Proof. For the AUC loss under the logistic surrogate loss function:

̸

̸

̸

where, w j = 1 /n i n j ∑ k = i 1 /n i n k = 1 /n j ∑ k = i 1 /n k and w i = ∑ k = i 1 /n k n i .

̸

This completed the proof.

The theorem indicates that minimizing a weighted version of contrastive loss can implicitly optimize the ovo logistic AUC loss. This paper adopts a more general form of AUC loss, in which various surrogate loss functions are explored.

̸

̸

̸

̸

̸

̸

̸

F Additional Experimental Settings

In this section, we make a supplementation to Section 5.1.

F.1 Datasets

We use three datasets in our experiments: Cityscapes, ADE20K, and COCO-Stuff 164K.

Cityscapes [19] is a dataset of urban road traffic scenes, with each image sized at 1024 × 2048 pixels. It consists of 5000 images with pixel-level annotations across 19 classes. The training, validation, and testing set numbers are 2975 , 500 , and 1525 , respectively.

ADE20K [109] is a benchmark for scene parsing with 150 class labels. It includes over 25000 images, with 20210 , 2000 , and 3352 images used for training, validation, and testing, respectively.

COCO-Stuff 164K [9] is a large-scale dataset with 164 K images. It is finely annotated across 171 classes.

We split the three datasets into head class, middle class, and tail class based on the proportion of pixels for each class in the training set. Table 8, Table 9, and Table 10 provide the details of these partitions.

F.2 Implementation Details

Network Architecture. We perform all experiments using mmsegmentation [18] on an NVIDIA 3090 GPU. For our model, we use SegNeXt [32] as the backbone and pretrain all encoders on the ImageNet-1K [22] dataset.

Data Augmentation. For Cityscapes, we resize the images to 1024 × 2048 , randomly crop them to 1024 × 1024 , and then apply random horizontal flips. For ADE20K and COCO-Stuff 164K, the resizing is set to 512 × 2048 , random cropping is done at 512 × 512 , and random horizontal flips are applied as well.

Training Strategy. We use Adam with Weight Decay (AdamW) [59] optimizer with an initial learning rate of 6 e -5 and a weight decay of 0 . 01 . We adopt the 'poly' learning rate policy, where the initial learning rate is multiplied by 1 -iter max _ iter . Moreover, a 'linear' warmup strategy is employed at the beginning of training, allowing the learning rate to increase from 1 e -6 to the initial learning rate within 1500 iterations. The batch size is set to 2 for the Cityscapes dataset and 4 for all the other datasets. The total number of iterations is 160000 on Cityscapes and ADE20K and 80000 on COCO-Stuff 164K.

Evaluation Metrics. Following the setup outlined by SegNeXt [32], we conduct experiments using the mean of Intersection over Union (mIoU) as the evaluation metric on the validation set.

F.3 Competitors

Here we give a more detailed summary of the competitors mentioned in the experiments.

We compared our method with 13 recent advancements and 6 long-tail approaches in semantic segmentation. The recent advancements include DeepLabV3+, EncNet, FastFCN, EMANet, DANet, HRNet, OCRNet, DNLNet, PointRend, BiSeNetV2, ISANet, STDC, and SegNeXt. The long-tail methods are VS, LA, LDAM, Focal Loss, DisAlign, and BLV, all based on SegNeXt. To ensure fairness, we re-implement the listed methods using their publicly shared code and test them on the same hardware.

For the semantic segmentation methods:

DeepLabV3+ [14] combines the advantages of the spatial pyramid pooling module and the encoderdecoder structure. It explores the Xception model and applies depthwise separable convolution to both the atrous spatial pyramid pooling and decoder modules, resulting in a faster and more robust encoder-decoder network.

EncNet [102] enhances semantic segmentation by utilizing a context encoding module that captures global contextual information to aid in the accurate segmentation of complex scenes.

FastFCN [80] employs a novel joint upsampling module called Joint Pyramid Upsampling, which transforms the task of extracting high-resolution feature maps into a joint upsampling challenge.

EMANet [55] enhances semantic segmentation by utilizing an EM-based attention mechanism that iteratively refines feature representations for more accurate segmentation.

DANet [27] improves scene segmentation by integrating both spatial and channel-wise attention mechanisms to capture rich contextual relationships across features.

HRNet [71] maintains high-resolution representations through the network and progressively adds lower-resolution subnetworks to enhance the learning of spatial hierarchies, significantly improving semantic segmentation.

OCRNet [97] enhances semantic segmentation by leveraging object-contextual representations, which aggregates contextual information around each pixel to improve segmentation accuracy.

DNLNet [93] improves performance on tasks like image classification by disentangling the traditional non-local operations into two separate streams for capturing spatial and channel dependencies separately.

PointRend [50] introduces a novel rendering-style algorithm that selectively refines segmentation predictions at adaptively sampled points, enhancing detail accuracy in image segmentation tasks.

BiSeNetV2 [95] utilizes an efficient architecture with inverted residuals and linear bottlenecks, enabling high-performance mobile vision applications with significantly reduced computational cost.

ISANet [98] improves semantic segmentation by using a novel interlaced sparse self-attention mechanism that efficiently captures long-range dependencies with fewer parameters and computational overhead.

STDC [25] addresses real-time semantic segmentation by proposing a novel and efficient structure that removes structural redundancy, reduces dimensions of feature maps gradually, and uses their aggregation for image representation.

SegNeXt [32] rethinks convolutional attention design for semantic segmentation by introducing an advanced network architecture, enhancing the model's ability to focus on relevant features for more accurate segmentation.

For the long-tail methods:

VS [49] proposes to leverage both multiplicative and additive logit adjustments to address label imbalance problems.

LA [61] advances the conventional softmax cross-entropy by ensuring Fisher consistency in minimizing the balanced error.

LDAM [10] improves the performance of tail classes by encouraging larger margins for tail classes.

Focal Loss [56] is a modified cross-entropy loss designed to address class imbalance by focusing more on hard-to-classify examples, reducing the relative loss for well-classified instances and thus boosting performance on imbalanced datasets.

DisAlign [104] introduces a unified framework for long-tail visual recognition by aligning feature distributions across different classes, using a novel distribution alignment technique that adjusts class-specific thresholds to mitigate the bias towards head classes and enhance recognition of tail classes.

BLV [77] addresses long-tailed semantic segmentation by dynamically adjusting the learning rates for the logits of different classes based on their frequency, effectively reducing the performance gap between head and tail classes.

G Additional Experimental Results

G.1 Per-tail-class Results

In Figure 6, we present the results for each tail class. We select 7 classes with the fewest training samples from the Cityscapes dataset as tail classes: truck, bicycle, bus, train, traffic light, rider, and motorcycle. Our method not only achieves the highest overall mIoU but also shows significant improvements in these tail classes. Specifically, it outperforms the current SOTA method by more than 1% in several tail classes.

Overall

Tail

Truck

Bicycle

Bus

Train

Traffic Light

Rider

Motorcycle

30

40

50

60

70

80

90

100

Classes Name

mIoU (%)

DeepLabV3+

EMANet

STDC

SegNeXt

AUCSeg (Ours)

G.2 Performance Differences Across Different Datasets

The performance gain depends on the degree of imbalance of the underlying dataset. To see this, we show the pairwise mean imbalance ratio r m (average the imbalance ratio of each class pair).

where C h represents the set containing the pixel counts of each head class, and C h denotes the set containing the pixel counts of each non-head class. The larger the r m value, the more imbalanced the dataset is.

In Table 11, we compare r m for ADE20K, Cityscapes, and COCO-Stuff 164K, along with the tail classes performance improvements of AUCSeg compared to the runner-up method.

The results suggest that the larger the imbalance degree the larger the improvements of our method. ADE20K has the largest degree imbalance, therefore gaining the most significant improvement.

G.3 More Qualitative Results

Here we present more qualitative results on the Cityscapes, ADE20K, and COCO-Stuff 164K validation sets.

Input

Ground Truth

DeepLabV3+

SegNeXt

AUCSeg (Ours)

Input

Ground Truth

DeepLabV3+

SegNeXt

AUCSeg (Ours)

Input

Ground Truth

DeepLabV3+

SegNeXt

AUCSeg (Ours)

G.4 Backbone Extension of Different Model Sizes

In this paper, we use the large version of SegNeXt because of its outstanding performance. We also provide the results for the tiny, small, and base versions of SegNeXt, as shown in Table 12. The experiments indicate that AUCSeg achieves better performance under any model size.

G.5 Backbone Extension of Different Pixel-level Long-tail Problems

Apart from semantic segmentation, salient object detection is also a pixel-level task. In salient object detection, the salient objects often exhibit a long-tailed distribution. We apply AUCSeg to this task, using the latest SOTA method SI-SOD-EDN [81, 53] as the backbone. The results are shown in Table 13.

Our AUCSeg achieves improvements across three commonly used evaluation metrics on three datasets, demonstrating that our method is highly versatile and extensible.

G.6 Spatial Resource Consumption

In this section, we thoroughly explore the spatial resource consumption of the T-Memory Bank.

Table 14 details the ablation experiments on the spatial resource use of the T-Memory Bank. We set the memory size S M to 5 and conduct experiments on the ADE20K dataset. It is no longer necessary for samples of all classes to appear in the mini-batch, but rather only a minimum of 5 tail class samples are needed. The results demonstrate that using AUC alone requires a batch size and graphics memory 5.5 times greater than the baseline to satisfy computational demands, which is a significant expense. However, the T-Memory Bank substantially reduces this cost, enabling more efficient training without an intolerant increase in graphics memory.

Moreover, we explore the effect of memory size S M on spatial resource consumption. The findings in Figure 8 show that the graphics memory occupation increases slightly as S M rises. However, a smaller memory size generally suffices for effective performance, indicating that AUCSeg can achieve significant improvements with a manageable graphics memory burden. As shown in Figure 5a, when S M = 5 , significant performance improvements can be achieved with lower spatial resource consumption.

S M

=

1

S M

=

3

S M

=

5

S M

=

8

S M

=

10

S M

=

20

2GB

4GB

6GB

8GB

10GB

12GB

14GB

16GB

18GB

G.7 Results of Different AUC Surrogate Losses and Calculation Methods

AUCSeg can adopt various surrogate losses. In the previous section, we use the square loss. In Table 15, we explore two other popular surrogate losses (hinge loss and exponential loss) for AUCSeg. Additionally, we include results for two AUC loss calculation methods (one-vs-one and one-vs-all) applied to AUCSeg using the square loss. The results are presented in Table 16.

The results indicate that AUCSeg shows improved performance with any of the surrogate functions. Among them, using square loss and the ovo calculation method delivers the best overall performance.

G.8 Results of the Comparison Between PMB and TMB

There are two differences between the Pixel-level Memory Bank (PMB) and our Tail-class Memory Bank (TMB). First , the PMB stores pixels from all classes, whereas TMB only stores pixels from tail classes. Second , in TMB, the storing and retrieving processes are conducted on an entire object (we ensure that the pasted pixel forms a meaningful object). However, the PMB typically focuses on a fixed number of pixels without structural information (regardless of whether these pixels can form a complete image).

Why do we only store tail class pixels instead of all pixels?

Table 17 shows the average number of pixels from head and tail classes per image in the ADE20K, Cityscapes, and COCO-Stuff 164K datasets.

It can be observed that the number of head class pixels in each image is 2 . 46 to 9 . 45 times greater than that of the tail classes, meaning that storing head class pixels would require significantly more memory.

Table 18 compares the performance differences between storing all and tail class pixels. It shows that the PMB, which incurs additional memory costs, performs almost the same as the TMB, and even shows a noticeable decline in the Cityscapes dataset. This is because head classes appear in almost every image (for example, in urban road datasets, it is hard to find an image without head class pixels like 'road' or 'sky'), so they do not need additional supplementation. Even if some images require supplementation of head classes, their larger pixel counts might cause them to overwrite the original tail class pixels when pasted, leading to a decline in performance. Thus, we only store tail class pixels.

Why it is not feasible to focus on a fixed number of pixels?

We conduct tests on the ADE20K dataset by supplementing a fixed number of tail class pixels ( 10000 / 20000 / 30000 / 40000 ) in each image and find that compared to AUCSeg, the performance differences are -3 . 00% / -1 . 93% / -0 . 86% / -0 . 73% . This is because supplementing a fixed number of pixels can result in incomplete images, such as only adding the front wheel of a bicycle, therefore loss of the structural information. The model is then unable to learn complete and accurate features. Therefore, in TMB, storing and retrieving are conducted on all pixels of an entire image.

G.9 Results of Different Memory Bank Update Strategies

In the previous section, we use the random replacement strategy to update the T-Memory Bank. We experiment with three other selection methods on the ADE20K dataset:

First-In-First-Out (FIFO) replacement : Prioritizes replacing the images that were first stored in the Tail-class Memory Bank.

Last-In-First-Out (LIFO) replacement : Prioritizes replacing the images that were last stored in the Tail-class Memory Bank.

Priority Used (PU) replacement : Prioritizes replacing images that have previously been selected by the retrieval branch.

The results are shown in table 19.

FIFO and PU both show better performance overall and on tail classes compared to random sampling. However, LIFO, by updating only the most recently added images in the T-Memory Bank, causes the earlier images to remain unchanged. This leads to overfitting and, consequently, a decline in performance.

While these complex strategies can improve performance, the gains are relatively limited. On the other hand, the random replacement method is easy to implement. Exploring more complex and effective replacement methods could be a promising direction for future work.

G.10 Detailed Results of the Ablation Study on Hyper-Parameters

The detailed results of the ablation study on hyper-parameters are shown in Table 20, Table 21, Table 22, and Table 23.

Explanation of the reasons why performance improvement decreases as memory size increases. This is a trade-off between diversity and learnability. When the memory size is too large, the probability of any single sample being effectively learned decreases. So the model fails to focus on important examples, and thus fails to capture their features, ultimately leading to underfitting. Conversely, if the memory size is too small, the diversity of samples is limited, which leads to overfitting. Hence, there is no free lunch for increasing the bank.

Therefore, we pursue a reasonable memory size. As shown in Table 20, we believe that a memory size of 5 is suitable in most cases. As the memory size increases/decreases, the performance slightly declines due to model overfitting/underfitting.

G.11 Results of the Ablation Study on the Impact of Batch Size

We conduct ablation experiments on the ADE20K dataset to evaluate the impact of batch size. The results are shown in Table 24:

The performance improves as the batch size increases. Moreover, our AUCSeg is consistently effective across different batch sizes.

H More Discussions About AUCSeg

Training and inference efficiency. During training, since AUCSeg ( 1 . 62 ± 0 . 2 s per iteration) adopts pairwise loss, it will inevitably suffer from extra complexity compared to the standard CE-based SegNeXt ( 0 . 76 ± 0 . 15 s per iteration). Besides, during inference, since AUCSeg does not modify the model's backbone, all algorithms with the same backbone achieve similar inference efficiency ( 60 ± 5 ms per image). Overall, AUCSeg could perform well with acceptable efficiency.

Performance trade-off between head and tail classes. We recognize that AUCSeg might slightly impair the performance of head classes due to an increased focus on tail classes. However, this often results in substantial improvements for tail classes, a trade-off that is generally beneficial since tail classes are typically more critical. For instance, on the Cityscapes dataset, AUCSeg exhibits a marginal decrease of 0 . 17% in head classes but gains 0 . 75% in tail classes compared to the runner-up, SegNeXt. Furthermore, we note that performance decreases in head classes mainly arise from misclassification at the blurry edges of objects. In contrast, gains in tail classes often stem from either the successful detection of smaller objects or the more complete detection of such objects. To summarize, on the Cityscapes datasets, detecting a new tail object like 'Traffic Lights' is far more significant than precisely detecting the edge pixels of a head class like 'sky'. Thus, we consider this trade-off highly beneficial.

I Broader Impact

We propose a general semantic segmentation method to deal with the potential bias toward long-tail objects. For fairness-sensitive scenarios, it might be helpful to improve fairness for long-tail groups.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We briefly summarize it in the abstract and detail the paper's contributions and scope in the introduction.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discuss this issue in Appendix H.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We provide complete proofs for each theoretical result in Appendix B and Appendix C.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We describe our algorithm in Section 4 and fully disclose all the information needed to reproduce the main experimental results of the paper in Section 5.1 and Appendix F.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We provide a link to the data and code in the abstract.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/pu blic/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https://nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We provide the completed experimental setting in Appendix F.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We report this issue in Section 5.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We provide sufficient information on the computer resources in Appendix F.2. Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research conducted in the paper conforms in every respect with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We discuss this issue in Appendix I.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper does not release data or models that have a high risk for misuse.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We cite the original paper that produced the code package and dataset.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: We provide the code with documentation in the link within the abstract.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing and research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing and research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors extend AUC optimization techniques to pixel-level long-tail semantic segmentation and propose a general pixel-level AUC loss function. They decompose the loss function into inner-image and inter-image terms to decouple the interdependency, and calculate bounds to theoretically prove the effectiveness of this loss. Additionally, they design a Tail-Classes Memory Bank to reduce memory demand. Qualitative and quantitative comparisons have been made on three benchmark datasets, showing the efficiency of their method.\n\nSTRENGTHS:\n1. The paper is clearly organized and well written.\n2. The general idea of the paper is clear, and the motivation is interesting.\n3. The authors provide detailed theoretical proofs.\n\nWEAKNESSES:\n1. In the tail-class memory bank module, how do you ensure that pasted tail pixels do not completely cover any category?\n2. As shown in Fig 3(b), authors argue that because images contain multiple labels, it is impossible to use stratified sampling techniques to ensure each mini-batch contains at least one sample from each class. However, I believe this view is incorrect; we can group images by category, different categories may include identical images due to multi-label nature. Then we can use stratified sampling technique for each category so that although it might sample identical images (the same sample being sampled twice as both head and tail), it still ensures each mini-batch contains at least one sample from every class,  and small batch sizes can also achieve this goal. Therefore I think there’s no need for designing a Tail-Class Memory Bank.\n3. How does Tail-Class Memory Bank store data? Does it directly use ground truth or predicted results? If using ground truth directly why design Store Branch? You could directly build memory bank using ground truth. If using predictions then why not just use ground truths?\n4. Pasting tail-class pixels onto original image might occlude original image classes; would this behavior affect performance?\n5. This method mainly relies on SegNeXt; while sometimes environmental factors prevent exact replication of other works' results. However, SegNeXt provides training logs & pre-trained weights, why did your reproduction show significant performance gaps compared with SegNeXt's original work (e.g., nearly 4% gap on ADE20K)?\n6. In Table 1, this method shows significant improvement only on ADE20K but limited gains across two other datasets suggesting poor real-world generalizability inconsistent with theoretical conclusions.\n7. Currently effective methods trained over small datasets often fail scaling up large-scale models. Hence the author should further provide the results with large-scale pretrained models like SAM/CLIP etc., demonstrating broader applicability/generalization potentiality thereof.\n\nQUESTIONS:\n1. I am confused about Tail-Class Memory Bank design, please refer weaknesses section above explaining them accordingly.\n2.  Authors should clarify certain experimental outcomes referring weakness points mentioned earlier.\n3. NeurIPS Paper Checklist lacks honest responses e.g., code provision/error bars reporting/new assets do not be contained, but all marked “YES”.\n\nLIMITATIONS:\nThe authors have addressed the limitation of this method.  This method might slightly impair the performance of head classes due to an increased focus on tail classes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper explores AUC optimization methods for pixel-level long-tail semantic segmentation, addressing complex dependencies and space complexity challenges. The authors propose a novel pixel-level AUC loss function and conduct a dependency-graph-based theoretical analysis to enhance generalization. They also introduce a Tail-Classes Memory Bank (T-Memory Bank) to manage memory demands, and experiments confirm the effectiveness of their AUCSeg method.\n\nSTRENGTHS:\n1. This paper presents a novel perspective by using AUC as the loss function.\n2. The paper has performed a tail class cache to boost the performance and control the memory demand.\n\nWEAKNESSES:\nPlease refer to the questions.\n\nQUESTIONS:\n1. For a long-tail semantic segmentation problem, which is also a pixel-wise classification task, I am uncertain why the authors chose this perspective. From the standpoint of the loss function, it emphasizes the contrast between the current class and other classes. Would a contrastive classification approach achieve similar results?\n2. The significance of Proposition 1 is not very clear. What is the meaning of the $\\Omega$? My understanding is that for a tail class, it is naturally difficult to randomly sample such a sample due to the scarcity. Therefore, re-sampling techniques are needed, which seems quite intuitive. What additional information is this proposition supposed to convey?\n3. The logic of using a memory bank in this paper seems weired. The authors argue that it is difficult to randomly sample a tail sample, hence the need for a memory bank. Most long-tail based work considers batch re-sampling a very conventional approach. However, this paper's method enhances the diversity of tail samples, thereby improving the model's learning ability for tail samples. Perhaps rephrasing this aspect would be more effective.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates AUC optimization within the context of pixel-level long-tail semantic segmentation (PLSS), a complex task due to intricate loss term coupling and extensive memory requirements. Initially, the authors demonstrate the potential of AUC for PLSS from a theoretical perspective by addressing the two-layer coupling issue across loss terms. Subsequently, they propose a novel Tail-Classes Memory Bank (T-Memory Bank) to manage the significant memory demands of AUC-oriented PLSS. Finally, comprehensive experiments show the effectiveness of the proposed method.\n\nSTRENGTHS:\n1. This paper addresses a compelling problem with clear motivations and significant contributions, offering insightful ideas for both the AUC and segmentation communities. \n\n2. Developing a theoretically grounded loss from an AUC perspective is novel, and the proposed T-memory bank effectively mitigates the memory burden associated with pixel-level AUC optimization. \n\n3. The performance of the proposed AUCSeg seems promising.\n\nWEAKNESSES:\nOverall, I believe this is a qualified paper for publication after addressing the following minor concerns:\n\n1. The notations in this paper should be carefully defined. Some key symbols are used repeatedly, such as $N$ representing both batch size (In Alg.1) and sample size (In Thm.1).\n\n2. This paper only examines the square AUC surrogate loss without considering two other popular surrogate losses for AUC optimizations (i.e., hinge and exponential losses). It is essential to provide empirical verification by applying the ignored losses to AUCSeg.\n\n3. In Alg.1, the batch size scale will also impact the final performance of long-tailed classifications. The authors are recommended to conduct an ablation study on this aspect.\n\nQUESTIONS:\nPlease refer to the weakness part.\n\nLIMITATIONS:\nNone", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-11"}

comments: SUMMARY:
This paper introduces AUC optimization into the domain of long-tailed semantic segmentation. Specifically, the authors developed a pixel-level AUC loss function tailored for long-tailed semantic segmentation tasks and introduced a tail-class memory bank to address the memory demands. Additionally, the authors utilized Rademacher complexity to provide a generalization bound for AUCSeg and theoretically analyzed the potential for AUCSeg to generalize to unseen data. 

The experimental evaluation in this paper assessed the performance of the proposed strategy on various datasets. Generally speaking, this paper is relatively detailed and comprehensive.

STRENGTHS:
- It is novel to introduce auc optimization in long-tailed semantic segmentation and give a detailed analysis over the feasibility of this method.

- The proposed method is validated on different datasets(ADE20k, CityScapes, COCO etc.) and has been proven to be effective in tail classes.

WEAKNESSES:
There are a few concerns about this paper:

- On COCO, the performance improvement of Tail Class seems to be limited. Could the author explain the reason?

- Fig. 6(a) shows the memory size reduction of the tail-class memory bank. The results show that as the memory size increases, the performance improvement decreases. What is the reason for this?

QUESTIONS:
I thank the author for their detailed work. Nevertheless, there are a few questions I wonder and I hope the author could respond.

- For Tail Class Memory Bank, what would happen if we use a pixel-level memory bank? For example, prototype or similar technology, would this lead to significant performance degradation?

- For the memory bank update problem, is there a more appropriate selection method instead of random replacement?

LIMITATIONS:
yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-12