Title: Revive Re-weighting in Imbalanced Learning by Density Ratio Estimation
Venue: NeurIPS 2024
Accepted: yes
Keywords: machine learning

density ratio estimation

optimization
Abstract: In deep learning, model performance often deteriorates when trained on highly imbalanced datasets, especially when evaluation metrics require robust generalization across underrepresented classes. To address the challenges posed by imbalanced data distributions, this study introduces a novel method utilizing density ratio estimation for dynamic class weight adjustment, termed as Re-weighting with Density Ratio (RDR). Our method adaptively adjusts the importance of each class during training, mitigates overfitting on dominant classes and enhances model adaptability across diverse datasets. Extensive experiments conducted on various large scale benchmark datasets validate the effectiveness of our method. Results demonstrate substantial improvements in generalization capabilities, particularly under severely imbalanced conditions.

Revive Re-weighting in Imbalanced Learning by Density Ratio Estimation

Abstract

In deep learning, model performance often deteriorates when trained on highly imbalanced datasets, especially when evaluation metrics require robust generalization across underrepresented classes. To address the challenges posed by imbalanced data distributions, this study introduces a novel method utilizing density ratio estimation for dynamic class weight adjustment, termed as Re-weighting with Density Ratio (RDR). Our method adaptively adjusts the importance of each class during training, mitigates overfitting on dominant classes and enhances model adaptability across diverse datasets. Extensive experiments conducted on various large scale benchmark datasets validate the effectiveness of our method. Results demonstrate substantial improvements in generalization capabilities, particularly under severely imbalanced conditions. The code is available here.

1 Introduction

In recent years, deep learning has made significant strides across various domains by utilizing complex architectures and large-scale datasets, setting new benchmarks for performance. However, these advancements often rely on well-curated datasets that ensure balanced class distributions [Russakovsky et al., 2015]. In contrast, real-world datasets typically exhibit a long-tailed distribution, where few classes dominate the majority of samples, while many others are underrepresented [Krizhevsky et al., 2009]. This imbalance leads to model biases favoring frequent classes, thereby reducing performance on the less common ones. Yet, in many applications-such as medical diagnostics and financial analysis-greater emphasis is placed on ensuring strong generalization for underrepresented classes. Addressing this challenge not only reduces data collection costs but also improves the robustness and fairness of the models.

Many excellent methods, such as re-sampling [Bowyer et al., 2011], re-weighting [Morik et al., 1999], decoupled learning [Kang et al., 2020], margin-based learning [Cao et al., 2019, Menon et al., 2020], transfer learning [Yin et al., 2019] and contrastive learning [Tian et al., 2021], have been proposed to tackle the issue of imbalanced data. Despite the simplicity of re-weighting, it falls behind in performance significantly compared with other directions of methods due to the inappropriate weighting coefficients during training. Cui et al. [2019] proposes a method for re-weighting by effective number, which accounts for potential overlaps among data samples and adjusts the weights for each category based on the actual effective number of samples. Chen et al. [2023b] leverages the effective area to re-weight, considering the actual spanned space of each class. However, such subsequent improvements can alleviate but still cannot effectively push that forward. Wang et al.

[2023] obtains a fine-grained generalization bound for re-weighting in imbalanced learning through the data-dependent contraction technique. Limited research has focused on the intrinsic limitations of the commonly employed re-weighting-based loss functions and the corresponding balancing mechanisms designed to enhance parity in class representation.

This study rethinks the characteristics of re-weighted loss and explores the question "Why is reweighting necessary under conditions of sample imbalance?" Under conditions of sample imbalance, the variation in weights of samples arises due to discrepancies between the distribution of collected data and a balanced data distribution. In scenarios where class balance exists, such discrepancies are absent, thus obviating the need for re-weighting. Conversely, in imbalanced settings, re-weighting becomes essential to bridge the gap between these distributions. The weights must therefore represent a suitable compromise between balanced and imbalanced distributions and necessarily reflect accurately on each sample. Additionally, as model training progresses dynamically, optimizing the fit to feature distributions, the weights applied to each sample should be continuously updated to maintain robust performance.

This research introduces a novel method, Reweighting with Density Ratio (RDR), designed to mitigate learning disparities in imbalanced distributions. In this method, a feature extractor is employed to discern the features from the training data. A more balanced feature distribution is approximated by continuously updating the momentum on the feature level. This enables real-time density ratio estimation with features learned under imbalanced distributions, thereby obtaining the sample-wise weights. Notably, as the learned features evolve, our method dynamically adjusts weights in response to observed shifts in class density throughout the training cycle, ensuring that the model remains adaptive and effective. This method significantly enhances the robustness and adaptability of the training process. By integrating density ratio estimation to evaluate the difference between the balanced and real data distributions, our approach more accurately reflects the underlying class distribution and improves the model's generalization capabilities across diverse datasets. The contributions are summarized as follows:

We explore the existing re-weighting techniques, and model the performance of various algorithms during training under different data distributions. This approach offers a novel perspective on understanding re-weighting methods in the scenarios of sample imbalance.

We introduce a novel methodology, Re-weighting with Density Ratio (RDR), which leverages the method of density ratio estimation to dynamically adjust class weights during model training. This approach not only addresses the limitations of prior re-weighting methods but also introduces a mechanism to continuously adapt to the changing importance of classes as learning progresses, thereby enhancing model robustness and adaptability.

Weconduct extensive experiments to validate the effectiveness of our proposed RDR method. These experiments are conducted across various large-scale, long-tailed datasets, demonstrating substantial improvements in handling class imbalance. Our results illustrate significant enhancements in generalization capabilities, particularly under severely imbalanced scenarios.

2 Related Work

2.1 Re-weighting Based Methods

Re-weighting methods for addressing class imbalance have evolved significantly over the years. Early techniques, such as [Zadrozny et al., 2003], employed inverse frequency techniques to address class imbalances but failed to consider deeper data distribution traits, leading to sub-optimal outcomes. Addressing these shortcomings, Huang et al. [2016] introduced a cost-sensitive learning framework that, beyond simple frequency adjustments, incorporated misclassification costs to achieve a more nuanced balance. However, this approach still struggled with complexities like class overlap and label noise. To further refine this approach, Lin et al. [2017] developed Focal Loss, which employs a modulation factor based on the prediction probability to adjust the loss function, thereby amplifying the impact of hard-to-classify samples while reducing the loss contribution of easy-to-classify samples. Cui et al. [2019] introduced Class-Balanced Loss, which adjusts loss by data overlap, calculating the effective number of each class. Advancements continued with methods based on training gradients, such as [Ren et al., 2020b, Wang et al., 2021a]. Chen et al. [2023b] proposed Adaptive Re-weighting via effective area, which enhances model accuracy by considering the spatial distribution and density

of data points within classes. Ma et al. [2023] introduced a re-weighting method that adjusts based on semantic richness and visual variability. However, no prior work has tackled the issue of sample imbalance by dynamically re-weighting based on the model's performance across training and test sets with differing distributions during the training process.

2.2 Non-re-weighting Based Methods

In addition to re-weighting, many other methods are available to address the issue of sample imbalance. Re-sampling techniques [Kubat and Matwin, 1997, Wallace et al., 2011, Han et al., 2005, Hong et al., 2024a] mitigate category imbalances by under-sampling dominant classes [Buda et al., 2018] or over-sampling minority classes [Bowyer et al., 2011]. However, under-sampling may degrade feature representation by discarding valuable majority class data, whereas over-sampling could cause overfitting by duplicating minority class samples. Decoupled training approaches, such as [Kang et al., 2020], challenge the traditional joint training model by separating representation learning from classification. Margin-based methods such as, LADM [Cao et al., 2019], LA [Menon et al., 2020] and VS [Kini et al., 2021], adjust training processes to increase minority class margins, therefore to obtain a more balanced decision boundary. More flexible and robust methods are proposed, including Transfer Learning [Yin et al., 2019, Liu et al., 2019], Contrastive Learning [Li et al., 2022, Chen et al., 2023a], Ensemble Learning [Wang et al., 2021b, Cai et al., 2021] and Self-supervised Learning [Liu et al., 2022, Zhou et al., 2023b]. Please refer to Appendix B for more discussions.

3 Method

3.1 Problem Setup

For a typical classification task in imbalanced learning, suppose given a training dataset S = ⋃ n i =1 { ( x i , y i ) } , where n is the total number of samples. Denote { n 1 , n 2 , ..., n d } as the sample number of each class. We assume, without loss of generality, that n i < n j , when i < j , with n d typically much larger than n 1 , reflecting a pronounced imbalance in class distribution. We use a typical loss function like l : W × X × Y → R + . Denote { π 1 , π 2 , ..., π d } as the proportions of each class, such that ∑ d i =1 π i = 1 . Define a family of deep learning models parameterized by ω ∈ W ⊆ R k . Typically, a model consists of a feature extractor f ( x ; ϕ ) and a classifier h ( z ; θ ) , with ω = ⋃ { ϕ, θ } . The notations used in this paper are summarized in Appendix A.

3.2 Motivation

Inspired by our review of prior methods, we observe a gap in the adaptation of dynamic class-weight adjustments during training phases. Building on the groundwork of static re-weighting strategies, we introduce a novel approach utilizing the method of density ratio estimation to dynamically recalibrate class weights. This innovation aims to provide a more refined adjustment by estimating real-time class density, thereby promoting an equitable influence of all classes throughout the training.

3.3 Dynamic Re-weighting with Density Ratio

In a typical training optimization problem, our objective is to minimize the empirical risk of the loss function, i.e. , R = 1 n ∑ n i =1 l ( x i , y i ; ω ) . However, in imbalanced datasets, where the frequency of samples across different classes varies, it is necessary to adjust for these gaps by applying different weights for the samples. We assume that the weight of each sample is denoted by α ( x, y ; ω ) , then the empirical risk can be formulated like R = 1 n ∑ n i =1 α ( x i , y i ; ω ) l ( x i , y i ; ω ) .

In naive re-weighting approaches, the weight α of class y is often set to 1 π y . This setting is based on the assumption that the distribution of the training set P and the distribution balanced data set P bal satisfy the equation P ( x | y ; ω ) = P bal ( x | y ; ω ) . However, in practical training scenarios, both training and test sets are subsets drawn from the actual distribution, leading to potential missing of feature patterns. Furthermore, classes with more complex features and lower sample frequencies tend to exhibit more pronounced missing of patterns. Therefore, in the training process, there exists a discrepancy between P ( x | y ; ω ) and P bal ( x | y ; ω ) . Wemeasure the extent of this discrepancy using the ratio r ( x | y ; w ) = P ( x | y ; ω ) /P bal ( x | y ; ω ) , incorporating it as a correction term into our weighting

scheme. Consequently, the empirical risk can be reformulated as follows

We can explain the rationality of this formula as follows. Considering each class i , where P bal ( y i ; ω ) ∝ π -1 i P ( y i ; ω ) and P bal ( x i | y i ; ω ) = r ( x i | y i ; ω ) P ( x i | y i ; ω ) , with conditional probability formula, we can derive:

Eq. (2) demonstrates that our approach aligns with the balanced risk of the loss function. Consequently, minimizing Eq. (1) also serves to minimize the balanced risk.

Let's take a closer look at r ( x | y ; w ) = P ( x | y ; ω ) /P bal ( x | y ; ω ) . The variable r represents the ratio of two different distributions. We approximate this ratio using methods of density ratio estimation. This problem can be solved by first-order moment matching approach. Our goal is to minimize

where ∥·∥ denotes the Euclidean norm. Recall that we capture the features of the input samples by the feature extractor f ( x ; ϕ ) in our model, and these features are a good reflection of what our model learned from the distribution of the input samples. Therefore, in order to capture more complex structures and patterns in raw data, we use f ( x ; ϕ ) to obtain a variant of Eq. (3). Our goal can be achieved by obtaining argmin r MM ′ ( r ) , where MM ′ ( r ) denotes

where MM stands for 'moment matching'. Let us ignore the irrelevant constant in MM ′ ( r ) , and define the rest as MM( r ) :

where ⟨· , ·⟩ denotes the inner product. In practice, as for the real-world imbalanced data distribution P , we denote Φ P to dynamically reflect the knowledge learned from the distribution P , that is Φ P = ( f ( x 1 ; ϕ ) , . . . , f ( x n ; ϕ )) . Remember that the output of feature extractor is z , i.e. , z = f ( x ; ϕ ) is a Z -dimensional vector, then Φ P would be a [ Z, n ] -dimensional vector. Similarly, we denote Φ i P for class i , which is a [ Z, n i ] -dimensional vector. As for the balanced data distribution P bal , we design a momentum mechanism to accumulatively estimate the expectation of features learned from balanced data distribution along with the training. Concretely, for each class, we maintain a prototype feature F for the entire training progress, using each batch's feature expectation for momentum updates. Therefore, we define F P bal as follows F P bal = ( F 1 , . . . , F d ) .

Since the total number of classes is d , F P bal would be a [ Z, d ] -dimensional vector. For each batch, we can obtain z = ( z 1 , . . . , z d ) , where z i the mean of z of all samples in class i . Then, the momentum updates works as follows

where m ∈ [0 , 1) is a momentum coefficient. Back to Eq. (5), replace the expectations over P bal and P by Φ P and F P bal , respectively. Then, take the derivative of MM( r ) with respect to r and set it to zero. Detailed derivations are provided in Appendix C.1. For each class i , we can obtain the estimation of density ratio in imbalanced learning as follows

Epoch

0.025

0.050

0.075

0.100

0.125

0.150

0.175

Value

B

′

Weight of

RDR

(a)

Many

Category

Epoch

0.090

0.092

0.094

0.096

0.098

0.100

0.102

Value

B

′

Weight of

RDR

(b)

Few

Category

Substitute Eq. (7) into Eq. (1), we can obtain our object to optimize

In our implementation, we introduced a warm-up phase to pre-adapt the feature distribution in F P bal , thereby mitigating excessive oscillations during the initial stages of training. Additionally, we employed a temperature coefficient γ to modulate the influence of weights, which is typically set to 1. When integrating with logit adjustment (LA) [Menon et al., 2020], we adhere to the same procedures outlined in [Wang et al., 2023] to ensure the fisher consistency. The framework and pseudo-code of our method are shown in Appendix C.2 and Appendix C.3.

3.4 Generalization Bound Analysis

Here, we use a formal generalization analysis to characterize the interesting point of our method.

Theorem 1. Given a model m ∈ M and the loss function l , for any δ ∈ (0 , 1) , with probability at least 1 -δ over the training set S , according to [Wang et al., 2023], the following generalization bound holds for the risk on the balanced distribution

where Φ( l, δ ) is positively correlated with the empirical re-weighting risk of the training set. C S ( M ) denotes the empirical complexity of the function set M . B y ( f ) denotes the minimal prediction on the ground-truth class y in the training set. w y refers to the weight of class y of the re-weighting loss.

Specifically, from the above generalization bound, we can find two inherent requirements for reweighting methods. 1) Why re-weighting is necessary : w y helps to re-balance the imbalanced term √ π y [1 -softmax ( B y ( m ))] to get a sharper bound. 2) Why dynamic re-weighting is necessary : The term B y ( m ) changes dynamically with model training. Therefore, we need a w y that can adapt dynamically to the changes of B y ( m ) . 3) Why RDR works : From Fig. 1, we can observe that the dynamic trend of the RDR weight aligns well with √ π y [1 -softmax ( B y ( m ))] , denoted as B ′ y . This shows that our RDR can adapt to the dynamics in B ′ y , maintaining a sharp bound during training.

3.5 Implementation and Complexity Analysis

At the end of each epoch, a global variable is maintained and updated using momentum, as described by Eq. (6). Within each minibatch, the weight of each sample is computed dynamically. Typical

optimization procedures for deep neural networks entail both forward and backward passes per minibatch, characterized by a computational complexity of O ( B Λ) , where B represents the batch size and Λ denotes the overall parameter size. Within the RDR framework, suppose the feature dimension used as input to the classifier is K , and the sample weights are computed according to Eq. (8). This computation for all d classes aggregates to a complexity of O ( ∑ d i =1 ( n i × K 2 + K 3 )) where n i is the sample count of class i in a minibatch. Given that K generally exceeds n i , the complexity predominantly stems from the matrix inversion, approximating to O ( dK 3 ) . The complexity for momentum updates is O ( BK ) . Notice that K and B are considerably minor relative to the scale of the model parameters, rendering the time overhead of this method manageable.

On the storage front, the memory cost of the RDR primarily arises from the matrix inversion step in Eq. (8), resulting in a space complexity of O ( K 2 ) . Given the scales of K is much lower than Λ , the extra memory usage is negligible when compared with the memory utilization of the model parameters. To this end, RDR imposes a relatively small computational or space cost, enabling its integration with existing approaches at a reduced cost. An empirical evaluation of the computational expense is presented in Fig. 2. For more discussions about limitations of RDR , please refer to Appendix E.

CE

RDR

SAM

SAM(RDR)

Method

0

500

1000

1500

2000

Time cost(s)

(a) Time Cost on CIFAR-10-LT

CE

RDR

SAM

SAM(RDR)

Method

0

500

1000

1500

2000

Time cost(s)

(b) Time Cost on CIFAR-100-

LT

4 Expriments

4.1 Experimental Setup

Datasets. We conduct experiments on four major long-tailed datasets, CIFAR-10-LT, CIFAR-100-LT, ImageNet-LT [Liu et al., 2019] and Places-LT [Liu et al., 2019]. CIFAR-10-LT and CIFAR-100-LT are two datasets sampled from the original CIFAR [Krizhevsky et al., 2009] dataset with a total of 10 and 100 classes, respectively. We conduct experiments with different imbalance factors IF = n max n min , where n max and n min denotes the number of the most and least frequent classes [Kang et al., 2020, Hong et al., 2023, 2024b]. Following the mainstream protocol [Wang et al., 2023], we set the imbalance factor as 100 and 10 for evaluation. ImageNet-LT has 115.8K training images covering 1000 classes, with imbalance factor being 256. The number of samples per class ranges from 1280 to 5 images. Places-LT contains 62.5K training images covering 365 categories, with imbalance factor being 996. The number of samples per class ranges from 4980 to 5 images.

Evaluation Protocol. In the task of long-tailed classification, all classes are treated equally during testing. Following [Rangwani et al., 2022, Zhou et al., 2023c], we also report accuracy on three splits of classes according to the number of training data. Since the number of samples per class increases by its class index, for CIFAR-10-LT dataset, class[0, 3), class[3, 7) and class[7, 10) are reported as Many , Medium and Few classes, respectively. Similarly, CIFAR-100-LT is splited as class[0, 35), class[35, 69) and class[69, 100). ImageNet-LT is splited as class[0, 390), class[390, 835) and class[835, 1000), while Places-LT is splited as class[0, 131), class[131, 288) and class[288, 365).

Baselines. Our method is combined with existing long-tailed classification methods to demonstrate the efficacy, including the baseline trained by cross-entropy loss (CE), focal loss (Focal) [Lin et al., 2017], class-balanced loss (CB) [Cui et al., 2019] and logit adjustment (LA) [Menon et al., 2020]. Recently, Sharpness-Aware minimization (SAM) [Foret et al., 2021] has been proved to be a powerful method in imbalanced learning, therefore we also adopt baseline including SAM [Foret et al., 2021],

0.3

0.4

0.5

0.6

0.7

0.8

0.9

78

79

80

81

82

83

84

85

Accuracy (%)

m

ImbSAM [Zhou et al., 2023a] and CCSAM [Zhou et al., 2023c], the latter two are also SAM-based methods. The strategies above have been demonstrated superior performance in imbalanced learning.

Implementation details. Our code is implemented with Pytorch 1.12.1. Experiments based on CIFAR-10-LT and CIFAR-100-LT are carried out on NVIDIA GeForce RTX 3090 GPUs, while experiments based on ImageNet-LT and Places-LT are carried out on NVIDIA A100 GPUs. For a fair comparison, we use ResNet32 on CIFAR-10-LT and CIFAR-100-LT, ResNet50 on ImageNet-LT and pre-trained ResNet-152 on Places-LT. We train each model with batch size of 128 (for CIFAR-10-LT and CIFAR-100-LT) / 256 (for Places-LT and ImageNet-LT), SGD optimizer with momentum of 0.9, weight decay of 0.0002. The initial learning rate is set to 0.1, with cosine learning-rate scheduling along training. The results of ImbSAM and CCSAM are obtained by implementing the official codes.

4.2 Comparison Results

Comparative analyses have been performed to evaluate the effectiveness of the proposed RDR. The results are presented in Table 1, Table 2 and Table 4. The metric employed to measure performance is the top-1 accuracy on the test sets.

Results on CIFAR-10-LT and CIFAR-100-LT. We first evaluate RDR on CIFAR-10-LT and CIFAR100-LT. We report the final accuracy of different methods with imbalance factor ratio {10, 100} in Table 1 and Table 2. We can observe that RDR significantly outperforms all baselines under different imbalance factor ratios across the two datasets. Our observations highlight that RDR consistently outperforms baselines across various class distributionsMany , Medium , and Few -particularly under severe imbalance ( IF =100).

In CIFAR-10-LT, combined with CE and LA, our method shows substantial improvement in the categories with fewer samples, increasing the accuracy by 19.8% and 9.5% respectively in the Few category under IF =100. This improvement is notable as it effectively addresses the challenge of learning from scarce data. With the inclusion of SAM, the performance of RDR is further enhanced. Under a less severe imbalance ( IF =10), where the results show less performance drop-off between categories, RDR combined methods still maintain high performance across all categories, suggesting scalability and reliability of our approach in different imbalance contexts.

In CIFAR-100-LT, where the data distributions are more diverse and challenging, RDR also enhances the overall performance, particularly for the Medium and Few categories. Under the imbalance factor of 10 and 100, RDR increases the accuracy in Few classes by 11.8% and 20.0% respectively, compared to the original CE loss. Furthermore, it is notable that techniques like ImbSAM and CCSAM, which especially focus on the Few categories, may heavily sacrifice the performance on Many classes. The results in both datasets show that RDR generally outperforms in the Many classes compared to the other two variants of SAM, indicating that RDR can efficiently address the overfitting issues for Few classes. For more experimental details, please refer to Appendix D.2 and Appendix D.1.

Flat minima of loss landscape. Key metrics associated with Eigen Spectral Density, such as the maximum and minimum eigenvalues ( λ max and λ min ) and the trace of the Hessian matrix ( Tr ( H ) ), effectively reflect the smoothness of the loss landscape. Lower values of λ max and Tr ( H ) indicate a smoother loss landscape. Rangwani et al. [2022] have demonstrated that smoother loss landscapes correlate with stronger model generalization, which is particularly crucial when dealing

Dataset

Loss

Method

IF=100

IF=10

Many

Med.

Few

All

Many

Med.

Few

All

CIFAR-10-LT

CE

SAM

94.8

74.5

60.6

76.4

95.8

85.9

86.5

89.3

ImbSAM

94.6

73.9

67.7

78.3

94.2

84.1

90.1

89.0

CCSAM

85.6

79.2

80.3

81.4

90.6

85.3

90.8

88.5

RDR+SAM

91.8

78.6

81.9

83.6

94.2

86.3

92.5

90.5

LA

SAM

90.8

78.1

82.9

83.3

92.6

86.5

91.8

89.9

ImbSAM

85.2

75.1

89.6

82.5

90.8

83.6

94.8

89.1

CCSAM

85.8

77.8

80.5

81.0

90.7

82.4

90.8

87.4

RDR+SAM

88.5

80.0

85.0

84.1

92.5

86.5

93.6

90.4

CIFAR-100-LT

CE

SAM

72.7

40.3

7.5

41.5

75.3

60.2

45.1

60.8

ImbSAM

71.5

40.6

17.7

44.3

72.5

57.9

53.5

61.7

CCSAM

61.5

50.8

29.7

48.0

62.8

59.3

54.5

59.0

RDR+SAM

63.4

51.9

30.5

49.3

68.0

61.6

58.3

62.8

LA

SAM

64.3

50.5

30.8

49.2

66.1

60.9

59.7

62.3

ImbSAM

58.8

45.4

40.1

48.4

62.9

56.6

64.2

61.2

CCSAM

57.7

48.9

29.0

45.8

61.0

57.8

51.9

57.1

RDR+SAM

63.9

52.4

30.5

49.6

67.6

61.7

60.1

63.2

150

100

50

0

50

100

150

Eigenvalue

10

7

10

6

10

5

10

4

10

3

10

2

10

1

10 0

10 1

Density (Log Scale)

: 179.21

min

: -165.97

(a) CE

40

20

0

20

40

Eigenvalue

10

7

10

6

10

5

10

4

10

3

10

2

10

1

10 0

10 1

Density (Log Scale)

: 45.76

min

: -44.48

(b) RDR

20

0

20

40

60

80

100

120

Eigenvalue

10

7

10

6

10

5

10

4

10

3

10

2

10

1

10 0

10 1

Density (Log Scale)

: 127.37

min

: -24.06

20

10

0

10

20

30

Eigenvalue

10

7

10

6

10

5

10

4

10

3

10

2

10

1

10 0

10 1

Density (Log Scale)

: 28.45

min

: -23.44

(d) RDR(SAM)

with imbalanced data. λ min also serves as a significant indicator of the loss landscape characteristics. Apreponderance of negative eigenvalues from the Hessian spectrum, resulting in smaller λ min values, empirically suggests convergence to saddle points. Saddle points typically represent regions in the loss landscape characterized by a plateau with some negative curvature. In non-convex settings, it has been shown that an exponential number of saddle points exist, and convergence to these points is indicative of poor generalization.

Fig. 4 illustrates the Eigen Spectral Density under different loss function training regimes. It is evident that combining our method with the CE technique significantly improves the loss landscape. On one hand, λ max is substantially reduced, indicating a flatter loss landscape. On the other hand, there is an increase in λ min , suggesting our method's effectiveness in escaping from saddle points.

Table 3 delves deeper into the changes in the loss landscape for all Few classes. It reveals that combining our method with CE and SAM results in average reductions in λ max by 58.5% and 66.1%, respectively, and increases in λ min by 49.2% and 7.6%, respectively. Furthermore, our method significantly reduces Tr ( H ) for minority classes (class7, class8 and class9). The average value of Tr ( H ) decreases by 55.7% and 66.0% when combined with CE and SAM, respectively. These findings underscore the efficacy of our method in improving the loss landscape and enhancing the generalization capability of the model [Dauphin et al., 2014].

Results on ImageNet-LT and Places-LT. Our experiments conducted on ImageNet-LT and PlacesLT, two large-scale datasets characterized by irregular and complex data distributions, demonstrate notable accuracy improvements through the application of RDR, as shown in Table 1 and Table 4.

Specifically, when combined with CE and LA on ImageNet-LT, our method achieves accuracy enhancements of 18.7% and 2% in Few classes, respectively. While the SAM technique combined with CE and LA offers limited accuracy improvements, its integration with our approach still results in an overall accuracy increase of approximately 2.3% compared to other methods. Notably, the LA method tends to suppress accuracy in Many classes more than the CE method; however, this side effect is effectively mitigated when LA is combined with RDR.

In the Places-LT dataset, the performance of RDR is even more pronounced. Combinations of our method with CE and LA result in accuracy gains of 10.1% and 2% in overall classes, respectively. Additionally, integrating SAM with our method also yields incremental improvements of 10.3% and 0.5% under CE and LA conditions, respectively. Our approach not only enhances accuracy in Few classes but also surpasses other methods in Medium classes, indicating its comprehensive efficacy across different categories. This broad applicability is particularly crucial for addressing the challenges of imbalanced learning.

Results on data with label noise. We further investigate the performance of our approach on datasets with label noise. Specifically, we evaluate two datasets: CIFAR-10-LT-NL and CIFAR-100-LT-NL, both of which exhibit class imbalance and label noise. Experiments are conducted with a noise ratio of 5%, and the results are presented in Fig. 5. As shown, our method demonstrates consistent and significant improvements on more datasets with label noise.

4.3 Ablation Study

In our study, we perform an ablation experiment to validate the efficacy of the multiple components that comprise our method. The outcomes from experiments across four datasets are delineated in Table 5. It is crucial to note that our weighting definition for each category i follows the formula w = r/n i . When r = 1 , our method simplifies to the traditional inverse frequency weighting w = 1 /n i . We explored the differences between our approach with and without the integration of SAM compared to this conventional weighting method.









































(a)

IF

=10 with CE































(b)

IF

=100 with CE





























(c)

IF

=10 with LA































(d)

IF

=100 with LA

The results depicted in Table 5 reveal that our dynamic weighting approach consistently outperforms the classic method under various scenarios. Without SAM, when combined with CE and LA, our method achieves accuracy improvements ranging from 0.4% to 4.3% and 0.3% to 2.3%, respectively. When integrated with SAM, the improvement in accuracy is particularly notable on large datasets. Specifically, the accuracy enhancements on ImageNet-LT and Places-LT reach 7.1% and 1.9% respectively when combined with LA. These results underscore the tangible benefits of our dynamic weighting strategy in enhancing model performance. The impact of momentum coefficient m in RDR is shown in Fig. 3. For more experimental details, please refer to Appendix D.3.

5 Conclusion

In this work, we have introduced RDR, a novel approach for mitigating model degradation in imbalanced learning scenarios by dynamically adjusting class weights using density ratio estimation. Our method dynamically adjusts class weights during training based on density ratio estimation, enhancing both model robustness and adaptability. Extensive experiments on diverse large-scale datasets demonstrate the effectiveness of RDR, particularly in severely imbalanced settings. Future work will focus on refining the dynamic adjustment mechanisms and exploring broader applicability across various domains and dataset complexities.

References

Kevin W. Bowyer, Nitesh V. Chawla, Lawrence O. Hall, and W. Philip Kegelmeyer. SMOTE: synthetic minority over-sampling technique. CoRR , abs/1106.1813, 2011. URL http://arxiv. org/abs/1106.1813 .

Mateusz Buda, Atsuto Maki, and Maciej A. Mazurowski. A systematic study of the class imbalance problem in convolutional neural networks. Neural Networks , 106:249-259, 2018. doi: 10.1016/J. NEUNET.2018.07.011. URL https://doi.org/10.1016/j.neunet.2018.07.011 .

Jiarui Cai, Yizhou Wang, and Jenq-Neng Hwang. Ace: Ally complementary experts for solving long-tailed recognition in one-shot. In Proceedings of the IEEE/CVF international conference on computer vision , pages 112-121, 2021.

Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning imbalanced datasets with label-distribution-aware margin loss. Advances in neural information processing systems , 32, 2019.

Mengxi Chen, Jiangchao Yao, Linyu Xing, Yu Wang, Ya Zhang, and Yanfeng Wang. Redundancy adaptive multimodal learning for imperfect data. arXiv preprint arXiv:2310.14496 , 2023a.

Xiaohua Chen, Yucan Zhou, Dayan Wu, Chule Yang, Bo Li, Qinghua Hu, and Weiping Wang. Area: adaptive reweighting via effective area for long-tailed classification. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 19277-19287, 2023b.

Yin Cui, Menglin Jia, Tsung-Yi Lin, Yang Song, and Serge Belongie. Class-balanced loss based on effective number of samples. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 9268-9277, 2019.

Yann N Dauphin, Razvan Pascanu, Caglar Gulcehre, Kyunghyun Cho, Surya Ganguli, and Yoshua Bengio. Identifying and attacking the saddle point problem in high-dimensional non-convex optimization. Advances in neural information processing systems , 27, 2014.

Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware minimization for efficiently improving generalization. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021 . OpenReview.net, 2021. URL https://openreview.net/forum?id=6Tm1mposlrM .

Yu Gong, Greg Mori, and Frederick Tung. Ranksim: Ranking similarity regularization for deep imbalanced regression. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato, editors, International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA , volume 162 of Proceedings of Machine Learning Research , pages 7634-7649. PMLR, 2022. URL https://proceedings.mlr.press/v162/ gong22a.html .

Hui Han, Wenyuan Wang, and Binghuan Mao. Borderline-smote: A new over-sampling method in imbalanced data sets learning. In ICIC , volume 3644 of Lecture Notes in Computer Science , pages 878-887. Springer, 2005.

Feng Hong, Jiangchao Yao, Zhihan Zhou, Ya Zhang, and Yanfeng Wang. Long-tailed partial label learning via dynamic rebalancing. In International Conference on Learning Representations . OpenReview.net, 2023.

Feng Hong, Yueming Lyu, Jiangchao Yao, Ya Zhang, Ivor W. Tsang, and Yanfeng Wang. Diversified batch selection for training acceleration. In International Conference on Machine Learning . OpenReview.net, 2024a.

Feng Hong, Jiangchao Yao, Yueming Lyu, Zhihan Zhou, Ivor W. Tsang, Ya Zhang, and Yanfeng Wang. On harmonizing implicit subpopulations. In International Conference on Learning Representations . OpenReview.net, 2024b.

Chen Huang, Yining Li, Chen Change Loy, and Xiaoou Tang. Learning deep representation for imbalanced classification. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016 , pages 5375-5384. IEEE Computer Society, 2016. doi: 10.1109/CVPR.2016.580. URL https://doi.org/10.1109/CVPR.2016.580 .

Bingyi Kang, Saining Xie, Marcus Rohrbach, Zhicheng Yan, Albert Gordo, Jiashi Feng, and Yannis Kalantidis. Decoupling representation and classifier for long-tailed recognition. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020 . OpenReview.net, 2020. URL https://openreview.net/forum?id=r1gRTCVFvB .

Mahsa Keramati, Lili Meng, and R. David Evans. Conr: Contrastive regularizer for deep imbalanced regression. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024 . OpenReview.net, 2024. URL https://openreview.net/ forum?id=RIuevDSK5V .

Ganesh Ramachandra Kini, Orestis Paraskevas, Samet Oymak, and Christos Thrampoulidis. Labelimbalanced and group-sensitive classification under overparameterization. In Marc'Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors, Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual , pages 18970-18983, 2021. URL https://proceedings.neurips.cc/paper/2021/hash/ 9dfcf16f0adbc5e2a55ef02db36bac7f-Abstract.html .

Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Handbook of Systemic Autoimmune Diseases , 2009.

Miroslav Kubat and Stan Matwin. Addressing the curse of imbalanced training sets: One-sided selection. In ICML , pages 179-186. Morgan Kaufmann, 1997.

Tianhong Li, Peng Cao, Yuan Yuan, Lijie Fan, Yuzhe Yang, Rogerio S Feris, Piotr Indyk, and Dina Katabi. Targeted supervised contrastive learning for long-tailed recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 6918-6928, 2022.

Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision , pages 2980-2988, 2017.

Hong Liu, Jeff Z. HaoChen, Adrien Gaidon, and Tengyu Ma. Self-supervised learning is more robust to dataset imbalance. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022 . OpenReview.net, 2022. URL https://openreview. net/forum?id=4AZz9osqrar .

Ziwei Liu, Zhongqi Miao, Xiaohang Zhan, Jiayun Wang, Boqing Gong, and Stella X Yu. Large-scale long-tailed recognition in an open world. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 2537-2546, 2019.

Yanbiao Ma, Licheng Jiao, Fang Liu, Yuxin Li, Shuyuan Yang, and Xu Liu. Delving into semantic scale imbalance. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023 . OpenReview.net, 2023. URL https://openreview.net/ pdf?id=07tc5kKRIo .

Aditya Krishna Menon, Sadeep Jayasumana, Ankit Singh Rawat, Himanshu Jain, Andreas Veit, and Sanjiv Kumar. Long-tail learning via logit adjustment. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021 , 2020.

Katharina Morik, Peter Brockhausen, and Thorsten Joachims. Combining statistical learning with a knowledge-based approach - A case study in intensive care monitoring. In Ivan Bratko and Saso Dzeroski, editors, Proceedings of the Sixteenth International Conference on Machine Learning (ICML 1999), Bled, Slovenia, June 27 - 30, 1999 , pages 268-277. Morgan Kaufmann, 1999.

Harsh Rangwani, Sumukh K. Aithal, Mayank Mishra, and Venkatesh Babu R. Escaping saddle points for effective generalization on class-imbalanced data. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 -December 9, 2022 , 2022. URL http://papers.nips.cc/paper_files/paper/2022/hash/ 8f4d70db9ecec97b6723a86f1cd9cb4b-Abstract-Conference.html .

Jiawei Ren, Cunjun Yu, Shunan Sheng, Xiao Ma, Haiyu Zhao, Shuai Yi, and Hongsheng Li. Balanced meta-softmax for long-tailed visual recognition. In Hugo Larochelle, Marc'Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual , 2020a. URL https://proceedings.neurips. cc/paper/2020/hash/2ba61cc3a8f44143e1f2f13b2b729ab3-Abstract.html .

Jiawei Ren, Cunjun Yu, Shunan Sheng, Xiao Ma, Haiyu Zhao, Shuai Yi, and Hongsheng Li. Balanced meta-softmax for long-tailed visual recognition. In Hugo Larochelle, Marc'Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual , 2020b. URL https://proceedings.neurips. cc/paper/2020/hash/2ba61cc3a8f44143e1f2f13b2b729ab3-Abstract.html .

Jiawei Ren, Mingyuan Zhang, Cunjun Yu, and Ziwei Liu. Balanced MSE for imbalanced visual regression. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022 , pages 7916-7925. IEEE, 2022. doi: 10.1109/CVPR52688. 2022.00777. URL https://doi.org/10.1109/CVPR52688.2022.00777 .

Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael S. Bernstein, Alexander C. Berg, and Li Fei-Fei. Imagenet large scale visual recognition challenge. Int. J. Comput. Vis. , 115(3):211-252, 2015. doi: 10.1007/S11263-015-0816-Y. URL https://doi.org/10.1007/s11263-015-0816-y .

Yonglong Tian, Olivier J. Hénaff, and Aäron van den Oord. Divide and contrast: Self-supervised learning from uncurated data. In 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021 , pages 10043-10054. IEEE, 2021.

Byron C. Wallace, Kevin Small, Carla E. Brodley, and Thomas A. Trikalinos. Class imbalance, redux. In ICDM , pages 754-763. IEEE Computer Society, 2011.

Jiaqi Wang, Wenwei Zhang, Yuhang Zang, Yuhang Cao, Jiangmiao Pang, Tao Gong, Kai Chen, Ziwei Liu, Chen Change Loy, and Dahua Lin. Seesaw loss for long-tailed instance segmentation. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021 , pages 9695-9704. Computer Vision Foundation / IEEE, 2021a. doi: 10.1109/CVPR46437.2021. 00957. URL https://openaccess.thecvf.com/content/CVPR2021/html/Wang_Seesaw_ Loss_for_Long-Tailed_Instance_Segmentation_CVPR_2021_paper.html .

Xudong Wang, Long Lian, Zhongqi Miao, Ziwei Liu, and Stella X. Yu. Long-tailed recognition by routing diverse distribution-aware experts. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021 . OpenReview.net, 2021b. URL https://openreview.net/forum?id=D9I3drBz4UC .

Zitai Wang, Qianqian Xu, Zhiyong Yang, Yuan He, Xiaochun Cao, and Qingming Huang. A unified generalization analysis of re-weighting and logit-adjustment for imbalanced learning. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors, Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023 , 2023. URL http://papers.nips.cc/paper_files/paper/2023/hash/ 973a0f50d43cf99118cdab456edcacda-Abstract-Conference.html .

Ziyan Wang and Hao Wang. Variational imbalanced regression: Fair uncertainty quantification via probabilistic smoothing. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors, Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023 , 2023. URL http://papers.nips.cc/paper_files/paper/ 2023/hash/612a56f193d031687683445cd0001083-Abstract-Conference.html .

Yuzhe Yang, Kaiwen Zha, Ying-Cong Chen, Hao Wang, and Dina Katabi. Delving into deep imbalanced regression. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event , volume 139 of Proceedings of Machine Learning Research , pages 11842-11851. PMLR, 2021. URL http://proceedings.mlr.press/v139/yang21m.html .

Xi Yin, Xiang Yu, Kihyuk Sohn, Xiaoming Liu, and Manmohan Chandraker. Feature transfer learning for face recognition with under-represented data. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 5704-5713, 2019.

Bianca Zadrozny, John Langford, and Naoki Abe. Cost-sensitive learning by cost-proportionate example weighting. In Proceedings of the 3rd IEEE International Conference on Data Mining (ICDM 2003), 19-22 December 2003, Melbourne, Florida, USA , page 435. IEEE Computer Society, 2003. doi: 10.1109/ICDM.2003.1250950. URL https://doi.org/10.1109/ICDM. 2003.1250950 .

Yixuan Zhou, Yi Qu, Xing Xu, and Hengtao Shen. Imbsam: A closer look at sharpness-aware minimization in class-imbalanced recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 11345-11355, 2023a.

Zhihan Zhou, Jiangchao Yao, Feng Hong, Ya Zhang, Bo Han, and Yanfeng Wang. Combating representation learning disparity with geometric harmonization. Advances in Neural Information Processing Systems , 36, 2023b.

Zhipeng Zhou, Lanqing Li, Peilin Zhao, Pheng-Ann Heng, and Wei Gong. Class-conditional sharpness-aware minimization for deep long-tailed recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 3499-3509, 2023c.

A Notations

In Table 6, we summarize the notations used in this paper.

B More Discussions of Related Work

Beyond imbalanced classification that has discretized label space, an noteworthy area, imbalanced regression that has a continuous label space is also very common in real applications [Yang et al., 2021, Gong et al., 2022]. In this direction, the empirical label distribution often does not accurately reflect the true label density in regression tasks, which limits the effectiveness of traditional re-weighting techniques [Yang et al., 2021, Wang and Wang, 2023]. Label Distribution Smoothing (LDS) [Yang et al., 2021] and Variational Imbalanced Regression (VIR) [Wang and Wang, 2023] propose using kernel smoothing and other techniques to estimate an accurate label density distribution. Ranking Similarity (Ranksim) [Gong et al., 2022] leverages local and global dependencies by encouraging the correspondence between the similarity order of labels and features. Balanced Mean Squared Error (Balanced MSE) [Ren et al., 2022] extends the concept of Balanced Softmax [Ren et al., 2020a] to regression tasks to achieve a balanced predictive distribution. Contrastive Regularizer (ConR) [Keramati et al., 2024] improves contrastive learning techniques to translate label similarities into the feature space.

C Algorithm Details

C.1 Detailed Derivations of Eq. (7)

Back to Eq. (5), replace the expectations over P bal and P by Φ P and F P bal , respectively. For each class i , We can obtain ̂ r i = ̂ MM( r ) , where

f

g

Re-weighted

Loss

Update

Update

Ordinary Process

w

-Min Pred.

Weight

Feature prototype

Momentum

update

Extracted features

-Min Pred.

Weight

Head class

Step

Step

Tail class

Weight

Constant re-weighting

does not match the

training dynamic

Constant

Re-weighting

Dynamic Re-weighting

Then, taking the derivative of ̂ MM( r ) with respect to r and setting it to zero, we can obtain the estimation of density ratio in imbalanced learning as follows

Solving equation above with respect to r i , we can obtain the solution as

C.2 Framework of RDR

We provide the framework of RDR, which is shown in Fig. 6.

C.3 Pseudo-code of RDR

We provide the pseudo-code of RDR to demonstrate the process of implementing our method in detail, as shown in Algorithm 1. In addition, we also provide pseudo-code that combines our method with the SAM method, as shown in Algorithm 2.

D Supplement for Experiments

D.1 Experiment with More Imbalanced Data

We conduct experiments on the more imbalanced CIFAR-10-LT and CIFAR-100-LT datasets, specifically with imbalance factors of 200 and 500. As shown in Table 7, our method consistently achieves significant improvements.

D.2 Dynamically Re-weighting Process

Our approach dynamically adjusts the weights assigned to each category throughout the training process. To gain more insights into RDR, we sampled the weights of each category during training. Fig. 7 presents the results of four samplings during the training processes at imbalance factors of 10 and 100, respectively.

The analysis of these results reveals a consistent trend in weight changes across different imbalance factors. For the Many classes, the weights of the categories consistently decrease during training. Specifically, under the IF of 10, the weights of class0, class1, and class2 (the three classes with the highest sample counts, in descending order) decrease by 5.6%, 6.7%, and 8.0%, respectively. Under the IF of 100, these decreases are more pronounced, with reductions of 10.3%, 15.4%, and 14.7%, respectively. For the Medium classes, the weight changes are less marked, with an average decrease

Algorithm 1 Training Paradigm of RDR.

1: Input: Training dataset S = ∪ n i =1 { ( x i , y i ) } , model M ω with feature extractor f ϕ and classifier h θ , loss function l , momentem coefficient m , learning rate α , weight decay coefficient λ , batch size b , temperature coefficient γ 2: Output: Trained parameters ϕ ∗ , θ ∗ 3: Initialize the model parameters ϕ and θ ramdomly, F P bal = ( F 1 , . . . , F d ) ← 0 4: for t = 1 to T do 5: B ← SampleMiniBatch ( S , b ) 6: z ← f ( x, ϕ t ) 7: output ← h ( z, θ t ) 8: if t < T 0 then 9: // warm up 10: w ← 1 11: else 12: for class i to d do 13: Φ i P ← ( z j ) where y j = i 14: compute w i via Eq. (8) and γ 15: end for 16: w ← normalize( w γ ) 17: end if 18: L ( ω t , B ) ← 1 b ∑ B w · l ( output, y ) 19: ω t = ω t -α t [ ∇L ( ω t , B ) + λω t ] 20: F P bal ← mF P bal +(1 -m ) z 21: Optional: anneal the learning rate α t 22: end for

Algorithm 2 Training Paradigm of RDR combined with SAM.

1: Input: Training dataset S = ∪ n i =1 { ( x i , y i ) } , model M ω with feature extractor f ϕ and classifier h θ , loss function l , momentem coefficient m , learning rate α , weight decay coefficient λ , batch size b , neighborhood size ρ , temperature coefficient γ 2: Output: Trained parameters ϕ ∗ , θ ∗ 3: Initialize the model parameters ϕ and θ ramdomly, F P bal = ( F 1 , . . . , F d ) ← 0 4: for t = 1 to T do 5: B ← SampleMiniBatch ( S , b ) 6: z ← f ( x, ϕ t ) 7: output ← h ( z, θ t ) 8: if t < T 0 then 9: // warm up 10: w ← 1 11: else 12: for class i to d do 13: Φ i P ← ( z j ) where y j = i 14: compute w i via Equation 8 and γ 15: end for 16: w ← normalize( w γ ) 17: end if 18: L 1 ( ω t , B ) ← 1 b ∑ B w · l ( output, y ) 19: ϵ t ← ρ ∇L 1 ( ω t , B ) |∇L 1 ( ω t , B ) | 20: L 2 ( ω t + ϵ t , B ) ← 1 b ∑ B w · l ( f ω t + ϵ t ( · ) , y ) 21: ω t = ω t -α t [ ∇L 2 ( ω t + ϵ t , B ) + λω t ] 22: F P bal ← mF P bal +(1 -m ) z 23: Optional: anneal the learning rate α t 24: end for

Method

IF=500

IF=200

CIFAR-10-LT

CIFAR-100-LT

CIFAR-10-LT

CIFAR-100-LT

CE

60.1

34.1

68.6

37.9

RDR+CE

69.7

39.0

76.7

43.1

SAM+CE

60.4

34.5

69.9

39.0

RDR+SAM+CE

71.1

40.0

80.0

44.8

LA

73.9

38.8

76.7

44.0

RDR+LA

75.8

40.1

80.8

45.0

SAM+LA

74.9

39.6

80.2

44.2

RDR+SAM+LA

76.4

40.6

81.6

45.1

Step

0.550

0.575

0.600

0.625

0.650

0.675

0.700

Weight

Class 0

Class 1

Class 2

(a) Many

Step

0.75

0.80

0.85

0.90

0.95

1.00

1.05

1.10

1.15

Weight

Class 3

Class 4

Class 5

Class 6

(b) Medium

Step

1.3

1.4

1.5

1.6

1.7

Weight

Class 7

Class 8

Class 9

(c) Few

Step

0.32

0.34

0.36

0.38

0.40

0.42

0.44

Weight

Class 0

Class 1

Class 2

(d) Many

Step

0.5

0.6

0.7

0.8

0.9

1.0

1.1

Weight

Class 3

Class 4

Class 5

Class 6

(e) Medium

Step

1.4

1.6

1.8

2.0

2.2

2.4

Weight

Class 7

Class 8

Class 9

(f) Few

Figure 7: Visualization of weight changes across different categories throughout the training process. Experiments conducted on CIFAR-10-LT. (a), (b), and (c) illustrate the weight changes for Many , Medium , and Few classes, respectively, under an imbalance factor of 10. (d), (e), and (f) correspondingly show the changes for each class type under an imbalance factor of 100.

of 3.7% at IF =10 and 0.3% at IF =100. For the Few classes, there exits a notable increase in weights during training; at IF =10, the weights of class7, class8, and class9 increase by 2.3%, 6.3%, and 9.4%, respectively, while at IF =100, they increase by 6.2%, 6.5%, and 3.2%.

These results suggest that our method increasingly focuses on minority classes as training progresses. Initially, our method effectively learns common features across all categories, while later in training, increasing the weights helps to target learning towards minority samples, thereby enhancing the model's generalizability. Wang et al. [2023] also corroborate these findings.

D.3 Ablation Study

We provide more detailed experimental results for each category in ablation study, as illustrated in Fig. 8 and Fig. 9. From these figures, we can find that across various datasets and different imbalance factors, our method significantly enhances the generalizability of both Few classes and Medium classes. Moreover, our method maintains superior performance when combined with the SAM method.

Many

Medium

Few

Category

0.45

0.50

0.55

0.60

0.65

0.70

0.75

CE

1/

n

RDR

(a) Accuracy under

IF

=10

Many

Medium

Few

Category

0.45

0.50

0.55

0.60

0.65

0.70

0.75

CE

1/

n

RDR

(b)

under

IF

=10 with

SAM

Many

Medium

Few

Category

0.1

0.2

0.3

0.4

0.5

0.6

0.7

CE

1/

n

RDR

(c) Accuracy under

IF

=100

Many

Medium

Few

Category

0.1

0.2

0.3

0.4

0.5

0.6

0.7

CE

1/

n

RDR

(d) Accuracy under

IF

=100 with

SAM

Many

Medium

Few

Category

0.2

0.3

0.4

0.5

0.6

CE

1/

n

RDR

(a) Accuracy on ImageNet-LT

Many

Medium

Few

Category

0.1

0.2

0.3

0.4

0.5

0.6

CE

1/

n

RDR

(b) Accuracy on ImageNet-LT with

SAM

Many

Medium

Few

Category

15

20

25

30

35

40

CE

1/

n

RDR

(c) Accuracy on Places-LT

Many

Medium

Few

Category

15

20

25

30

35

40

45

CE

1/

n

RDR

(d)

Accuracy on Places-LT with

SAM

E More Discussions about Limitations

While our approach demonstrates promising results, there are potential challenges that warrant further attention. In particular, as the number of classes increases to a very large scale, especially in certain tasks such as face recognition, retail product recommendation, or landmark detection, there could be concerns regarding computational efficiency. It is important to consider lightweight techniques to ensure that scalability does not compromise practical applicability. Additionally, in addressing imbalanced learning, care must be taken to avoid excessive rebalancing toward minority groups, as this could unintentionally affect the learning performance of the majority class, which would not align with the broader goals of fairness. Rebalancing should be conducted within a reasonable framework, mindful of avoiding misuse or overcompensation that could arise from improper manipulation by any group. Ensuring fairness for all remains a critical consideration.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The abstract and introduction have stated the claims.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: See Section 3.5 and Appendix E.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: This paper does not include theoretical results.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: See Section 4 and Appendix C.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [NA]

Justification: The code of this paper will be released after anonymized review.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: See Section 4.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We report error bars for CIFAR-10-LT and CIFAR-100-LT dataset. For the large scale ImageNet-LT and Places-LT, we do not report error bars due to computational constraints.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: See Section 3 and Section 4.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: This paper is conducted with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: Our proposed method can enhance the efficiency and robustness of imbalance learning, thereby increasing productivity.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: This paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: This paper does not use such assets.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: This paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a dynamic re-weighting method for imbalanced learning. The author defines the ratio of the balanced data set distribution to the training set distribution, and tries to estimate it with an iterative update method. The effectiveness of this method is proved by experiments.\n\nSTRENGTHS:\n1.\tThis paper points out a problem with distribution differences, which leads to the potential missing feature patterns in general re-weighting methods.\n2.\tThis paper proposes a new method, which approximates the ratio of the balanced data set distribution to the training set distribution using methods of density ratio estimation. As far as I know, a dynamic re-weighting strategy is novel in this field.\n3.\tThe experimental introduction of this paper is clear, and extensive experiments have been carried out, which validates the effectiveness of the proposed method.\n\nWEAKNESSES:\n1.\tThe formula derivation in Sec. 3.3 can be more detailed. It is suggested to explain how formula (7) is obtained in the appendix.\n2.\tThe introduction may have overlooked some key articles. For example, the article mentions Wang et al. 's article at the end of Sec.3.3, but does not discuss this paper in the introduction section.\n3.\tDoes the new method enjoy the same theoretical boundaries as the general reweighting method? It is recommended to provide more analysis.\n4.\tBesides, there are some typos in the details:\n  - In the experimental section, 'class[390,385] 'may be a typo.\n  - In table 3, the interpretation of Tr_{Few} is supposed to be there.\n\nQUESTIONS:\nPlease refer to Weaknesses.\n\nBesides, I am also curious about some experimental details. Do the authors use other techniques such as RandAug or mixup?\n\nLIMITATIONS:\nNo. Although the authors say they clarify the limitations in Sec.3.4, I find they mainly highlight the efficiency of the proposed method. More discussion about limitations and potential negative societal impact is recommended.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a novel approach called Re-weighting with Density Ratio (RDR) to address the challenges posed by imbalanced data distributions in machine learning.  The RDR approach aims to mitigate overfitting on majority classes and enhance adaptability across diverse datasets by continuously updating the weights in response to observed shifts in class density.  Extensive experiments on various large-scale, long-tailed datasets demonstrate that the RDR method significantly improves the model's generalization capabilities, particularly under severely imbalanced conditions.  The analysis of the weight changes during training reveals that the method increasingly focuses on minority classes as training progresses, initially learning common features across all categories and then targeting learning towards minority samples to enhance generalizability.  The paper also provides an ablation study to further validate the effectiveness of the proposed approach.\n\nSTRENGTHS:\n1. The paper introduces a novel approach called Re-weighting with Density Ratio (RDR) to address the challenges posed by imbalanced data distributions in machine learning. \n\n2. Extensive experiments on various large-scale, long-tailed datasets demonstrate that the RDR method significantly improves the model's generalization capabilities, particularly under severely imbalanced conditions. \n\n3. The analysis of the weight changes during training reveals that the method increasingly focuses on minority classes as training progresses, initially learning common features across all categories and then targeting learning towards minority samples to enhance generalizability. \n\n4. The paper provides an ablation study to further validate the effectiveness of the proposed approach. \n\n5. The results show that RDR generally outperforms other methods, including Inverse Frequency (1/n) and SAM variants, in both the Many and Few classes, indicating that RDR can efficiently address the overfitting issues for Few classes.\n\nWEAKNESSES:\n- The paper does not provide a detailed theoretical analysis or justification for the proposed Re-weighting with Density Ratio (RDR) method, beyond the intuition that it can mitigate overfitting on majority classes and enhance adaptability across diverse datasets.\n\n- I am interested in how RDR might perform in the presence of extreme imbalance, noisy data, or other challenging scenarios. The current experiment is well-established but dataset itself is relatively simple.\n\n- The paper discusses reweighting/non-reweighting for classification problems. I suggest the authors also briefly discuss reweighting methods in imbalanced regression problems, e.g., VIR [1] for reweighting problems and ConR [2] for non-reweighting problems.\n\n[1] Variational Imbalanced Regression: Fair Uncertainty Quantification via Probabilistic Smoothing, NeurIPS 2023\n\n[2] ConR: Contrastive Regularizer for Deep Imbalanced Regression, ICLR 2024\n\n**Summary** I think the theoretical analysis or at least insights is needed for acceptance, so my suggest score is 5, as the experiment part is excellent in this paper.\n\nQUESTIONS:\nsee above\n\nLIMITATIONS:\nauthors discussed in sec 3.4", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a weighting strategy in order to handle class imbalance. Contrary to existing method, they propose to adapt the weight throughout the training procedure. \n\nTheir method estimates the discrepancy between the sample distribution and the balanced sample distribution for parameterization w and updates the estimate through the training.\n\nThe authors use two resnet architectures to evaluate their contribution on multiple datasets. They also compare to other baselines and show significant gain.\n\nSTRENGTHS:\n* The paper develops a novel approach for handling class imbalance.\n* The methodology is derived theoretically from the problem formulation\n* The authors propose an analysis of the complexity of the method and empirically evaluate the training time.\n* The methodology is evaluated on multiple datasets and compared to multiple baselines.\n\nWEAKNESSES:\n* The paper is sometimes difficult to read:\n  * Row 125, the authors refer to the distribution of training set, which get parameterized by w. Thus, my understanding is that the authors refer to the distribution of the training set \"captured by the model\". \n  * row 134, P_bal = pi P.. P_bal is the distribution of y in the balanced case ? But should therefore be 1/number classes... and P, should just be the class proportion and we should have P = pi P_bal ?\n  * LDAM and LA terms are not defined at first. First definition of LA is at row 208\n  * row 212 \"trategies\" => strategies\n\nQUESTIONS:\n* Could you clarify the term, P, etc. You often refer to it as \"real world data distribution\", but the distribution of x does not depends on any other classes (imbalanced or not) ?\n\nLIMITATIONS:\nThe paper would benefit from a clarification in notification (what is the true data distribution, what is the feature distribution, what is an estimate of what quantity, etc.). I believe the contribution is novel and worth it.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-08"}