Title: A Unified Generalization Analysis of Re-Weighting and Logit-Adjustment for Imbalanced Learning
Venue: NeurIPS 2023
Accepted: yes
Keywords: Imbalanced Learning

Re-weighting

Logit Adjustment

Genralization Analysis
Abstract: Real-world datasets are typically imbalanced in the sense that only a few classes have numerous samples, while many classes are associated with only a few samples. As a result, a naive ERM learning process will be biased towards the majority classes, making it difficult to generalize to the minority classes. To address this issue, one simple but effective approach is to modify the loss function to emphasize the learning on minority classes, such as re-weighting the losses or adjusting the logits via class-dependent terms. However, existing generalization analysis of such losses is still coarse-grained and fragmented, failing to explain some empirical results. To bridge this gap between theory and practice, we propose a novel technique named data-dependent contraction to capture how these modified losses handle different classes. On top of this technique, a fine-grained generalization bound is established for imbalanced learning, which helps reveal the mystery of re-weighting and logit-adjustment in a unified manner. Furthermore, a principled learning algorithm is developed based on the theoretical insights. Finally, the empirical results on benchmark datasets not only validate the theoretical results but also demonstrate the effectiveness of the proposed method.

A Unified Generalization Analysis of Re-Weighting and Logit-Adjustment for Imbalanced Learning

Abstract

Real-world datasets are typically imbalanced in the sense that only a few classes have numerous samples, while many classes are associated with only a few samples. As a result, a naïve ERM learning process will be biased towards the majority classes, making it difficult to generalize to the minority classes. To address this issue, one simple but effective approach is to modify the loss function to emphasize the learning on minority classes, such as re-weighting the losses or adjusting the logits via class-dependent terms. However, existing generalization analysis of such losses is still coarse-grained and fragmented, failing to explain some empirical results. To bridge this gap, we propose a novel technique named data-dependent contraction to capture how these modified losses handle different classes. On top of this technique, a fine-grained generalization bound is established for imbalanced learning, which helps reveal the mystery of re-weighting and logit-adjustment in a unified manner. Furthermore, a principled learning algorithm is developed based on the theoretical insights. Finally, the empirical results on benchmark datasets not only validate the theoretical results but also demonstrate the effectiveness of the proposed method.

1 Introduction

In recent years, machine learning has achieved great success with the help of well-collected datasets, where the number of samples is artificially balanced among classes [1, 2]. However, the real-world datasets are generally imbalanced in the sense that only a few classes have numerous samples ( i.e. , the majority ones), while the others are associated with only a few samples ( i.e. , the minority ones) [3-5]. Owing to this issue, a naïve Empirical Risk Minimization (ERM) learning process will be biased towards the majority classes, and the generalization on the minority ones becomes challenging. Hence, the imbalanced learning problem has attracted increasing attention in recent years [6-9].

One simple yet effective approach for imbalanced learning is to modify the naïve loss function, such that the learning process can pay more attention to the minority classes (Please refer to Appendix A for more orthogonal approaches). In this direction, existing approaches generally fall into two categories: re-weighting [10, 11] and logit-adjustment [12-16]. The former category assigns larger weights to

the losses of the minority classes. Although intuitive, this approach might lead to difficulties and instability in optimization [11, 12, 17]. To tackle this issue, Cao et al. [12] propose an effective scheme named Deferred Re-Weighting (DRW), where the re-weighting approach is applied only during the terminal phase of training. The latter category adjusts the logits by class-dependent terms. For example, the Label Distribution Aware Margin (LDAM) loss enforces larger margins for minority classes to achieve strong regularization [12]. The Logit-Adjustment (LA) loss [13] and the Class-Dependent Temperatures (CDT) loss [14] utilize additive and multiplicative terms to adjust the logits, respectively. Most recently, Kini et al. [16] combine the two types of terms and proposes a unified loss named Vector-Scaling (VS) for imbalanced learning.

Although existing loss-modification methods have achieved promising performance, the theoretical insights are still fragmented and coarse-grained. To be specific, Cao et al. [12] and Ren et al. [18] utilize the classic margin theory to explain the necessity of the additive terms in the LDAM loss. However, the theory fails to explain the significant improvement induced by the DRW scheme. Menon et al. [13] analyzes the Fisher consistency property [19] of the additive terms in the LA loss, while providing no further generalization analysis. Kini et al. [16] provides a generalization analysis of the VS loss, but the results can only explain the role of the multiplicative terms under the assumption that a linear model is trained on linearly separable data. Besides, we find that the VS loss is rather incompatible with the DRW scheme, which is also out of the scope of existing theory. Hence, a gap still exists between the theory and the practice of the loss-modification approaches.

To bridge this gap, this paper provides a systematical and fine-grained analysis of loss-modification approaches. After revisiting prior arts, we find that the only property of the loss function utilized in existing proofs is the classic Lipschitz continuity [19, 20]. However, this property is global in nature such that the whole analysis provides no insight into how the losses handle different classes. Inspired by this observation, we extend the classic Lipschitz continuity with a local technique. In this way, the local Lipschitz constants on different classes exactly correspond to the class-dependent terms of the modified loss functions. And a fine-grained generalization bound is established by a novel technique named data-dependent contraction. By applying this bound to the VS loss, the mystery of re-weighting and logit-adjustment is finally uncovered. Last but not least, a principled learning algorithm is proposed based on our theoretical insights.

To sum up, the main contributions of this paper are listed as follows:

New technique. We extend the classic Lipschitz continuity and propose a novel technique named data-dependent contraction to obtain a fine-grained generalization bound for imbalanced learning.

Theoretical insights. Based on the fine-grained bound, a systematical analysis succeeds in explaining the role of re-weighting and logit-adjustment in a unified manner, as well as some empirical results that are out of the scope of existing theories.

Principled Algorithm. A principled algorithm is proposed based on the insights, where the reweighting term is aligned with the generalization bound, and the multiplicative logit-adjustment term is removed during the DRW phase to avoid the incompatibility between terms.

Empirical Validation. The empirical results on multiple benchmark datasets not only validate the theoretical results, but also demonstrate the superiority of the proposed method.

2 Preliminary

We first introduce the basic notations and the imbalanced learning problem in Sec.2.1. Then, we briefly review existing generalization analysis for imbalanced learning in Sec.2.2.

2.1 Notations and Problem Definition

We assume that the samples are drawn i.i.d. from a product space Z = X × Y , where X is the input space and Y = { 1 , · · · , C } is the label space. Let S = { ( x ( n ) , y ( n ) ) } N n =1 be the imbalanced training set sampled from the imbalanced distribution D defined on Z , S y = { x | ( x , y ) ∈ S} be the set of samples from the class y , N y := |S y | denote the size of S y , and π y := N y /N . Without loss of generality, we assume that N 1 ≥ N 2 ≥ · · · ≥ N C .

Let D bal be the balanced distribution defined on Z . Specifically, a class y is first uniformly sampled from Y , and then the input x is sampled from the class-conditional distribution D y := P [ x | y ] .

Then, our task is to learn a score function f : X → R C to minimize the risk defined on the balanced distribution:

where R y is the risk defined on the class y , and M : R C ×Y → R + is the measure that evaluates the model performance at z ∈ Z . For example, one of the most popular choices is to check whether the top-1 prediction is right: M ( f ( x ) , y ) = 1 [ y / ∈ arg max y ′ ∈Y f ( x ) y ′ ] , where 1 [ · ] is the indicator function. Since M is generally non-differential and thus hard to optimize, one has to select a differential surrogate loss L : R C ×Y → R + , which induces the following surrogate risk:

Let G := { L ◦ f : f ∈ F} denote the hypothesis set. Next, we consider a family of loss functions named Vector-Scaling (VS) [16]:

The advantage behind this loss family is two-fold. On one hand, the VS loss generalizes popular re-weighting and logit-adjustment methods. For example, when α y = 1 , β y = 1 , ∆ y = 0 , it becomes the traditional CE loss [19]. When β y = 1 , ∆ y = 0 , re-weighting terms α y = π -1 y and α y = (1 -p ) / (1 -p N y ) , p ∈ (0 , 1) recover the classic balanced loss [10] and Class-Balanced (CB) loss [11], respectively. α y = 1 , β y = 1 , ∆ y = τ log π y , τ > 0 yield the LA loss [13]. When α 1 = 1 , β y = ( N y /N 1 ) γ , ∆ y = 0 , γ > 0 , we can deduce the CDT loss [14]. On the other hand, an ideal surrogate loss should be Fisher consistent such that minimizing R L bal ( f ) not only can put more emphasis on minority classes, but also helps bound R bal ( f ) [19, 21]. Fortunately, prior arts [13] have shown that a subset of the VS loss family satisfies such a property:

glyph[negationslash]

where δ y is an arbitrary positive constant.

2.2 Existing Generalization Analysis for Imbalanced Learning

In balanced learning, we can directly minimize the empirical balanced risk defined on the balanced datasets S bal sampled from D bal :

Then, the generalization guarantee is available by traditional concentration techniques [19]. However, in imbalanced learning, we can only minimize the empirical risk on the imbalanced dataset S :

To handle this issue, Cao et al. [12] and Ren et al. [18] aggregate the class-wise generalization bound directly with a union bound over class-wise results [19]:

Proposition 1 (Union bound for Imbalanced Learning [12]) . Given the function set F and a loss function L : R C × Y → [0 , M ] , then for any δ ∈ (0 , 1) , with probability at least 1 -δ over the training set S , the following generalization bound holds for all g ∈ G :

where ˆ R L y ( f ) is the empirical risk on S y ; ˆ C S ( G ) := E ξ [sup g ∈G 1 N ∑ N n =1 ξ ( n ) g ( z ( n ) )] denotes the empirical complexity of the function set G , and ξ := ( ξ (1) , ξ (2) , · · · , ξ ( N ) ) are sampled from independent distributions such as the uniform distribution with { 1 , -1 } ; glyph[precedesorequal] denotes the asymptotic notation that omits undominated terms, that is, f ( t ) glyph[precedesorequal] g ( t ) ⇐⇒∃ a constant C > 0 , f ( t ) ≤ C · g ( t ) .

To further bound the complexity term ˆ C S y ( G ) , Cao et al. [12] assume that the loss function L satisfies the Lipschitz continuity and applies the traditional contraction lemma [20]:

Definition 1 (Lipschitz Continuity) . Let ‖ · ‖ denote the 2-norm. Then, we say the loss function L ( f, y ) is Lipschitz continuous with constant µ if for any f, f ′ ∈ F , x ∈ S ,

Lemma 1 (Contraction Lemma) . Assume that the loss function L ( f, x ) is Lipschitz continuous with a constant µ . Then, the following inequality holds:

Finally, the standard margin-based generalization bound [22] is directly applied to obtain the upper bound of ˆ C S y ( F ) . However, this union bound has the following limitations:

Theoretically, this generalization bound is coarse-grained and not sharp enough. To be specific, the differences among different loss functions lie in the choice of α y , β y , ∆ y . However, the Lipschitz continuity, which is the only property of L utilized in the proof, is global in nature and thus obscures these differences. Although the margin theory can provide some theoretical insights into the role of ∆ y , the roles of α y , β y are still a mystery. Besides, since

a sharper bound might be available if we can bound R L bal ( f ) directly.

Empirically, although the induced LDAM loss outperforms the CE loss, the improvement is not so significant. Fortunately, when combining the Deferred Re-Weighting (DRW) technique [12], where α y = (1 -p ) / (1 -p N y ) , p ∈ (0 , 1) [11] during the terminal phase of training, the improvement becomes much more impressive. However, Eq.(7) fails to explain this phenomenon.

Recently, Kini et al. [16] provide a generalization analysis for the VS loss. However, the results, which only hold for linear models with linearly separable data, can only explain the roles of β y . For the role of ∆ y , they resort to analyzing the gradient of the VS loss and provide a coarse-grained analysis.

To sum up, existing generalization analysis for imbalanced learning is coarse-grained and fragmented. Next, we aim to build a more fine-grained and systematical generalization bound that can unify the roles of both re-weighting and logit-adjustment.

3 Fine-Grained Generalization Analysis for Imbalanced Learning

In Sec.3.1, we first establish a sharp generalization bound based on a novel technique named datadependent contraction. Then, in Sec.3.2, we apply this generalization bound to the VS loss to provide a series of theoretical insights. Finally, in Sec.3.3, a principled algorithm is proposed based on the theoretical insights.

3.1 Generalization Bound Induced By Data-Dependent Contraction

Different from Eq.(7), we hope to build a direct bound between R L bal ( f ) and ̂ R L ( f ) . To this end, our analysis is based on the following lemma, whose proof can be found in Appendix B:

Lemma 2. Given the function set F and a loss function L : R C × Y → [0 , M ] , then for any δ ∈ (0 , 1) , with probability at least 1 -δ over the training set S , the following generalization bound holds for all g ∈ G :

where Φ( L, δ ) := 1 Cπ C [ ̂ R L ( f ) + 3 M √ log 2 /δ 2 N ] contains the empirical risk on S and the δ term.

Remark 1. Recall that π C := N C /N,N 1 ≥ N 2 ≥ · · · ≥ N C . Hence, this lemma reveals how the model performance depends on the imbalance degree of the data.

As shown in Sec.2.2, the fine-grained analysis is unavailable due to the global nature of the classic Lipschitz continuous property. In view of this, we extend this traditional definition with a local technique [23]:

Definition 2 (Local Lipschitz Continuity) . We say the loss function L ( f, y ) is local Lipschitz continuous with constants { µ y } C y =1 if for any f, f ′ ∈ F , y ∈ Y , x ∈ S y ,

Then, the following data-dependent contraction inequality helps us obtain a sharper bound, whose proof is given in Appendix C.

Assumption 1. Next, we assume that ˆ C S ( F ) ∼ O (1 / √ N ) . Note that this result holds for kernelbased models with traditional techniques [19] and neural networks with the latest techniques [24, 25]. And the prior arts also adopt this assumption [12].

Lemma 3 (Data-Dependent Contraction) . Assume that the loss function L ( f, x ) is local Lipschitz continuous with constants { µ y } C y =1 . Then, the following inequality holds under Asm.1:

Combining Lem.2 and Lem.3, we have the following theorem:

Theorem 1 (Data-Dependent Bound for Imbalanced Learning) . Given the function set F and a loss function L : R C ×Y → [0 , M ] , for any δ ∈ (0 , 1) , with probability at least 1 -δ over the training set S , the following generalization bound holds for all f ∈ F :

At the first glance, Eq.(14) seems a little loose since ∑ C y =1 √ π y > 1 . In fact, this intuition holds when local Lipschitz continuity degenerates to Def.1. However, when µ y is decreasing w.r.t. π y , a shaper bound might be available. To build an intuitive understanding, we present the following proposition, whose proof can be found in Appendix D.

Proposition 2. Assume that µ y ∝ N -κ y , κ > 0 . Then, when κ > 1 , the data-dependent bound presented in Thm.1 is sharper than the union bound defined in Prop.1.

3.2 Application to the VS Loss

Next, we apply Thm.1 to the VS loss to reveal the role of both re-weighting and logit-adjustment. To this end, it is necessary to analyze the local Lipschitz property of the VS loss, whose proof is presented in Appendix E.

Lemma 4. Assume that the score function is bounded. Then, the VS loss is local Lipschitz continuous with constants { µ y } C y =1 , where

˜ β y := √ β 2 y + ( ∑ y ′ = y β y ′ ) 2 ; softmax ( · ) denotes the softmax function; B y ( f ) denotes the minimal prediction on the ground-truth class y , i.e., B y ( f ) := min x ∈ S y f ( x ) y .

glyph[negationslash]

glyph[negationslash]

Remark 2. B y ( f ) is closely related to the minimal margin defined by margin ↓ y := min x ∈ S y ( f ( x ) y -max j = y f ( x ) j ) . It is not difficult to check that B y ( f ) -margin ↓ y ≤ max x ∈ S y ,j = y f ( x ) j . Hence, as we improve the model performance on class y , the RHS of the above inequality, i.e., the gap between B y ( f ) and margin ↓ y will decrease, and both the minimal margin and B y ( f ) will increase.

glyph[negationslash]

Then, combining Thm.1 and Lem.4, we have the following proposition, which reveals how the existing loss-oriented methods improve generalization performance by exploiting the data priors.

Proposition 3 (Data-Dependent Bound for the VS Loss) . Given the function set F and the VS loss L VS, for any δ ∈ (0 , 1) , with probability at least 1 -δ over the training set S , the following generalization bound holds for all f ∈ F :

From Eq.(16), we have the following insights, whose empirical validation can be found in Sec.4.2.

(In1) Why re-weighting and logit-adjustment are necessary? Due to the term √ π y and B y ( f ) , the generalization bound is also imbalanced among classes. Both re-weighting and logit-adjustment can obtain a sharper generalization bound by assigning different weights to the classes with different √ π y and B y ( f ) . In this process, α y mainly rebalances the generalization performance among classes, i.e. , √ π y , while β y and ∆ y focus on adjusting the imbalance of the terms B y ( f ) among classes.

(In2) Why the deferred scheme is necessary? As pointed out in [11, 17], weighting up the minority classes will cause difficulties and instability in optimization, especially when the distribution is extremely imbalanced. To fix this issue, Cao et al. [12] develops a deferred scheme, where α y = 1 and (1 -p ) / (1 -p N y ) , p ∈ (0 , 1) during the initial and terminal phase of training, respectively. Although this scheme shows significant improvement, there is still a lack of theoretical explanation.

Fortunately, Prop.3 can give us some inspiration. Specifically, although a weighted loss can boost the optimization on the minority classes, it is harmful to the further improvement on the majority classes, as shown in Fig.3. Hence, the majority/minority classes will have relatively small/large B y ( f ) respectively, and the generalization bound becomes even looser. By contrast, in the DRW scheme, we have α y = 1 during the initial phase of training. Such a warm-up phase will encourage the model to focus on the majority classes and induce a small B y ( f ) for both majority and minority classes after weighting up the minority classes. On top of this, the generalization bound can become sharper, which explains the effectiveness of the deferred scheme.

(In3) How does our result explain the design of existing losses? On one hand, for re-weighting losses, α y should decrease as π y increases, which is consistent with the balanced loss with α y = π -1 y [10] and α y = (1 -p ) / (1 -p N y ) , p ∈ (0 , 1) [11]. On the other hand, from the insight (In2) , we know that when α y = 1 , B y ( f ) will be increasing w.r.t. π y . Hence, for logit-adjustment losses, both β y and ∆ y should increase as π y increases. This insight is consistent with the LDAM loss ( ∆ y ∝ -N -1 / 4 y ) [12], the logit-adjusted loss ( ∆ y = τ log π y ) [13], and the CDT loss ( β y = ( N y /N 1 ) γ ) [14].

(In4) Are re-weigting and logit-adjustment fully compatible? (a) Unfortunately, the answer is negative. To be specific, the re-weighting term α y is decreasing w.r.t. π y , whereas the multiplicative logit-adjustment term β y is increasing w.r.t. π y . As a result, ˜ β y will weaken the effect of α y . (b) Fortunately, α y is compatible with the additive logit-adjustment term ∆ y since both terms can induce a sharper generalization bound.

3.3 Principled Learning Algorithm induced by the Theoretical Insights

In this part, we present a principled learning algorithm induced by the theoretical insights in Sec.3.2. First, according to (In1)-(In3) , it is crucial to comprehensively utilize re-weighting, logit-adjustment, and the DRW scheme, as they all contribute to improving the generalization bound. Second, according to (In4) , we propose a Truncated Logit-Adjustment (TLA) scheme to avoid the conflict between α y and β y . In this scheme, β y still increases w.r.t. π y during the initial phase of training but is truncated to 1 during the terminal phase of training. Third, we set α y ∝ π -ν y , ν > 0 to align α y with √ π y , which we name Aligned DRW (ADRW). Note that such a re-weighting scheme also follows the Fisher consistency property presented in [13]. Finally, the overall algorithm is summarized in Alg.1, where the logit-adjustment methods mentioned in Sec.2.1 are all reasonable options for β y , ∆ y in the line 5 and ∆ y in the line 7.

Algorithm 1: Principled Learning Algorithm induced by the Theoretical Insights

Require: Training set S = { ( x i , y i ) } N i =1 and a model f parameterized by Θ . 1: Initialize the model parameters Θ randomly. 2: for t = 1 , 2 , · · · , T do 3: B ← SampleMiniBatch ( S , m ) glyph[triangleright] A mini-batch of m samples 4: if t < T 0 then 5: Set α = 1 , β y , ∆ y glyph[triangleright] Adjust logits during the initial phase 6: else 7: Set α y ∝ π -ν y , β y = 1 , ∆ y , ν > 0 glyph[triangleright] TLA and ADRW 8: end if 9: L ( f, B ) ← 1 m ∑ ( x ,y ) ∈B L VS ( f ( x ) , y ) glyph[triangleright] Calculate the loss 10: Θ ← Θ -η ∇ Θ L ( f, B ) glyph[triangleright] One SGD step 11: Optional: anneal the learning rate η . glyph[triangleright] Required when t = T 0 12: end for

(a) CIFAR-10 LT

(b) CIFAR-10 Step

(c) CIFAR-100 LT

(d) CIFAR-100 Step

Balanced Accuracy

80

Balanced Accuracy

BalancedAccuracy

43

BalancedAccuracy

46

78

76

42

44

76

72

40

42

74

CE+None

CE+ADRW

68

39

40

72

LDAM+None

LDAM+ADRW

64

38

38

3.0

3.0

3.0

(a) CIFAR-10 LT

(b) CIFAR-10 Step

Average

Performance

08

80

Average

Performance

80

79

78

79

78

78

76

1.5

1.5

1.25

1.25

T

1.0

1.0

0.75

0.75

0.5

0.5

0.150.2

0.25

0.5

0.75

1.01.251.52.0

78

Average

80

0.05

0.1

0.15

0.2

0.25

0.5

0.75

1.0

1.251.5

2.0

78

Average

80

A

Performance

Performance

4 Experiments

4.1 Experiment Protocols

Here, we briefly introduce the experiment protocols, and more details can be found in Appendix F.

Datasets. We conduct the experiments on four popular benchmark datasets for imbalanced learning. (a) CIFAR-10 and CIFAR-100 : Following the protocol in [26, 11, 12], we consider two types of imbalance: long-tailed imbalance (LT) and step imbalance (Step). For both imbalance types, we

(a) CIFAR-100 LT (

ρ

= 100

)

(b) CIFAR-100 LT (

ρ

= 100

)

(c) CIFAR-100 LT (

ρ

= 100

1.00

Min. / Maj. Train Acc

4

CE+None

CE+DRW

Test Accuracy

40

Train Acc

0.75

3

0.50

30

CBMajor

2

CB Minor

0.25

CE+DRWMajor

CE+DRW Minor

M

1

20

CE+DRW

CE+None

0.00

0

0

50

100

150

200

0

50

100

150

0

50

100

150

)

(a) CIFAR-10 LT

(b) CIFAR-10 Step

(c) CIFAR-100 LT

(d) CIFAR-100 Step

88.8

Balanced Accuracy

BalancedAccuracy

60.0

口

89.0

57.5

88.6

55.0

88.5

55.0

Balanced

88.4

88.0

52.5

Balanced

50.0

88.2

88.0

VS+DRW

VS+None

87.5

87.0

口

VS+None

VS+DRW

47.5

50.0

口

VS+DRW

VS+None

45.0

40.0

口

VS+None

VS+DRW

0.05

0.10

0.15

0.20

0.25

0.15

0.20

0.25

0.05

0.10

0.15

0.20

0.25

0.05

0.10

0.15

0.20

0.25

Y

report the balanced accuracy averaged over 5 random seeds with an imbalance ratio ρ := N 1 /N C ∈ { 10 , 100 } . (b) ImageNet-LT and iNaturalist : We use the long-tailed version of the ImageNet dataset 2 [2] proposed by [27], and iNaturalist 3 [5] is a real-world long-tailed dataset.

Baselines and Competitors. For the CIFAR datasets, we aim to validate the theoretical results and the performance gain induced by the proposed method. Hence, we select the following baselines: the CE loss (CE) [19], the LDAM loss (LDAM) [12], LDAM+DRW [12], and the VS loss (VS) [16] that generalizes the LA loss [13] and the CDT loss [14]. We tune all the hyperparameters according to the suggestions in the original papers. For the ImageNet-LT and iNaturalist datasets, we select state-of-the-art methods, listed in Tab.2, as the competitors to validate the effectiveness of the method.

Implementation Details. We implement three instances of the proposed learning algorithm: the CE loss equipped with the ADRW scheme ( CE+ADRW ), the LDAM loss equipped with the ADRW scheme ( LDAM+ADRW ), and the VS loss equipped with the TLA and the ADRW scheme ( VS+TLA+ADRW ). We tune the hyperparameter ν , and the other hyperparameters follow those used in the baselines. In addition, we incorporate the Sharpness-Aware Minimization (SAM) technique [28] to facilitate the optimization of the minority classes, allowing them to escape saddle points and converge to flat minima [29].

4.2 Theory Validation

In this part, we aim to validate our theoretical insights presented in Sec.3.2 on the CIFAR datasets. Some more empirical results can be found in Appendix G.

0.05

0.10

Validation of (In1) and (In4-b). We report the model performance of the baselines w.r.t. the hyperparameters in Fig.1 and Fig.2. From these results, we can find that (1) Both CE+ADRW and LDAM perform better than CE. In other words, either re-weighting or logit-adjustment can boost the model performance. (2) LDAM+ADRW outperforms both CE+ADRW and LDAM, and for VS+ADRW, increasing the hyperparameters ν and τ appropriately can also bring performance gains. All these results validate the compatibility between re-weighting and the additive logit-adjustment.

Validation of (In2). We present a series of results in Fig.3, where the training accuracy of different classes represents the corresponding B f ( y ) . To be specific, Fig.3(a) demonstrates the trend of training accuracy on the majority classes and the minority classes. For the CB loss, the learning process only focuses on the minority classes and hinders the performance improvement on the majority classes (CB Major v.s. CB Minor). Hence, an extremely imbalanced B y ( f ) induces a poor generalization performance. By contrast, the DRW scheme first focuses on the majority classes (CE+DRW Major v.s. CE+DRW Minor with the training epoch t ≤ T 0 = 160 ) and then pays more attention to the minority classes during the terminal phase of training (CE+DRW Major v.s. CE+DRW Minor with the training epoch t > T 0 = 160 ). Benefiting from the DRW scheme, both the majority classes and the minority classes are well-trained and thus have a balanced term B y ( f ) (Fig.3(b)), leading to a corresponding improvement on the test accuracy (Fig.3(c)). Even if we remove the re-weighting term (CE+None), the imbalance degree of B y ( f ) is still consistent with the test performance. Note that the learning rate of CE+None is also decreased at the corresponding epoch T 0 , making the line in Figure 3(b) and 3(c) not constant.

Validation of (In4-a). In Fig.4, we present the model performance of the VS loss w.r.t. the hyperparameter of the multiplicative logit-adjustment γ . We can find that VS+DRW performs inferior to VS, especially when γ is large. This phenomenon validates the incompatibility between re-weighting and multiplicative logit-adjustment.

4.3 Performance Comparison

We report the empirical results on the CIFAR datasets in Tab.1, where the imbalance ratio ρ ∈ { 10 , 100 } . From these results, we have the following observations: (1) The proposed learning algorithm consistently outperforms the baselines, especially when the dataset is more imbalanced.

Such performance gains validate the effectiveness of the proposed methods (2) Both re-weighting and logit-adjustment can improve the model performance, which is consistent with the theoretical insight (In1) and (In4-b) . (3) When ρ = 10 or on the CIFAR-100 dataset, VS+DRW performs inferior to VS. Fortunately, when equipped with the proposed TLA scheme, VS+TLA+DRW outperforms both VS and VS+DRW. These results again validate our theoretical insight (In4-a) . (4) When ρ = 10 , CE+ADRW outperforms LDAM+ADRW, and similar counter-intuitive phenomena are also observed in [29]. We conjecture that in this case, re-weighting is enough to rebalance the generalization bound, and the additional LDAM loss might induce other issues such as inconsistency.

We present the overall balanced accuracy on the ImageNet-LT and iNaturalist datasets in Tab.2, where SAM and Ours denotes LDAM+DRW+SAM and VS+TLA+ADRW+SAM, respectively. These results demonstrate that the proposed learning algorithm outperforms the competitors, especially the one-stage ones, which again confirms the effectiveness of the proposed learning algorithm.

5 Conclusion and Future Work

In this work, with the proposed local Lipschitz property and the data-dependent contraction technique, we present a unified generalization analysis of the loss-modification approaches for imbalanced learning. Benefiting from this fine-grained analysis, we not only reveal the role of both re-weighting and logit-adjustment approaches but also explain some empirical phenomena that are out of the scope of existing theories. Moreover, a principled learning algorithm is proposed based on the theoretical insights. Finally, extensive experimental results on benchmark datasets validate our theoretical analysis and the effectiveness of the proposed method.

Theoretically, one important future work is to provide a systematical Fisher consistency analysis for the VS loss, providing more insights to design re-weighting and logit-adjustment terms. Methodologically, it might be a promising direction to design an adaptive scheme that can automatically determine the hyperparameters of the learning algorithm.

Acknowledgments

This work was supported in part by the National Key R&D Program of China under Grant 2018AAA0102000, in part by National Natural Science Foundation of China: 62236008, U21B2038, U2001202, 61931008, 62122075, 61976202 and 62206264, in part by the Fundamental Research Funds for the Central Universities, in part by Youth Innovation Promotion Association CAS, in part by the Strategic Priority Research Program of Chinese Academy of Sciences (Grant No. XDB28000000) and in part by the Innovation Funding of ICT, CAS under Grant No. E000000.

References

Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. pages 1-60, 2009.

Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael S. Bernstein, Alexander C. Berg, and Li FeiFei. Imagenet large scale visual recognition challenge. Int. J. Comput. Vis. , 115:211-252, 2015.

Mark Everingham, Luc Van Gool, Christopher K. I. Williams, John M. Winn, and Andrew Zisserman. The pascal visual object classes (VOC) challenge. Int. J. Comput. Vis. , 88:303-338, 2010.

Tsung-Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: common objects in context. In European Conference on Computer Vision , pages 740-755, 2014.

Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alexander Shepard, Hartwig Adam, Pietro Perona, and Serge J. Belongie. The inaturalist species classification and detection dataset. In IEEE Conference on Computer Vision and Pattern Recognition , pages 8769-8778, 2018.

Haibo He and Edwardo A. Garcia. Learning from imbalanced data. IEEE Trans. Knowl. Data Eng. , 21:1263-1284, 2009.

Grant Van Horn and Pietro Perona. The devil is in the tails: Fine-grained classification in the wild. CoRR , abs/1709.01450, 2017.

Kemal Oksuz, Baris Can Cam, Sinan Kalkan, and Emre Akbas. Imbalance problems in object detection: A review. IEEE Trans. Pattern Anal. Mach. Intell. , 43:3388-3415, 2021.

Yifan Zhang, Bingyi Kang, Bryan Hooi, Shuicheng Yan, and Jiashi Feng. Deep long-tailed learning: A survey. CoRR , abs/2110.04596, 2021.

Katharina Morik, Peter Brockhausen, and Thorsten Joachims. Combining statistical learning with a knowledge-based approach - A case study in intensive care monitoring. In International Conference on Machine Learning , pages 268-277, 1999.

Yin Cui, Menglin Jia, Tsung-Yi Lin, Yang Song, and Serge J. Belongie. Class-balanced loss based on effective number of samples. In IEEE Conference on Computer Vision and Pattern Recognition , pages 9268-9277, 2019.

Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Aréchiga, and Tengyu Ma. Learning imbalanced datasets with label-distribution-aware margin loss. In Annual Conference on Neural Information Processing Systems , pages 1565-1576, 2019.

Aditya Krishna Menon, Sadeep Jayasumana, Ankit Singh Rawat, Himanshu Jain, Andreas Veit, and Sanjiv Kumar. Long-tail learning via logit adjustment. In International Conference on Learning Representations , 2021.

Han-Jia Ye, Hong-You Chen, De-Chuan Zhan, and Wei-Lun Chao. Identifying and compensating for feature deviation in imbalanced deep learning. CoRR , abs/2001.01385, 2020.

Jingru Tan, Changbao Wang, Buyu Li, Quanquan Li, Wanli Ouyang, Changqing Yin, and Junjie Yan. Equalization loss for long-tailed object recognition. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11659-11668, 2020.

Ganesh Ramachandra Kini, Orestis Paraskevas, Samet Oymak, and Christos Thrampoulidis. Label-imbalanced and group-sensitive classification under overparameterization. In Annual Conference on Neural Information Processing Systems , pages 18970-18983, 2021.

Chen Huang, Yining Li, Chen Change Loy, and Xiaoou Tang. Learning deep representation for imbalanced classification. In IEEE Conference on Computer Vision and Pattern Recognition , pages 5375-5384, 2016.

Jiawei Ren, Cunjun Yu, Shunan Sheng, Xiao Ma, Haiyu Zhao, Shuai Yi, and Hongsheng Li. Balanced meta-softmax for long-tailed visual recognition. In Annual Conference on Neural Information Processing Systems , 2020.

Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar. Foundations of Machine Learning . The MIT Press, 2012.

Michel Ledoux and Michel Talagrand. Probability in Banach Spaces: isoperimetry and processes , volume 23. Springer Science & Business Media, 1991.

Aditya Krishna Menon, Harikrishna Narasimhan, Shivani Agarwal, and Sanjay Chawla. On the statistical consistency of algorithms for binary classification under class imbalance. In International Conference on Machine Learning , pages 603-611, 2013.

Sham M. Kakade, Karthik Sridharan, and Ambuj Tewari. On the complexity of linear prediction: Risk bounds, margin bounds, and regularization. In Annual Conference on Neural Information Processing Systems , pages 793-800, 2008.

Peter L. Bartlett, Olivier Bousquet, and Shahar Mendelson. Localized rademacher complexities. In Annual Conference on Computational Learning Theory , pages 44-58, 2002.

Noah Golowich, Alexander Rakhlin, and Ohad Shamir. Size-independent sample complexity of neural networks. In Conference On Learning Theory , pages 297-299, 2018.

Philip M. Long and Hanie Sedghi. Generalization bounds for deep convolutional neural networks. In International Conference on Learning Representations , 2020.

Mateusz Buda, Atsuto Maki, and Maciej A. Mazurowski. A systematic study of the class imbalance problem in convolutional neural networks. Neural Networks , 106:249-259, 2018.

Ziwei Liu, Zhongqi Miao, Xiaohang Zhan, Jiayun Wang, Boqing Gong, and Stella X. Yu. Large-scale long-tailed recognition in an open world. In IEEE Conference on Computer Vision and Pattern Recognition , pages 2537-2546, 2019.

Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware minimization for efficiently improving generalization. In International Conference on Learning Representations , 2021.

Harsh Rangwani, Sumukh K. Aithal, Mayank Mishra, and R. Venkatesh Babu. Escaping saddle points for effective generalization on class-imbalanced data. In Advances in Neural Information Processing Systems , pages 22791-22805, 2022.

Liuyu Xiang, Guiguang Ding, and Jungong Han. Learning from multiple experts: Self-paced knowledge distillation for long-tailed classification. In European Conference on Computer Vision , pages 247-263, 2020.

Boyan Zhou, Quan Cui, Xiu-Shen Wei, and Zhao-Min Chen. BBN: bilateral-branch network with cumulative learning for long-tailed visual recognition. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 9716-9725, 2020.

Bingyi Kang, Saining Xie, Marcus Rohrbach, Zhicheng Yan, Albert Gordo, Jiashi Feng, and Yannis Kalantidis. Decoupling representation and classifier for long-tailed recognition. In International Conference on Learning Representations , 2020.

Yin-Yin He, Jianxin Wu, and Xiu-Shen Wei. Distilling virtual examples for long-tailed recognition. In IEEE/CVF International Conference on Computer Vision , pages 235-244, 2021.

Songyang Zhang, Zeming Li, Shipeng Yan, Xuming He, and Jian Sun. Distribution alignment: A unified framework for long-tail visual recognition. In IEEE Conference on Computer Vision and Pattern Recognition , pages 2361-2370, 2021.

Shaden Alshammari, Yu-Xiong Wang, Deva Ramanan, and Shu Kong. Long-tailed recognition via weight balancing. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, , pages 6887-6897, 2022.

Kaihua Tang, Jianqiang Huang, and Hanwang Zhang. Long-tailed classification by keeping the good and removing the bad momentum causal effect. In Annual Conference on Neural Information Processing Systems , 2020.

Dvir Samuel and Gal Chechik. Distributional robustness loss for long-tail learning. In IEEE/CVF International Conference on Computer Vision , pages 9475-9484, 2021.

Lu Yang, He Jiang, Qing Song, and Jun Guo. A survey on long-tailed visual recognition. Int. J. Comput. Vis. , 130:1837-1872, 2022.

Miroslav Kubat and Stan Matwin. Addressing the curse of imbalanced training sets: One-sided selection. In International Conference on Machine Learning , pages 179-186, 1997.

Nitesh V Chawla, Kevin W Bowyer, Lawrence O Hall, and W Philip Kegelmeyer. Smote: synthetic minority over-sampling technique. Journal of artificial intelligence research , 16: 321-357, 2002.

Inderjeet Mani and I Zhang. knn approach to unbalanced data distributions: a case study involving information extraction. In Workshop on Learning from Imbalanced Datasets, International Conference on Machine Learning , pages 1-7, 2003.

Chris Drummond, Robert C Holte, et al. C4. 5, class imbalance, and cost sensitivity: why under-sampling beats over-sampling. In Workshop on Learning from Imbalanced Datasets II, International Conference on Machine Learning , pages 1-8, 2003.

Justin M. Johnson and Taghi M. Khoshgoftaar. Survey on deep learning with class imbalance. J. Big Data , 6:27, 2019.

Guillem Collell, Drazen Prelec, and Kaustubh R. Patil. Reviving threshold-moving: a simple plug-in bagging ensemble for binary and multiclass imbalanced data. CoRR , abs/1606.08698, 2016.

Jiarui Cai, Yizhou Wang, and Jenq-Neng Hwang. ACE: ally complementary experts for solving long-tailed recognition in one-shot. In IEEE/CVF International Conference on Computer Vision , pages 112-121, 2021.

Xudong Wang, Long Lian, Zhongqi Miao, Ziwei Liu, and Stella X. Yu. Long-tailed recognition by routing diverse distribution-aware experts. In International Conference on Learning Representations , 2021.

Jiequan Cui, Zhisheng Zhong, Shu Liu, Bei Yu, and Jiaya Jia. Parametric contrastive learning. In IEEE/CVF International Conference on Computer Vision , pages 695-704, 2021.

Zhisheng Zhong, Jiequan Cui, Shu Liu, and Jiaya Jia. Improving calibration for long-tailed recognition. In IEEE Conference on Computer Vision and Pattern Recognition , pages 1648916498, 2021.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In IEEE Conference on Computer Vision and Pattern Recognition , pages 770-778, 2016.

Ilya Sutskever, James Martens, George E. Dahl, and Geoffrey E. Hinton. On the importance of initialization and momentum in deep learning. In International Conference on Machine Learning , pages 1139-1147, 2013.

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Z. Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems , pages 8024-8035, 2019.

Charles R. Harris, K. Jarrod Millman, Stéfan van der Walt, Ralf Gommers, Pauli Virtanen, David Cournapeau, Eric Wieser, Julian Taylor, Sebastian Berg, Nathaniel J. Smith, Robert Kern, Matti Picus, Stephan Hoyer, Marten H. van Kerkwijk, Matthew Brett, Allan Haldane, Jaime Fernández del Río, Mark Wiebe, Pearu Peterson, Pierre Gérard-Marchant, Kevin Sheppard, Tyler Reddy, Warren Weckesser, Hameer Abbasi, Christoph Gohlke, and Travis E. Oliphant. Array programming with numpy. Nat. , 585:357-362, 2020.

Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, Jake VanderPlas, Alexandre Passos, David Cournapeau, Matthieu Brucher, Matthieu Perrot, and Edouard Duchesnay. Scikit-learn: Machine learning in python. J. Mach. Learn. Res. , 12:2825-2830, 2011.

Sébastien Marcel and Yann Rodriguez. Torchvision the machine-vision package of torch. In Proceedings of the 18th International Conference on Multimedia 2010, Firenze, Italy, October 25-29, 2010 , pages 1485-1488, 2010.

Peer reviews:
comments: SUMMARY:
For class-imbalanced learning, many studies modified the loss function to emphasize the learning on minority classes by reweighting or logit adjustment. These studies showed high epirical classification performance, but existing generalization analysis of such studies is not unified. In this paper, the authors propose a data-dependent contraction to capture how the modified losses affect each class. With the data-dependent contraction technique, a fine-grained generalization bound is established and the authors analyze existing class imbalanced learning studies in a unified manner by applying the generalization bound to the VS loss. The authors propose two principled algorithm : TLA and ADRW based on the theoretical insights, and verify effectiveness of the two proposed algorithm through extensive experiments on benchmark datasets.

STRENGTHS:
Well written paper, it is easy to follow the paper.

Great theoretical analysis and insights are given in this paper.

Novel algorithms are proposed based on the analysis and the insights.

Extensive experiments were conducted.

WEAKNESSES:
Code is not submitted. Thus I can not reproduce the experiment results in the paper.

Recently proposed class imbalanced learning algorithms based on multiple experts, contrastive learning, knowledge distillation are not reviewed, and also not compared in the section 4.3. It seems that the proposed algorithms achive lower classification performance than recently proposed algorithms such as PACO, RIDE and ACE.

QUESTIONS:
I have no questions.

LIMITATIONS:
The proposed algorithms (TLA, ADRW) improved classification performance, but the improvement seems not that significant under some settings (specifically between VS and VS+TLA+ADRW under CIFAR-10-LT, step w/sam and CIFAR-100-LT w/sam).
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-06-09

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors study the problem of class imbalance. To be specific, they identify that there is a gap between the generalization theory of re-weighting & logit adjustment techniques and the practice. To be specific, they identify that the existing generalization bounds fail to account the imbalance among classes. The authors propose to close the gap by introducing an imbalance-specific bound, and then perform an analysis with existing methods to gain more insights about the effects of certain design choices. Finally, they introduce a variation on existing loss functions that perform better than existing approaches.\n\nAfter the rebuttal:\n\nWith the rebuttal, the authors addressed my minor comments about clarifications on certain details. This is a solid NeurIPS paper. I recommend accepting the paper.\n\nSTRENGTHS:\n1. The paper provides theoretical as well as practical results & insights.\n\n2. The four insights are very valuable.\n\n3. Strong improvements over the baselines.\n\n4. Generally easy to follow text.\n\nWEAKNESSES:\nI am generally happy with the paper. My only concern is that it contains many typos and grammatical errors. Moreover, I would find it easier if Fig 1, 2 and 3 analyzed the performances with respect to the imbalance ratio as commonly performed in the literature.\n\n\nMinor comments:\n\n- \"[12] proposes an effective scheme\" => Reference numbers should not be a part of a sentence. A correct way to write this is \"Cao et al. [12] propose an effective scheme\". \n\n- The paper uses many many acronyms without introducing them.\n\n- Lines 18-45: Already in these lines you should start with formulations (of ERM, margin theory etc.) since this paper is making theoretical contributions.\n\n- Line 47: \"loss function utilized in existing proof\" => \"loss function utilized in existing proofs\"?\n\n- Line 111: \"Let ||.|| denotes\" => \"Let ||.|| denote\".\n\n- Line 114: \"constants \\mu\" => \"constant \\mu\".\n\n- Line 207: \"Do re-weigting and logit-adjustment fully compatible?\" => \"Are re-weigting and logit-adjustment fully compatible?\"\n\nQUESTIONS:\nNone.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-03"}

comments: SUMMARY:
This paper proposes a sharpened generalization bound of imbalance learning by directly bounding the balanced empirical risk. The authors achieve this by generalizing the Lipschitz Continuity to the Local Lipschitz Continuity with a group of constants, which, in VS Loss, is parameterized by a re-weighting term, a generalization term, and a logit adjustment term. By adjusting the above three terms with their proposed algorithm TLA and ADRW, the authors achieve keeping the balance between class balance and generalization of a model.

STRENGTHS:
This paper unified re-weighting and logit adjustment, two common methods for solving the imbalance problem, which I think is a good contribution to this field. Moreover, the authors' entry point is novel, providing sophisticated proof for their theory, the paper is well-written, and the experiments also well demonstrate their claims.

WEAKNESSES:
The authors only perform their experiment on the ResNet family for all the baselines and their proposed method. I hope the authors can provide the experiment result of their proposed method on different network structures in future work.

QUESTIONS:
I have no questions from the authors.

LIMITATIONS:
The authors have discussed their limitations in the form of future work in Section 5.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-04

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper provides a unified generalization analysis of the loss-modification approaches for imbalanced learning. It analyzes the gap between balanced accuracy and empirical loss (the loss may involve re-weighting and logit-adjustment approaches). It further provides empirical analysis that matches the theoretical insights. And a new method induced by the theoretical results with better performance is also proposed.\n\nSTRENGTHS:\n1. This paper directly establishes a theoretical connection between balanced accuracy and empirical loss, which is not attained in previous work. \n2. In this paper, the existing methods are systematically reviewed, and according to the new theoretical results, the existing methods are well classified and discussed. \n3. The charts and tables are beautifully formatted and laid out.\n\nWEAKNESSES:\n1. What we would like to bound is the balanced accuracy $\\mathcal{R}_{bal}$, where the loss is measured by $M$. However, in the imbalanced learning setting, $L$ is not just a differential surrogate version of $M$, but instead is an adjusted version that puts more emphasis on the small classes. In such circumstances, what is the meaning of bounding $\\mathcal{R}_{bal}^L$? Measuring under a class-balanced distribution while small classes are still emphasized in the loss? (By the way, please check the mapping of L in line 84.)\n2. How is ‘data-dependent’ exhibited in the theoretical result? Is it just about the sample size in each class, i.e., label distribution? Is there anything more than label distribution?\n3. In Eq.(14), what is By(f)? I checked the supplementary and I see it is a constant. What is the meaning of it? Is it the minimal margin on class y? I think these points should be made clear in the main body of the paper. And at the moment, I am also doubtful about the argument ‘balanced / imbalanced By(f)’ in the paragraph of line 192-200, which may involve the mechanisms in By(f). \n4. Maybe I misunderstood it, I suppose in Figure 3, T0 marks the start of DRW, and the total epoch T is fixed. Then why the line ‘CE+None’ is not constant in Figure 3(b) and 3(c)？\n5. In section 3.3, there is no mention of the setting of $\\Delta_y$. So does the setting of $\\beta_y$ in the $t<T_0$ stage. And in experiments, it would be better if provided more details on how these hyperparameters are tuned.\n\nQUESTIONS:\nSEE Weaknesses.\n\nLIMITATIONS:\nNAN.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}