SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Zitai Wang $^{1,2}$   
Yuan He $^{5}$
Qianqian Xu $^{3*}$   
Xiaochun Cao $^{6}$
Zhiyong Yang4 Qingming Huang4,3,7*
$^{1}$  SKLOIS, Institute of Information Engineering, CAS
$^{2}$  School of Cyber Security, University of Chinese Academy of Sciences Lab. of Intelligent Information Processing, Institute of Computing Tech., CAS
$^{4}$  School of Computer Science and Tech., University of Chinese Academy of Sciences, 5 Alibaba Group
$^{6}$  School of Cyber Science and Tech., Shenzhen Campus of Sun Yat-sen University, BDKM, University of Chinese Academy of Sciences
wangzitai@ie.ac.cn xuqianqian@ict.ac.cn yangzhiyong21@ucas.ac.cn heyuan.hy@alibaba-inc.com
caoxiaochun@mail.sysu.edu.cn qmhuang@ucas.ac.cn
Real-world datasets are typically imbalanced in the sense that only a few classes have numerous samples, while many classes are associated with only a few samples. As a result, a naive ERM learning process will be biased towards the majority classes, making it difficult to generalize to the minority classes. To address this issue, one simple but effective approach is to modify the loss function to emphasize the learning on minority classes, such as re-weighting the losses or adjusting the logits via class-dependent terms. However, existing generalization analysis of such losses is still coarse-grained and fragmented, failing to explain some empirical results. To bridge this gap, we propose a novel technique named data-dependent contraction to capture how these modified losses handle different classes. On top of this technique, a fine-grained generalization bound is established for imbalanced learning, which helps reveal the mystery of re-weighting and logit-adjustment in a unified manner. Furthermore, a principled learning algorithm is developed based on the theoretical insights. Finally, the empirical results on benchmark datasets not only validate the theoretical results but also demonstrate the effectiveness of the proposed method.
In recent years, machine learning has achieved great success with the help of well-collected datasets, where the number of samples is artificially balanced among classes [1, 2]. However, the real-world datasets are generally imbalanced in the sense that only a few classes have numerous samples (i.e., the majority ones), while the others are associated with only a few samples (i.e., the minority ones) [3-5]. Owing to this issue, a naïve Empirical Risk Minimization (ERM) learning process will be biased towards the majority classes, and the generalization on the minority ones becomes challenging. Hence, the imbalanced learning problem has attracted increasing attention in recent years [6-9].
One simple yet effective approach for imbalanced learning is to modify the naïve loss function, such that the learning process can pay more attention to the minority classes (Please refer to Appendix A for more orthogonal approaches). In this direction, existing approaches generally fall into two categories: re-weighting [10, 11] and logit-adjustment [12-16]. The former category assigns larger weights to
the losses of the minority classes. Although intuitive, this approach might lead to difficulties and instability in optimization [11, 12, 17]. To tackle this issue, Cao et al. [12] propose an effective scheme named Deferred Re-Weighting (DRW), where the re-weighting approach is applied only during the terminal phase of training. The latter category adjusts the logits by class-dependent terms. For example, the Label Distribution Aware Margin (LDAM) loss enforces larger margins for minority classes to achieve strong regularization [12]. The Logit-Adjustment (LA) loss [13] and the Class-Dependent Temperatures (CDT) loss [14] utilize additive and multiplicative terms to adjust the logits, respectively. Most recently, Kini et al. [16] combine the two types of terms and proposes a unified loss named Vector-Scaling (VS) for imbalanced learning.
Although existing loss-modification methods have achieved promising performance, the theoretical insights are still fragmented and coarse-grained. To be specific, Cao et al. [12] and Ren et al. [18] utilize the classic margin theory to explain the necessity of the additive terms in the LDAM loss. However, the theory fails to explain the significant improvement induced by the DRW scheme. Menon et al. [13] analyzes the Fisher consistency property [19] of the additive terms in the LA loss, while providing no further generalization analysis. Kini et al. [16] provides a generalization analysis of the VS loss, but the results can only explain the role of the multiplicative terms under the assumption that a linear model is trained on linearly separable data. Besides, we find that the VS loss is rather incompatible with the DRW scheme, which is also out of the scope of existing theory. Hence, a gap still exists between the theory and the practice of the loss-modification approaches.
To bridge this gap, this paper provides a systematical and fine-grained analysis of loss-modification approaches. After revisiting prior arts, we find that the only property of the loss function utilized in existing proofs is the classic Lipschitz continuity [19, 20]. However, this property is global in nature such that the whole analysis provides no insight into how the losses handle different classes. Inspired by this observation, we extend the classic Lipschitz continuity with a local technique. In this way, the local Lipschitz constants on different classes exactly correspond to the class-dependent terms of the modified loss functions. And a fine-grained generalization bound is established by a novel technique named data-dependent contraction. By applying this bound to the VS loss, the mystery of re-weighting and logit-adjustment is finally uncovered. Last but not least, a principled learning algorithm is proposed based on our theoretical insights.
To sum up, the main contributions of this paper are listed as follows:
- New technique. We extend the classic Lipschitz continuity and propose a novel technique named data-dependent contraction to obtain a fine-grained generalization bound for imbalanced learning.
- Theoretical insights. Based on the fine-grained bound, a systematical analysis succeeds in explaining the role of re-weighting and logit-adjustment in a unified manner, as well as some empirical results that are out of the scope of existing theories.
- Principled Algorithm. A principled algorithm is proposed based on the insights, where the reweighting term is aligned with the generalization bound, and the multiplicative logit-adjustment term is removed during the DRW phase to avoid the incompatibility between terms.
- Empirical Validation. The empirical results on multiple benchmark datasets not only validate the theoretical results, but also demonstrate the superiority of the proposed method.
We first introduce the basic notations and the imbalanced learning problem in Sec.2.1. Then, we briefly review existing generalization analysis for imbalanced learning in Sec.2.2.
2.1 Notations and Problem Definition
We assume that the samples are drawn i.i.d. from a product space  $\mathcal{Z} = \mathcal{X}\times \mathcal{Y}$ , where  $\mathcal{X}$  is the input space and  $\mathcal{V} = \{1,\dots ,C\}$  is the label space. Let  $\mathcal{S} = \{(x^{(n)},y^{(n)})\}_{n = 1}^{N}$  be the imbalanced training set sampled from the imbalanced distribution  $\mathcal{D}$  defined on  $\mathcal{Z}$ ,  $\mathcal{S}_y = \{\pmb {x}\mid (\pmb {x},\pmb {y})\in \mathcal{S}\}$  be the set of samples from the class  $y$ ,  $N_{y}\coloneqq |\mathcal{S}_{y}|$  denote the size of  $\mathcal{S}_y$ , and  $\pi_y\coloneqq N_y / N$ . Without loss of generality, we assume that  $N_{1}\geq N_{2}\geq \dots \geq N_{C}$ .
Let  $\mathcal{D}_{\mathrm{bal}}$  be the balanced distribution defined on  $\mathcal{Z}$ . Specifically, a class  $y$  is first uniformly sampled from  $\mathcal{Y}$ , and then the input  $\pmb{x}$  is sampled from the class-conditional distribution  $\mathcal{D}_y \coloneqq \mathbb{P}[\pmb{x} \mid y]$ .
Then, our task is to learn a score function  $f: \mathcal{X} \to \mathbb{R}^C$  to minimize the risk defined on the balanced distribution:
$$
\mathcal {R} _ {\text {b a l}} (f) := \frac {1}{C} \sum_ {y = 1} ^ {C} \mathcal {R} _ {y} (f) = \frac {1}{C} \sum_ {y = 1} ^ {C} \underset {\boldsymbol {x} \sim \mathcal {D} _ {y}} {\mathbb {E}} [ M (f (\boldsymbol {x}), y) ], \tag {1}
$$
where  $\mathcal{R}_y$  is the risk defined on the class  $y$ , and  $M: \mathbb{R}^C \times \mathcal{Y} \to \mathbb{R}_+$  is the measure that evaluates the model performance at  $z \in \mathcal{Z}$ . For example, one of the most popular choices is to check whether the top-1 prediction is right:  $M(f(x), y) = 1$  [ $y \notin \arg \max_{y' \in \mathcal{Y}} f(x)_{y'}$ ], where  $\mathbf{1}[\cdot]$  is the indicator function. Since  $M$  is generally non-differential and thus hard to optimize, one has to select a differential surrogate loss  $L: \mathbb{R}^C \times \mathcal{Y} \to \mathbb{R}_+$ , which induces the following surrogate risk:
$$
\mathcal {R} _ {\text {b a l}} ^ {L} (f) := \frac {1}{C} \sum_ {y = 1} ^ {C} \mathcal {R} _ {y} ^ {L} (f) = \frac {1}{C} \sum_ {y = 1} ^ {C} \underset {\boldsymbol {x} \sim \mathcal {D} _ {y}} {\mathbb {E}} [ L (f (\boldsymbol {x}), y) ]. \tag {2}
$$
Let  $\mathcal{G} \coloneqq \{L \circ f : f \in \mathcal{F}\}$  denote the hypothesis set. Next, we consider a family of loss functions named Vector-Scaling (VS) [16]:
$$
L _ {\mathrm {V S}} (f (\boldsymbol {x}), y) = - \alpha_ {y} \log \left(\frac {e ^ {\beta_ {y} f (\boldsymbol {x}) _ {y} + \Delta_ {y}}}{\sum_ {y ^ {\prime}} e ^ {\beta_ {y ^ {\prime}} f (\boldsymbol {x}) _ {y ^ {\prime}} + \Delta_ {y ^ {\prime}}}}\right). \tag {3}
$$
The advantage behind this loss family is two-fold. On one hand, the VS loss generalizes popular re-weighting and logit-adjustment methods. For example, when  $\alpha_{y} = 1, \beta_{y} = 1, \Delta_{y} = 0$ , it becomes the traditional CE loss [19]. When  $\beta_{y} = 1, \Delta_{y} = 0$ , re-weighting terms  $\alpha_{y} = \pi_{y}^{-1}$  and  $\alpha_{y} = (1 - p) / (1 - p^{N_{y}})$ ,  $p \in (0,1)$  recover the classic balanced loss [10] and Class-Balanced (CB) loss [11], respectively.  $\alpha_{y} = 1, \beta_{y} = 1, \Delta_{y} = \tau \log \pi_{y}, \tau > 0$  yield the LA loss [13]. When  $\alpha_{1} = 1, \beta_{y} = (N_{y} / N_{1})^{\gamma}, \Delta_{y} = 0, \gamma > 0$ , we can deduce the CDT loss [14]. On the other hand, an ideal surrogate loss should be Fisher consistent such that minimizing  $\mathcal{R}_{\mathrm{bal}}^{L}(f)$  not only can put more emphasis on minority classes, but also helps bound  $\mathcal{R}_{\mathrm{bal}}(f)$  [19, 21]. Fortunately, prior arts [13] have shown that a subset of the VS loss family satisfies such a property:
$$
L _ {\text {F i s h e r}} (f (\boldsymbol {x}), y) := \frac {\delta_ {y}}{\pi_ {y}} \log [ 1 + \sum_ {y ^ {\prime} \neq y} \frac {\delta_ {y ^ {\prime}}}{\delta_ {y}} e ^ {f (\boldsymbol {x}) _ {y ^ {\prime}} - f (\boldsymbol {x}) _ {y}} ], \tag {4}
$$
where  $\delta_y$  is an arbitrary positive constant.
2.2 Existing Generalization Analysis for Imbalanced Learning
In balanced learning, we can directly minimize the empirical balanced risk defined on the balanced datasets  $S_{\mathrm{bal}}$  sampled from  $D_{\mathrm{bal}}$ :
$$
\widehat {\mathcal {R}} _ {\text {b a l}} ^ {L} (f) := \frac {1}{N} \sum_ {(\boldsymbol {x}, y) \in \mathcal {S} _ {\text {b a l}}} L (f (\boldsymbol {x}), y). \tag {5}
$$
Then, the generalization guarantee is available by traditional concentration techniques [19]. However, in imbalanced learning, we can only minimize the empirical risk on the imbalanced dataset  $S$ :
$$
\widehat {\mathcal {R}} ^ {L} (f) := \frac {1}{N} \sum_ {(\boldsymbol {x}, y) \in \mathcal {S}} L (f (\boldsymbol {x}), y). \tag {6}
$$
To handle this issue, Cao et al. [12] and Ren et al. [18] aggregate the class-wise generalization bound directly with a union bound over class-wise results [19]:
Proposition 1 (Union bound for Imbalanced Learning [12]). Given the function set  $\mathcal{F}$  and a loss function  $L:\mathbb{R}^C\times \mathcal{V}\to [0,M]$ , then for any  $\delta \in (0,1)$ , with probability at least  $1 - \delta$  over the training set  $\mathcal{S}$ , the following generalization bound holds for all  $g\in \mathcal{G}$ :
$$
\mathcal {R} _ {b a l} ^ {L} (f) = \frac {1}{C} \sum_ {y = 1} ^ {C} \mathcal {R} _ {y} ^ {L} (f) \precsim \frac {1}{C} \sum_ {y = 1} ^ {C} \left(\hat {\mathcal {R}} _ {y} ^ {L} (f) + \hat {\mathfrak {C}} _ {\mathcal {S} _ {y}} (\mathcal {G}) + 3 M \sqrt {\frac {\log 2 C / \delta}{2 N _ {y}}}\right), \tag {7}
$$
where  $\hat{\mathcal{R}}_y^L (f)$  is the empirical risk on  $S_{y}$ ;  $\hat{\mathfrak{C}}_{\mathcal{S}}(\mathcal{G})\coloneqq \mathbb{E}_{\pmb{\xi}}[\sup_{g\in \mathcal{G}}\frac{1}{N}\sum_{n = 1}^{N}\xi^{(n)}g(\pmb{z}^{(n)})]$  denotes the empirical complexity of the function set  $\mathcal{G}$ , and  $\pmb {\xi}\coloneqq (\xi^{(1)},\xi^{(2)},\dots ,\xi^{(N)})$  are sampled from independent distributions such as the uniform distribution with  $\{1, - 1\}$ ;  $\precsim$  denotes the asymptotic notation that omits undominated terms, that is,  $f(t)\precsim g(t)\iff \exists a$  constant  $C > 0,f(t)\leq C\cdot g(t)$ .
To further bound the complexity term  $\hat{\mathfrak{C}}_{S_y}(\mathcal{G})$ , Cao et al. [12] assume that the loss function  $L$  satisfies the Lipschitz continuity and applies the traditional contraction lemma [20]:
Definition 1 (Lipschitz Continuity). Let  $\| \cdot \|$  denote the 2-norm. Then, we say the loss function  $L(f,y)$  is Lipschitz continuous with constant  $\mu$  if for any  $f,f^{\prime}\in \mathcal{F},\boldsymbol {x}\in \mathcal{S},$
$$
\left| L (f, y) - L \left(f ^ {\prime}, y\right) \right| \leq \mu \cdot \| f (\boldsymbol {x}) - f ^ {\prime} (\boldsymbol {x}) \|. \tag {8}
$$
Lemma 1 (Contraction Lemma). Assume that the loss function  $L(f, \pmb{x})$  is Lipschitz continuous with a constant  $\mu$ . Then, the following inequality holds:
$$
\hat {\mathfrak {C}} _ {\mathcal {S}} (\mathcal {G}) \leq \mu \cdot \hat {\mathfrak {C}} _ {\mathcal {S}} (\mathcal {F}). \tag {9}
$$
Finally, the standard margin-based generalization bound [22] is directly applied to obtain the upper bound of  $\hat{\mathfrak{C}}_{\mathcal{S}_y}(\mathcal{F})$ . However, this union bound has the following limitations:
- Theoretically, this generalization bound is coarse-grained and not sharp enough. To be specific, the differences among different loss functions lie in the choice of  $\alpha_y, \beta_y, \Delta_y$ . However, the Lipschitz continuity, which is the only property of  $L$  utilized in the proof, is global in nature and thus obscures these differences. Although the margin theory can provide some theoretical insights into the role of  $\Delta_y$ , the roles of  $\alpha_y, \beta_y$  are still a mystery. Besides, since
$$
\operatorname {B o u n d} \left(\mathcal {R} _ {\text {b a l}} ^ {L} (f)\right) = \frac {1}{C} \operatorname {B o u n d} \left(\sum_ {y} \mathcal {R} _ {y} ^ {L} (f)\right) \leq \frac {1}{C} \sum_ {y} \operatorname {B o u n d} \left(\mathcal {R} _ {y} ^ {L} (f)\right), \tag {10}
$$
a sharper bound might be available if we can bound  $\mathcal{R}_{\mathrm{bal}}^L (f)$  directly.
- Empirically, although the induced LDAM loss outperforms the CE loss, the improvement is not so significant. Fortunately, when combining the Deferred Re-Weighting (DRW) technique [12], where  $\alpha_{y} = (1 - p) / (1 - p^{N_{y}}), p \in (0, 1)$  during the terminal phase of training, the improvement becomes much more impressive. However, Eq.(7) fails to explain this phenomenon.
Recently, Kini et al. [16] provide a generalization analysis for the VS loss. However, the results, which only hold for linear models with linearly separable data, can only explain the roles of  $\beta_{y}$ . For the role of  $\Delta_y$ , they resort to analyzing the gradient of the VS loss and provide a coarse-grained analysis.
To sum up, existing generalization analysis for imbalanced learning is coarse-grained and fragmented. Next, we aim to build a more fine-grained and systematical generalization bound that can unify the roles of both re-weighting and logit-adjustment.
3 Fine-Grained Generalization Analysis for Imbalanced Learning
In Sec.3.1, we first establish a sharp generalization bound based on a novel technique named data-dependent contraction. Then, in Sec.3.2, we apply this generalization bound to the VS loss to provide a series of theoretical insights. Finally, in Sec.3.3, a principled algorithm is proposed based on the theoretical insights.
3.1 Generalization Bound Induced By Data-Dependent Contraction
Different from Eq.(7), we hope to build a direct bound between  $\mathcal{R}_{\mathrm{bal}}^L (f)$  and  $\widehat{\mathcal{R}}^L (f)$ . To this end, our analysis is based on the following lemma, whose proof can be found in Appendix B:
Lemma 2. Given the function set  $\mathcal{F}$  and a loss function  $L: \mathbb{R}^C \times \mathcal{Y} \to [0, M]$ , then for any  $\delta \in (0, 1)$ , with probability at least  $1 - \delta$  over the training set  $\mathcal{S}$ , the following generalization bound holds for all  $g \in \mathcal{G}$ :
$$
\mathcal {R} _ {b a l} ^ {L} (f) \preceq \Phi (L, \delta) + \frac {1}{C \pi_ {C}} \cdot \hat {\mathfrak {C}} _ {S} (\mathcal {G}), \tag {11}
$$
where  $\Phi(L, \delta) := \frac{1}{C\pi_C} [\widehat{\mathcal{R}}^L(f) + 3M\sqrt{\frac{\log 2 / \delta}{2N}}]$  contains the empirical risk on  $\mathcal{S}$  and the  $\delta$  term.
Remark 1. Recall that  $\pi_C\coloneqq N_C / N,N_1\geq N_2\geq \dots \geq N_C$ . Hence, this lemma reveals how the model performance depends on the imbalance degree of the data.
As shown in Sec.2.2, the fine-grained analysis is unavailable due to the global nature of the classic Lipschitz continuous property. In view of this, we extend this traditional definition with a local technique [23]:
Definition 2 (Local Lipschitz Continuity). We say the loss function  $L(f, y)$  is local Lipschitz continuous with constants  $\{\mu_y\}_{y=1}^C$  if for any  $f, f' \in \mathcal{F}, y \in \mathcal{Y}, x \in \mathcal{S}_y$ ,
$$
\left| L (f, y) - L \left(f ^ {\prime}, y\right) \right| \leq \mu_ {y} \cdot \| f (\boldsymbol {x}) - f ^ {\prime} (\boldsymbol {x}) \| \tag {12}
$$
Then, the following data-dependent contraction inequality helps us obtain a sharper bound, whose proof is given in Appendix C.
Assumption 1. Next, we assume that  $\hat{\mathfrak{C}}_S(\mathcal{F})\sim \mathcal{O}(1 / \sqrt{N})$ . Note that this result holds for kernel-based models with traditional techniques [19] and neural networks with the latest techniques [24, 25]. And the prior arts also adopt this assumption [12].
Lemma 3 (Data-Dependent Contraction). Assume that the loss function  $L(f, \mathbf{x})$  is local Lipschitz continuous with constants  $\{\mu_y\}_{y=1}^C$ . Then, the following inequality holds under Asm.1:
$$
\hat {\mathfrak {C}} _ {S} (\mathcal {G}) \precsim \hat {\mathfrak {C}} _ {S} (\mathcal {F}) \sum_ {y = 1} ^ {C} \mu_ {y} \sqrt {\pi_ {y}}, \tag {13}
$$
Combining Lem.2 and Lem.3, we have the following theorem:
Theorem 1 (Data-Dependent Bound for Imbalanced Learning). Given the function set  $\mathcal{F}$  and a loss function  $L:\mathbb{R}^C\times \mathcal{V}\to [0,M]$ , for any  $\delta \in (0,1)$ , with probability at least  $1 - \delta$  over the training set  $\mathcal{S}$ , the following generalization bound holds for all  $f\in \mathcal{F}$ :
$$
\mathcal {R} _ {b a l} ^ {L} (f) \preceq \Phi (L, \delta) + \frac {\hat {\mathfrak {C}} _ {\mathcal {S}} (\mathcal {F})}{C \pi_ {C}} \sum_ {y = 1} ^ {C} \mu_ {y} \sqrt {\pi_ {y}}. \tag {14}
$$
At the first glance, Eq.(14) seems a little loose since  $\sum_{y=1}^{C} \sqrt{\pi_y} > 1$ . In fact, this intuition holds when local Lipschitz continuity degenerates to Def.1. However, when  $\mu_y$  is decreasing w.r.t.  $\pi_y$ , a shaper bound might be available. To build an intuitive understanding, we present the following proposition, whose proof can be found in Appendix D.
Proposition 2. Assume that  $\mu_y \propto N_y^{-\kappa}, \kappa > 0$ . Then, when  $\kappa > 1$ , the data-dependent bound presented in Thm.1 is sharper than the union bound defined in Prop.1.
3.2 Application to the VS Loss
Next, we apply Thm.1 to the VS loss to reveal the role of both re-weighting and logit-adjustment. To this end, it is necessary to analyze the local Lipschitz property of the VS loss, whose proof is presented in Appendix E.
Lemma 4. Assume that the score function is bounded. Then, the VS loss is local Lipschitz continuous with constants  $\{\mu_y\}_{y=1}^C$ , where
$$
\mu_ {y} = \alpha_ {y} \tilde {\beta} _ {y} [ 1 - \operatorname {s o f t m a x} (\beta_ {y} B _ {y} (f) + \Delta_ {y}) ], \tag {15}
$$
$\tilde{\beta}_y \coloneqq \sqrt{\beta_y^2 + \left(\sum_{y' \neq y} \beta_{y'}\right)^2}$ ; softmax ( $\cdot$ ) denotes the softmax function;  $B_y(f)$  denotes the minimal prediction on the ground-truth class  $y$ , i.e.,  $B_y(f) \coloneqq \min_{\mathbf{x} \in S_y} f(\mathbf{x})_y$ .
Remark 2.  $B_{y}(f)$  is closely related to the minimal margin defined by  $\text{margin}_y^{\downarrow} \coloneqq \min_{\mathbf{x} \in S_y} (f(\mathbf{x})_y - \max_{j \neq y} f(\mathbf{x})_j)$ . It is not difficult to check that  $B_{y}(f) - \text{margin}_y^{\downarrow} \leq \max_{\mathbf{x} \in S_y, j \neq y} f(\mathbf{x})_j$ . Hence, as we improve the model performance on class  $y$ , the RHS of the above inequality, i.e., the gap between  $B_{y}(f)$  and  $\text{margin}_y^{\downarrow}$  will decrease, and both the minimal margin and  $B_{y}(f)$  will increase.
Then, combining Thm.1 and Lem.4, we have the following proposition, which reveals how the existing loss-oriented methods improve generalization performance by exploiting the data priors.
Proposition 3 (Data-Dependent Bound for the VS Loss). Given the function set  $\mathcal{F}$  and the VS loss  $L_{VS}$ , for any  $\delta \in (0,1)$ , with probability at least  $1 - \delta$  over the training set  $\mathcal{S}$ , the following generalization bound holds for all  $f \in \mathcal{F}$ :
$$
\mathcal {R} _ {b a l} ^ {L} (f) \precsim \Phi \left(L _ {V S}, \delta\right) + \frac {\hat {\mathfrak {C}} _ {\mathcal {S}} (\mathcal {F})}{C \pi_ {C}} \sum_ {y = 1} ^ {C} \alpha_ {y} \tilde {\beta} _ {y} \sqrt {\pi_ {y}} [ 1 - \text {s o f t m a x} \left(\beta_ {y} B _ {y} (f) + \Delta_ {y}\right) ]. \tag {16}
$$
From Eq.(16), we have the following insights, whose empirical validation can be found in Sec.4.2.
(In1) Why re-weighting and logit-adjustment are necessary? Due to the term  $\sqrt{\pi_y}$  and  $B_{y}(f)$ , the generalization bound is also imbalanced among classes. Both re-weighting and logit-adjustment can obtain a sharper generalization bound by assigning different weights to the classes with different  $\sqrt{\pi_y}$  and  $B_{y}(f)$ . In this process,  $\alpha_{y}$  mainly rebalances the generalization performance among classes, i.e.,  $\sqrt{\pi_y}$ , while  $\beta_{y}$  and  $\Delta_y$  focus on adjusting the imbalance of the terms  $B_{y}(f)$  among classes.
(In2) Why the deferred scheme is necessary? As pointed out in [11, 17], weighting up the minority classes will cause difficulties and instability in optimization, especially when the distribution is extremely imbalanced. To fix this issue, Cao et al. [12] develops a deferred scheme, where  $\alpha_{y} = 1$  and  $(1 - p) / (1 - p^{N_y}),p\in (0,1)$  during the initial and terminal phase of training, respectively. Although this scheme shows significant improvement, there is still a lack of theoretical explanation.