SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Fortunately, Prop.3 can give us some inspiration. Specifically, although a weighted loss can boost the optimization on the minority classes, it is harmful to the further improvement on the majority classes, as shown in Fig.3. Hence, the majority/minority classes will have relatively small/large  $B_{y}(f)$  respectively, and the generalization bound becomes even looser. By contrast, in the DRW scheme, we have  $\alpha_{y} = 1$  during the initial phase of training. Such a warm-up phase will encourage the model to focus on the majority classes and induce a small  $B_{y}(f)$  for both majority and minority classes after weighting up the minority classes. On top of this, the generalization bound can become sharper, which explains the effectiveness of the deferred scheme.
(In3) How does our result explain the design of existing losses? On one hand, for re-weighting losses,  $\alpha_{y}$  should decrease as  $\pi_y$  increases, which is consistent with the balanced loss with  $\alpha_{y} = \pi_{y}^{-1}$  [10] and  $\alpha_{y} = (1 - p) / (1 - p^{N_{y}}),p\in (0,1)$  [11]. On the other hand, from the insight (In2), we know that when  $\alpha_{y} = 1$ ,  $B_{y}(f)$  will be increasing w.r.t.  $\pi_y$ . Hence, for logit-adjustment losses, both  $\beta_{y}$  and  $\Delta_y$  should increase as  $\pi_y$  increases. This insight is consistent with the LDAM loss  $(\Delta_y\propto -N_y^{-1 / 4})$  [12], the logit-adjusted loss  $(\Delta_y = \tau \log \pi_y)$  [13], and the CDT loss  $(\beta_{y} = (N_{y} / N_{1})^{\gamma})$  [14].
(In4) Are re-weighting and logit-adjustment fully compatible? (a) Unfortunately, the answer is negative. To be specific, the re-weighting term  $\alpha_{y}$  is decreasing w.r.t.  $\pi_y$ , whereas the multiplicative logit-adjustment term  $\beta_{y}$  is increasing w.r.t.  $\pi_y$ . As a result,  $\tilde{\beta}_{y}$  will weaken the effect of  $\alpha_{y}$ . (b) Fortunately,  $\alpha_{y}$  is compatible with the additive logit-adjustment term  $\Delta_{y}$  since both terms can induce a sharper generalization bound.
3.3 Principled Learning Algorithm induced by the Theoretical Insights
In this part, we present a principled learning algorithm induced by the theoretical insights in Sec.3.2. First, according to (In1)-(In3), it is crucial to comprehensively utilize re-weighting, logit-adjustment, and the DRW scheme, as they all contribute to improving the generalization bound. Second, according to (In4), we propose a Truncated Logit-Adjustment (TLA) scheme to avoid the conflict between  $\alpha_{y}$  and  $\beta_{y}$ . In this scheme,  $\beta_{y}$  still increases w.r.t.  $\pi_{y}$  during the initial phase of training but is truncated to 1 during the terminal phase of training. Third, we set  $\alpha_{y} \propto \pi_{y}^{-\nu}, \nu > 0$  to align  $\alpha_{y}$  with  $\sqrt{\pi_{y}}$ , which we name Aligned DRW (ADRW). Note that such a re-weighting scheme also follows the Fisher consistency property presented in [13]. Finally, the overall algorithm is summarized in Alg.1, where the logit-adjustment methods mentioned in Sec.2.1 are all reasonable options for  $\beta_{y}, \Delta_{y}$  in the line 5 and  $\Delta_{y}$  in the line 7.
Algorithm 1: Principled Learning Algorithm induced by the Theoretical Insights
Require: Training set  $S = \{(x_i,y_i)\}_{i = 1}^N$  and a model  $f$  parameterized by  $\Theta$ .  
1: Initialize the model parameters  $\Theta$  randomly.  
2: for  $t = 1,2,\dots ,T$  do  
3:  $\mathcal{B}\gets \mathrm{SampleMiniBatch}(S,m)$  ▷A mini-batch of  $m$  samples  
4: if  $t < T_0$  then  
5: Set  $\alpha = 1,\beta_y,\Delta_y$  ▷Adjust logits during the initial phase  
6: else  
7: Set  $\alpha_{y}\propto \pi_{y}^{-\nu},\beta_{y} = 1,\Delta_{y},\nu >0$  ▷TLA and ADRW  
8: end if  
9:  $L(f,\mathcal{B})\gets \frac{1}{m}\sum_{(\boldsymbol {x},\boldsymbol {y})\in \mathcal{B}}L_{\mathrm{VS}}(f(\boldsymbol {x}),\boldsymbol {y})$  ▷Calculate the loss  
10:  $\Theta \gets \Theta -\eta \nabla_{\Theta}L(f,\mathcal{B})$  ▷One SGD step  
11: Optional: anneal the learning rate  $\eta$ . ▷Required when  $t = T_0$   
12: end for
(a) CIFAR-10 LT
(b) CIFAR-10 Step
(c) CIFAR-100 LT
(d) CIFAR-100 Step
Figure 1: The balanced accuracy of the CE loss and the LDAM loss w.r.t.  $\alpha_{y} \propto \pi_{y}^{-\nu}$  on the CIFAR datasets, where the imbalance ratio  $\rho = 100$ . Both re-weighting and logit-adjustment boost the model performance, which is consistent with the theoretical insight (In1) and (In4-b).
(a) CIFAR-10 LT
Figure 2: Sensitivity analysis of VS+ADRW w.r.t.  $\alpha_{y} \propto \pi_{y}^{-\nu}$  and  $\Delta_{y} = \tau \log \pi_{y}$  on the CIFAR-10 dataset, where the imbalance ratio  $\rho = 100$ . Both re-weighting and logit-adjustment boost the model performance, which is consistent with the theoretical insights (In1) and (In4-b).
4.1 Experiment Protocols
Here, we briefly introduce the experiment protocols, and more details can be found in Appendix F.
Datasets. We conduct the experiments on four popular benchmark datasets for imbalanced learning. (a) CIFAR-10 and CIFAR-100: Following the protocol in [26, 11, 12], we consider two types of imbalance: long-tailed imbalance (LT) and step imbalance (Step). For both imbalance types, we
(a) CIFAR-100 LT  $(\rho = 100)$
(b) CIFAR-100 LT  $(\rho = 100)$
(c) CIFAR-100 LT  $(\rho = 100)$
Figure 3: (a) Training accuracy of CE+DRW  $(T_0 = 160)$  and the CB loss w.r.t. training epoch. (b)  $\widehat{Acc}_{\mathrm{min}} / \widehat{Acc}_{\mathrm{maj}}$  w.r.t. the DRW epoch  $T_{0}$ , where  $\widehat{Acc}_{\mathrm{min}}$  and  $\widehat{Acc}_{\mathrm{maj}}$  denote the training accuracy of the best model on the minority/majority classes, respectively. (c) The test accuracy of the best model w.r.t. the DRW epoch  $T_{0}$ . We can find that the DRW scheme balances the training accuracy between the majority classes and the minority classes and thus improves the model performance on the test set, which is consistent with the theoretical insight (In2).
(a) CIFAR-10 LT
(b) CIFAR-10 Step
Figure 4: The balanced accuracy of the VS loss w.r.t.  $\beta_{y} = (N_{y} / N_{1})^{\gamma}$  on the CIFAR datasets, where the imbalance ratio  $\rho = 10$ . We can find that VS+DRW performs inferior to VS+None, especially when  $\gamma$  is large, which is consistent with the theoretical insight (In4-a).
report the balanced accuracy averaged over 5 random seeds with an imbalance ratio  $\rho \coloneqq N_1 / N_C\in$ $\{10,100\}$ . (b) ImageNet-LT and iNaturalist: We use the long-tailed version of the ImageNet dataset2 [2] proposed by [27], and iNaturalist3 [5] is a real-world long-tailed dataset.
Baselines and Competitors. For the CIFAR datasets, we aim to validate the theoretical results and the performance gain induced by the proposed method. Hence, we select the following baselines: the CE loss (CE) [19], the LDAM loss (LDAM) [12], LDAM+DRW [12], and the VS loss (VS) [16] that generalizes the LA loss [13] and the CDT loss [14]. We tune all the hyperparameters according to the suggestions in the original papers. For the ImageNet-LT and iNaturalist datasets, we select state-of-the-art methods, listed in Tab.2, as the competitors to validate the effectiveness of the method.
Implementation Details. We implement three instances of the proposed learning algorithm: the CE loss equipped with the ADRV scheme (CE+ADRW), the LDAM loss equipped with the ADRV scheme (LDAM+ADRW), and the VS loss equipped with the TLA and the ADRV scheme (VS+TLA+ADRW). We tune the hyperparameter  $\nu$ , and the other hyperparameters follow those used in the baselines. In addition, we incorporate the Sharpness-Aware Minimization (SAM) technique [28] to facilitate the optimization of the minority classes, allowing them to escape saddle points and converge to flat minima [29].
4.2 Theory Validation
In this part, we aim to validate our theoretical insights presented in Sec.3.2 on the CIFAR datasets. Some more empirical results can be found in Appendix G.
<https://www.cs.toronto.edu/~kriz/cifar.html. Licensed MIT.
$^{2}$ https://image-net.org/index.php. Licensed MIT.
<sup>3</sup>https://github.com/visipedia/inat_comp/tree/master/2017. Licensed MIT.
Table 1: The balanced accuracy averaged over 5 random seeds on the CIFAR datasets. The best and the runner-up method for each protocol are marked with red and blue, respectively. The best baseline model is marked with underline.
Dataset	CIFAR-10	CIFAR-100
Imbalance Type	LT	Step	LT	Step
Imbalance Ratio	100	10	100	10	100	10	100	10
w/o SAM
CE	71.5±0.4	87.0±0.2	64.8±0.9	85.1±0.3	38.3±0.4	56.7±0.4	38.6±0.2	54.4±0.3
LDAM	73.8±0.4	86.4±0.4	65.8±0.6	85.0±0.3	39.9±0.7	55.7±0.5	39.2±0.0	50.5±0.2
VS	78.8±0.2	88.7±0.1	76.1±0.7	88.3±0.1	41.8±0.7	58.4±0.2	46.2±0.3	59.9±0.2
CE+DRW	75.8±0.3	87.9±0.3	72.2±0.8	88.0±0.3	40.8±0.6	58.1±0.3	45.4±0.4	59.1±0.3
LDAM+DRW	77.7±0.4	87.5±0.2	77.8±0.5	87.8±0.3	42.7±0.5	57.5±0.3	45.3±0.6	56.9±0.2
VS+DRW	80.1±0.1	88.6±0.1	78.2±0.2	88.1±0.1	41.3±0.4	57.6±0.3	44.0±0.3	58.0±0.3
CE+ADRW	78.6±0.5	88.2±0.3	75.5±0.6	88.5±0.2	41.8±0.6	58.3±0.4	46.5±0.3	59.2±0.3
LDAM+ADRW	79.1±0.2	87.6±0.2	78.5±0.4	88.1±0.2	43.0±0.2	58.0±0.1	45.8±0.2	57.6±0.3
VS+TLA+DRW	80.8±0.2	88.8±0.1	80.0±0.1	89.2±0.1	43.0±0.4	58.9±0.1	46.8±0.1	60.0±0.3
VS+TLA+ADRW	81.1±0.2	89.0±0.2	80.9±0.2	89.3±0.1	43.4±0.6	59.2±0.2	47.8±0.1	60.5±0.3
w/ SAM
CE+DRW	80.5±0.2	89.8±0.2	79.5±0.3	90.2±0.2	44.7±0.6	60.7±0.4	48.5±0.3	61.7±0.2
LDAM+DRW	81.6±0.2	89.4±0.2	81.2±0.7	89.4±0.1	45.2±0.3	59.9±0.2	49.1±0.2	59.3±0.2
VS	82.6±0.2	90.0±0.1	83.2±0.4	90.5±0.1	45.9±0.3	61.0±0.3	47.4±0.3	61.6±0.3
CE+ADRW	82.6±0.2	90.1±0.1	82.8±0.9	90.2±0.3	44.9±0.6	61.0±0.4	48.9±0.2	62.1±0.2
LDAM+ADRW	83.0±0.1	89.7±0.1	82.4±0.3	90.0±0.2	46.3±0.4	60.3±0.3	49.3±0.4	60.3±0.2
VS+TLA+ADRW	83.6±0.2	90.3±0.2	83.8±0.1	90.8±0.1	46.4±0.6	61.9±0.3	49.1±0.2	62.3±0.3
Validation of (In1) and (In4-b). We report the model performance of the baselines w.r.t. the hyperparameters in Fig.1 and Fig.2. From these results, we can find that (1) Both CE+ADRW and LDAM perform better than CE. In other words, either re-weighting or logit-adjustment can boost the model performance. (2) LDAM+ADRW outperforms both CE+ADRW and LDAM, and for VS+ADRW, increasing the hyperparameters  $\nu$  and  $\tau$  appropriately can also bring performance gains. All these results validate the compatibility between re-weighting and the additive logit-adjustment.
Validation of (In2). We present a series of results in Fig.3, where the training accuracy of different classes represents the corresponding  $B_{f}(y)$ . To be specific, Fig.3(a) demonstrates the trend of training accuracy on the majority classes and the minority classes. For the CB loss, the learning process only focuses on the minority classes and hinders the performance improvement on the majority classes (CB Major v.s. CB Minor). Hence, an extremely imbalanced  $B_{y}(f)$  induces a poor generalization performance. By contrast, the DRW scheme first focuses on the majority classes (CE+DRW Major v.s. CE+DRW Minor with the training epoch  $t \leq T_0 = 160$ ) and then pays more attention to the minority classes during the terminal phase of training (CE+DRW Major v.s. CE+DRW Minor with the training epoch  $t > T_0 = 160$ ). Benefiting from the DRW scheme, both the majority classes and the minority classes are well-trained and thus have a balanced term  $B_{y}(f)$  (Fig.3(b)), leading to a corresponding improvement on the test accuracy (Fig.3(c)). Even if we remove the re-weighting term (CE+None), the imbalance degree of  $B_{y}(f)$  is still consistent with the test performance. Note that the learning rate of CE+None is also decreased at the corresponding epoch  $T_0$ , making the line in Figure 3(b) and 3(c) not constant.
Validation of (In4-a). In Fig.4, we present the model performance of the VS loss w.r.t. the hyperparameter of the multiplicative logit-adjustment  $\gamma$ . We can find that VS+DRW performs inferior to VS, especially when  $\gamma$  is large. This phenomenon validates the incompatibility between re-weighting and multiplicative logit-adjustment.
4.3 Performance Comparison
We report the empirical results on the CIFAR datasets in Tab.1, where the imbalance ratio  $\rho \in \{10, 100\}$ . From these results, we have the following observations: (1) The proposed learning algorithm consistently outperforms the baselines, especially when the dataset is more imbalanced.
Table 2: The balanced accuracy on the ImageNet-LT and iNaturalist datasets.
Method	One stage	ImageNet-LT	iNaturalist
Many	Med.	Few	All	Many	Med.	Few	All
OLTR [27]	×	43.2	35.1	18.5	35.6	59.0	64.1	64.9	63.9
LFMR [30]	×	47.1	35.0	17.5	37.2	-	-	-	-
BBN [31]	×	-	-	-	-	49.4	70.8	65.3	66.3
cRT [32]	×	61.8	46.2	27.3	49.6	69.0	66.0	63.2	65.2
τ-norm [32]	×	59.1	46.9	30.7	49.4	65.6	65.3	65.5	65.6
DiVE [33]	×	64.1	50.4	31.5	53.1	70.6	70.0	67.6	69.1
DisAlign [34]	×	61.3	52.2	31.4	52.9	69.0	71.1	70.2	70.6
WB [35]	×	62.5	50.4	41.5	53.9	71.2	70.4	69.7	70.2
CE [32]	✓	65.9	37.5	7.7	44.4	72.2	63.0	57.2	61.7
CE+CB [11]	✓	39.6	32.7	16.8	33.2	53.4	54.8	53.2	54.0
Focal [11]	✓	36.4	29.9	16.0	30.5	-	-	-	61.1
De-confound [36]	✓	62.7	48.8	31.6	51.8	-	-	-	-
DRO-LT [37]	✓	64.0	49.8	33.1	53.5	-	-	-	69.7
SAM [29]	✓	62.0	52.1	34.8	53.1	64.1	70.5	71.2	70.1
Ours	✓	62.9	52.6	37.1	54.1	64.7	70.7	72.1	70.7
Such performance gains validate the effectiveness of the proposed methods (2) Both re-weighting and logit-adjustment can improve the model performance, which is consistent with the theoretical insight (In1) and (In4-b). (3) When  $\rho = 10$  or on the CIFAR-100 dataset, VS+DRW performs inferior to VS. Fortunately, when equipped with the proposed TLA scheme, VS+TLA+DRW outperforms both VS and VS+DRW. These results again validate our theoretical insight (In4-a). (4) When  $\rho = 10$ , CE+ADRW outperforms LDAM+ADRW, and similar counter-intuitive phenomena are also observed in [29]. We conjecture that in this case, re-weighting is enough to rebalance the generalization bound, and the additional LDAM loss might induce other issues such as inconsistency.
We present the overall balanced accuracy on the ImageNet-LT and iNaturalist datasets in Tab.2, where SAM and Ours denotes LDAM+DRW+SAM and VS+TLA+ADRW+SAM, respectively. These results demonstrate that the proposed learning algorithm outperforms the competitors, especially the one-stage ones, which again confirms the effectiveness of the proposed learning algorithm.
In this work, with the proposed local Lipschitz property and the data-dependent contraction technique, we present a unified generalization analysis of the loss-modification approaches for imbalanced learning. Benefiting from this fine-grained analysis, we not only reveal the role of both re-weighting and logit-adjustment approaches but also explain some empirical phenomena that are out of the scope of existing theories. Moreover, a principled learning algorithm is proposed based on the theoretical insights. Finally, extensive experimental results on benchmark datasets validate our theoretical analysis and the effectiveness of the proposed method.
Theoretically, one important future work is to provide a systematical Fisher consistency analysis for the VS loss, providing more insights to design re-weighting and logit-adjustment terms. Methodologically, it might be a promising direction to design an adaptive scheme that can automatically determine the hyperparameters of the learning algorithm.
This work was supported in part by the National Key R&D Program of China under Grant 2018AAA0102000, in part by National Natural Science Foundation of China: 62236008, U21B2038, U2001202, 61931008, 62122075, 61976202 and 62206264, in part by the Fundamental Research Funds for the Central Universities, in part by Youth Innovation Promotion Association CAS, in part by the Strategic Priority Research Program of Chinese Academy of Sciences (Grant No. XDB28000000) and in part by the Innovation Funding of ICT, CAS under Grant No. E000000.
[52] Charles R. Harris, K. Jarrod Millman, Stefan van der Walt, Ralf Gommers, Pauli Virtanen, David Cournapeau, Eric Wieser, Julian Taylor, Sebastian Berg, Nathaniel J. Smith, Robert Kern, Matti Picus, Stephan Hoyer, Marten H. van Kerkwijk, Matthew Brett, Allan Haldane, Jaime Fernandez del Río, Mark Wiebe, Pearu Peterson, Pierre Gérard-Marchant, Kevin Sheppard, Tyler Reddy, Warren Weckesser, Hameer Abbasi, Christoph Gohlke, and Travis E. Oliphant. Array programming with numpy. Nat., 585:357-362, 2020.
[53] Fabian Pedregosa, Géel Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, Jake Vander-Plas, Alexandre Passos, David Cournapeau, Matthieu Brucher, Matthieu Perrot, and Edouard Duchesnay. Scikit-learn: Machine learning in python. J. Mach. Learn. Res., 12:2825–2830, 2011.
[54] Sébastien Marcel and Yann Rodriguez. Torchvision the machine-vision package of torch. In Proceedings of the 18th International Conference on Multimedia 2010, Firenze, Italy, October 25-29, 2010, pages 1485-1488, 2010.