Exploring Weight Balancing on Long-Tailed Recognition Problem
Authors: Naoya Hasegawa
Issei Sato
Venue: ICLR 2024 poster (2024)
Keywords: long-tailed recognition
imbalanced learning
weight decay
regularization
neural collapse
simplex ETF
machine learning
learning theory
Primary area: representation learning for computer vision, audio, language, and other modalities
TL;DR: We theoretically and empirically analyze weight balancing in long-tailed recognition, which leads to the further improvement of performance.

Abstract: Recognition problems in long-tailed data, in which the sample size per class is heavily skewed, have gained importance because the distribution of the sample size per class in a dataset is generally exponential unless the sample size is intentionally adjusted. Various methods have been devised to address these problems.
Recently, weight balancing, which combines well-known classical regularization techniques with two-stage training, has been proposed. Despite its simplicity, it is known for its high performance compared with existing methods devised in various ways.
However, there is a lack of understanding as to why this method is effective for long-tailed data. In this study, we analyze weight balancing by focusing on neural collapse and the cone effect at each training stage and found that it can be decomposed into an increase in Fisher's discriminant ratio of the feature extractor caused by weight decay and cross entropy loss and implicit logit adjustment caused by weight decay and class-balanced loss. Our analysis enables the training method to be further simplified by reducing the number of training stages to one while increasing accuracy. Code is available at https://github.com/HN410/Exploring-Weight-Balancing-on-Long-Tailed-Recognition-Problem.

Published as a conference paper at ICLR 2024
EXPLORING WEIGHT BALANCING ON LONG-TAILED RECOGNITION PROBLEM
Naoya Hasegawa & Issei Sato
The University of Tokyo
[hasyyl etc.//e..cc..utokkyo.
ABSTRACT
Recognition problems in long-tailed data, in which the sample size per class is heavily skewed, have gained importance because the distribution of the sample size per class in dataset generally exponential unless the sample size is intentionally adjusted. Various methods have been devised to address these problems. Recently, weight balancing, which combines well-known classical regularization techniques with two-stage training, has been proposed. Despite its simplicity, it is known for its high performance compared with existing methods devised in various ways. However, there is a lack of understanding as to why this method is effective for long-tailed data, In this study, we analyze weight balancing by focusing on neural collapse and the cone effect at each training stage and found that it can be decomposed into an increase in Fisher's discriminant ratio of the feature extractor caused by weight decay and cross entropy loss and implicit logit adjustment caused by weight decay and baasn-balaneed loss. Our analysis enables the training method be further simplified by reducing the number of training stages one while increasing accuracy. Code available at Weiihh--Blanncig-onn-Loooo lailed haaarng-ooo-ooooo-oall--------ee laaledd-ecooguiioon-oobeeer
INTRODUCTION
Datasets with an equal number of samples per class, such MNIST (Lecun et al. 1998) and CIFAR100 (Krizhevsky 2009). are often used, when we evaluate classification models and training methods in machine learning. However, it is empirically known that the size distribution in the real world often shows a type exponential distribution called Pareto distribution [Reed] 2001), and the same is true for the number of per-class samples in classification problems (Li ct al. 2017 Spain & Perona 2007). Such distributions are called sangttailed data due to the shape of the distribution since some classes (head classes) often sampled and many others (tail classes) are not sampled very often. Long-tailed recognition (LTR) is used to attempt to improve the accuracy of classification models uniform distribution when training data shows such distribution. There is problem in LTR that the head classes have large sample size; thus, the output is biased toward them. This reduces the overall and tail class accuracy because tail classes make up the majority (Zhang al. (021).
Various methods have been developed for LTR, such as class-balanced loss (CB) (Cui et al. 2019). augmenting samples of tail classes (Wang et al. (021), two-stage learning (Kang et al. 22220), and enhancing feature extractors (Liu et al. 2023] Yang et al. 2022); see Appendix A.2 for more related research. Alshammani et al.) (2022) proposed simple method, called weight balancing (WB), that empirically outperforms previous complex -fate-of-the-art methods. WB simply combines two classic techniques, weight decay (WD) (Hanson & Pratt] 1989) and MaxNorm (Hinton al. 2012). with two-stage learning. WD and MaxNorm is known to prevent overlcarning (Hinton 1989) however, is not known why WB significantly improves the LTR performance.
Contribution In this work, we analyze the effectiveness of WB in LTR focusing on neural collapse (NC) (Papyan et al. 2020) and the cone effect (Liang al. 2022). We first decompose WB into five components: WD, MaxNorm, cross entropy (CE), CB, and tto-stage learning. We then show that each component has the following useful properties.

---

Published as a conference paper at ICLR 2024
1st stage: WD and CE increase the Fisher's discriminant ratio (FDR) (Fisher 1936 of features.
Degrade the inter-class cosine similarities (Theorem I in Sec. 4.2
Decrease the scaling parameters of batch normalization (BN) (Sec. 4.3)
This has a
positive effect on feature training.
Facilitate improvement of FDR as features pass through layers (Sec 4.4)
1st stage: WD increases the norms of features for tail classes (Sec. 4.5
2nd stage: WD and CB perform implicit logit adjustment (LA) by making the norm of classifier's weights higher for tail classes, MaxNorm facilitates this effect. This stage does not work well for datasets with small class number (Theorem in S.c. 5.1
The above analysis is useful in the following points: 1. provides guideline for the design of learning in LTR; 2. our theorem offers an insight into how to suppress the cone effect (Liang et al.) 2022) that negatively affects deep learning classification models; 3. WB can be further simplified by removing the second stage. Specificallyy we only need to learn in LTR by WD, feature regularization (FR), and an equiangular tight frame (ETF) classifier for the linear layer to extract features that are more linearly separable and adjusting the norm of classifier's weights by LA after the training.
RELATED WORK
Papyan al. (2020) investigated the set of phenomena that occurs when the terminal phase of training, which they termed NC. NC can be briefly described as follows. Feature vectors converge to their class means (NC1); the class means of feature vectors converge simplex ETF (Strohmer & Heath 2003) (NC2); the class means of feature vectors and the corresponding weights of the linear classifier converge in the same direction (NC3); and when making predictions, models converge to predict the class, the mean feature vector of which is closest to the feature in Euclidean distance (NC4).
Papyan et al. 2020)) claimed that NC leads increased generalization accuracy and robustness to adversarial samples. The number of studies have been conducted on the conditions under which NC
occurs (Han al. 2022 Ji et 2022 Lu & Steinerberger 2021 Papyan al. 2020). Rangamani
Banburski-Fahey 2022 theoretically and empirically showed that the occurrence of NC needs WD.
There have also been studied on NC when the model trained with imbalanced data. Fang al.) (202T) theoretically and experimentally proved "minority collapse" in which features corresponding to classes with small numbers of samples tend converge in the same direction, even if they belong to different classes. Yang et al. (2022) attempted to solve this problem by fixing the weights of the linear layers to an ETF, Thrampoulidis al. (2022) tried to generalize NC to imbalanced data by interpolation (SELI), an extension of ETF in the terminal phase training. They also demonstrated that minority collapse does not occur under the appropriate regularization including WD.
demonstrating that the features converge towards smmplex-encoed--lle
WD is a regularization method often used for deep neural networks (DNNs) with BN (Ifff & Szegedy 2015), which is scale invariant. /Zhan et al. (2019) revealed that WD increases the effective learning rate which facilitates regularization in such models. Training of networks containing BN layers with WD is often studied terms of training dynamics (Li & Aroral 2020 Lobacheva et al.) 2021 Wan et al. 2021). Summeer & Dinneen (2020) and Kim et al. (2022) studied whether to apply WD to the scaling and shilting parameters of the BN for such networks. WD also known have
many other positive effects on DNNs, e.g., smoothing loss landscape (Li et al. 2018 Lyu et al.
2022), causing NC (Rangamani & Banburski-Fahey. (202)), and making filters sparser (Mehta et
2019 and low-ranked (Galanti et al. (003). WD is also effective for imbalanced data (Alshammari
ct al.
2022 Thrampoulidis et al. (022).
Liang et al. 2022) found phenomenon they termed "cone effect" observed in neural networks. The phenomenon is a tendency in which features from DNNs with activation functions are prone to have high cosine similarities. It is widely observed regardless of the modality of data, the structure of models, and whether the models are trained. It also indicates that features from different classes tend to exhibit high cosine similarity. Thus, it has negative impact on DNNs for classification since features with lower inter-class cosine similarity are more linearly separated. We also reconfirmed such phenomena in Sec. 4.2 Our analysis provides guidelines prevent the effect.
2

---

Published as a conference paper at ICLR 2024
3 PRELIMINARIES
This section defines the notations used in the paper and presents the strategies for LTR. See Appendix A.1 for the table of these notations. Suppose multiclass classification with C' classes by samples X C Ip" and labels ) =
C). The training dataset D = {(x, yr ]x E X, yi E y/ consists of D2 {(x,, Mt) D Ur = k). We define N as ||D] and N₂ as |D the number of samples. Without loss of generality, assume the classes are sorted in descending order by the number of samples. In other words, Vk (1,2, C- 1], N₂ 2 N&+1 holds. Imbalance factor N1 max, N2 indicates the extent which the training dataset is imbalanced and p 1 holds mine Nu
p
Ne
in LTR. Therefore, the number of samples in each class N2 satisfies N₂ = Nip e- Define N as
Ek=1 N₂ the harmonic mean of the number of samples per class. In LTR, the test dataset D' used for accuracy evaluation is dlass-balanced i.e., each class has the same number of samples.
Consider network f(;; : X' + CC parameterized by Q {(1)- It outputs logit z; f(x;; θ) The network is further divided into a feature extractor g( 2 : XX + Rd and a classifier h((:On) IRd + RCC with d as the number of dimensions for features. This means fx;; 2) = + (xx, (xx) be the ((g(x;; On): We often abbreviate gxx; θ₂) to g(x)). Let μ4 Nr
inner-class mean of the features for class k and μ t Ee=i p4 be the mean of the inner-class mean of the features. We used a linear layer for the classifier. In other words, h(v; θ₂) = WT holds for a feature v E Rd with W RdxC'' as weight matrix. Denote the kth column vector of W by Wa; thus, O = [w The loss function is denoted by E(z,, Ut) RC RC + R. Let tCE and fCB be the loss function of CE and CB, respectively. By rewriting (); = N 1 (xf(x;; θ) W.). parameters @ are optimized as follows the absence of regularization:
arg min D)
We evaluated the method by accuracy on test dataset and FDR. FDR is the ratio of the inter-class variance to the inner-class variance and indicates the ease of linear separation of the features. For
example, the FDR of training features is Tr(Sw) SB) where SB = Ex k= N& - [) pT)
and Sw = kk= Ex;ED, (x: [xx)((== - _ The FDR of test features similar.
Regularization Methods We implemented WD as L2 regularization with a hyperparameter A because we used stochastic gradient descent (SGD) for the optimizer. Thus, optimization is written as O' = arg ming F(O; D) + EeEy |/ 11. MaxNorm is regularization that places restriction on the upper bound of the norm of weights. As with Allhammari et al.) (2022) , for the convenience of hyperparameter tuning, the weights to be constrained are only the those belonging to the classifier O O Thus, we compute θ* = arg mine F(O; D), s.t. A We E O,: w 22 S ni where m a hyperparameter. Since constrained optimization difficult to solve in neural networks, we implemented it using projected gradient descent. In our implementation, this projects the weights that do not satisfy the constraint the range where the constraint is satisfied by updating Wa to min [[aalaa) Wk.
Weight Balancing WB adopts two-stage training (Kang et al. 2020). In the first stage, the parameters of the entire model θ are optimized with CE using WD. In the second stage, O, is fixed and only θ is optimized with CB using WD and MaxNorm. For simplicity, let 33 of CB be 1 in
this paper. Thus, LCB(Z,, 91) No
NV log
exp((a.)x)) ) ES_ exp((z.))
Therefore, in the second stage of WB,
FWD (: = EN (cn(z,, v.) consider Max in this paper, as we reveal that it only changes the initial values and do not impose any intrinsic constraints on the optimization presented in Appendix.
3 Eey |[ 11 is optimized for W. Note that we do not
A.5
Logit Adjustment Multiplicative LA (Kim Kim 2020) changes the norm of the per-class weights of the linear layer. The weights of the linear layer corresponding to class k. w', are adjusted
In the paper of Cui al. 2011)), NV is set to be 1, but in their implementation, N is equal to the harmonic mean. We adopt the latter since our experiments were based on their implementation.
3

---

Published as a conference paper at ICLR 2024
Table 1: FDRs for each dataset of models trained with each method. A higher FDR indicates that features are more easily linearly separable. For all datasets, the method with WD or CE produces a higher FDR and using both results in the highest FDR.
CIFAR 10-LT
CIFAR100-LT
mini-ImageNet-IIT
Method
Train
Test
Train
Test
Train
Test
CE w/o WD
8.17 x 10º
2.17 x 10
1.28 x 10a
4.16 x 10
7.56 x 10 4.28 x 10
CB w/o WD
4.43 x 10
1,50 x 10
8.17 x 10a
2.42 x 10
4.68 x 10a
2.93 x 10
CE w/ WD
2.60 x 10a
3.89 x 10 2.87 x 10
1.07 x 10 6.58 x 10a 1.01 x 10
CB w/ WD
3.39 x 10
3.04 x 10
2.12 x 104
6.74 x 10
4.84 x 102 6.84 x 10'
WD w/o BN
2.19 x 102
3.42 x 10a
5.77 x 102
7.95 x 10a
1.61 x 102
6.73 x 101
WD fixed BN 1.63 x 10
4.16 x 10t 2.04 x 104
1.05 x 10
3.94 x 10
1.04 x 10
to P(Y=Ry² I[wal22' where 7 > 0 is a hyperparameter. Kim & Kim (020) trained using projected gradient descent so that the weights of the linear layer always satisty Vk. || = 1, but in our study, for comparison, we trained with regular gradient descent, normalized the norm post-hoc, and then adjusted the norm. Additive LA (Menon et al. 2020) a method to adjust the ouiput to minimize the average per-class error by adding different constant for each class to the logit at prediction. That is, the logit z; at prediction is adjusted to z as (z))x = (z:)x log P(Y = k)), where T is a hyperparameter. LA generally increases the probability of classifying inputs into tail classes.
ETF Classifier NC indicates that the matrix of classifier weights in deep learning models converges to an ETF when trained with CE (Papyan al., 2020). The idea of ETF classifiers is train only the feature extractor by fixing the linear layer to an ETF from the initial step. ETF classifiers' weights
W E R satisfy W = EE // ( (Ic-)Icc1)) where U E R is a matrix such that U is an identity matrix, Ia E RCxC is an identity matrix, and le is a C-dimensional vector, all elements in which are 1. Also, Ew is hyperparameter, which was set in experiments.
4 ROLE OF FIRST TRAINING STAGE IN WB
First, we analyze the effect of WD and CE in the first training stage. In Sec. 4.1 datasets and models used in the analysis. Then, we examine the cosine similarity and FDR of the features from the models trained with each training method for the first stage of training in Secs. 4.2 [.4 In Sec. 4.5, we identify the properties of training features trained with WD, which is the key to the success of WB. The results of the experiments that is not included in the main text presented in Appendix A.5)
we present the
4.3,
4.1 SETTINGS
We used CIFARIO, CIFAR100 Krizhevsky 2009). mini-ImageNee (Vinyals et al.] 2016 and
ImageNet (Deng et al. 2009) as the datasets and tollowed Cui et al.) (2019), Vigneswaran et al. (021)) and Liu et al. (2019) to create long-taine datasets, CIFARIO-LT, CIFAR100-LT, mini- Maaiii-mmeeee---- and LT. We also experimented with a tabular dataset, Helena (Guyon et al., (019)) to verify the effectiveness our method for other than image dataset. Each class is classified into one of three groups, Many, Medium, and Few, depending on the number of training samples Ng.
We used ResNeXt50 (Xie et al., 2017) for ImageNet-L.T and ResNet34 (He et al.) (016) for the other datasets. In WB, parameters were trained in two stages (Kang et al.) (020); in the first stage, trained the weights of the entire model O while the second stage, we trained the weights of the classifiers O with the weights of the feature extractors θ₂ fixed.
To investigate the properties of neural network models, we also experimented with models using multilayer perceptron (MLP) and residual block (ResBlock) (He et al.) 2016). The number of blocks is added after the model name to distinguish them (e.g.. MLP3). We trained and evaluated these models using MNIST (Lecun al. 1998).
4

---

Published as a conference paper at ICLR 2024
CIFARIO-LT
CIFAR100-LT
CE wya WD
GE W/ WD
L00
w/o WD
CE WD
1.00
Fenture's class
0.75
30
30-
5
0.50
0.50
0.25
0.25
300
300
0.00
Feature's class
Foature's
CB w/o WD
CB w/ WD
WD
-0.25
-0.75
90
-0.75
-2.00
-1.00
Figure I: Heatmaps showing average cosine similarities of training features between two classes. WD maintains high cosine similarity between the same classes and reduces the cosine similarity between the different classes.
For training losses, we used CE unless otherwise stated. See Appendix A.4 for detailed settings.
4.2 WD AND CE DEGRADE INTER-CLASS COSINE SIMILARITIES
We first examined the role of WD in the first stage of training. We trained the model in two ways: without WD (w/o WD) and with WD (w/ WD). In addition, we compared two types of loss: one using CE and the other using CB, for total of four training methods. We investigated the FDR, the inter-class and inner-class mean cosine similarity of the features, and the norm of the mean features per class. Due to space limitations, some of results of the mini-ImageNet-LTT excluding FDR are presented in Appendix A.S
Table lists the FDRs of the models trained with each method, and Figure shows the cosine similarities of the training features per class. It can be seen that the combination of WD and CE achieves the highest FDR. This fact is consistent with the conclusion that WD is necessary for the NC in balanced datasets, shown by Rangamani & Banburski- 2022) and suggests that this may also hold for long-tailed data.
When WD is not applied, the cosine similarity is generally higher even between features of different classes. This is natural because neural networks have the cone effect found in Liang et al. (022). The cone effect is phenomenon in which features from DNN tend to have high cosine similarities to each other even if they belong to different classes, However, the methods with WD result in lower cosine similarities between features of different classes and higher cosine similarities between features of the same classes, leading to a higher FDR. This shows that WD prevents the cone effect, as supported by the following theorem.
Theorem 1. For all (xi, vi), (xj, Wj) S.I. Ve + Uú W is an ETF and there exists and L s.s. and [1g(x;)l12, (lx(x)) I L s 2V2log (C 1), the following
atcr
dtem 0g(x,)
0g(x.)
holds:
cos
S 25 12
where 5 I. C-1 log (IC-))(-=)) E / 1] and cos(:.)) means cosine similarity of the two vectors.
This theorem states that the cone effect is suppressed when the following two conditions hold: 1. the weight matrix of the linear layer is an ETF; the norms of the features are sufficiently small. Sufficient training causes NC and makes the weight matrix of the linear layer converge ETF
5

---

Published as a conference paper at ICLR 2024
Table 2: Means and standard deviations of all scaling parmeters in BN layers of models trained with each method for each dataset. WD greatly reduces the average scaling parameter.
Method
CIFARIO-LT
CIFARIO0-LT mini-ImageNet-IT
CE w/o WD 1.00 = 1.00
0.0218 1.00 = 0.0289 CE w/ WD 0.0395 = 0.0273 0.0649 = 0.0548
0.998 = 0.0498 0.0863 + 0.0707
Table 3: Means and standard deviations of all shifting parmeters in BN layers of models trained with each method for each dataset.
Method
CIFARIO-LT
CIFAR 100-LT
mini-ImageNet-LI
CE w/o WD -0.0236 + 0.0255 -0.0444 + 0.0348 -0.0904 + 0.0698
CE w/ WD -0.00310 + 0.0244 -0.00667 + 0.0309 00.0186 0.0448
Paapyn et al. 2020 ETF Classifiers are also effective: thery are methods to fix the weight matrix of the linear layer to ETF from the beginning to satisfy this condition. As for the second condition, WD contributes to satisfy the condition. In fact, as shown in Figure 2 the norm of the features by regularizing the weights causing low inter-class cosine similarity. This also suggests that explicit FR leads to better feature extractor training. We consider that in Sec. 5.2
WD also indirectly degrades
4.3 WD AND CE DECREASE SCALING PARAMETERS OF BN
Next, we split the effect of WD on the model into that the convolution layers and that the BN layers. To examine the impact on convolution, we analyzed the FDR of the features from the models trained with WD applied only to the convolution layers (WD w/o BN). The bottom halves of Table I shows that the FDRs of the methods with restricted WD application. These results indicate that when WD is applied only to the convolution layers, the FDR improves sufficiently, which means that enabling WD for the convolution layers is essential for improving accuracy. One reason for this may be an increase in the effective learning rate (Zhang al. 2019).
Small scaling parameters of BN facilitates feature learning We also examined the effect on BN layers and found that WD reduces the mean of the BN's scaling parameters; see Tables and 3 Note that the standard deviation of the shifting parameters remains almost unchanged. We investigated how the FDR changes when models are trained with the scaling parameters of BN fixed to one common small value (WD fixed BN). this method, we applied WD only to the convolution layers and fixed the shifting parameters of BN to O. We selected the optimal value for the scaling parameters from (0.01, 0.02, , 0.20) using the validation dataset. Table shows that small value of the scaling parameters of BN is crucial for boosting FDR; even setting the same value for all scaling parameters in the entire model also works. In this case, the scaling parameters BN does not directly affect FDR, it only multiplies the features uniformly by constant. This suggests that the improvement in FDR caused by applying WD to BN layers is mainly because smaller scaling parameters have a positive effect on the learning dynamics and improve FDR. Although Kim et al. (2022) attribute this to the increase in the effective learning rate of the scaling parameters, they have a positive effect on FDR even when the scaling parameters are not trained, suggesting that there are other significant effects. For example, smaller scaling parameters reduce the norm of the feature, promoting the effects discussed in Sec. 4.2 We also examined the effect of high relative variance in shifting parameters of
BN in Appeindix A.5
4.4 WD AND CE FACILITATE IMPROVEMENT OF FDR AS FEATURES PASS THROOGH LAYERS
Table 4| compares the FDR of two sets of features; ones from each model trained with each method (before) and ones output from the model, passed through a randomly initialized linear layer, then ReLU-applied (after). This experiment revealed that the features obtained from the models trained with WD have a bias to improve their FDR regardless of the class imbalance. The detailed experimen- tal set-up is described in Appendix A.5 While training without WD does not increase the FDR in

---

Published as a conference paper at ICLR 2024
Table 4: FDRs of features from each model trained with each method and features passed through randomly initialized linear layer and ReLU after model. C10, C100, and mlm are the abbreviations for CIFARIO, CIFARI00, and mini-ImageNet respectively. Red text indicates higher values than before, blue text indicates lower values. The FDR of features obtained from models trained with WD improves by passing the features through a random initialized layer and ReLU.
CE
WD w/o BN
WD
Model
Dataset
before
after
before
after
before
after
C100
7.51 x 10 7.11 x 10 2.29 10 2.34 x 102 3.98 102 4.10 x 102
C100-LT
4.16 x 10 4.02 10 7.95 10' 7.89 x 10 1.07
102 1.13 x 10
C10
4.77 x 10 5.56 x 10a 7.96 x 10a 1.02 x 10a 1.81
x 10 2.35 x 10a
ResNet34
C10-LT
2.17 x 10 2.36 x 10a 3.42
10 3.94 x 10a 3.89
10a 4.30 10t
mlm
7.34 x 10t 6.66 x 10 1.81
102 1.86 x 10a 3.63
102
3.93 x 10
mIm-LT
4.28 x 10t 3.97 x 10a 6.73 x 10a 6.30 x 10a 1.01 x 102 1.08 x 10a
MLP3
1.58 x 10 1.54 102 2.48
102 3.53 x 10 2.36
102
7.96 10
MLP4
1.98 x 10a 1.96 x 10a 3.60 x 10 4.97 x 102 4.12 x 10 1.43 x 10
ML.5
MNIST
2.44 x 10a 2.37 x 102 4.91
102 6.76 x 102 6.10
102 2.47 103
ResBlock1
1.39 102 1.36
102 2.20
102 2.69 x 102 2.44
102
5.81 x 10
ResBlock2
1.66 x 10a 1.59 x 10 2.97 x 10 3.65 x 10 4.44 10
9.55 X 10
CIFAR1O-LT
CIFAR100-LT
w/o 6I0
CB w/o WD
C3 avo WD
50
WD
Ngrm
40
w/
WD
Norm
60-
WD
30
20
20-
19
91
Class number
Class number
Figure 2: Norm of mean per-class training features produced from models trained with each method. Features learned with methods with WD all demonstrate that the norms of the Many classes' features tend to be smaller than those of the Few classes.
most cases, the features from models with WD-applied convolution improve the FDR over the before FDR in most cases. Moreover, the fiter-FDR increases significantly when WD is applied to the whole model. Note that this phenomenon is valid when the features go through only one randomized layer. Experiments have also shown that passing features through multiple layers of randomly initialized linear layers and ReLUs does not necessarily improve FDR; see Appendix [A.5] These results mean that only increasing randomized layers does not enhance FDR. However, features trained with WD are less likely to be degraded even when going through one randomized layer. This suggests that WD and CE facilitate the training of models.
4.5 WD INCREASES NORMS OF FEATURES FOR TAIL CLASSES
We also investigated the properties appearing the norm of the training features from each model investigated in Sec. 4.2 Sec. 5. reveals that this property is crucial to the effectiveness of WB. Figure 2 shoow the norm of the per-class mean training features. Note that this phenomenon does not occur with test features. The method without WD shows almost relationship between the number of samples and the norm of the features for each class. However, when WD is applied, the norms of the Many-class features drop significantly, to between one-half and one-fifth that of Few classes. While this phenomenon has been observed in step-imbalanced data (Thrampoulidis et al. (022), we found that it also occurs in mniled data.
7

---

Published as a conference paper at ICLR 2024
5 ROLE OF SECOND TRAINING STAGE IN WB
In this section, we thcoretically analyze how the second stage of training operates in Sec. 5.1 leads to further simplification and empirical performance improvement of WB in Sec. 3.2
which
5.1 WD AND CB PERFORM IMPLICIT LOGIT ADJUSTMENT
We found that WB encourages the model to output features with high FDR in the first stage of training. How does the second stage of training improve accuracy? AAlshammari et al. (2022) found that WB trains models so that the norm of the linear layer becomes more significant for the tail classes in the second stage of training but does not explicitly train in this way. We found the following theorem that shows the second stage of WB is equivalent to multiplicative LA under certain assumptions. In the following theorem, we assume per-class mean features follow ETF or SELI (Papyan et al., 2020 Thrampoulidis al. 2022). See Appendix. A.3 for the theorem and proof when this does not hold.
O, which is valid when NC occurs, i.c. when
Theorem 2. Assume μ
0. For any k E V, if there exists w S.I.
#FWI
0 and
dws
We W
[wtll2 l2 0 (
(pp))
we have
N AN M
= -xxpO (XP))))
vk E y
W
(3)
where O a Bachmann-Landal O-notation. For example, the notation in the form O (+) indicates
N that the term is at most constant multiple of if is sufficiently large. Note that x is independent of k. This theorem states that if the number of classes or the imbalance factor is sufficiently large, and if NC has occurred in the first stage, there exists linear layer weights that are constant multiples of the corresponding features and sufficiently close to the stationary point in the optimization. In other words, if the previous conditions are satisfied, the norm of the linear layer is proportional the norm of the corresponding per-class mean feature. We show that the norm of training features are larger for Few classes in Sec. 4.5 Thus, the linear layer is trained to larger for the Few classes. Besides, the features and the corresponding linear layers' weights are aligned when NC occurs, so the orientation of the linear layers' weights hardly changes during this training. This is the same operation as when multiplicative LA makes adjustments to long-tailed data, increasing the norm of the linear layer for the tail class (Kim & Kim 2020). Indeed, if there exists some co and Yo. and for all k, || LA. In this theorem, we do not consider MaxNorm because we found that MaxNorm makes the norm of the weights in the linear layer closer to the same value before training, only facilitating convergence. For more details, see Appendix A.5
cy = k) holds, then the second-stage training is equivalent to multiplicative
Note also that this theorem does not guarantee that the second stage of WB is operation equivalent to multiplicative LA when the number of classes is small. In fact, even though the norm of the training features is larger for Few classes (Figure 2). WB fails make the norm of the weights greater for Few classes in CIFARIO-LT: see Appendix A.5 This suggests that replacing the second stage of training with LA would be more generic and we confirm validity in the following section.
5.2 IS TWO-STAGE LEARNING REALLY NECESSARY?
We experimentally demonstrated the role of each training stage of WB indicated by Theorems 1 and 2. For this purpose, we develop a method with a combination of essential operations and verify its high accuracy and FDR. In the first stage of training, FR and an ETF classifier are used to satisfy the conditions of Theorem 1. We fix the linear layer and train the feature extractor with CE, WD. and FR: then adjust the norm of the linear layer with multiplicative LA. To show that these are effective operations implicitly carried out with WB, we compare the combination with WB. For the ablation study, we experimented by comparing WB with the following methods.
Training with CE or CB only (CE, CB).
Training with WD and CE (WD).
8

---

Published as a conference paper at ICLR 2024
Table 5: FDRs and accuracy of models trained with each method for CIFAR100-LTT In addition to WD, training with FR and an ETF classifier further improves the FDR. However, only using these does not greatly improve accuracy; LA increases the accuracy of Medium and Few classes and enhances the average accuracy significantly as result. In all cases, multiplicative LA improves the accuracy more than additive LA.
FDR
Accuracy (%)
Method
LA
Train
Test
Many
Medium
Few
Average
CE
N/A
1.28x10
4.16x10 +1.0
64.6
36.9
11.94a.T
38.5
10
CB
N/A
8.17x10
2.42x10
47.6+1.0
23.2+0.6
5..1
26. 10.4
8.0x10"
10.6x109
N/A
2.87x10 10
1.07x102
75.9
45.3+0.6
13.9
46.0204
-0.2 10
WD
Add
2.87x10 10
1.07x10
70.7
45.710.9
30.940.9
49.6
101
Mult
2.87x10* 6.0x10
1.07x10*
72.620.7
48.540.8
29.540.9
50.8
002x101
WB
N/A
2.94x10 10
1.07x102
73.8+0.7
50.240.6
25.641.0
50.620.2
1.22x
N/A
3.33x10 10
1,13x10
76.3
46.0.0.4
15.5+0.6
46.9=0.2
101
WD&ETF
Add
3.33x10
2.0x10
1,13x10
0.1x10'
73.8+0.4
48.9
25.8+0.1
50.2=0.2
Mult 3.33x10
1.13x10
70.4
51.440.3
31.7+0.6
51.7=0.3
-2.0x 103
0.11
N/A 8.81x10
1.22x10
77.9
46.8+1.
15.340.3
47.6=0.5
WD&FR
100x
0.1x10
Add 8 8.85x10
1.22x10 0.1x10'
75.1
49.3+10
26.2
50.9=0.3
&ETF
200x103
Mult 8.85x10
1.22x10
74.2+0.3
52.9
29.940.8
53.0 53.0+0.3
10
101x10
Using an ETF classifier the linear layer and training only the feature extractor with WD and CE (WD&ETF).
Using an ETF classifier as the linear layer and training only the feature extractor with WD, FR, and CE (WD&FR&ETFF.
As for the bottom three methods, we also compared with additive LA (Menon et al. 2022), with multiplicative LA (Kim Kim 2020), and without LA. To ensure that the model correctly classifies both samples of the head classes and tail classes, we consider the average of the per-class accuracy of all classes and each group, i.e., Many, Medium, and Few.
Table the FDRs and accuracy of each method for CIFAR100-LT. We show the results on the other datasets including tabular data CIFARIO-LT, mini- Sini-ImageNet-LTT ImageNet-LT, and Helena) and the other model (ResNeXt50) in Appendix A.S First, training with the ETF classifier in addition to WD increases the FDR for both training and test features. Using FR further improves the FDR. However, it does not boost the average accuracy much. In contrast, LA enhances the classification accuracy of Medium and Few classes in all cases; thus, it also increase the final average accuracy significantly. It was observed that multiplicative LA improved accuracy a greater extent than additive LA and that WD& FR&ET&&& S.EETF&Muliippiatiiii LA outperformed WB on average accuracy despite our method requiring only one-stage training.
CONCLUSION
We theoretically and experimentally demonstrated how WB improves accuracy in LTR by investigat- ing each components of WB. In the first stage of training, WD and CE cause following three effects, decrease in inter-class cosine similarity, reduction the scaling parameters of BN, and improvement of features in the ease of increasing FDR, which enhance the FDR of the features. In the second stage, WD, CB, and the norm of features that increased in the tail classes work as multiplicative LA, by improving the classification accuracy of the tail classes. Our analysis also reveals training method that achieves higher accuracy than WB by improving each stage based on the objectives. This method is simple, thus recommend trying it first. A limitation is that experiments were limited to MLP, ResNet, and ResNeXt and the usefulness in other models such as VIT (Dosovitskiy et al. (02I) is unknown. Future research will include experiments and development of useful methods for such models.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENTS
We would like to express my sincere gratitude to the members of Issei Sato Laboratory for their invaluable discussions, We are also grateful to the reviewers for their diligent critique, which has significantly improved the quality of this paper. This work was supported by JSPS KAKENHI Grant Number 20H04239 Japan.
REFERENCES
Shaden Alshammari, Yu-Xiong Wang, Deva Ramanan, and Shu Kong. Long- Tailed Recognition via Weight Balancing. c2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition
(CVPR), pp. 6887-6897 March 2022. ISSN 10636919, doi: 10.1 109/cvpr52688..... 2022.00677. URL
https://aaxtt org/ abs 2203. 12223.11199 14197 arXiv: 2203. 14197 ISBN: 0781665469463.
Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learn- ing Imbalanced Datasets with Label-Distrbuttinn-AAarr Margin Loss. In Ad- vances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL. Inttps::/proceediinng wtwpooeeeeeiinnn.n.inn.ieeee neurips cccpppprr/011100 (ateerr///9////tt/ 9/hash/ 62146 2a110aat0caafffff06ccc2222221281 -bstt:9 html
Abst
Peng Chu, Xiao Bian, Shaopeng Liu, and Haibin Ling. Feature Space Augmentation for Long-Tailed Data. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm (eds.), Computer Vision ECCV 2020, Lecture Notes in Computer Science, pp. 694-710, Cham, 2020. Springer International Publishing. ISBN 978-3-030-55526-66 doi: 10.10077978-33030--555266
Yin Cui, Menglin Jia, Tsung Yi Lin, Yang Song, and Serge Belongie. Class-balanced loss based on effective number of samples. Proceedings of the IEEE Computer Society Conference Computer Vision and Pattern Recognition, 2019-Jume:9200-02 9269, Jamuary 2019. ISSN 10636919.
doi: 10L.1////./.2000 WwW.PP..21..000900 209999999 URL http: http:/aaxivvv/.. org. abs 1901 05555 arXiv:
1901.05555 ISBN: 9781728132938.
Jia Deng. Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large- scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248-255. IEEE, June 2009. ISBN 978-1 1--224449992-2444 doi: 10.1.099CVPR.2009. 5206848.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. In International Conference on Learning Representations, January 2021. URL. https //openreview. net./ forum? www.::idddiiiiddhdoccmmom
Cong Fang, Hangfeng He, Qi Long, and Weijie J. Su. Exploring deep neural networks via layer-peeled model: Minority collapse imbalanced training. Proceedings of the National Academy of Sciences of the United States of America, 118(43), January 2021. ISSN 10916490. doi: 10. 10.1073/pm 107a// 2103091118, URL URLhttps: http::://axxvvv.orrr.. wttp:////p/aa///.o///////// 112.001...69999 699 arXiv: 2101.
R. A. Fisher. The Use of Multiple Measurements in Taxonomic Problems. Annals of Eugenics, 7(2): 179-188, 1936. doi: 10.1111/1 1469) 1809. 196....9...
Tomer Galanti, Zachary S. Siegel, Aparna Gupte, and Tomaso Poggio. SGD and Weight Decay Provably Induce a Low-Rank Bias in Neural Networks, January 2023. URL http::/aaa org/abs 2206. 057 194 7xXiv:220600579. [cs, stat].
Aditya Sharad Golatkar, Alessandro Achille, and Stefano Soatto. Time Matters in Regularizing Deep Networks: Weight Decay and Data Augmentation Affect Early Learning Dynamics, Matter Little Near Convergence. In Advances in Neural Information Processing Systems, volume 32. Cur- ran Associates, Inc., 2019. URL https: http////sss/ppppeedean.a.rrrrraa paper/2019/1i 20 8 /tt.a//aeo.o....aaaaaaaa0333333.
files/ httm)
10

---

Published as a conference paper at ICLR 2024
Isabelle Guyon, Lisheng Sun-Hosoya, Mare Boullé, Hugo Jair Escalante, Sergio Escalera, Zhengying Liu, Damir Jajetic, Bisakha Ray, Mehreen Saeed, Michèle Sebag. Alexander Statnikov, Wei- Wei Tu, and Evelyne Viegas. Analysis of the AutoML Challenge Series 2015-2018. In Frank Hutter, Lars Kotthoff, and Joaquin Vanschoren (eds.), Automated Machine Learning: Methods, Systems, Challenges, The Springer Series on Challenges in Machine Learning, pp. 177-219. Springer Inter- national Publishing, Cham, 2019, ISBN 978 978-3-030-05318833 doi: 10. 10007/978-3-030-0551-----8-8
URL http:///o.oo..///...///////
030 05 50
R. Hadsell, S. Chopra, and LeCun. Dimensionality Reduction by Learning an Invariant Mapping. In 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR (06), volume 2, pp. 1735-1742, June 2006. doi: 10.1 10.1109/CVPR.2006 100. ISSN: 1063-6919.
X. Y. Han, Vardan Papyan, and David L. Donoho. Neural Collapse Under MSE Loss: Proximity to and Dynamics on the Central Path. In International Conference on Learning Representations, January 2022. URL https: http://ppprrreiieee//eee / httprrvveewwneeeeeeeeeeeeeeeee wwtp://nee/ooummmm//// sasrumm?idwwUubdyy
R3
Stephen José Hanson and Lorien Pratt. Comparing Biases for Minimal Network Construction with Paok-PPoppaattoon Advances in neural information processing systems 1, 1:177-185,
1989. /tt.tttp::/ /ttp:::////rrallll/ wttp//otrall...ooooooooo. wam.moooo. pttgg/iiaaiionoo intaatiioncom cfm? d= 89872 Publisher: Morgan- Kaggman ISBN: 1 1-558-60015-9.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, volume 2016-Decem, pp. 770-778, December 2016. ISBN 978-1-4673- 8850-4. doi: 10.1 1119///PR.2006..009 URL htt.:::::/aaxi..trrrrter 1512.03385 ISSN: 10636919.
/1t../111..001 03385 arXiv:
Geoffrey E. Hinton. Connectionist learning procedures. Artificial Intelligence, 40(1):185-234, September 1989. ISSN 0004-3702 doi: 1010016000044770269999900000 URL https://ww/
Geoffrey E. Hinton, Nitish Srivastava, Alex Krizhevsky, Ilya Sutskever, and Ruslan R. Salakhutdinov. Improving neural networks by preventing co-adaptation of feature detectors. July 2012. doi: arXiv: 1207.0580.
10,48550/arxiv. wtw.:eiiii.i.1...... 207.0580.. URL /tRL//tc:://aarvv..rrraaa/////// 0580
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning. pp. 448-456 pmlr, 2015.
Wenlong Ji, Yiping Lu, Yiliang Zhang, Zhun Deng, and Weijie J. Su. An Unconstrained Layer- Perspective on Neural Collapse. In International Conference on Learning Representations, January 2022. URL TtLLhttps: htttps://openrevi.e.. wwt.eeewweeeeeeeeeeeeeeeeeeeeeee www.nneeetooommmmm-amma2... https////pppperreeeeeeeeer/rrrrs
Jhccoo
Arlind Kadra, Marius Lindauer, Frank Hutter, and Josif Grabocka. Well-tuned Simple Nets Excel on Tabular Datasets. In Advances in Neural Information Processing Systems, November 2021. URL
net https: http://reeeeeeettt.ooorrr..dddd. Tooummmdd=d33381110
Bingyi Kang, Saining Xie, Marcus Rohrbach, Zhicheng Yan, Albert Gordo, Jiashi Feng, and Yannis Kalantidis. Decoupling Representation and Classifier for Long- Tailed Recognition. In International Conference on Learning Representations, 2020. URL https: BtRLLtttss///pereveiiie http:///pppeeeeeeeeemeeeee www.eeeffee nwt/forumm
id=rlcm Ad-rlgRTccn gRRCVVEVB
Bingyi Kang, Yu Li, Sa Xie, Zehuan Yuan, and Jiashi Feng. Exploring Balanced Feature Spaces for Representation Learning. International Conference on Learning Representations, March 2023.
URL
net. /eee:fooroe forum? wit.ggdtggggtttttto.e. htt.:/oorrmmmdddgtttaaabbbbb
Bum Jun Kim, Hyeyeon Choi, Hyeonah Jang, Dong Gu Lee, Wonseok Jeong, and Sang Woo Kim. Guidelines for the Regularization of Gammas in Batch Normalization for Deep Residual Networks, May 2022. URL http::ttpa/aattt
2205 77 MrXiv:2205.07260 [cs].
Byungju Kim and Junmo Kim. Adjusting decision boundary for class imbalanced learning. IEEE Access, 1881674-81688 December 2020. ISSN 21693536. doi: 10. D0.110//CCESSS.22002220 URL htw.:http://aaaa
000.991...118811 01857
arXiv: 1912.01857.

---

Published as a conference paper at ICLR 2024
Alex Krizhevsky. Learning Multiple Layers of Features from Tiny Images. Science Department, University of Toronto, Tech. pp. 1-60, 2009. ISSN 1098-6596. doi: 10.1.1.222.9220. arXiv: 1011.1669v3 ISBN: 9788578110796.
Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, (6(11):2278-2 2324,, January 1998. ISSN 1558-2256. doi: 10.1109/5.72679. Conference Name: Proceedings of the IEEE.
Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the Loss Land- scape of Neural Nets. In Advances Neural Information Processing Systems, volume 31. Cur- ran Associates, Inc., 2018, URL htLLhtpss::/
http::///ooceeedeeeeeeoe neurips httpppscccppppppp wwtpaperppppppeeeeeee http:////./aaaaaaeeeooeeeeeeeee. ltb. otg.://cc................a.a..a. Abstract.
Tianhong Li, Peng Cao, Yuan Yuan, Lijie Fan, Yuzhe Yang, Rogerio S. Feris, Piotr Indyk, and Dina Katabi. Targeted Supervised Contrastive Learning for Long-Tailed Recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6918 6928, 2022. URL httht::////// hpwp:////ppppccceeeeeeesec thecvf. com/ content/ CVPR2022 http/////mmmmm/lmmm Li Targeted Stppreiiiddpeeeieeeeeeeeee Contrastive Learning_Boo Long-Taa ed weweeconnntinncveeeeee
2acll/_pppee._. html
Wen Li, Limin Wang, Wei Li, Eirikur Agustsson, and Luc Van Gool. WebVision Database: Visual Learning and Understanding from Web Data, August 2017.
1708. 02862
arXiv: 1708. 02862 [cs].
Zhiyuan Li and Sanjeev Arora. An Exponential Learning Rate Schedule for Deep Learning. In Inter- national Conference on Learning Representations, March 2020. URL htttps:://ppeeeeiiiiii
met/forne SiruummdddUU@@SSSS
Weixin Liang, Yuhui Zhang. Yongchan Kwon, Serena Yeung, and James Zou. Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning. Advances in Neural Information Processing Systems, 2022. URL htttp://arxiii org/abs /2203
02053
arXiv: 2203.0205:
Jialun Liu, Yifan Sun, Chuchu Han, Zhaopeng Dou, and Wenhui Li. Deep representation learning on long-tailed data: A Iearnable embedding augmentation perspective. In Proceedings of the IEFE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 2967-2976. February 2020. doi: 10. 10.11090 0.304.0 URL.https:: htt.ttsss:/axii.....tttt abs/
2002. 10826
arXiv: 2002.10826 ISSN: 10636919.
Xuantong Liu, Jianfeng Zhang. Tianyang Hu, He Cao, Yuan Yao, and Lujia Pan. Inducing Neural Collapse in Deep Long-tailed Learning. In International Conference on Artificial Intelligence and Statistics, pp. 11534-11544, PMLR, 2023.
Ziwei Liu, Zhongqi Miao, Xiaohang Zhan, Jiayun Wang, Boqing Gong, and Stella X. Yu. Scrge-Scale Long-Tailed Tailed Recognition in an Open World. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). April 2019. doi: 10.48550/arxiv. http:/aawx..1...00.... 190...0044..0 arXiv: 1904.05160.
URL https: //as://araa rxiv. org. abs/ 904. 05160
Ekaterina Lobacheva, Maxim Kodryan, Nadezhda Chirkova, Andrey Malinin, and Dmitry P. Vetrov. On the periodic behavior of neural network training with batch normalization and weight decay. Advances in Neural Information Processing Systems, 34:21 14:221555 12111555515566 2021.
Alexander Long, Wei Yin, Thalaiyasingam Ajanthan, Vu Nguyen, Pulak Purkait, Ravi Garg, Alan Blair, Chunhua Shen, and Anton van den Hengel. Retrieval Augmented Classification for Long- -ail Visual Recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6959-6969, 2022. URL https: httt:s:///ppeaccesse. hptp//ppsccccs..eeeppp theevf. com/ content/ wwtp:////ecv222222m.cccc Long Retrieval Augmented Classification. For Long- Tail Visual Recognition CVPR 2022 paper.html
Ilya Loshchilov and Frank Hutter. SGDR: Stochastic gradient descent with warm restarts. In 5th International Conference Learning Representations, ICLR 2017 Conference Track Proceedings. August 2017. doi: 10.448550/rxxiv1166 100.0000......03883...
URL https: :/tpps/i////x....//// ozg.
etq/abb//// abs/ fabb/1508
03 99.3
arXiv:
12

---

Published as a conference paper at ICLR 2024
Ilya Loshchilov and Frank Hutter. Decoupled Weight Decay Regularization. September 2018. URL
Jianfeng Lu and Stefan Steinerberger. Neural Collapse with Etoss-Entropy Loss, January 2021. URL http://arxiv. http://awwivv.ogg///////222222// wtw.o///..//.........//o./ 08465 MrXiv:2012.08466 [cs, math].
Kaifeng Lyu, Zhiyuan Li, and Sanjeev Arora. Understanding the Generalization Benefit of Normal-
ization Layers: Sharpness Reduction. In Advances Neural Information Processing Systems, October 2022. URLhttpss ppwnnevieeeeeeeeeeeeeeeeeee net. forum 2tt.:mmmmmxxpp00xXXx
Teli Ma. Shijie Geng, Mengmeng Wang, Jing Shao, Jiasen Lu, Hongsheng Li, Peng Gao, and Yu Qiao. A Simple Long- Tailed Recognition Baseline via Vision-Language Model, November 2021. URL
hittss://arxtne org/ httpr///////11111 ..111..44441
Yanbiao Ma, Licheng Jiao, Fang Liu, Yuxin Li, Shuyuan Yang, and Xu Liu. Delving into Seman- tic Scale Imbalance. In The Eleventh International Conference on Learning Representations, September 2022. BtLhtttp::/ epwereeiee...........e.ee....... .ot./forumm.d-oTTccc.....
Dushyant Mehta, Kwang In Kim, and Christian Theobalt. On implicit filter level sparsity in convo- lutional neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 520-528, 2019.
Aditya Krishna Menon, Sadeep Jayasumana, Ankit Singh Rawat, Himanshu Jain, Andreas Veit, and Sanjiv Kumar. Long-tail learning via logit adjustment. International Conference on Learning Representations, July 2020. doi: 484550//axi.220000733444 URL
abs 2007. , 07314 arXiv: 2007,07314.
Vardan Papyan, X. Y. Han, and David L.. Donoho. Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences of the United States of America, (77440):2652-2246633 August 2020. ISSN 10916490 doi: 10. 1073paasss.. 2015509117. htthhttp:A/ /ta.tp://axxiv....o org. abs /208 08186 arXiv: 2008.08186.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning Transferable Visual Models From Natural Language Supervision. In Proceedings of the 38th International Conference on Machine Learning, pp. htt.://///raaodddeeeoomm
PMLR, July 2021. URL htm_
mlr
ISSN: 2640-3498.
Akshay Rangamani and Andrzej Banburski-Fahey. Neural Collapse in Deep Homogeneous Classifiers and The Role of Weight Decay. In /CASSP, IEEE International Conference on Acoustics, Speech and Signal Processing- Proceedings, volume 2022-May, pp. 4243-4244 2022. ISBN 978-1-
66540-540-9, doi: 10. LLL.://CIAASSSP2222222202222
ISSN: 15206149.
William Reed. The Pareto, Zipf and other power laws. Economics Letters, 74(1):15-19, December 2001. ISSN 01651765. doi: 10.1016/S0165- 170200655-176(0005555554540 Publisher: North-Holland.
Merrielle Spain and Pietro Perona. Measuring and Predicting Importance of Objects in Our Visual World, November 2007.
beattts::/resolweee www.ech. www.ealltccaaaaaaaeeaaaaa
CNS-TR-2007-0CC Num Pages: 8 Place: Pasadena, CA Publisher: California Institute of Technology.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: A Simple Way to Prevent Neural Networks from Overfitting. Journal of Machine Learning Research, 19696):199-1158, 2014. ISSN 1533-7928. URL http:/ 3m1 org/
paperssv15/srivastaaaae 4a. html
Thomas Strohmer and Robert W Heath. Grassmannian frames with applications to coding and communication. Applied and Computational Harmonic Analysis, 14(3):257-755, 2003. ISSN 10635203. doi: ctt.://ccaeeeeettteeeee///333330
URL httpps://// www.wwwwwwww.ww..........w.....w eeceenneeirerr
arXiv: math/0301135.
Cecilia Summers and Michael J. Dinneen. Four Things Everyone Should Know to Improve Batch Normalization. In International Conference on Learning Representations, March 2020. URL
13

---

Published as a conference paper at ICLR 2024
Christos Thrampoulidis, Ganesh R. Kini, Vala Vakilian, and Tina Behnia. Imbalance Trouble: Revisit- ing Neural-Collapse Geometry, Advances in Neural Information Processing Systems, August 2022. doi: 10. 10.485500/mxx..2208.55553 2208.05512.
URL
otg.//ii.irrg/ooooooooooo abs 2208. 05512, arXiv:
Changyao Tian, Wenhai Wang, Xizhou Zhu, Jifeng Dai, and Yu Qiao. VL-LTR:
Learning Class-wise Visual-Linguisic Representation for Long-Tailed Visual Recognition. In Shai Avidan, Gabriel Brostow, Moustapha Cissé, Giovanni Maria Farinella, and Tal Hassner (eds.), Computer Vision ECCV 2022, Lecture Notes in Computer Science, pp. 73-91, Cham, 2022. Springer Nature Switzerland. ISBN 978-3-031-1980699 doi: 10. (0.1000/978-3-000 00079997-33-31--19966-99993
Mariya Toneva, Adam Trischler, Alessandro Sordoni, Yoshua Bengio, Remi Tachet Des Combes, and Geoffrey J. Gordon. An empirical study example forgetting during deep neural network learning. 7th International Conference Learning Representations, ICLR 2019, December 2019. doi: 4485500/arxiv11112.055..3 URL http:://axxi..oot// http://awwwwww.rr/bbbbas/a 812. 05159. arXiv:
1812.05159.
Rahul Vigneswaran, Marc T. Law, Vineeth N. Balasubramanian. and Makarand Tapaswi. Feature generation for long-tail classification. In Proceedings of the Twelfth Indian Conference Computer Vision, Graphics and Image Processing, ICVGIP 221, pp. 1-9, New York, NY, USA, 2021. Association for Computing Machinery. ISBN 9778---453--776----99 doi: 10.1145/3490033344900000
URL http:tpp:/ddaacc..rrrr////////// http:///w.........///////0/.11..
3490300
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, koray kavukcuoglu, and Daan Wierstra. Matching Networks for One Shot Learning. In Advances in Neural Information Processing Systems, vol- ume 29. Curran Associates, Inc., 2016.
http:/ppppprr...p.cccppppp cttpp//prs..ncpccppppp.p.e.eeeee f html
Ruosi
Wan, Zhanxing Zhu, Xiangyu Zhang, and Jian Sun. Spherical Motion Dynamics: Learning Dynamics of Normalized Neural Network using SGD and Weight Decay. In Ad- vances in Neural Information Processing Systems, volume 34, pp. 3380-6391, Curran Asso-
ciates, Inc., 2021. URL httpttss//pproeeediinnse .ew.:eeeedddnnn...e............. cc ptcp/apper222//aahh 37658005o00011556664666633333333
Abstract html
Jianfeng Wang, Thomas Lukasiewicz, Xiaolin Hu, Jianfei Cai, and Zhenghua Xu. RSG: A Simple but Effective Module for Learning Imbalanced Datasets. In 202/ IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 3783-3792, June 2021. doi: 10.1109/CVPR46437,
2021.00378. ISSN: 2575-7075.
Saining Xie, Ross Girshick, Piotr Dollar, Zhuowen Tu, and Kaiming He. Ag- gregated Residual Transformations for Deep Neural Networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1492-1500, 2017. URL https:://pensceesss tww.tt...ccooooooo cittoo/coottttttcoooo cvpr_ aaprr_207//hmae Xie Aggregatee orgaraaeeeeetacaaaaaaaoaaaoroonr Transsformattonss_cvvr.... www.eo111pppppppee.. html
Yibo Yang, Shixiang Chen, Xiangtai Li, Liang Xie, Zhouchen Lin, and Dacheng Tao. Inducing Neural Collapse in Imbalanced Learning: Do We Really Need Learnable Classifier at the End of Deep Neural Network? In Advances in Neural Information Processing Systems, October 2022.
URL https:://ppereviiiiie
n forum? 12281. URL for.mrid=dmmmmm Aww
Guodong Zhang, Chaoqi Wang, Bowen Xu, and Roger Grosse. Three mechanisms of weight decay regularization. 7th International Conference on Learning Representations, ICLR 20/9, October 2019. doi: 10.448550/rrxv.188111 arXiv: 1810.12281.
Yifan Zhang, Bingyi Kang, Bryan Hooi, Shuicheng Yan, and Jiashi Feng. Leng Leaming A Survey. October 2021. doi: 10.48550/arxiv.211 10.04596. UR..httpp://axxiivorrr/... 2110 0. 04596 arXiv: 11.0.459..
14

---

Published as a conference paper at ICLR 2024
A APPENDIX
A.1 ACRONYM AND NOTATION
Tables 6 and summarizes the abbreviation and notations used this paper, respectively.
Table 6: Table of abbreviation.
Abbreviation
Definition
BN
batch normalization (loffe & Szegedy 2015
CB
class-balanced loss (Cui al. 2019
CE
cross entropy
DNN
deep neural network equiangular tight frame
ETF
FDR
Fisher's ' discriminant ratio (Fisher 1936
FR
feature regularization
LA
logit adjustment (Kim & Kim 2020 Menon et al.
2020
LTR
long-failed recognition
MLP
multilayer perceptron
NC
neural collapse (Papyan
2020
ResBlock
residual block (He et al. 2016)
SELI
simplex-encodede -ecodddd- interpolation stochastic gradient descent
SGD
WB
weight balancing (Alshammari et al. 2022
WD
weight decay
Table 7: Table of notations.
Variable
Definition
number of classes
d
number of dimensions for features number of samples of class l
N / Na
N
harmonic mean of number of samples per class
hyper parameter of weight decay / feature regularization
imbalance factor: max, No.
p
miny N,
X/Y
domain of samples labels
x/y/z
sample label logit
D/Ds,
dataset of all classes / class A
f/g/h h
neural network of whole model / feature extractor linear classifier
g(x) gg(x)
feature of sample normalized feature of sample
/1/ pL
mean inner-class mean features / inner-class mean of features for class k of parameters of f //g h parameter in +
O 0/0,/0)
0
W /wk
linear layers' weight matrix / vector of class le
E / Ice / tcB
loss function / of CE / of CB objective function in the second training stage WB cosine similarity of two vectors
F FwB
cos(:;)
T/7
hyperparameter of additive LA multiplicative LA
A.2
RELATED WORK
A.2.1
LONG-TAILED RECOGNITION
There are three main approaches to LTR: "Class Re-balancing, Information Augmentation, and Module Improvement" (Zhang et al, 202T). "Class Re-balancing" adjusts the imbalance in the number of samples per class various stages prevent accuracy deterioration. It includes logit
15

---

Published as a conference paper at ICLR 2024
Table Comparison of existing LTR methods with our simplified method. A symbol indicates that the component is required, indicates that the component is not required. We regard the deferred ee-balancing optimization schedule (Cao et al. 2019) as two-stage training. Our method does not require complex technique like many other methods.
2022)
Kim & Kim 2020
Menon et aL. (2020)
Yang et al. (022
Ma et al. 2022
Liu el aL (2023
Alshammart et al.
Cao et 2 2019
Kang et al. (023
Long et al (2022
Ma et al. 2021
Tian et al. 2022
Li Li 2022
Ours
Components \
Methods
Number of training stages
3
2
2
2
Devising loss functions Resampling
V V
Contrastive learning
Training of linear layer Extra text encoder Extra image encoder
Extra datasct
adjustment (LA) (Kim & Kim, 2020 Menon et al. 2020) and balancing the loss function such as CB and dynamic semantic-scales balaneee loss (Ma et al. 2022). "Information Augmentation" prevents the accuracy from degrading by supplementing the information of the tail classes that lacks the number of samples (Chu et al. 2020; Liu et al., 2020 Wang et al.) (021). "Module Improvement" improves accuracy by increasing the performance of each module of the network individually, e-g..
training feature extractors and classifiers separately (Kang et al., 2020), fixing the linear layer the
ETF classifier for the better feature extractor (Yang et al., 2022), and regularizing to occur NC (Liu
al. 2023) In recent years, as Kang et al. (223) found contrastive learning (Hadsell al.
2006
effective for imbalanced data, many methods have been proposed to use (Kang et al.) 2023
Li et al.
2022
Ma et al. 2021] Tian al. 2022). Ma et al. (2021), Long al. (2022), and Tian et (2022)
also include vision-language models such as CLIP (Radford et al. 2021) and leverage text features to improve accuracy. However, these usually have problems such slow convergence and complex models (Liu et al. 2203).. WB is a combination of "Class Re-balancing" and "Module Improvement", g compares our simplification of WB with typical existing methods, including those that have achieved SOTA. While many methods use complex innovations, we aimed to improve on a simple structure.
Table
Two-stage learning Two-stage learning (Kang al. 2020) is a method for improving accuracy by dividing LTR training into two stages: feature extractor training and classifier training. It is used in numerous methods (Cao et al. 2019 Ma al. 2021 2022: Li et al. [022 Tian al. 2022 Liu et al. 2023 Kang et al.] 2023), including WB, because its simple but significant improvement in accuracy. Note that since work analyzes WB, the formulation of two-stage learning is based WB. For example, the classifier weights are initialized randomly at the start of the second stage of training in Kang al. 2020) but not in WB.
"This method does not require linear layers.
"This method requires the I.GR head to be trained Tian al.
2022
16

---

Published as a conference paper at ICLR 2024
A.3
PROOF
A.3.1
PROOF OF THEOREM T
Proof. Define pa (g(x)) as
exp(w, g(x:))
For x E {xi; X, we have
Di=1 exp(pi (wi 9(x.1)
OCcr 0g(x) 1
OlcF 0g(x)
<
(8g(x)' Wyy )
OlCE
||wyl2 2
12
(1 (x(g + [p((s(x)) Wy
izy
- (1 - pv(g(x)))
- Σ p(((()))
= C-1 (1 - ((1-p(gg(x)) p)(g(x)))
by
(1 Pr(g(x))).
(4)
Therefore,
dccE S
(1 pr(g(x))) <
0g(x) 2
pv(g(x)) 1
(5)
For Py (x)),, using Jensen's inequality, we get
exp( (xp(w]g(xx) g(x))
exp(w,]g(x) gx())
pr(g(x))
Σ exp (w, g(x))
exp(wTg(()) + (C- ot exp (w, g(x))
exp(w__g(x)
xxp(w]g(x) + -C (ct1 Eiey g(x)
exp(w g()))
exp(w g(x) + (C' - 1) exp
T- w (-o-_w g(x)
-C 1+ - 1) exp
-- w g(x))
C
Define g(x,) as
[x(x) By 55) and (6)
g(x.)
we have
C
(C - 1)exp
C-1
1-c
1
C-1
(C - (16)))
w]]g(x= 2
log
[g(x)]l
C
C
-
(C-1)(1-0)) - o) (C)
(C-
wx](()== 2
log
= d.
L C
Note that < S e. L. < 2 log (C- 1) < 2 (C - 1) E which means 5 >
1 -z Also, 8 is a
lower bound of the inner product of the two unit vectors. Therefore, we have
1
8 EE
1
(8)
2
17

---

Published as a conference paper at ICLR 2024
Let <(s,t) arccos(cos(s, t)), which represents the angle between vectors and t. For example, we have L(Wy.: Wyy) arccos
(-c-1) Using (8), the following holds.
g(xx))))(x())) 2 <( Wys) L(w y (xx)) 2( g(x)))
L( (Wy, Wy,) max ([(wy.g(x))
xE(xxx,)
N 0.
(9)
Rewrite <(Wy, Way) and m (<( ((x)())))) and respectively. Since the cosine is monotonically decreasing in [0, mT. we get
(g(x) S cos(a - 2,3
cos cos² / - 1) + 2 a sin 3 cos /
cos a cos? $ 1) + 2 cos cos 8
(10)
Note that rv1 - r is monotonically decreasing and 2x > holds in
1 V2
1]
Therefore, from (7
and 10 the following holds.
cos (g(xx)..(xx)) s 26 82.
(11)
A.3.2 PROOF OF THEOREM 2
Define W& be following theorem.
N Mgg We prove strict theorem. You easily derive Theorem p4.
from the
Theorem 3. For any k E V, if there exists w s.t.
8FWE dwL
0 and |/ O 0
Apc
we
WW=WE
have
N AN
Vk V. w S
1lll
+
(12)
X²p2
Before proving the theorem, first present the following lemmas. Lemma 1. For the dataset of Imbalance rate p. the following holds.
N
(13)
N
pc
Proof. Since N₂ np c-1 holds, the harmonic mean and the number of all samples are
C
nC
N
(14)
k=
NF
Σ
C C
C
C
N
N₂
Σ
n
(15)
8
k=1
k=1
Therefore,
(pot - 1)²
pc
N
PC
(16)
N
(po--
18

---

Published as a conference paper at ICLR 2024
From 16), N = 0 (p + oo) is obvious. In addition, the following holds.
NC
pC
pC²
lim -000
N
lim C
A-1
S lim
00000
C
=
(17)
k==1
p
N Thus, N = (t) (C + o0) is also satisfied.
Lemma 2. When Wi
N XN ph is satisfied, we have
Wk V y, OFWB N Nu-
(x/===
1
- 0
(18)
Ow
Ap²C2
Proof of Lemma
For any V, holds that
Fww
N
Σ exp(w
NV
(x:J)D, E exp (wi g(x:)) g(x:))
N pa + Awk-
(19)
Owk
N
NJ
In addition, using Lemma]
! we derive the following:
exp(w[_g(x))
exp(W]g((x))
E exp (w/ g(x:))
(w]]g(x))
C
E 1=1 exp (w/ g(x:)
C
1
1 1w g(xx))(x))) g(x:))
1
+
0
E (1 O(w g(x,)))
110
Apc ))
1
1 0
1
-1)
C
0
+0
(20)
C
ApC
Using this, the assumption and
19
we get
FFwB
N
+ +0
1 (10c2).
Owk I
(21)
N
Proof of Theorem 3
0FwB
FFwB
FFwB
Owk
Owk
Ow
ww-
w.=
A(we
wi)
N
Σ
exp(w, g(x))
1-1 exp (wi g(xx)) s(xi))
exp(w: g(x:))
N
ll exp (w, g(x:))
(XiJ)ED
(22)
Here, similar to 20
the following can be derived;
exp(WTg(xx)
exp(wi g(x;))
C +0 ( (1022) =0 1
1
(2/c²)
EC exp (w/ g(x:))
Σ exp (wi gxx:)
(23)
19

---

Published as a conference paper at ICLR 2024
Using 22 and 23
0FwB
A(WE
w/)
Owk
Xp²C2
wi ww=wkk
N
M O
(24)
N
Ap2C22
Therefore,
N
|llp + +0 (
(22202 (113202)
W wxll2 - AN
(25)
A.4 SETTINGS
Our implementation is based on Alshammari et al. (2022) and Vigneswaran et al. (202) For comparison, many of the experimental settings also follow Alshammari et al. (2022).
A.4.1 DATASESTS
We created validation datasets from the portions of the training datasets because CIFARIO and CIFARION 100 have only training and test data. As with Liu et al. (2019), only 20 samples per class were taken from the training dataset compose the validation dataset, and the training dataset was composed of the rest of the data. We set N1 to 4980 for CIFARIO and 480 for CIFARI00, In our experiments, we set imbalance factor p to 100. We call the class Many if the number of training samples satisfies 1000 < N& I 4980 (resp. 100 < N₂ I 480), Medium if the number of training samples fullfills 200 < N₂ S 1000 (resp. 20 S Ng S 100), and Few otherwise for CIFAR 10-LT (resp. CIFARI00-LT) For mini-ImageNet-LIT and ImageNet-LT, the same applies as for CIFAR100-1.1
A.4.2 MODELS
As for MLPs, one module block consists of three layers: linear layer outputting 1024 dimensional features, a BN layer, and ReLU layer. The layers are stacked in sequence and the blocks composed of them are as well. As for ResBlocks, each block has the same structure as He et al. (2016), except that the linear layer outputs 1024 dimensional features. These blocks are combined into sequential, and at the bottom of it, we further concatenate an MLP block. In other words, the input first passes through the linear layer, BN, and ReLU before flowing into the residual blocks. In both cases, classifier consisting of a linear layer and a softmax activation layer on the top and does not count as one block. For example, in MLP3, the features pass through the three blocks and the classifier in sequence.
A.4.3 EVALUATION METRICS
Unless otherwise noted, we used the following values for hyperparameters for the ResNet. The optimizer was SGD with momentum 0.9 and cosine learning rate scheduler (Loshchilov & Hutter 2017) to gradually decrease the learning rate from 0.01 to O. The batch size 64, and the number of epochs was 320 for the first stage and 10 for the second stage. As loss functions, we used naive CE and CB. As for CB, we used class- balanced CE with 3 0.9999. We set À of WD 0.005 in the first stage and 0.1 in the second stage. For mini-ImageNet-LI1 we set À for the first stage to 0.003. We set ( of FR to 0.01. We calculated the MaxNorm's 's threshold as in Alshammari et al. (0022). We searched the optimal and for the LA by cross-validation using the validation data. We chose the optimal value of T from (1.00, 1.05, 2.00} and y from {0.00, 0.05. 1.00].
For ImageNet-LT, We searched hyperparameters using the validation dataset for the ones not published in Alshammarri et al. (2022) and used the following values. We reduced the learning rate gradually from 0.05 to 0. The number of epochs was 200 for the first stage. We set À of WD to 0.00024 in the first stage and 0.00003 in the second stage. We set Ç of FR to 0,0001. The other hyperparameters were the same in CIFAR 100-LTT.
20

---

Published as a conference paper at ICLR 2024
Table 9: FDRs and accuracy of models trained with ETF classifier for CIFARTOO-LT.
FDR
Accuracy (%)
Method
Train
Test
Many
Medium
Few
Average
WD&ETF 3.33x102 10
1.13x10 2
46.0 10.3
10.4 15.5 46.9
101x100
WD&ETF&DR 8.25x10 8.25x1017.0x100 17.0x104
9.46x10 40.8 100
12.9
43.0 20.0.0.9 46.0
We trained MLPs and ResBlocks with À set to 0.01 and the number of epochs set to 150. Other parameters were the same as above. FDRs and accuracy reported in our experiments were obtained by averaging the results from five training runs with different random seeds. We conducted experiments on an NVIDIA A100.
A.4.4 WHY WE DO NOT USE DR LOSS
This section explains why we use ETF classifiers but not the dot-regression loss (DR) proposed in Yang et al. (2022). Table compares FDRs and accuracy when we use WD and an ETF Classifier for the linear layer with two options, using CE or DR for the loss. This table shows that CE is superior in the test FDR and average accuracy. Theorem also claims that training by CE and WD prevents the cone effect. does not apply to DR, which is squared loss function. Thus, it is difficult to prove whether DR is equally effective in preventing the cone effect.
A.5 EXPERIMENTS
A.5.1 EXPERIMENTS OF SEC 4.2
The right half of Figure 4 show the result of 4.2 for mini-ImageNett-I.T We also examined the forget score Toneva et al. 2019) of each method in Sec 4.2 Phenomena similar to the ones we showed in Sec 4.2 can be observed in the forgetting score of features trained with CB: Figure shows that the forgetting score is higher on average with CB than with CE. In the absence of WD, this phenomenon is particularly evident for the Many classes, possibly because CB gives each image in Many classes less weight during training, which inhibits learning. These may be some of the reasons for the poor accuracy of CB compared with CE in training feature extractors reported by Kang et al. (2020).
A.5.2 EXPERIMENTS OF SEC.
High relative variance in shifting parameters of BN temporarily degrades FDR In See Sec investigated the effect of the small scaling parameters of BN. What then is the impact of higher relative variance in the shifting parameters of BN? Our experiments have shown that they temporarily worsen the FDR in Figure 6 In the experiments, we trained MLPs with MNIST using three different methods: without WD (Naive), with WD (WD all), and with WD except for any shifting parameters of BN (WD w/o shift). We retrieved the output features of untrained and these trained models for each layer, applied ReLU to them, and examined their FDRs. We refer to these as ReLUFDRs. Figure 6 shows the increase and decrease in the ReLUFDRs of the intermediate outputs from the MI.ss trained with each method. This figure indicates that the ReL.UFDRs of the models trained without WD monotonically and gradually increases as the features pass through the layers. However, this is not the case for the models trained with WD. In particular, ReLUFDRs decrease drastically when train the shifting parameters and apply the scaling and shifting parameters the BN in the blocks close to the last layer. In this case, the ReLUFDRs increase when the features pass through the linear layer more than the ReLUFDRs decrease when the features pass through the BN later. These results suggest that applying WD the scaling and shifting parameters of BN have positive effect training of the linear layer.
21

---

Published as a conference paper at ICLR 2024
CIFAR1O-LT
with WD
without WD
40
WD & CE
40
Naive & CE
WD CB
Naive & CB
Forget scores
30
30
20
20
10
10
0
0
2
4
5
8
9
10
2
4
5
8
9
10
Class number
CIFAR100-LT
with WD
without WD
WD CE
Naive & CE
WD & CB
Naive & CB
Forget scores
30
30
20
10
10
01
0
20
40
60
80
100
20
40
60
80
100
Class number
mini-ImageNet-LT
with WD
without WD
60
60
WD & CE
Naive & CE
50
WD CB
50
Naive &
Forget scores
30
30
10
10
0
0
20
40 Class number
80
100
20
60
80
100
Figure Average forgetting scores per class when models are trained with each method. These indicate higher forgetting scores when the models are trained with CB; this is particularly noticeable in the Many classes without WD.
22

---

Published as a conference paper at ICLR 2024
mini-ImageNet-IT
CE wHO WD
CP WD
40-
0.50
mini-ImageNet-IT
3.25
60
1B: mio WD
we
30 4
N. 20200
1010
WD
Norm
WD
-0.25
4.50
20
-100
20
40
60
80
100
Class number
180
Figure 4: Results for mini ImageNet-LI LT. (Left) Norm of mean per-class training features produced from models trained with each method. (Right) Heatmaps showing average cosine similarities of training features between two classes.
CIFAR10-LT
CIFAR100-LT
WB
Ratio of norms
0.975
CR
Ratio of norms
1.4
WD WD & Norm CB ....
M
0.950
0.925
Mm
0.875
100
0.850
60 Class number
40
Class number
mini-ImageNet-LT
WB
Ratio norms
M
0.3
mdy W
1.2
m
40
80
100
Class number
Figure 5: Norm ratio of mean per-class training features produced from models trained with each dataset and each method. Note that the vertical axis shows the ratio of the norm of the weights for each class with the one for the class of which sample size is the largest. Models trained with both methods have almost identical linear layer norms.
23

---

Published as a conference paper at ICLR 2024
MLP Bllock
MLP (Block
MLP (Block 5)
Not trained
10"
No: trained
Naive
Naive
WD all
WD
WD
WD w/o bios
ReWUFR
107
Not trained
felie
162
WD all
10a
WD w/o shift
Input
UN normellzee ized
(NN scaled shilled
BN Lineer
BB scaled & shifted
Uneel
BN norma lacd
shitted
Input
Unzar
BN normal zod
scaled & shimtd
Unaar
BN nommal zed
scated 5. shimed
Linar
BN normal zod
BM scated shimad
Unaar
BN normal zed
scated shimed
Inpar
Linear
BM narmalized scaled shifte
Linear BN normalizea & shifted
Linea
BN nunmalizzn scaled & shiff ad
Linear
DN nonmalized scaled K. shifted
Linear
UN narmalized
EN scaled & shitted
Block
Block Layer
Black
Block 1
Block 2 Block 3
Block a
Black
Block Block Block
Figure 6: ReLUFDR of intermediate outputs of each MLP trained with each method. Training with WD decreases ReLUFDR significantly when features pass through the scaling and shifting layer in the BN of the final layer.
Table 10: Change in FDR as learned features pass through randomized linear layers and ReLUs multiple times. Red text indicates an increase compared the FDR of the feature when the number of layers passed is one less, while blue text indicates decrease.
Number of times through randomized linear layers
Method
0
2
3
CE
2.44 x 102 2.37 x 102 2.17 x 10a
1.96 x 102
WD w/o BN 4.91 x 102 6.76 x 102 6.56 x 102
6.22
102
WD
6.10 10 2.47 x 10a 2.34 x 103
1.60
10
A.5.3 EXPERIMENTS OF SEC. 4.4
This subsection describes the experimental setup of Sec. 4.4 and additional experiments. We trained models for the balanced datasets using the same parameters settings as in Sec. 4.2, The FDRs of "before" are measured from the features obtained by models trained in the same way as the experiments in Sec. 4.2 These features are passed through an extra linear layer, and FDRs of them are measured as the "atter". We initialized the weight (E RRdxd) and bias (E Rdx1) of the extra
linear layer with random values following uniform distribution in Va Va and fixed them.
Table 10 shows the relationship between the FDRs and numbers of times features from MLP5 are applied by randomly initialized linear layers and ReLUs. The FDRs increase the first time, but gradually decrease the second and subsequent times.
A.5.4 EXPERIMENTS OF SEC.
First, we experimentally show MaxNorm is not necessary for the second stage of WB. In the second training stage in WB, we observed that normalizing the norm of the weights to one before training (WD WD & Norm & CB) instead of applying MaxNorm every epoch gives almost identical results as shown in Figure 5
This phenomenon is consistent with the fact that the regularization loses its effect beyond certain epoch (Golatkar al. 2019)
Figure 5 also shows that WB has difficulty doing implicit LA when the number of classes is small. While the weights of the linear layer are larger for Few classes in the datasets with sufficiently large number of classes, they remain lower for Few classes in CIFAR1O- These results are consistent with the conclusions drawn from Theorem
24

---

Published as a conference paper at ICLR 2024
Table 11: FDRs and accuracy of models trained with restricted WD and ETF Classifier for CIFART00-
LT.
FDR
Accuracy (%)
Method
LA
Train
Test
Many
Medium
Few
Average
WD
N/A
1.94x10 10
5.89x10
11.0x10 100
74.640.5
13.4 44.4
45.1
w/o BN
Add
1.94x10 108x101
5.89x10
110x100
69.3
49.4
30.4
50.3
& ETF
Mult
1.94x10 10
5.89x10
67.640.9
51.040.5
31.5
50.640.7
1.0x 10"
WD
N/A 5.47x10
8.0x10
1.05x10
0.2x10'
75.2+0.6
45.4
16.2+0.4
46.540.1
fixed BN
Add 5.42x10
1.05x10
71.9+0.6
44.8
26.540.5
48,4+0.3
8.0x10
0.2x10
&ETF
Mult 5.42x10
1.05x10
72.240.8
45.7.0.5
26.9
48.940.3
10
10
Table 12: FDRs and accuracy of models trained with each method for CIFARIO-LT.
FDR
Accuracy (%)
Method
LA
Train
Test
Many
Medium
Few
Average
CE
N/A
8.17x10
2.17x10
67 67.2+3.1 87.7
47.4+2.
69.4+1.0
11
10
CB
N/A
4.43x10
1.50x10
81.7
60.642.0
42.0+4.7
63.5+1.3
11
10
N/A
2.97x10 303
4.21x10
89.1 76.6
61.545.0
77.1
10
WD
Add
3.40x10 4.32x10
4.21x10
81.6=3.8
79.5.1.2
79.743.1
80.4
+0.7x100
Mult
3.40x10 103
4.21x10
86.222.9
80.3
74.0+3.9
80.8
1.7
WB
N/A
1.82x10
3.86x10
87.9
77.6
67.943.2
78.8
108
1.9
N/A 3.90x10
4.38x10
89.7+1.9
73.9
59.1+1.8
75.8
10
10"
WD&ETF Add 3.59x10
4.38x10
87.223.4
77.1
72.542.6
79.8+1.1
1.18x104
10.2x100
Mult 3.59x10
4.38x10
86.9 79.0
73.2
80.4
4.18x104
102
N/A 5.56x10
4.78x10
90.9 76.7+1.
56.2+2.0
76.2
WD&FR
4
12.0x10"
Add 7.58x10 6.06x10
4.78x10 10"
88.8
78.7
70.3
80.2 80.2
&ETF
Mult 7.58x10
4.78x10
89.9 1.9
80.8 11.3
66.3
80.1
6006x105
100
WD
N/A 9.17x10 +3.00x
2.95x101
+2.8x 100
88.3
75.613.6
59.1
75.7
w/o BN
Add
9.26x10 10
2.95x10
1055114428x100 12.8x109
85.922.
79.0
74.3+
80.3
& ETF
Mult 9.26x10
2.95x101 1.95x10+28xx10 12.8x100
85.5125
79.0
75.8
80.711.8
10'
WD
N/A 7.02x10 x4.59 10a
4.29x10 122xx00428xx00 +2.8x
89.5.1.
74.313.2
58.9
75.7
fixed BN
Add
6.60x10 1.28x
4.29x10
79.3
76.9.3.6
78.7
78.413.2
10
&ETF
Mult
6.60x10
4.29x10 128x109
84.5 78.5
77.1 80.5
10
A.5.5 EXPERIMENTS OF SEC 5.2
We show the FDRs and accuracy of methods with restricted WD for CIFARI00-LT in Table 11
Table 22 13 and Table 14 show the results for CIFARI0-LI mini-ImageNerr-LT, and ImageNet-LT respectively. We set Ç for CIFARIO-LT, mini-ImageNet--T, and ImageNet-LT to 0.02, 0.001, and 0.0001 respectively.
We also experimented with the ResNeXt50 Xie al. 2017) as with Alshammari al. 2022 We used the same hyperparameters as the ResNet. Table 15 presents the results for CIFARTON (00
25

---

Published as a conference paper at ICLR 2024
Table 13: FDRs and accuracy of models trained with each method for mini Mini-lmmaeNNet-LL
FDR
Accuracy (%)
Method
LA
Train
Test
Many
Medium
Few
Average
CE
N/A
7.56x10
4.28x10
72.1
34.1
18.3
42.7+0.2
106
+0.3x100
CB
N/A
4.68x10
2.93x10
59.440.5
25.040.4
15.7+0.1
34.3+0.2
0077x100
10.
N/A
6.58x10 2. 10
1.01x10
81.710.4
10.
20.310.7
49.810.3
x
WD
Add
6.51x10
10
1.01x10
101
80.510.3
42.1
39.3
54.7=0.3
Mult
6.51x10
1.01x10 10.1x101
79.940.2
41.640.8
40.3+0.3
54.6-0.4
103
WB
N/A
6.52x10 3.0x
1.00x10
10
80.1 10.2
13.
38.1
54.8
N/A
8.20x10
1.09x10
80.7 42.9
21.6+1.2
50.1+0.2
6.5x
10
WD&ETF Add
8.27x10 8.5x10
1.09x10
79.6 43.0.0.6
37.520.7
54.1+0.2
10'
Mult 8.27x10
1.09x10 10
78.8 42.711.
39.411.0
54.3
8.5x 10
N/A 1.32x10*
1.20x10
81.7 43.1
20.8
49.9
1.2x10
102xx11
WD&FR
Add
1.31x10 1.1x10
1.20x10 101
79.6
43.520.
34.9
53.640.6
&ETF
Mult 1.31x10
1.20x10
78.8
43.5
37.9
5.52
+1.1x102
+0.2x 101
WD
N/A 1.21x10 0011x00
5.85x101
10
78.640.4
41.7
21.7
48.640.3
w/o BN
Add 1.21x10 0.1x10
5.85x10
1093x109
75.110.2
40.3
41.710.6
52.9
& ETF
Mult
1.21x10
=0.1x10
5.85x10
109
76.2+0.3
41.0+0.5
39.640.6
52.940.3
WD
N/A
8.55x10
+2.0x10
1.25x10
011x10
80.640.2
42.8+0.5
20.3
49.3
fixed BN
Add
8.56x10
2.1x10
1.25x10
0.1x101
78.7
41.6
36.0
52.9
&ETF
Mult
8.56x10
1.25x102
80.5
43 43.610.3
25.6.0.5
51.140.1
2.1x10
10.1x10*
Table 14: FDRs and accuracy of models trained with each method for ImageNet-LT.
FDR
Accuracy (%)
Method
LA
Train
Test
Many
Medium
Few
Average
CE
N/A
1.35x102
1.58x102 10.2 101
54.7
30.1
12.1
37.1
10!
CB
N/A
1.00x10
10.8100'
1.34x10 10'
47.9
24.142.0
8.19+1.1
31.1=2.0
N/A
3.79x102 10
2.99x102
67.4
42.0
15.210.3
48.1
+0.3x 10a
WD
Add
3.77x10 10!
2.99x10
62.9
48.840.5
35.210.3
52.4
101
Mult
3.77x10 3.78x102 N/A
10.8100
2.99x10
10.
62.8
49.3106
34.410.4
52.540.
WB
10
2.99x10
10.
62.4
50.0.0.5
31.1
52.2
N/A
6.98x10* 1.1x
4.56x10
68.4
43.640.4
17.740.7
49.620.4
10.
WD&ETF Add 6.95x10
1.0x10
4.56x10
156x10000.71100
10.
64.0+0.3
49.610.
35.510.4
53. 23.24
Mult 6.95x10
4.56x10
63.410.4
50.0104
35.1
53. 13.2
100
107
N/A 1.03x10
5.46x10
68. 68.110.2
43.0
17.040.5
49.1
WD&FR
1.4x
106
Add 1.03x10 +0.4x10
5.46x10
63.3
48.740.
36.440.7
52.6-0.2
&ETF
10
Mult 1.03x10 10
5.46x10
62.4
49.3102
36.010.5
52.6 0.2
10.
26

---

Published as a conference paper at ICLR 2024
Table 15: FDRs and accuracy of ResNeXt5O trained with each method for CIFARIOO-LT.
FDR
Accuracy (%)
Method
LA
Train
Test
Many
Medium
Few
Average
CE
N/A
2.04x10
7.32x101
59.9
32.8
9.51
34.8
10.6x10
10.
CB
N/A
1.37x10 10!
5.19x10
44.8
20.710.9
4.62.0.4
23.9
N/A
8.36x10 47.1100
2.06x10
77.940.2
48.3
14.9
48.0=0.3
00
WD
Add
8.41x101 10
2.06x10 003x10
2.06x102
20.3x00000
13.4
48.610.8
32.9
52.2
Mult
8.41x101 17.9x 47.9x I0s
73.7
49.2
33.811.5
52.8
008x
WB
N/
3.19x10
-2.0x10
2.06x10
0.2x10
77.5+0.2
51.2+0.9
21.1+0.6
50.8
N/A 1.67x10 10
2.02x10
77.710.6
48.810.9
17.4
48.9
003x10
WD&ETF
Add 1.67x10
104
2.02x10
003x10
72.7
50.341.0
31.1
52.0=0.6
Mult 1.67x10
2.02x102 =003x100
74.6
54.0=11.
30.9
53.8+0.4
-2
003
N/A 3.19x10
2.06x102
77.4
49.810.5
18.210.4
49.4
WD&FR
+1.8x10
00.2110
Add 3.17x10
2.06x10
76.3
51.940.4
26.210.4
52.2
&ETF
1.8x104
0.2x10
Mult 3.17x10
2.06x10
72.9
53.8
33.4+0.6
54.0+0.3
+18x10
1022110
A.5.6 EXPERIMENTS ON TABULAR DATA
We considered analyzing tabular data an experiment for data with completely different character- isties and features from image data. We used Helena, a dataset for classification with 100 classes. Since the data is not divided for validation and test, we randomly extracted 20 samples per class without duplicates. The distribution of the training data is similar to that of long-tailed data with P = 40. We call the class & Many if the number training samples satisfies 500 < Ng. Medium the number of training samples fullfills 200 < N₂- < 500, and Few otherwise for this dataset.
Kadra al. (202)) show that even an MLP with carefully tuned regularization outperforms state- of-the-art models in the classification of tabular data. Following them, used 9-layer MLP with 512 dimensions per layer for the feature extractor. The model is trained by Adam (Loshchilov & Hutter 2018) for 400 epochs in the first stage and 10 epochs in the second stage. Thus, WD is not implemented in L2 regularization in this dataset, but is built into the optimizer. By a hyperparameter search with the validation data, set the hyperparameters
follows. We used dropout (Srivastaval jet al., 2014) and set the hyperparameter to 0.15. The initial learning rate was 0.001 for the first stage and 0.0004 for the second stage. We set of WD to 0.15 for the first stage and 0. 0.0003 for the second stage. We set ( of FR to 0.001.
Table presents the results. As in the experiment with image data, the proposed method outperforms the existing methods in both test FDR and average accuracy. Note that we used Adam W for the optimizer instead of SGD and that the proposed method also succeeds this case. This result indicates that the proposed method works for optimizers other than SGD.
A.6 BROADER IMPACTS
Our research provides theoretical and experimental evidence for the effectiveness an existing ad-hoc method. We also show that the original method can be simplified based this theory. The theory is valid for general DNNs and does not concern any social issues such as privacy. On the contrary, it reduces the number of training stages while maintaining higher level of accuracy, thus reducing the computational cost and the negative impact on the environment.
27

---

Published as a conference paper at ICLR 2024
Table 16: FDRs and accuracy of ML. trained with each method for Helena.
FDR
Accuracy (%)
Method
LA
Train
Test
Many
Medium
Few
Average
CE
N/A
5.40x10
6.48x10
34.7
20.8
24.3+0.5 17.2+0.5
5.7x100
1.1xx10
CB
N/A
4.31x10 7.1x100
6.53x10
26.420.8
27.5+1.0
25. 25.1+1.
26.3
10
N/A
7.61x10 +10x100P
6.53x10
36.310.6
20.840.4
16.6+1.1
24.7
10.
WD
Add
7.61x10 11.0x100
6.53x10
10.6x100
33.4=0.1
26.340.
24.5+0.8
28.1+0.2
Mult
7.61x10 +1.0x10
6.53x10
31.620.3
27.140.5
21.9+1.0
26.9
10
WB
N/A
7.63x101
6.55x10
35.0=o.5
26.04n.
22.7+1.4
28.0+0.5
+08xx100
10.7x10P
N/A 7.73x10
6.86x10
36.240.6
22.6411
17.7+0.3
25.6
17.21100
109
WD&ETF Add 7.73x10
6.86x10
32.6=0.9
28.840.6
25.540.9
29.0+0.4
17.2x109
10"
Mult 7.73x10
6.86x10
32.1.1.0
30.1
25.440.9
29.2
+7.2x109
10P
N/A 8. 8.49x10
7.04x10
36.1.13
21.610.6
17.411.
25.2
WD&FR
14.9x100
10%
Add 8.49x10
7.04x10
31.3-1.0
28.740.6
27.5+0.6
29.2+0.4
&ETF
14
10"
Mult 8.49x10 11
7.04x10
31.9-1.1
29.6.0.9
25.640.5
29.1 19.1
10
28


Reviews:
[{"review_id": "hw84OMhPof", "reviewer": "Reviewer_dLbF", "summary": "The author analyzed the weight balancing method for long-tailed classification problems from the perspectives of neural collapse and the cone effect, and provided some insights.", "questions": "See above", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The problem of imbalanced classification is undeniably a highly practical and crucial research issue in the field of machine learning.\n2. The authors provided an analysis of weight balancing to a certain extent and offered insightful perspectives on the topic.", "weaknesses": "1. This paper appears to resemble an appendix on Weight Balancing to some extent and the technical innovation is rather limited.\n2. Given that Weight Balancing is not the best-performing method in the field of imbalanced learning, the significance of this paper in the field remains debatable.\n3. Considering that Weight Balancing involves implicit constraints at the parameter level (compared to direct correction in other long-tail classification methods), its extension to address broader distribution shift issues should hold greater value.\n4. Sec 5.1\"the second stage of WB is equivalent to multiplicative LA\". Why not just use explicit LA?\n\n\nupdate: After reading the authors' response and other reviewers' comments, I would like to increase my score to weak accept.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "n7pyPTapGH", "reviewer": "Reviewer_7mhz", "summary": "This paper studies the problem of long-tailed recognition (LTR) and presents theoretical analysis regarding the two-stage training of LTR. The main findings include two theorems showing 1) how neural collapse and the cone effect are affected by weight balancing at each training stage; 2) how weight decay contributes to an increased in Fisher's discriminant ratio of the feature extractor and implicit logit adjustment. In addition to those theoretical results, authors also report extensive experimental results as supporting evidence. The paper is well-written and easy to follow. The technical contributions of this work are expected to sharpen our understanding of the LTR problem, which might inspire other attacks to LTR than weight balancing.", "questions": "1. What do blue and red colors in Table 4 and Table 9 highlight? Some explanations can be added to the caption of those tables. \n2. Table 1 includes experimental results for WD without and with fixed batch normalization (BN). Any plausible explanation for these results? Why does BN further improve the performance of WD?\n3. In Table 5, the accuracy performance of LA (N/A) is noticeably higher than add/mult for the category of \"many\". Why does LA only work for the Medium and Few classes?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1. The problem formulation is well motivated and sensible. Developing a theory for weight balancing in LTR has been under-researched in the literature. This work makes a timely contribution to this important topic.\n2. The technical contributions in Sec. 4 and 5 are solid. Both theorems 1 and 2 are well presented and their rigorous proof have been included in the Appendix. The generalized result of Theorem 2 (Theorem 3 in Appendix) is commendable. \n3. In addition to the theoretical analysis, this paper also reported extensive experimental results as supporting evidence. Those figures and tables have greatly facilitated the understanding of the underlying theory.", "weaknesses": "1. The difference between weight balancing (WB) and weight decay (WD) needs to be make clearer. Sec. 3 only reviews WB and overlooks WD. Historically, WD was proposed much earlier than WB. It will be a good idea to include some review of WD in Sec. 3, I think. Note that WD is already present in Table 1 on page 4 (right after Sec. 3).\n2. For those who are less familiar with two-stage training of LTR, it might be a good idea to include a concise review of two-stage training methods in the Appendix. Note that CVPR2022 and ICLR2020 have different formulation of two-stage training. Please clarify that the model analyzed in this paper is based on the CVPR2022 work even though it cited the ICLR2020 as the original source of two-stage training.\n3. There are many acronyms in this paper. It might be a good idea to provide a table summarizing them in the Appendix A.1 (Notation and Acronym).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "u5480T8NJj", "reviewer": "Reviewer_GF5W", "summary": "This paper primarily investigates why the two-stage WD method could perform well in long-tailed tasks. It analyzes the WB by focusing on\nneural collapse and the cone effect at each training stage and found that it can be decomposed into an increase in Fisher’s discriminant ratio of the feature extractor caused by weight decay and cross-entropy loss and implicit logit adjustment caused by weight decay and class-balanced loss. Then the paper proposes the simplify the WD by reducing the number of training stages into one with the combination of WD, FR, and ETF.", "questions": "1. What's the meaning of O in Eq.3 and could the author explain more about Theorem 2?\n2. Could the author explain why the WD&FR&ETF performs worse than the WD&ETF on the ImageNet-LT dataset in Table 13? And are there any experiments conducted on large-scale datasets, such as iNaturalist 2018?\n3. Existing long-tail solutions often rely on expert systems to improve the performance of tail classes, such as RIDE[1] and SADE[2]. Is the proposed method in this paper compatible with them?\n\n[1] Wang, Xudong, et al. \"Long-tailed recognition by routing diverse distribution-aware experts.\" arXiv preprint arXiv:2010.01809 (2020).\n[2] Zhang, Yifan, et al. \"Test-agnostic long-tailed recognition by test-time aggregating diverse experts with self-supervision.\" arXiv e-prints (2021): arXiv-2107.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper provides an in-depth analysis of the reasons behind the success of WD in long-tail scenarios, demonstrating thoughtful insights. From the perspective of neural collapse and the cone effect, it explains the WD well.\n2. This paper has a well-organized structure which makes it easy for readers to understand the research.\n3. Extensive experimental results confirm the validity of the analysis.", "weaknesses": "1. The paper only discusses the related work of NC and WD but the related work of the long-tail is also necessary.\n2. Some concerns which I will mention in the following section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "A3s8mWNctJ", "reviewer": "Reviewer_2xQy", "summary": "The paper aims to analyze weight balancing by examining neural collapse and the cone effect at each training stage. The analysis reveals that weight balancing can be broken down into an increase in Fisher's discriminant ratio of the feature extractor due to weight decay and cross entropy loss, as well as implicit logit adjustment caused by weight decay and class-balanced loss. This analysis allows for a simplified training method with only one training stage, while improving accuracy.", "questions": "1. Besides the analysis metrics mentioned in the paper, what other commonly used metrics exist? Why was the choice of metrics in the paper considered?\n2. If we consider balanced datasets, the analysis in the paper can still hold true. The only difference lies in the performance based on the sota models. What distinguishes this type of analysis from conventional methods when dealing with long-tailed data distributions? What are the innovative aspects of this paper?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. As an experimental and analytical paper, the logical flow of the entire article is smooth, providing a good reading experience.\n2. Weight Decay, as a simple yet effective model, is thoroughly explained in this paper with targeted explorations and explanations at each step. The argumentation is well-grounded and convincing.\n3. The feasibility of single-stage training is explored based on the analysis of the original method, which represents a certain breakthrough.", "weaknesses": "1. The analysis solely based on one particular model method has certain limitations, as it lacks consideration of other methods. Exploring why Weight Decay performs exceptionally well indeed raises a thought-provoking question in the long-tail domain. However, the favorable properties of Weight Decay have already been extensively explored in balancing datasets, and its effectiveness can be considered widely recognized.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)