Decoupled Training for Long-Tailed  Classification With Stochastic Representations
Authors: Giung Nam
Sunguk Jang
Juho Lee
Venue: ICLR 2023 poster (2023)
Keywords: long-tailed learning
stochastic weight averaging

Abstract: Decoupling representation learning and classifier learning has been shown to be effective in classification with long-tailed data. There are two main ingredients in constructing a decoupled learning scheme; 1) how to train the feature extractor for representation learning so that it provides generalizable representations and 2) how to re-train the classifier that constructs proper decision boundaries by handling class imbalances in long-tailed data. In this work, we first apply Stochastic Weight Averaging (SWA), an optimization technique for improving the generalization of deep neural networks, to obtain better generalizing feature extractors for long-tailed classification. We then propose a novel classifier re-training algorithm based on stochastic representation obtained from the SWA-Gaussian, a Gaussian perturbed SWA, and a self-distillation strategy that can harness the diverse stochastic representations based on uncertainty estimates to build more robust classifiers. Extensive experiments on CIFAR10/100-LT, ImageNet-LT, and iNaturalist-2018 benchmarks show that our proposed method improves upon previous methods both in terms of prediction accuracy and uncertainty estimation.

Published as a conference paper at ICLR 2023
DECOUPLED TRAINING FOR LONG-TAILED CLASSIFI- CATION WITH STOCHASTIC REPRESENTATIONS
Giung Nam I- Sunguk Jang 2-1 Juho Lee'.2
1 Korea Advanced Institute of Science and Technology (KAIST), PAITRICS
(ggiugg, juholee) @@k iiaaist... st. kr.
ABSTRACT
Decoupling representation learning and classifier learning has been shown to be effective in classification with long-tailed data. There are main ingredients in constructing a decoupled learning scheme; 1) how to train the feature extractor for representation learning so that it provides generalizable representations and 2) how to re-train the classifier that constructs proper decision boundaries by han- dling class imbalances in long- long-tailed data. In this work, we first apply Stochastic Weight Averaging (SWA), an optimization technique for improving the general- ization of deep neural networks, to obtain better generalizing feature extractors for long-tailed classification. We then propose a novel classifier re-training al- gorithm based on stochastic representation obtained from the SWA-Gaussian, a Gaussian perturbed SWA, and a self-distillation strategy that can harness the di- verse stochastic representations based on uncertainty estimates to build more ro- bust classifiers. Extensive experiments iNaturalist-200 2018 benchmarks show that our proposed method improves upon pre- vious methods both in terms of prediction accuracy and uncertainty estimation.
CIFARI 10/100-LI ImageNet-LT, and
INTRODUCTION
While deep neural networks have achieved remarkable performance on various computer vision benchmarks (e.g., image classification (Russakovsky al., 2015) and object detection (Lin et al., 2014)), there still are many challenges when it comes to applying them for real-world applications. One of such challenges that the sal-world classification data are long-tailed the distribution of class frequencies exhibits long tail, and many of the classes have only a few observations belonging to them. As a consequence, the class distribution of such data is extremely imbalanced, degrading the performance of a standard classification model trained with the balanced class assumption due to a paucity of samples from tail classes (Van Horn et al., 2018; Liu et al., 2019). Thus, it is worth exploring novel technique dealing with long-tailed data for eral works have diagnosed the performance bottleneck of long-tailed recognition as distinct from balanced one (e.g.. improper decision boundaries over the representation space (Kang et al., 2020), low-quality representations from the feature extractor (Samuel and Chechik, 2021)), the shared de- sign principle of them giving tail classes a chance to compete with head classes.
deployments. While sev-
Decoupling (Kang et al., 2020) is one of the learning strategies proven to be effective for loig-tailed data, where the representation learning via the feature extractor and classifier learning via the last classification layer are decoupled. Even for classification network failing on long-tailed data, the representations obtained from the penultimate layer can flexible and generalizable, provided that the feature extractor part is expressive enough (Donahue et al., 2014; Zeiler and Fergus, 2014; Gir- shick et al., 4). The main motivation behind the decoupling is that the performance bottleneck of the long-tailed classification is due to the improper decision boundaries set over the representa- tion space. Based on this, Kang et al. (2020) has shown that a simple re-training of the last layer parameters could significantly improve the performance.
Equal contribution
The work was done while the author was graduate student KAIST.

---

Published as a conference paper at ICLR 2023
Self- distillation
Fa(
F Jaswa (a:)
Fo, (a)
Jose (x)
Joswa (x)
Enhannn
Fo
Foswa
Forg(O)
FosWA
x
Figure 1: A schematic diagram depicting the overall composition of the paper. Left: We first apply SWA obtain more generalizing feature extractor (Section 3). Right: We then propose novel self-distillation strategy distilling SWAG into SWA to obtain more robust classifier (Section 4).
The success of decoupling naturally motivates obtaining more informative representations from which the classifier re-training can benefit. To this end, we first investigate Stochastic Weight Aver- aging (SWA; Izmailov et al., 2018), which improves the generalization performance of deep neural networks by seeking flat minima in loss surfaces. Although SWA has been successful for various tasks involving deep neural networks (for instance, supervised learning (Izmailov et al., 2018), semi- supervised learning (Athiwaratkun et al., 2019) and domain generalization (Cha al., 2021)), the best of our knowledge, it has never been explored for long-tailed classification problems. In Sec- tion 3, we empirically show that naive application of SWA for long-tailed classification would fail due to a similar bottleneck issue, but when combined with decoupling, SWA significantly improves the classification performance due its property obtain generalizable representations.
Confirming that SWA can benefit long-tailed classification, we take a step further and propose novel classifier re-training strategy. For this, we first obtain stochastic representations, the output of penultimate layers computed with multiple feature extractor parameters drawn from an approximate posterior. where we construct the approximate parameter with SWA-Gaussian (SWAG; Maddox et al., 2019); SWAG is Bayesian extension of SWA, adding Gaussian noise to the parameter ob- tained from SWA to approximate posterior parameter uncertainty. In Section 3.2, we first show that the diverse stochastic representations obtained from SWAG samples well reflect the uncertainty of inputs. Hinging on this observation, we propose a novel self-distillation algorithm where the stochastic representations are used to construct ensemble of virtual teachers, and the classifier re-training is formulated as a distillation (Hinton et al., 2015) from the virtual teacher.
Fig. 1 depicts the overall composition of this paper diagram. Using CIFAR 10/R100100011 (Cao al., 2019), ImageNet-LT (Liu et al., 2019), and iNaturalist-201 2018 (Van Horn et al.. 2018) benchmarks for long-tailed image classification, we empirically validate that our proposed method improves upon previous approaches both in terms of prediction accuracy and uncertainty estimation.
2 PRELIMINARIES
2.1 DECOUPLED LEARNING FOR LONG-TAILED CLASSIFICATION
Let Fe RR. + R be neural network parameterized by 0 that produces L-dimensional outputs for given D-dimensional inputs. For the K-way classification problem, an output from Fo first transformed into K -d-imensional logits via a linear classification layer parameterized by Φb = (wE RR,, by E IR. { and then turned into classification probability with the softmax function,
exp [w) Fo(x) + by)
p (x) ④ Σイ exp ( Fo(x)) + b,))
for k 1,..., K,
where e = 000 Φ) denotes a set of trainable parameters. Given a training set D consisting of pairs of input a: E TR and corresponding label y E (1,..., , K), O is trained to minimize the cross-entropy
2

---

Published as a conference paper at ICLR 2023
loss over D,
θ -0",0"") "²) arg min B(aa)) [- log p (x; 6)
Throughout the paper, we call Fø with parameters with parameters Φ as classifier. Also, we refer to basic algorithm training the feature extractor and classifier together with Stochastic Gradient Descent (SGD; Robbins and Monro, 1951) as SGD.
feature extractor, and the last linear layer
Previous works (Liu et al., 2019; Van Horn et al., 2018) have shown that the vanilla SGD suffers from the data paucity of tail classes when applied long-tailed classification tasks. Notably, Kang et al. (2020) showed that simple re-tra re-training procedure on the classifier effectively resolves this problem. For instance, using pre-trained 0* from Eq. (2), we can re-train the classifier as
arg min LE (a.y))pPpn log p(v) (æ; (0", +))
(3)
where DcB is the class-balancee balanced training dataset, ie., the probability of sampling a data (x,y) is given by PDe. (<x,)) 1/(K x ny), where ny is the number of training examples for class y- This classifier Re-Training (cRT; Kang et al., 2020) method effectively improves the classification accuracy tail classes without any changes in the feature extractor For
2.2 STOCHASTIC WEIGHT AVERAGING (SWA)
Stochastic Weight Averaging (SWA; Izmailov et al., 2018) is an optimization method to improve generalization performance of deep neural networks. Given a loss function C(O), the conventional SGD steps towards local minima by following the gradient direction, where denotes step size,
θ 0 [Ve(()))--e....
(4)
forming parameter trajectory {O, SWA constructs moving average of parameters for periodically sampled subset this trajectory, starting from S = 0,
(5) where , is usually sampled at the end of every training epochs. This averaging in the weight space implicitly seeks flat minima in the loss surface, and thus enhances generalization. In practice, the averaging phase defined in Eq. (5) starts after the SGD trajectory falls into the basin of the loss function (e.g., after the 75% training epochs), and the learning rate during the averaging phase is set as high values to encourage exploration in the loss surface.
SWA = (nOswa + (n)//nn+))
2.3 SWA-GAUSSIAN FOR A APPROXIMATE BAYESIAN INFERENCE
SWA-Gaussian (SWAG; Maddox et al., 2019) conducts Bayesian inference using a Gaussian ap- proximation to the posterior distribution over the model parameters. With slight abuse of notation writing the element-wise square as @² SWAG maintains the second moment for model parameters in addition the first moment defined in Eq. (5), starting from 6' 0,
O'SAA (n SWA + (n)
(6)
to compute a diagonal covariance matrix ESWAG = diag 'KSWA - SwS)) approximating the sample covariance of parameters captured during SWA. The approximate posterior for the parameters is then constructed as Gaussian, ( = (e; Sww, Eswag). As suggested in Maddox et al. (2019), one also consider higher-rank rank approximation for the covariance, but in this paper, only consider the diagonal approximation.
3 LONG-TAILED CLASSIFICATION WITH STOCHASTIC WEIGHT AVERAGING
After the success supervised image classification tasks (Izmailov et al., 2018), SWA has been further validated for others, including semi-supervised learning (Athiwaratkun et al., 2019) and domain generalization (Cha et al., 2021). this section, we first study whether the success SWA continues in the long-tailed classification (Section 3.1). Then introduce the concept of stochastic representation constructed with SWAG and empirically justify how such stochastic representations can benefit the long-tailed classification (Section 3.2).
3

---

Published as a conference paper at ICLR 2023
Comparison between SGD and SWA before and after applying cRT on benchmarks. Blue denotes a clear improvement of SWA over SGD, while red denotes deterioration.
Table 1:
ImageNet-LT
iNaturalist-2018
Many
Medium
Few
All
Many
Medium
Few
All
Before applying cRT:
SGD
66.84=0.26
40.78=0.24
12.05+0.25
40.21
40.52
67.89+0.18
62.24
66.52=0.05
SWA 67.71
40.74.0.15
11.0
47.08
77.2610:25
28.23
61.87
66.65=0010
After applying cRT:
SGD 22.83-20:25
46.9220.96
26.3320.16
50.25
73.03-a0.57
69.0922.10
66. 14.00.2
68.33.20.04
SWA 63.5410.18
47 17.68
26.851..26
50.9510.12
73.3010.73
69, 69.22.10.19
66.7410.25
68.66110.15
3.1 SWA NEEDS THE CLASSIFIER RETRAINING
Following Liu et al. (2019), we report classification accuracy (ACC) on three splits: Many (a set of classes each with over 100 training examples), Medium (a set of classes each with 20-100 training examples), and Few (a set of classes each with under 20 training examples). Table compares SGD and SWA on large-scale benchmarks for long-tailed image classification, including ImageNet-LT and 1Naturalist-201. The experimental results 'Before applying cRT displayed Table 1 show that SWA does not bring significant performance gain for long-tailed classification tasks, unlike the previous results on other tasks. Compared to the SGD, 1) SWA does not significantly improve performance, and 2) SWA even degrades performance for the 'Few" split, raising question about the efficacy of SWA for the long-tailed classification. We diagnose the performance bottleneck of SWA from the perspective of decoupled learning. Specifically, we adopt cRT (Kang et al., 2020) method to verify whether the SWA inherently hinders the quality of the feature extractor. If SWA shows worse performance than SGD even after the classifier re-training, we can conclude that SWA is not preferable for representation learning for long-tailed classification. The results 'After applying cRT shown in Table demonstrate the classification accuracy for SGD and SWA models after their classifiers are re- re-trained with Eq. (3). The result shows that SWA improves performance for all splits, indicating that SWA actually enhances the quality of the feature extractor, but the classification layer is acting as bottleneck as in SGD, and this can be fixed with cRT.
3.2 STOCHASTIC REPRESENTATION CAPTURE PREDICTIVE UNCERTAINTIES
Now let the feature extractor parameter be constructed from SWA procedure as OSWA and the re- trained classifier parameter as dSwA, that is,
ØSWA = arg min ((x.)))ppe log (x) (OSWA, +)
(7)
While the parameters (OSWA, @SWA) may generalize better than the one obtained from SGD, is still a point estimate without fully capturing the uncertainty of the predictions. To overcome this limitation, we apply SWAG for the feature extractor parameters 0, approximating the posterior of with Gaussian distribution (f(D) = ((0l0swa EsWA). Given (D) predictive distribution for an input x can be approximated as
p(yl ax, D; @swa)) R plu) (a:; (0, (ss)a)a
/
1 M
Σp (a;; (x); (0m, &swA)),
(8)
(). Here, we are computing the model average from multiple predic- tions evaluated with the multiple feature extractor parameters )) to better capture epistemic uncertainty. During that process, we implicitly compute the stochastic representations æ, that is,
where OI....,.. 0m S.i.d
Fm.()) == = (x), for m = I,..,,M. M.
We hypothesize that these stochastic representations reflect the epistemic uncertainty of x, so it is important to consider them for the classifier re- re-training stage. For instance, if is hard example (e.g.. a sample from tail classes) that is likely to be misclassified, the corresponding stochastic representations are expected to produce predictive distribution having high variances.
4

---

Published as a conference paper at ICLR 2023
PCC-0.233
(probabiiity Speec)
PCC-0.385
disperson
0.030
dispersion
0.400
0.020
0.200
0.010
0.000
Q1
Q2
Q3
Q4
Q1
Q2
Q3
Q4
group exaamies
group examples
Figure Box-and-whisken plots for dispersion values in four different groups of examples. We also compute the Pearson Correlation Coefficient (PCC) value to clarify the positive correlation between negative log-likee gellikklihoodd (NLL) and dispersion.
deterministic represenattion
repnesenration)
51.2
2.37
0.13
50.9
2.34
0.11
30.6
2.31
0.09
30.3
2.29
0.07
50.0
2.25
0.05
30
Figure 3: Results for ensemble of stochastic representations over re-trained decision boundaries. Along with classification accuracy (ACC; left), we plot uncertainty estimates, including negative log-likelihood (NLL; center) and expected calibration error (ECE; right). We evaluate models on the test split of ImageNet-LT and colorize standard deviations over four seeds.
Measuring per-instance dispersion. We empirically validate our hypothesis on stochastic repre- sentations by measuring their dispersion over different inputs. The average cosine distance of the elements the set centroid in the L-dimensional representation space can quantify such disper- sion!. To be concrete, we consider the following per- per-instance dispersion for each instance a in the representation space,
M Σ m==1
F(x) T Fo.n (x) |F
Dispersion (z; (0m) m=1)
(10)
M
Fo (x)
where F(x) = ΣM Fe.. (x) MM is the centroid of M stochastic representations {Fe... (x) ) Jm=1
for an imput æ. Furthermore, we also measure the Jensen-Shannon Divergence (JSD) for set of predictions (p(x;; (0m, @swa))) }
Dispersion (æ:; ((0m, &swx. > JSD ((p(:: (0m, (
(11)
to quantify the per-instanee dispersion for each instance æ in the probability space.
Fig. 2 shows the box-an whisker plots for dispersion values over 20,000 validation examples from ImageNer-LT. We split them into four disjoint groups consisting of 5,000 instances each group based the Negative Log-L than the first quartile), Q2 (a set instances having NLLs in the range between the first and second quartiles), Q3 (a set of instances having NLLs in the range between the second and third quartiles), and Q4 (a set of instances having NLLs higher than the third quartile). We confirm that there exists positive correlation between NLL and dispersion (0.233 for Eq. (10) and 0.385 for Eq. (11)), that is, a hard example (i.e., higher NLL) tends to have more dispersed stochastic representations (i.e., higher dispersion). The higher correlation in Eq. (11) compared Eq. (10) motivates to harness the diversity in the probability space instead of the representation space. Empirical results Section 6.1 will show this indeed leads improvements. Moreover, we refer the reader to the first paragraph in Appendix B.5 for further analysis of the per-class dispersion in the context of long-tailed recognition.
(NLL) values: Q1 (a set of instances having NLLs lower
We use the cosine distance instead of Fuclidean distance since it more reasonable choice for the inner- Product-hased representation space trained with Eq. (1).
5

---

Published as a conference paper at ICLR 2023
Ensembling predictions with stochastic representations. We further provide empirical idence showing that the uncertainty captured by the stochastic representations can be harnessed by the classifier for robust classification. Specifically, compute the classification probabilities ensembled over stochastic representations using (8). Note that the classifier parameters ΦswA is re-trained with SSwa with (7), not with the sampled parameters (f. m=1- in principle, there is no guarantee that ΦswA is compatible with the stochastic representations computed from (0m) mmmm= Still, the stochastic representations properly capture the uncertainty of in the feature space, the ensembled classifier would provide well-calibrated predictions.
Fig. demonstrates how classification accuracy and uncertainty estimates, including NLL and Expected Calibration Error (ECE), vary along with the number of stochastic representations for ensembling defined Eq. (8). While the clas- sification accuracy remains at similar level, uncertainty estimates improve as the number stochastic representations increases, indicating that the stochastic representations indeed captures the uncertainty of inputs that are helpful for ro- bust predictions.
00 Fomns (2)
deciston boundary
Fe., (n), where O. g(0) degree of dispersion
We would like to emphasize that the classifier Eq. (8) is a proof-of-concept model to validate our hypothesis, and there still is room for im- provement. First of all, as we mentioned ear- lier, the classifier parameters ΦswA needs to be retrained according to the stochastic representa- tions. Second, even with such re-training, the cur- rent form requires multiple forward pass through the feature extractor (with multiple Om)) incur- ring undesirably heavy inference cost. In Sec- tion 4, we propose a novel re-training algorithm to resolve these issues.
Figure 4: A schematic diagram depicting two-dimensional representation space after the first representation learning stage. Left: The vanilla approach for the second classifier learn- ing stage re-trains the decision boundary be- tween two classes using deterministic represen- tations. Right: Our proposed methods consider stochastic representations having different de- grees of dispersion for each input to build more robust decision boundary.
DECOUPLED LEARNING WITH STOCHASTIC REPRESENTATION
4.1 RE-TRAINING CLASSIFIER WITH STOCHASTIC REPRESENTATIONS
A straightforward way re- re-train the classifier with stochastic representations would be minimizing
Φ- arg min E ((
(v) [m)] B log Po.
arg min (( (C)(( w)), (12)
(3) where Po. (00)(())))))) : (0, k)). 0 .I...., Om i.dd g(0 and
LCE(0,)) v) =
1
(2) log Po m (x))
(13)
M
Fig. 4 depicts two-dimensional representation space during the classifier re-training stage. While learning from deterministic feature extractor (i.e., minimizing Eq. (7)) computes the deterministic representations, our proposed method minimizing Eq. (12) builds more robust decision boundaries since it accounts for predictive uncertainties estimated from the stochastic representations.
4.2 SELF-DISTILLATION WITH STOCHASTIC REPRESENTATION
While the re-training with (12) helps build a robust classifier, there still room for improvement, Af- ter all, what we ultimately interested in is the uncertainty in set of predictions {Po... (z) (t) M not the stochastic representations {Fo... (x) {x))M M themselves. The objective (12) learns the classifier parameters with the mean loss (13), it does not consider the diversities of the predictions made

---

Published as a conference paper at ICLR 2023
by different stochastic representations. Moreover, pointed out earlier, in the current form, we need M forward passes through the feature extractor with (0m)m=1 to make a prediction. To resolve these issues, present self-distillatien algorithm that can enhance the robustness of the classifier by transferring diversities in the multiple predictions made with stochastic representations reducing the inference cost to be the same single model.
Setting up teachers and a student. We first set a set of teachers constructed from the set of stochastic representations. Formally, the mh teacher Tm produces prediction probability for an input x as PTm. (x) == Pem.ol (2t) where Om ~ (( For a student, we first fix the feature extractor parameter as OSWA, which is a natural choice considering that the mean of the distribution ((O(D) is OSWA. The goal is to learn the classifier parameters Φ such that the student prediction Poena db (z)) can maximally absorb the diversities in the teacher predictions (Pr... (z:))
Distillation objective. Following Ryabinin et al. (2021), instead of directly distilling from the mean of ensemble predictions Pr(x) := Σm PT... (T) (z)/M as the original knowledge distilla- tion (Hinton et al., 2015), we distill from the distribution of the ensembled predictions. Specifically, we assume that the teacher prediction probabilities (PT... (x))) are Dirichlet distributed with pa- rameter ((x)), that is.
PTi .T (a) iid Dir(B(a))-
(14)
The parameter ((x) can be estimated via the approximate maximum likelihood procedure whose solution is given in closed-form (Minka, 2000);
(K-1)/2
B (x)) = PT (xr
(x)
(15)
5==1 PJ (x) log pf (x) A EM
((())
M log log PT (z))
for k I,... K. Having computed 3(x), we assume that the output probabilities from student model (our original model to be re-trained for classifier parameters) are also Dirichlet distributed with parameter (xx). We constraint all components in parameters be greater than 1 (i.e., B <- B + 1 and a + a + 1) and minimize the reverse KL divergence Dkr. [ (Dir(Drr)))Diiiii [[)r(())) for stable training (Ryabinin et al., 2021). Expanding the KL divergence, we obtain the distillation loss as
K
(KD v) -E Dir((a
Σ k=1
(1) PT (x) (k)
ΣØ= 30 DKL (Dr((a))) (16)
where we compute ax (xx) E R for Φ = (wxx by K
k=1
(k) (x) == exp [w] Josan (x) + bb) for h: 1,.... K.
(17)
After learning ax(r), we can take the mean of the student Dirichlet distribution as a representative
classification probability for æ. By the property of the softmax, this actually coincides with the
Pa (z)). That is. (p-Di((((m)) [p [ (R)
(x) Σ a ai) (x))(())))))))))))())) =P(t) Po (x))
(k)
The final version of our algorithm combines the two loss terms LCe and LKD for the classifier training (see Appendix A.4 for more details). While the first term builds decision boundaries over the stochastic representations, the second term further adjusts decision boundaries by distilling diverse predictions from the stochastic representations. We name this procedure as a self-distillation in the sense that the model distills probabilistic outputs within the network itself (Zhang et al., 2019). We empirically validate the effectiveness of the oeff-disiillatin in Section 6.
5 RELATED WORKS
Decoupled learning. Recent works have shown that the performance bottleneck of deep neu- ral classifiers on long-tailed datasets is improper decision boundaries (Kang et al., 2020; Zhang et al., 2021). The vanilla training with instance-balanced sampling gives generalizable representa- tions (Kang et al., 2020), and thus simple adjusting strategy for the classifier can alleviate such bottleneck. For instance, re- re-training the classifier from scratch or normalizing the classifier weights with class-balanced sampling (Kang et al., 2020), adjusting the classifier biases with the empirical class frequencies on the training dataset (Menon et al., 2021), and training additional module which performs input-dependent adjustment of the original classifier (Zhang et al., 2021).

---

Published as a conference paper at ICLR 2023
Table Ablation studies of proposed methods on ImageNet-LT: classification accuracy (ACC), negative log-likelihooe (NLL), and expected calibration error (ECE). These results are with Class Balanced Sampling (CBS); refer Appendix B. the results for the other balancing strategies.
Method
Note
ACC (1)
NLL (L)
ECE (U)
SGID w/ classifier re- training
Section 2.1
50.25=0-18
2.36422.006
0.1 1020.000
introducing SWA the representation learning
Section
50.95=0.12
2.353:0.012
0.120
(b) classifier re-training w/ stochastic representation
Section 4.1
51.33
2.340=0.012
0.125
(c) classifier re-training w/ sell-distillatio
Section 4.2
51.6620-15
2.203200000
0.07420.002
Knowledge distillation. The seminal work of Hinton et al. (2015) has shown that distilling knowl- edge in deep neural networks is an effective way to obtain better generalizing models. One of the common practices in knowledge distillation is employing ensembles et al., 2020; Ryabinin et al., 2021) based on the superior performance of the ensemble of deep neu- ral networks (Lakshminarayanan al., 2017; Ovadia et al., 2019), and several works already applied this to long-tailed recognition (Iscen et al., 2021; Wang et al., 2021). Nevertheless, we would like to clarify that there a clear difference from them in that our method does not require a costly teacher model (i.e., we do not require an independently trained teacher model).
teacher model (Malinin
6 EXPERIMENTS
We present extensive experimental results on long-tailed image classification benchmarks for the family of residual networks (He et al., 2016), i.e., ResNet-32 on CIFARI0/100-LT (Cao et al., 2019) and ResNet-50 on ImageNet-LT (Liu et al., 2019) and iNaturalist-2018 (Van Horn et al., 2018). See Appendix C for more details. While some previous literature only reports classification accuracy (ACC), we also provide Negative Log-Likelihood (NLL) and Expected Calibration Error (ECE) that further evaluate the calibration of classification models. See Appendix A.3 for definitions of each metric. Unless specified, we report numbers in Avg. Esid. over four random seeds.
6.1 ABLATION STUDIES OF PROPOSED METHODS
In this section, we empirically verify that our proposed method progressively improves upon the previous baseline. More precisely, we test the following step-by-step, (a) introducing SWA for the representation learning from Section 3, (b) introducing stochastic representation for the classifier re-training from Section 4.1, and (c) introducing self-distillation strategy from Section 4.2.
Here, we consider the following balaneing strategies during the classifier re-training: 1) Class- Balanced Sampling (CBS; Kang al., 2020), 2) Generalized Re- Re-eeight (GRW; Zhang et al., 2021), and 3) Logit Adjustment (LA; Menon et al., 2021). Such strategies to overcome the difficulty of imbalanced data distribution essential to re-train proper decision boundaries in the decoupled learning scheme. Refer to Appendix A.1 1 for more details on balancing strategies.
Table 2 shows the evaluation results for each step ImageNet-LT when we CBS. To sum- marize, (a) the results displayed in the first and second rows show that SWA improves the SGD) baseline with a decoupling scheme,
discussed before in Section 3.1. (b) Moreover, the results displayed in the second and third rows show that using stochastic representations to re-train the clas- sifier improves the classification accuracy. It indicates that the stochastic representations contribute to building more robust decision boundaries, as depicted in Fig. 4. However, our experimental re- sults up to this point show that the improvements in uncertainty estimates are not significant. (c) Finally, our proposed approach in Section 4.2 significantly outperforms previous baselines for every metric we measured. Notably, the improvement consistently appears for all balancing strategies we considered (see Appendix B.1 for the full results). In particular, definite improvement in uncertainty estimates confirms the effectiveness of the proposed dit--isiillation method.
6.2 RESULTS ON IMAGE CLASSIFICATION TASKS
We compare approach to the existing methods for long-tailed learning; cRT (Kang al., 2020), LWS (Kang et al., 2020), LA (Menon et al., 2021), and DisAlign (Zhang et al., 2021). See Ap- pendices A. and A.2 for more details on existing methods. Since the representation learning using
8

---

Published as a conference paper at ICLR 2023
Table 3: Results on ImageNet-LT and iNaturalist-2018. classification accuracy (ACC), negative log-
likelihood (NLL), and expected calibration error (ECE). Full results are available in Appendix B.1.
ImageNet.
iNaturalist 2018
Method
ACC (?)
NLL (
(1)
ACC (1)
NLL (L)
(.)
SWA
47.08so.12
2.631saom
a.187
66.65sm.10
1.568
0.071
CRT (Kang et al., 2020)
50.95=0.12
2.353
0.120=0.002
08.66=0.16
1.546==0.002
0.061
LWS (Kang al,, 2020)
51.00
2.189
0.077=0.00
70.53=0.00
1.370 no0
0.049 001
(Menon 2021)
51.62.00s
2.2061
6.077..0.00
69.631...0
1.466.0.005
0.038.10000
DisAlign (Zhang et al., 2021)
52.18 bo.11
2.673
0.215
70.81
1.410
0.0761.0000
SReer (ours)
52.12.0.00
70.79:0.17
0.036+0.002
Table Further comparisons in classification accuracy (ACC) with state-of-the-art methods on ImageNet-LT. Results for the basclines came from the corresponding paper. TTis actually requires training cost of 600 epochs since rwSAM doubles forward and backward passes for each iteration.
Method
Epoch
Network
Training details
ACC(
KCL (Kang ot al., 2021)
200
ResNer-50
51.5
SRope (ours)
ResNer-50
MiSLAS (Zhong al,, 2021)
180
ResNer-50
mixup (Zhang al. 2018) mixup (Zhang al.,
52.7
SWA SRepe (ours)
ResNet-50
LADE (Hong et al,, 2021)
180
ResNeXt-50 (Xic al. 2017)
53.0
SWA SRepe (ours)
180
ResNeXi-50 (Xie 2017)
54.6
Liu al. (2021)
300
ResNet-50
MoCo v2 (He et al., 2020)
55.0
SWA SRepr (ours)
300
ResNet-50
mixup (Zhang al., 2018)
55,1
Liu et al. (2021)
300
ResNer-50
MoCo v2 (He et al., 2020) + rwSAM (Foret al., 2021)
55.5
SWA SRepe (ours)
400
ResNet-50
mixup (Zhang al,,
55.7
SWA, we introduced in Scction 3, is also compatible with previous classifier re-training methods, hereby report the results built upon this for fair comparison.
Table 3 presents the evaluation results on ImageNet-LT and 2000 SWA + SRepr denotes the final version of our proposed method, previously discussed in Section 6.1, i.e,, self-distillation using stochastic representation and balanced by logit adjustment. demonstrates that the proposed SRepr outperforms baselines, even if existing methods strengthen with SWA. While DisAlign, im- proved by our proposed SWA, achieves competitive accuracy, SRepr still shows better uncertainty estimates. Refer Appendix B. for a full table, including classification accuracy for each split (i.c., Many, Medium, and Few) and baseline results without applying SWA. Appendix B.2 also provides the results on CIFARIO// 10/R100000011 which show the same tendency. Besides, Appendix B.5 provides further analysis of ours, which validates the effectiveness of the proposed method.
Further comparisons with state-of-the-art methods. One can claim the performance gap be- tween ours and the state-of-ther the-thh-rt methods. This issue appears due to the shorter training epochs (i.e., 100 training epochs) and the vanilla data augmentation strategy (i.e., random resized cropping and horizontal flipping) in our experimental setups. Table 4 further clarifies that our proposed ap- proach outperforms the state-of-the-arr methods (Kang et al., 2021; Zhong et al., 2021; Hong al., 2021; Liu et al., 2021) when the training costs get line (see Appendix B.3 for more details). More- over, Appendix B.4 further verifies the compatibility between ours and the existing state-of-the-ar art methods ( (Wang et al., 2021; Park et al., 2022; Zhang et al., 2022; Zhu al., 2022), which clarifies ours can be combined with existing algorithms taking different approaches.
CONCLUSION
In this paper, we proposed a simple yet effective classifier re- traiiiin strategy for long-tailed learning in decoupled learning scheme. We first showed that successful representation learning is achievable by SWA without any complex training methods. To the best of our knowledge, this is the first attempt to introduce SWA into long-tailed learning. While just combining existing classifier re-training methods with the representation learned by SWA shows better performance than with the vanilla SGD, we further proposed a novel self-distillation strategy that significantly improves uncertainty estimates of the final classifier.

---

Published as a conference paper at ICLR 2023
ETHIC AND REPRODUCIBILITY STATEMENTS
For societal impacts, there can be potential negative impacts associated with face recognition (e.g., privacy threats) since face data often exhibit long-tailed distribution over entities (Zhang et al., 2017). For reproducibility, Appendix C contains all the experimental setup including datasets and hyperparameters.
ACKNOWLEDGEMENTS
This work was partly supported by Institute Information Communications Technology Planning & Evaluation (ITP) grant funded by the Korea government (MSIT) (No. 2019-0-00075. Artificial Intelligence Graduate School Program (KAIST)), Samsung Electronies Co., Ltd (I02012144 20011211-81111 14-08176- 01), and the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (No. 0022R1A577089008
REFERENCES
Alshammari, Y.-X. Wang, D. Ramanan, and Kong. Long-tailed recognition via weight bal- ancing. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2022. 29
B. Athiwaratkun, M. Finzi, Izmailov, and A. G. Wilson. There are many consistent explanations of unlabeled data: Why you should average. In International Conference on Learning Represen- tations (ICLR), 2019. 2,3
J. Bradbury, R. Frostig, P. Hawkins, M. J. Johnson, C. Leary, D. Maclaurin, G. Necula, A. Paszke, VanderPlas, Wanderman-Milne, and Q. Zhang. JAX: composable transformations of Python+NumPy programs, 2018. URL http: //ggttut..///////////////...thh. com/ http://oolll/aaaaao 20
K. Cao, C. Wei, A. Gaidon, N. Arechiga, and T. Ma. Learning imbalanced datasets with label- distribution-aware margin loss. In Advances in Neural Information Processing Systems 32 (NeurIPS 2019), 2019. 8,
Cha, Chun, K. Lee, H.-.. Cho, S. Park, Y. Lee, and Park. Swad: Domain generalization by seeking flat minima. In Advances in Neural Information Processing Systems 34 (NeurIPS 2021), 2021. 2,3
Cui, Z. Zhong, S. Liu, B. Yu, and Jia. Parametric contrastive learning. In Proceedings of the IEEE/CVF International Conference Computer Vision, 2021. 17
J. Donahue, Y. Jia, O. Vinyals, J. Hoffman, N. Zhang, E. Tzeng, and T. Darrell. Decaf: A deep convolutional activation feature for generic visual recognition. In Proceedings of The 31st Inter- national Conference on Machine Learning (ICML 2014), 2014.
Foret, A. Kleiner, H. Mobahi, and Neyshabur. Sharpness-awee aaaree minimization for efficiently im- proving generalization. In International Conference on Learning Representations (ICLR), 2021.
R. Girshick, J. Donahue, T. Darrell, and J. Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2014.
C. Guo, G. Pleiss, Y. Sun, and K. Q. Weinberger. On calibration of modern neural networks. In International conference on machine learning, pages 1321-1330. PMLR, 2017. 19
K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2016. 8, 21
K. He, H. Fan, Y. Wu. S. Xie, and R. Girshick. Momentum contrast for unsupervised visual rep- resentation learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020. 9
10

---

Published as a conference paper at ICLR 2023
J. Heek, A. Levskaya, A. Oliver, M. Ritter, B. Rondepierre, A. Steiner, and M. Zee. Flax: A neural network library and ecosystem for JAX, 2020. URL httptt////gthhti com/ googlo/ flax. 20
M. Hessel, D. Budden, F. Viola, M. Rosca, E. Sezener, and T. Hennigan. Optax: composable gradi- ent transformation and optimisation, in jax!, 2020. URL http: / http:///gtthbee //ggttthbbb.... com/ deepmind/
optax. 20
G. E. Hinton, O. Vinyals, and Dean. Distilling the knowledge in a neural network. In Deep Learning and Representation Learning Workshop, NIPS 2014, 2015. 2, 7,
Y. Hong, S. Han, K. Choi, S. Sco, B. Kim, and B. Chang. Disentangling label distribution for long- tailed visual recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021. 9, 17
A. Iscen, A. Araujo, B. Gong, and C. Schmid. Class-balanced distillation for long-tailed visual recognition. In The British Machine Vision Conference (BMVC), 2021. 8
P. Izmailov, D. Podoprikhin, T. Garipov, D. Vetrov, and A. G. Wilson. Averaging weights leads to wider optima and better generalization. In 34th Conference on Uncertainty in Artificial Intelli- gence 2018, UAI 2018, 2018. 2,3,22
B. Kang, S. Xie. M. Rohrbach, Z. Yan, A. Gordo, J. Feng, and Y. Kalantidis. Decoupling rep- resentation and classifier for long-tailed recognition. In International Conference on Learning Representations (ICLR), 2020. 3, 4, 7, 8, 9, 14, 16, 17, 18, 19
B. Kang, Y. Li, S. Xie, Z. Yuan, and J. Feng. Exploring balanced feature spaces for representation learning. In International Conference on Learning Representations (ICLR), 2021. 9
A. Krogh and J. Hertz. A simple weight decay can improve generalization. In Advances in Neural Information Processing Systems (NIPS 1991), 1991. 21
B. Lakshminarayanan, A. Pritzel, and Blundell. Simple and scalable predictive uncertainty esti- mation using deep ensembles. In Advances in Neural Information Processing Systems 30 (NIPS 2017), 2017. 8, 18
T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollar, and C. L. Zitnick. Microsoft coco: Common objects in context. In European conference computer vision, 2014.
H. Liu, J. Z. HaoChen, A. Gaidon, and T. Ma. Self-supervised learning is more robust to dataset imbalance. In International Conference on Learning Representations (ICLR), 2021.
Z. Liu, Z. Miao, X. Zhan, J. Wang. B. Gong, and S. Yu. Large-scale long-tailed recognition in an open world. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019. 1, 3, 4, 8, 16,
W. J. Maddox, Izmailov, T. Garipov, D. P. Vetrov, and A. G. Wilson. A simple baseline for bayesian uncertainty in deep learning. In Advances in Neural Information Processing Systems 32 (NeurIPS 2019), 2019. 2,3
A. Malinin, B. Mlodozeniec, and M. F. Gales. Ensemble distribution distillation. In International Conference on Learning Representations (ICLR), 2020. 8
A. K. Menon, S. Jayasumana, A. Rawat, H. Jain, A. Veit, and S. Kumar. Long-tail learning via logit adjustment. In International Conference on Learning Representatinss (ICLR), 2021. 7, 8,
9, 14, 16, 17, 18
Minka. Estimating dirichlet distribution, 2000.
Y. Ovadia, E. Fertig, J. Ren, Z. Nado, D. Sculley. S. Nowozin, J. V. Dillon, B. Lakshminarayanan, and J. Snoek. Can you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift. In Advances in Neural Information Processing Systems 32 (NeurIPS 2019), 2019. 8, 18
11

---

Published as a conference paper at ICLR 2023
S. Park, Y. Hong, B. Heo, S. Yun, and J. Y. Choi. The majority can help the minority: Context-rich minority oversampling for long-tailed classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattemn Recognition, 2022. 9, 18
Ren, C. Yu, X. Ma, Zhao, S. Yi, et al. Balanced meta-softmax for long-tailed visual recognition. In Advances in Neural Information Processing Systems 33 (NeurIPS 2020), 2020. 19
H. Robbins and S. Monro. A stochastic approximation method. The annals of mathematical statis-
tics, pages 400-407, 1951. 3
O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bemsstein, A. C. Berg, and L.. Fei-Fei. Imagenet large scale visual recognition challenge. International Journal of Computer Vision (ICV),, 2015. 1,
M. Ryabinin, A. Malinin, and M. Gales. Scaling ensemble distribution distillation to many classes with proxy targets. In Advances in Neural Information Processing Systems 34 (NeurIPS 2021), 2021. 7,8
D. Samuel and G. Chechik. Distributional robustness loss for long-tal tail learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021. 1
G. Van Horn, O. Mac Aodha, Y. Song, Y. Cui, C. Sun, A. Shepard, H. Adam, P. Perona, and Be- longie. The inaturalist species classification and detection dataset. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition (CVPR), 2018. 1, 2, 3, 8, 21
X. Wang, L. Lian, Z. Miao, Z. Liu, and S. Yu. Long-tailed recognition by routing diverse distribution-awarr experts. In International Conference on Learning Representations (ICLR), 2021. 8, 9, 17, 18
A. G. Wilson and P. Izmailov. Bayesian deep learning and a probabilistic perspective of generaliza- tion. In Advances in Neural Information Processing Systems 33 (NeurIPS 2020), 2020. 18
L. Xiang, G. Ding, and Han. Learning from multiple experts: Self-paced knowledge distillation for long-tailed classification. In European Conference on Computer Vision, 2020. 17
S. Xie, R. Girshick, P. Dollár, Z. Tu, and K. He. Aggregated residual transformations for deep neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017. 9, 17
M. D. Zeiler and R. Fergus. Visualizing and understanding convolutional networks. European conference on computer vision, 2014. 1
H. Zhang, M. Cisse, Y. N. Dauphin, and Lopez-Paz. mixup: Beyond empirical risk minimization. In International Conference on Learning Representations (ICLR), 2018. 9, 17
L. Zhang. J. Song, A. Gao. J. Chen, C. Bao, and K. Ma. Be your own teacher: Improve the per- formance of convolutional neural networks via self distillation. In Proceedings of the IEFF/CVF International Conference Computer Vision, 2019.
Zhang, Z. Li, S. Yan, X. He, and Sun. Distribution alignment: A unified framework for long- tail visual recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021. 7, 8, 9, 14, 15, 16, 17,
X. Zhang, Z. Fang, Y. Wen, Z. Li, and Y. Qiao. Range loss for deep face recognition with long-tailed training data. In Proceedings of the IEEE International Conference on Computer Vision, 2017. 10
Y. Zhang, B. Hooi, L. Hong, and J. Feng. Self-supervised aggregation of diverse experts for test- agnostic long-taiil tailed recognition. In Advances in Neural Information Processing Systems 35 (NeurIPS 2022), 2022. 9, 18
Z. Zhong, J. Cui. S. Liu, and J. Jia. Improving calibration for long tailed recognition. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2021. 9, 17
12

---

Published as a conference paper at ICLR 2023
B. Zhou, Q. Cui, X.-S. Wei, and Z.-M. Chen. Bbn: haateral-branch network with cumulative learning for long- toiglt visual recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020. 17
J. Zhu, Z. Wang, Chen, Y.-P. Chen, and Y.-G. Jiang. Balanced contrastive learning for long- tailed visual recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022. 9, 17, 18
13

---

Published as a conference paper at ICLR 2023
A SUPPLEMENTARY MATERIALS
A.1 BALANCING STRATEGIES FOR RE-TRAINING CLASSIFIER
In Section 6.1, we considered the following balancing strategies for the classifier re-training stage. Such strategies for re-balancing the training data having long-tailed distribution over classes is es- sential to construct proper decision boundaries over the representation space.
Blass-Balances Sampling (CBS) is the most straightforward way to re-balance long-tailed data distribution by sampling. The probability of sampling training example from class k is 1/K, where K is the number of training classes. More precisely, the softmax cross- entropy loss over the Mlss-balancee training dataset DeR is
exp (w, Fo(ar) + by)
(( ((ppp)ppPen
log Σラ= exp (w) Fo( + bb)
(18)
where the probability of sampling data (æ. y) is given by PDcs ((r, 1/(K ny) and Ny denotes the number of training examples for class y.
Generalized Re- Reiieinhn (GRW) performs loss re-weighting using the empirical class frequencies {T | the training dataset (Zhang et al., 2021). More precisely, the re- weighted version of the softmax cross- cross-entropy loss over the training dataset D
(1/Tg)² log (1/Tg) Σラ=1 exp [w] Fo(xx) + b,)
exp (w. Fo(x)) + by)
E E(Dyy)D
(19)
where p 0 is hyper-parameter that controls the scale of per-class weighting coeff- cients, e.g., reduces to the instance-balanced re-weighting with = 0.0, and to the class-balanced re-weighting with p 1.0. We use p 1.0 throughout the experiments.
Logit Adjustment (LA) is simple but efficient way to minimize the balanced error (i.e., average of per-class error rates) using the empirical class frequencies { {== on the train- ing dataset (Menon et al., 2021). More precisely, the logit adjusted version of the softmax cross-entropy loss over the training dataset ' is
exp [ J, Fo Folx) + by plog Ty)
log
(20)
Σ/= exp [w) Fo( + by + log TTy)
where p 2 is a syper-paramettr that controls the scale of offset to each of the logits. We use p = 1.0 throughout the experiments.
A.2 CLASSIFIER RE-TRAINIIG METHODS
Assume that we have pre-trained parameters e* (0", (0",""" xb) after the first representation learning stage. From this, the classifier re-training methods aim to find a new classifier parameters ø (wn by k= while the feature extractor parameters 0" frozen.
K
Classifier Re-Training (cRT) the most straightforward way re-train the classifier from scratch (Kang et al., 2020). Specifically, it first randomly re-initializes the classifier parameters & and re-trains them on the class-balanced training dataset DCB.
arg min E (2.v)~/P0E log p (v) (x: (0", +))]
(0*,)))]
(21)
+
Learnable Weight Scaling (LWS) only re-trains the scale of pre-trained weights (w while the direction is kept (Kang et al., 2020). Specifically, it introduces trainable parameter T E IR which controls the intensity of the weight normalization,
a (T) = (wt/ |" b bi)K
(22)
and finds x²= = = on the cllanebalanced training dataset DCB, where
7" arg min E (x
[ log (1) (0", ((r)))]
(23)
14

---

Published as a conference paper at ICLR 2023
Distribution Alignment (DisAlign) trains additional modules while the original classifier parameters Ø is kept (Zhang et al., 2021). Specifically, it introduces trainable parameters
{ag E R, 33 E fk=1 (y R . IR] which performs distribution calibration,
za (a) i Fo. (a) + lig-
(24)
a [x) k Si T zx (x)) + 6),
(25)
&x (x) レ (x) (ag= (a: + 3x) + (1 (1-o(z)) =x)), for k 1,.., K, (26)
where 24 and 2g respectively denote the original logits and the calibrated logits. While both O" and Ø are fixed, it finds the optimal value of {ax, 3,)k (7, = {aj 9h 1 3"* on the training dataset D with Generalized Re-Weighting (GRW).
A.3 EVALUATION METRICS
The problem we addressed this paper is the K K-way classification problem. Let P x + p be a model that outputs a categorical probability p [0, 1] for a given input æ. Following metrics are reported for all of the methods using our implementation.
Accuracy (ACC; higher is better):
ACC(P, D) ((a))E arg max (()) (x))
(27)
where inner [-] denotes the Iverson bracket.
Negative likeliiel (NLL; lower is better):
(P, D) = E(a.))DD [ log p (x))
(28)
which is equivalent to the softmax cross-entropy loss used in training.
Expected calibration error (ECE; lower better):
8n (1Bml/121),
(29)
n=1
where {B1, By) } is partition of D,
B, (x,v) v) E D max p (x)
12 -
n
for I,..,,N
(30)
N
N
and o, denotes calibration error for the rith bin B,,.
8n ACC(P, B.) (E,) max (kk) (x)
for = I,......
(31)
We used N = 15 bins in this paper.
A.4 DETAILED IMPLEMENTATION OF THE PROPOSED METHOD
Loss function During the optimization, simply fix M 10 and use equal weights for two terms in the final objective, i.e., the actual implementation is
0.5 0.5LkD
(32)
cross-entropy
self-distillation
Also, we blocked the gradient flow through the target Dirichlet distribution the self-distillation term, i.e. jax. lax. stop-gradient (B) in JAX library. Our training diverges when using the typical softmax outputs due to the sharp target Dirichlet distribution. Thus, we use the following temperature-scalee scaled softmax outputs to stabilize the optimization on the self-distillation term ÊKD.
exp (w (k) (x) + bt)/TTD)
(k) Pp (x:
for k = I,..., K.
(33)
Σ exp (w) F (x) + b,)/TKD)
Table 5 shows results for the final version of our algorithm (i.e., SRepr in Section 6.2) on the val- idation split of ImageNet-LT swept over TKD E {1,2,5, 5, 10, 20). Training losses were unstable for 21,5,5) and thus we fix TKD 20 throughout all experiments.
15

---

Published as a conference paper at ICLR 2023
Table Validation results of SRepr with varying temperatures (i.e., TKD in Appendix A.4) on ImageNet-LT. "N/A" denotes the training diverges.
ACC (1)
ImageNet-L.T (Val)
Many
Medium
Few
All
NLL (L)
ECE (L)
TKD = 1.0
N/A
N/A
N/A
N/A
N/A
N/A
TRD = 2.0
N/A
N/A
N/A
N/A
N/A
N/A
RD = 5.0
66.86=0.10 10
41 11.58=0.09
7.48=0.09
46.67=0.05
4.238=0.009
0.294
TDD 10.0
64.21..0.2
50.51
32.70
53.36 09
2.032.10.001
0.034110000
TDD 20.0
64,20 18
50.52=0.16 14
32.89-0.40
53.40=00188
2.059 049
0.030.10.005
Table 6: Ablation study of proposed methods with various balancing strategies on ImageNet-LT: classification accuracy (ACC), negative log-likelihood (NIL),, and expected calibration error (ECE).
Proposed Methods
Repr. learning (Section 3)
Classißier learning stochastic repr. (Section 4.1)
Classifier learning with self-distillation (Section 4.2)
Balancing Strategy
ACC
NLL (3)
ECE
46.91
0.158 2.546=0.0m
Without re-training classifier
47.0810.19
2.631 20.00
0,18720002
50,25
2..04
0,1 10.100000
CBS (Kang al., 2020)
50.95
51.3310017
2.340.0.012
2.353
0, 1o0000
002
51,66:20.33
2.203
0.074
50.77a9.13
0.02610000 2.243
11.33
0.041 2.220
GRW (Zhang et al., 2021)
51.73g9.11
2.20620000s
0,056.0000
/
52.08
2.133
0.019=0.00
50.97
2.231 200004
51.6210.0
2.206110000
0.077
LA (Menon 2021)
51, 14.844
2.208
0.090
4
52.12.0000
2.130.0.000
0.037 10.001
Pseudo-code for the proposed method Algorithm I summarizes the proposed method in pseudo- code. Note that training SWA-Gaussian with a diagonal covariance has virtually no additional cost conventional training with SGD. It only requires extra space for storing the first and second moments of the backbone parameters (i.e., lines 1-10). Instead, an additional training cost of our approach compared with the existing methods (Kang al., 2020; Zhang al., 2021) come from multiple forward passes of the backbone network during the classifier re-training stage (i.e., lines 11-17). Please refer to Appendix B.5 for further investigation regarding this issue.
B ADDITIONAL EXPERIMENTS
B. FULL TABLE OF RESULTS
Ablation studies of proposed methods (Section 6.1). Table 6 is an extended version of Table 2. It provides the results when apply the following balancing strategies; CBS, GRW, and LA. The arguments we discussed Section 6.1 consistently hold for all balancing strategies we considered.
Results on image classification tasks (Section 6.2). Table 7 is an extended version of Table 3. Here, we also provide detailed classification accuracy on three splits introduced in (Liu al., 2019): Many (a set of classes each with over 100 training examples), Medium (a set of classes each with 20- 100 training examples), and Few (a set of classes each with under 20 training examples).
B.2 ADDITIONAL RESULTS ON CIFAR10/100-LT
We also provide the experimental results on CIFARIO-LT and CIFARIOO-LT. Table shows that our approach outperforms the baselines in terms of every metric measured. It clearly demonstrates that the proposed approach consistently benefits from training robust decision boundaries regardless of the scale of datasets.
16

---

Published as a conference paper at ICLR 2023
Table 7: Full restuls on ImageNet-LT and INaturalist-2018. classification accuracy (ACC), negative log-likelihook (NLL), and expected calibration error (ECE).
ACC (1)
ImageNet-LT
Many
Medium
Few
NLL (J)
ECE (L)
SGD
66.84.2..8 26
40.78
12.05
46.91
2.54620.000
0.158
cRT (Kang ct al., 2020)
62.83=0022
46.92=0 26
16.33++0.1
50.25+0.18
2.364=00.00
0.110+0.001
LWS (Kang et al., 2020)
63.23..0.20
17.57.10.2
20.78.0.29
50,91
0.05410.001 2.197
LA (Menon et al,, 2021)
60.79:00:20
48.11
33.20020.44
50,97:0.13
0.063:+0:001 2.004
DisAlign (Zhang al., 2021)
61.63=0.39
48.68=0.11
32.71+0.45
51.49=0.15
2.596=0.012
0.202++0302
SWA (ours)
60.71
40.74=2.15
+1.01
47.08=0.12
2.631 00.00
0.187=+0.022
+ cRT' (Kang et al., 2020)
63.54:10.18
47, 67.68811.1 16
26.85
50.95
0.120:10.002 2.35
LWS (Kang et al., 2020)
48.53 63.51
07
28.6610.45
51.60
0.077.10.002 2.189
LA (Menon et al., 2021)
48.70=0.00 61.60=0.07
33.68+0.34
51.62+0.05
2.206=00000
0.077+00302
DisAlign (Zhang al., 2021)
62.43.0..0
49,48 10
32.6510.43
52.18
0.215 2..031
SRepr (ours)
02.52=0.20
49 49.44 18
12.14
52.12.00.6.
2,130
0.037=+0.001
ACC (1)
Naturalist-2018
Many
Medium
Few
All
NLL (1)
ECE (L)
SGD
76.31_0.52
67.8910.15
62.24.0.11
66.52.10.05
1.568
0.048
cRT (Kang et al., 2020)
73.03=0.57
69.09220.10
14.0220
68.33=0.04
1.537=00000
0.037:0.002
LWS (Kang et al., 2020)
72.35
70.11
70.19 69.73+0.40
1.386=00.00
0.030+0.001
LA (Menon et al., 2021)
69. 19.90.10.50
69.34.10.14
69.66610119
69 19.49
1.477
0.01510.002
DisAlign (Zhang ct al., 2021)
71.68=0.31
70.73=2025
+0.30
70.35+0.21
1,428=0.006
0.064=00001
SWA (ours)
27.26=0.22
68 68.23==0.25
61.87#0.18
66.65#0.10
1.568
0,071
cRT (Kang et al., 2020)
73.30.0.77
69.22.10.1 10.0
66. 66.77
68. 18..6610.15
1.5462..002
0,061 110.002
LWS (Kang et al., 2020)
72.82=00.45
70.43=0025
70.06=0.11
70.53+00.00
1.370=0.002
0.049+0.001
LA (Menon et al., 2021)
69.70=0..7
69.47:10111
69.82.10.68
69.63
1. 1.466=0.000
0.038
DisAlign (Zhang et al., 2021)
72.34:20.57
71.27:10.1 10
18
70.81 100
1.410
0.076:20.000
SRepr (ours)
70.70=0.31
70.83 20
70.76+0.32
70.79+0.17
1.353=0.00
0.036++0.002
B.3 FURTHER COMPARISONS WITH STATE-OP-THE-AN METHODS
Increasing the number of training epochs. Throughout the main text, we trained all the com- peting methods for 100 training epochs on ImageNet-LT, which sufficient validate the efficacy of our approach. However, state-of-the-ar performances reported in other papers are typically from long training epochs than we set in our experiments. We thus provided the results when the number of training epochs gets in line in Table 4. It clearly demonstrates that ours outperforms baselines by a wide margin. Besides, we also tested our method for ResNeXt-50 architecture (Xie et al., 2017) for a fair comparison with LADE (Hong et al., 2021).
Applying mixup augmentation. Zhong et al. (2021) employed mixup augmentation (Zhang et al., 2018) in decoupled training, which is actually one of the main ingredients to achieving that level of performance they reported. However, the mixup augmentation not exclusive to their method but applicable to generic approaches, including ours. In Table 4, our approach enhanced by the mixup augmentation indeed outperforms the previous baseline utilizing the mixup (Zhong et al., 2021).
B.4 COMBINING OURS WITH THE EXISTING STATE-OF-THE-ARE METHODS
Apart from the decoupled learning scheme (Kang et al., 2020) we mainly considered in the main text, there are other groups of methods achieving state-of-the-art performances: a) utilizing multiple experts (Zhou et al., 2020; Xiang et al., 2020; Wang et al., 2021), or b) applying contrastive learning algorithms (Cui et al., 2021; Zhu et al., 2022) for dealing with long-tailed data.
Although our proposed method based on the decoupled learning scheme, we would like to elarify that we can combine ours with the existing state-of-the-art frameworks due to its simplicity (i.e.. it only needs SWAG posterior and classifier re- te-trrining.. To this end, we tested ours upon exist- ing ffate-of-the-as art code bases by simply 1) obtaining SWAG posterior from the publicly available pre-trained checkpoint with a few SGD iterations (e.g., 10 training epochs) and 2) re-training the classification layer we proposed in Section 4.2. We adapted the following implementations:
17

---

Published as a conference paper at ICLR 2023
Table 8: Results on CIFARIO-LT and CIFARI00-LT: classification accuracy (ACC), negative log- likelihood (NLL), and expected calibration enror (ECE).
ACC (t)
CIFARIO-LT
Many
Medium
Few
All
NLL ())
ECE (4)
SGD
72.02=0.a:
1.443=0.102
0.207
+ cRT (Kang et al., 2020)
80.8010.03
0.598_0.011
0.041
LWS (Kang et al., 2020)
80.31 +0.07
0.615=0.002
0.055
LA (Menon et al., 2021)
77.36::0.1
0.947=00071
0.146
DisAlign (Zhang al., 2021)
81.37
0.600=2.022
0.054
SWA (ours)
74.86 10.23
1.201
0.1 0.17610.007
cRT (Kang et al., 2020)
81. 81.72-00.05
0.566=00.00
0.048
(Kang et ul., 2020)
80.53+s0.11
0.603=20006
0.056=0.002
(Menon et al., 2021)
81.7810.07
0.733
1.109
DisAlign (Zhang al., 2021)
81.96=0.05
0.574=0.011
0.052+0.000
SRepr (ours)
82.06
0.542-00.000
0.021
ACC (1)
CIFARIOO-LT
Many
Medium
Few
All
NLL (L)
ECE (L)
SGD
68.66.0058
38.9420.22
9.5020.57
40.31
3.202=200880
0.333=20010
cRT (Kang et al., 2020)
66.05=u.06
42.14+0.15
15.76=0.21
42.59+0.18
3.271=0.010
0.317=0.002
LWS (Kang ct al., 2020)
66.24=u.06
41.49+0.15
16.95=0.16
42.79+0.11
3.971
0.377#0.001
LA (Menon et al., 2021)
60.0910.08
43.3410.17
27.27
44.37
2.636 00.003
0.23110.001
DisAlign (Zhang al., 2021)
66.90=z0.12
42.51
16.94-20.1
43.37=0011
4.685=0.004
0.413+0.003
SWA (ours)
72.36=0.19
+1.51
7.97
42.34=0.188
2.924=0.030
0.301 +00007
cRT (Kang et al., 2020)
67.37.0.20
46.81
20.11
46.0010.09
2,953 0.0000
0.28211.000
(Kang et al., 2020)
67.09.0.07
48.1010.07
23.33
47.31
3,404
0.327
(Menon et al., 2021)
62.05=0.11 1
17.11
31.43=20.24
47.63:2007
2.224=0.020
178
DisAlign (Zhang al., 2021)
67.68=0.22
48.36+0.02
23.17=0.40
47.56+0.05
3.889=0.00
0.358+0.000
SRepr (ours)
66.69
49.9110.01
23.3110.11
10.81
2.148
0.149 10.002
Table 9: Results for combining ours with the existing state-of-the-ard methods on CIFARI00-LT and ImageNet-LTY LT: classification accuracy (ACC), negative log-likelihood (NLL), and expected cal- ibration error (ECE).
CIFAR100-LT
ImageNet-LT
Method
ACC
NLL
ECE
ACC
NLL
ECE
RIDE (Wang et al., 2021)
48.3
2.183
0.168
53.9
2.487
0.196
SWA SRepr (ours)
49.2 (40%)
2.176 (aom
0.159 (4.0))
54.8
2.471 (40016)
0.199
RIDE CMO (Park 2022)
48.5
2.161
0.155
54.1
2.425
0.187
SWA SRepr (ours)
49.1 (+0.6)
2.051 (Q.110)
0.118 (000))
55.0 (+0.9)
2,421 (0004)
0.189 (=0.002)
SADE (Zhang al., 2022)
49.9
2.015
0.120
59.0
1.826
0.060
SWA SRepr (ours)
50.1 (40.2)
2,008 (-0.00))
0.117 (-000))
59.2 (40.2)
1.824 (20022)
0.059 (200001)
BCI. (7hu et al., 2022)
52.0
1.941
0.192
$7.2
1.871
0.062
SWA SRepr (ours)
52.4 (+0.4)
(-0.042)
0.102 (-0.190)
57.5 (+1.3)
1.857 (-0014)
0.036
(-0.026)
Table 9 demonstrates that ours indeed improves the existing state-of-the-art methods. A marginal improvement upon multi-expert models (i.e., RIDE, RIDE + CMO, and SADE) is probably due to the ensemble model already providing well-calibrated predictions (Lakshminarayanan et al., 2017; Ovadia et al., 2019). Even if SWA-Gaussian well captures a single modality, naive ensembling significantly benefits from the multi-mode modal Bayesian marginalization (Wilson and Izmailov, 2020). Nevertheless, we believe the clear improvements upon the contrastive learning approach (i.c., BCL), which uses only single model, bear out the value of the proposed method.
18

---

Published as a conference paper at ICLR 2023
0.040
0.400
0.035
(represent dispersion
dispersion (probabilty bibiibltty
0.300
M M Mew MM A Mmn
0.030
Wy -1 yarnin M/ Ammaaahh
0.200
0.025
0.100
0.020
0.000
80D
1000
200
1000
cless indices sorted by the frequency
class indices sorted by the frequency
Figure 5: The per-class dispersion along with class indices in the representation space (left) and the probability space (right). The results are with ResNet-50 on ImageNet-LT.
per-class weight norm
per-class marginal likelihood
reliability diagram
2.50
10->
0.40
SWA
SWA cRT
marginal likal loodd
confidence accuracy
0.30
2.00
SWA SRepr
10
0.20
welght
1.50
0.10
10-4
0.00
1.00
-0.10
0.50
-0.20
250
500
1000
0.4
class Indices sorted by the frqquency
class Indices sorted by the frequency
confidence
Figure 6: The per-class weight norm of the classifier (left), the per-class marginal likelihood of the test predictions (middle), and the reliability diagram for the test predictions (right). The results are with 5e on ImageNet-LT.
B.5 FURTHER ANALYSIS ON PROPOSED METHODS
Mcasuring dispersion along with class indices While we already confirmed positive correla- tion between per-instance NLL and dispersion Section 3.2, would be worth further investigating it in the context of saiild recognition. To this end, Fig. 5 depicts the per-class dispersion along with class indices sorted by the number of training examples for each class. It clearly shows that the head class tends to have smaller dispersion (especially in the probability space dispersion), which indicates that our method is suitable for long-tailed recognition.
Per-class weight norms and marginal likelihoods after classifier re-training Following previ- ous works (Kang al., 2020; Ren et al., 2020; Alshammari et al., 2022), we further investigate how our proposed approach affects 1) the per-class weight norm, i.e., ||,,|| for y 1,..,, K, and [x)) for y = I,.., K, where Drest balanced test split. Fig. 6 depicts the per-class weight norm (left) and the per-class marginal likelihood (middle), along with class indices in x-axes sorted by the number of training examples for each class. We also plot the reliability diagram (right; Guo et al., 2017), showing whether the classification model produces well-calibrated predictions.
2) the per-class marginal likelihood, i.e., p(y) = (e (=, [P (x)]
Alshammari et al. (2022) argued that balanced weight norms of the classifier give tail classes chance to compete with head classes and produce the ideal marginal likelihood following uniform distribution. However, Fig. 6 demonstrates that our re-training method (i.e., SRepr) achieves both the uniform marginal likelihood (middle) and the well-calibrated prediction (right), even though it does not balance weight norms (left). This phenomenon suggests that our proposed approach works distinctly from the existing works balancing the weight norms for dealing with long-tailed data.
Training costs compared with vanilla decoupled training. Our approach (i.e., SRepr) requires M forward passes of the backbone network during the classifier re-training stage. However, the additional training cost due to these multiple forward passes is not huge bottleneck since we only
19

---

Published as a conference paper at ICLR 2023
Table 10: Training costs compared with vanilla decoupled training.
Method
Epoch
Wall-clock time
ACC
NLL
ECE
SWA cRT
20
647.2 sec
52.28
2.309
0.132
SWA SRepr (ours)
10
580.9
53.76
2.072
0.059
SWA cRT
40
1294,4 sec
52,47
2.284
0,124
SWA SRepr (ours)
20
1161.8
53.81
2.069
0.052
Table 11: Ablation study on alternative way to generate stochastic representation. (m) denotes the Th stochastic representation for given input æc
Method
How generate stochastic representation?
ACC
NLL
ECE
SWA ERT
52.28
2.309
0.132
SWA + SRepr (ours)
changing parameters, ie., Fi (m) Fe.. (3r), where L..... Our 1:4
(()).
53.81
2.069
0.052
SRepr (ours)
changing inputs, (..( Fer. (3 where zn mith augmented version zz
53.35
2.115
0.068
backpropagate through the last classifier layer while holding the backbone network frozen. To be concrete, Table 10 presents the wall-clock time of re-training classifier with and without SRepr for ResNet-50 on ImageNet-LT when M is 10. While SRepr requires x times compared to the vanilla classifier re-training (i.e., 32.36 sec/epoch vs. 58.09 sec/epoch), the performance gain achieved by SRepr outweighs the additional training time. That is, even if we run cRT for twice more training epochs than SRepr, it is worse than the performance that SRepr could achieve with the half number of training epochs. We also note that cRT cannot reach the numbers obtained by SRepr even if is ran for longer training epochs.
Generating stochastic representation by random augmentation. We can also generate stochas- tic representations by changing inputs (i.e., random augmentation) instead of model parameters (i.e., SWA-Gaussian, as we introduced in Eq. (9)). Table 11 shows that the proposed SRepr also works with random augmentation (i.e., random crop augmentation), but it is worse than one we originally proposed. One notable advantage of generating stochastic representations using SWAG is that we do need design data-dependent augmentation strategies. This helps when are to apply our method to the domain for which no straightforward data augmentation strategies are available (e.g.. text, graph, and speech data).
Ablation study on mixup augmentation and our approach. We also present the ablation study on the mixup augmentation and our approach (i.e., SRepr) since both methods improve the cal- ibration of the classification model. Table 12 clarifies our effectiveness distinct from the mixup augmentation for ResNet-50 on ImageNet-LT: (I) The performance gain most significant when we use them together. (2) While both the mixup and SRepr improve all metrics upon the baseline, the contribution from SRepr more significant than that from the mixup.
C EXPERIMENTAL DETAILS
Code is available https://github.cc haammm/s-giggllon--tae JAX (Bradbury et al., 2018), , Flax (Heek et al., 2020), and Optax (Hessel et al., 2020). These libraries available under the Apache-2.0 license? For ImageNet-LT and INaturalist-2018. conduct all experiments on 8 TPUv3 cores, supported by TPU Research Cloud?.
Our implementations are built on
C.I DATASETS
Net-LNe It is available at http://gthhb..ommzzzaiaaooooooo et al., 2019). consists 11,,866 train examples, 20,000 validation examples and 50,000 test
-OLTR (Liu
20

---

Published as a conference paper at ICLR 2023
Table 12: Ablation study on mixup augmentation and our approach.
mixup
SRepr (ours)
ACC
NLL
ECE
52.28
2.309
0.132
53.64 (+1.66)
2.131 (4.178)
0.079 (-0.053)
53.81 (+1.53)
2.069 (40.240)
0.052 (-0.080)
54.43 (+2.15)
1.990 (-0.119)
0.022 (+0.110)
examples from 1,000 classes. While the train split is imbalanced, with maximally 1,280 images per class and minimally 5 images per class, the validation and test splits are balanced.
We follow the standard data augmentation policy which consists of random resized cropping with an images size of 224 x 224 x 3 and random horizontal flipping. All images are standardized by subtracting the per-channel mean and dividing the result by the per-channel standard deviation. For per-channel mean and standard deviation values, we stay consistent with the values from the full ImageNet- Ik dataset (Russakovsky et al., 2015), i.e., mean of (0.485, 0.456, 0.406) and standard deviation of (0.229, 0.224, 0.225) in RGB order.
comp (Van Horn al., 2018). consists of 437,513 train examples, 24,426 validation examples and 149,394 test examples from 8. 122 classes. Since the ground-truth labels of the test split publicly available, instead use the balanced validation split as the test split. We apply the same data augmentation for training as that of ImageNet-LT.
INaturalist-2018. It available
https//github.coo//iiiipppeeee
CIFAR10/100-LT. It is available at https://gttub..com/kiid com/aaiic/LDDM-MMRY (Cao et al., 2019). consists of 10,847 train examples and 10,000 test examples from 10/100 classes when expo- nential decay with an imbalance factor of 0.01 is applied. We use simple data augmentation policy which consists of random padded cropping and random horizontal flipping (He al., 2016). All images are standardized with the mean of (0.4914, 0.4822, 0.4465) and standard deviation of (0.2023, 0.1994, 0.2010) RGB order.
C.2 OPTIMIZATION
ImageNet-LT and INaturalist-2018. Throughout the main experiments ImageNet-LT and iNaturalist-2018, we use an SGD optimizer with batch size 256, Nesterov momentum 0.9, and a single-cycle cosine decaying learning rate starting from the base learning rate of 0.1. Unless spec- ified, the optimization for the representation learning stage terminates after 100 training epochs for ImageNet-LT and 200 training epochs for iNaturalist-2018. For the classifier re-training, intro- duce an additional 10% training epochs to re-train the classifier.
CIFARI0/100-LT. Throughout the additional experiments CIFARI0/100-LT in Appendix B.2, we apply the same optimization strategy as that of ImageNet-LT and NNturalist-20118 except for the batch size of 128 and the baseline learning rate of 0.5.
C.3 WEIGHT DECAY (WD)
Weight Decay (WD; Krogh and Hertz, 1991) is the standard regularization technique for training deep neural networks. For instance, we additionally introduce the WD term of Awd | in Eq. (2), where Awd > is a hyperparameter to control the impact of the WD term. Table 13 shows valida- tion accuracy of SGD swept over And E {0.0001, 0.0002, 0.0003. 0.0004, 0.0005). We empirically found that tuning weight decay exerts a strong influence on long-tailes classification performance, Alshammari et al. (2022) reported. Throughout the paper, we apply Aud 0.0003 for ImageNet- LT, And= 0.0001 for NNtturaiit-2011 and Aud 0.0005 for CIFAR 10/100-LT.
21

---

Published as a conference paper at ICLR 2023
Validation accuracy of SGD with varying weight decay coefficients (i.e., Awd in Ap- pendix C.3) on ImageNet-LT and NNtuuaaisst-201.
Table 13:
ImageNet-LT
iNaturalist-2018
Many
Medium
Few
All
Many
Medium
Few
All
And 0.0001
64.72
39.27
14,66
45.73
76.31
67.89 18
62.2420.17
66.52:0005
Awd 0.0002
67.99
41.38
13.93
47.90
77.4820.59
68.5320.14 14
61.00 61.0720.35
666,50=0.17
And 0.0003
69.08 20.08 36
41.6611035
12.67.2..32
48.28.10.2
77.91
67.37
58.46
64,93
And 0.0004
69.29
41.16
10.50
47.83
77.16
65.54
54.37
62.32
Awd 0.0005
69.44
40.24
8.93
47.23
77.20
63.70
50.85
60.01
Table 14: Validation accuracy of SWA with varying SWA learning rates (i.e., 1/SWA in Appendix C.4) on ImageNet-LT and |Naturalist-2018.
ImageNet-ITT
iNaturalist-201
Many
Medium
Few
Many
Medium
Few
UswA 0.020
70.16
40.69
10.50
47.94
JswA 0.010
70.02+0.49
41.63+0.34
11.55+0.28
48.47
77.47
67.69
61.49
66.25
UswA 0.005
69.62
41.55
12.03
48.35
27.26
68.23
61.8720.18
66.65-0.10
7SWA 0.001
76.60
67.86
61.71
66.33
C.4 STOCHASTIC WEIGHT AVERAGING (SWA)
Stochastic Weight Averaging (SWA; Izmailov al., 2018) has three hyper-parameters: 1) when does the averaging phase start? 2) how frequently update the moving average? and 3) how to set the learning rate during the averaging phase? In response, we follow the instruction from (Izmailov et al., 2018); 1) the averaging phase starts at 75% of the training epoch, 2) we capture parameters at each epoch for averaging, and 3) we use the high constant learning rate V/swa during the averaging phase. Table 14 shows validation accuracy of SWA swept I/sWA {0.001, 0.005, 0.010, 0.020) on ImageNet-LT. Throughout the paper, we uSe 7WA 0.010 for ImageNet-LT, rJsWA = 0.005 for iNaturalist-200 2018, and IJSWA 0.1 for CIFAR 1 0/100-LTT
22

---

Published as a conference paper at ICLR 2023
Algorithm Decoupled training w/ SWA + SRepr (ours).
Require: Train dataset D { y:) H= the feature extractor F parameterized by 0, and the linear classification layer parameterized by Ø (wk, by)
Require: The number of representation learning steps Ti. the number of classifier retraining steps T2, the learning rate schedule 7)t, moment update start time TswA and update frequency JswA- Parameters (f", x² "b*),, trained in decoupled learning scheme.
Ensure:
// Representation learning stage
2: Initialize the parameters O (0o, Φo) to random values.
3: Initialize the first and second moments Sww 0, Osswa = 0, and viswA = 0.
4: for t E {1, .... Ti do
5: Sample mini-batch B C D.
6: Update the parameters with stochastic gradients,
θ f 0 Vo((() 0-0
(34)
where the loss is defined as
[ log (ogg(()l (x:: O)
(()) E E
(x:
(35)
7:
i Tswx and (O, / 0 then Update the first and second moments via moving average,
8:
+ SWA + (nswA O SWA + @x)/(nswA + 1),
(36)
O SWA + (NISWa O OSSA + 07)/( + 1),
(37)
NSSW
TOSWA + 1.
(38)
9: end if
10: end for
// Classifier retraining stage
12: We have the pre-trained parameters Or where Or +- Oswx if we applied SWA.
We also have the approximate posterior over the feature extractor parameters, i.e.,
D) = (e OSWA, EsWAG), where EsWAG = diag(6's (6sWA Wxx)).
(39)
14: for t E (T +1 1,.., T, + T2) do Sample mini-batch B D.
15:
16:
Update the parameters with stochastic gradients (with some balancing strategy),
Φ + Φ 7% C(Ø) "=
(40)
where the loss is defined as
L(Φ)= E(m,y)~B log (x: (0), .6))
(41)
L(Φ) E(
0.5LLE + 0.5kk)] if we applied SRepr.
(42)
Here, the first term is
iCE = [(aw))~((0
log log (v) (x; (0, 6))
(43)
Refer to Eq. (16) for the detailed definition of the second term.
17: end for
18: Return 0" =0,; and x +T, trained in decoupled learning scheme.
23


Reviews:
[{"review_id": "yBHA2KD7mS0", "reviewer": "Reviewer_fqWB", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper focuses on improving features to be more generalizable to achieve better performance in Long-Tailed Recognition (LTR). In particular, the paper adopts Stochastic Weight Averaging (SWA) for improving the generalization of deep neural networks. Moreover, the paper proposes a classifier re-training algorithm based on the learned representation. The algorithm uses a Gaussian perturbed SWA, and a self-distillation strategy that can harness the diverse stochastic representations based on uncertainty estimates to build more robust classifiers. Experiments on two benchmarks show that the proposed method improves upon previous methods in terms of both prediction accuracy and uncertainty estimation.", "summary_of_the_review": "The paper focuses on using Stochastic Weight Averaging (SWA) to improve features used for long-tailed recognition (LTR). The motivation is to learn more generalizable features for LTR. SWA is an existing method that learns more general features and the paper claims that \"to the best of our knowledge, this is the first attempt to introduce SWA into long-tailed learning\". However, SWA seems weak as it achieves only marginal improvement given the large added computation burden. Therefore, the rating of the paper is \"5: marginally below the acceptance threshold\"", "strength_and_weaknesses": "Strength\n- Studying how generalized features improves LTR is interesting.\n- Studying uncertainty estimation in the context of LTR is interesting.\n\n\nBelow are some weaknesses.\n\n- While \"the success of decoupling naturally motivates obtaining more informative representations from which the classifier re-training can benefit\", using SWA seems like less well-motivated. Instead, this motivation should suggest learn more generalized features using more data sources (e.g., external data) and more general loss functions (e.g., contrastive learning). I would like the authors to discuss this further, otherwise the paper just seems incremental especially when it states \"to the best of our knowledge, it has never been explored for long-tailed classification problems.\", and in the conclusion \"to the best of our knowledge, this is the first attempt to introduce SWA into long-tailed learning.\"\n\n- While the paper is motivated to learn \"generalizable\" features, a recent work [R1] shows that learning a backbone which has \"balanced-norm\" filters significantly boosts LTR performance. Moreover, [R1] simply tunes weight decay to regularize backbone's norms and achieves the state-of-the-art. This is an open questions -- whether to learn \"balanced backbone\" or more \"generalized backbone\" to achieve better LTR performance? Can authors discuss and show insights?\n\n[R1] Alshammari, et al., \"Long-Tailed Recognition via Weight Balancing\", CVPR 2022\n\n- While it is interesting to study uncertainty estimation in the context of LTR, the paper does not sufficiently analyze the results. For example, it is unclear how good uncertainty estimation for tail classes vs. head classes. Authors are encouraged to dive into this part and provide insightful discussions.\n\ntypos\n- Figure 2 caption: \"asdfsadf\".", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "fD5C6L5qD6", "reviewer": "Reviewer_1FXP", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors study the problem of long-tailed visual classification. To be specific, they propose obtaining a stochastic model for representations through stochastic weight averaging (SWA) and retraining a robust classifier by distilling these representations.", "summary_of_the_review": "Novel approach but there are concerns about the evaluation.", "strength_and_weaknesses": "trengths:\n+ A novel approach to obtaining better representations / decision boundaries for long-tailed visual recognition.\n+ Positive results over the baselines.\n\nWeaknesses:\n\n1. The paper identifies the main source of imbalanced learning as the decision boundary. However, the recent view identifies the culprit as the representation learning stage and suggests that features of the under-represented class are somehow sub-optimal, which leads to worse decision boundaries for the under-represented class. I suggest the authors to tone down \"the main bottleneck is the decision boundary\" claim.\n\n2. Fig 2: It is not clear why the dispersion of models along the training trajectory should be correlated with NLL. \n\n3. The proposed approach is computationally too expensive. One can easily obtain a set of alternative representations through dropout &  perform distillation. Would this not be cheaper? A discussion on complexity is necessary. \n\n4. The experimental evaluation needs improvement.\n\n4.1. The SOTA comparison in Table 4 seems to have unfair aspects: In some cases, less training epochs are used with the compared baseline (e.g. the comparison against KCL) and more are used in some others (Liu et al. (2021)). \n\n4.2. The experimental evaluation is missing important comparisons with more recent approaches from 2022. No approach is compared against from 2022.\n\n4.3. The experimental evaluation is missing comparisons on CIFAR10LT & CIFAR100LT. This is important to get a better feeling about how the method performs in a small-scale problem (CIFAR10LT) and a middle/large-scale problem such as CIFAR100LT. In fact, CIFAR100LT is surprisingly challenging for some problems and you can obtain unexpected results.\n\n\nMinor comments:\n- \"Confirming that SWA can benefit to long-tailed classification\" => \"Confirming that SWA can provide benefit to long-tailed classification\".\n- Eq 2: \"x\" is forgotten after the dot.\n- \"parameter θ\" => \"parameters θ\".\n- \"parameter ϕ\" => \"parameters ϕ\".\n- Eq 4: A minor detail but this should include a step size.\n- Eq 6: Θ′ => not introduced.\n- Below Eq 9: \"predictive uncertainty of x,\" => Predictive or epistemic? They are not the same. Two lines above Eq 9, you said epistemic.\n- Figure 2: \"asdfsadf.\" => Agreed :)\n- Fig 3: ERM not introduced. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "DLvsqtaesR", "reviewer": "Reviewer_eyhi", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies the effect of stochastic representations in long-tail recognition scenarios and designs a self-distillation method to adapt stochastic representations to long-tail learning. ", "summary_of_the_review": "This paper overall is a good contribution to the community for its detailed experiments on the effect of stochastic representations on long-tailed learning, and the idea of using stochastic representations as a measurement of confidence. However, the proposed method is relatively complicated and yields limited improvements over state-of-the-art methods. ", "strength_and_weaknesses": "I think this paper is very well-written and well-organized. The preliminary experiments on the effect of stochastic representation (Stochastic Weight Average, SWA) on decoupled long-tail learning methods are detailed and intuitive. They start with testing directly applying stochastic representations to single stage and two stage training and acquired results that promote further development of SWA on two-stage/decouple methods. The idea of using Gussian SWA (SWAG) as a measurement of prediction certainty, although not new, but directly connected to the intuition that tail class predictions are usually less certain. A robust certainty measurement is always a challenge in long-tail recognition, and I think using SWAG is an interesting proposition. The discussion and explanation of the limitations of direct usage of SWAG are clear, because it needs multiple runs to get a stochastic distribution. And this leads to the proposition of the proposed self-distillation method, which has multiple teachers during training for a SWAG distribution, and has a student, that learns from the stochastic teachers. Again, this multi-teacher single student framework is not new (e.g., mean teachers in semi-supervised learning). But it aligns well with the challenges mentioned in direct usage of SWAG. The experiments are extensive. Even the state-of-the-art multi-expert methods like  RIDE are compared. However, the improvements, especially the improvements over multi-expert methods, are limited (<1% on average). The biggest drawback of this method is that it is relatively too complicated (multiple teach networks need to be trained for SWAG to work), and, thus, not compatible with existing multi-expert models, which leads to limited improvements. The proposed multi-teacher single student framework itself is already a multi-expertish model. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "0Shtq3W6ZL", "reviewer": "Reviewer_DCKB", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper aims to solve the fundamental problem --- long-tailed recognition. It analyzes SWA/SWAG with long-tailed data. \n           \n（1) Classifier re-training can make better use of SWA for long-tailed data.    \n  (2) The positive correlation between NLL and dispersion is observed. Based on this phenomenon, a self-distillation strategy is developed to reduce predictive variance.   \n  (3) Experiments on ImageNet-LT and iNaturalist 2018 show some improvements.    \n", "summary_of_the_review": "The paper investigates SWA/SWAG for long-tailed data. However, some important comparisons are missed.    \nIf the concerns are addressed, I'm very glad to raise my score.    ", "strength_and_weaknesses": "Strength    \n\n(1) The paper writes clear and is easy to follow.    \n\n(2) The method should be general and is expected to work well with previous methods.   \n\nWeakness  \n\n(1) This paper use SWA to enhance the representation ability of deep models and a distillation method is proposed to reduce predictive variance. However, comparisons with most related work are missed.     \n      I. The state-of-the-art methods, like PaCo [1].     \n      II. distillation-based methods, like CBD [2].     \n\n\n[1] Parametric Contrastive Learning. ICCV 2021.     \n[2] Class-Balanced Distillation for Long-Tailed Visual Recognition. BMVC 2021.     \n\n(2) About the distillation objective.  Why do the teacher prediction probabilities obey the Dirichlet distribution?    \n\n(3) As shown in Figure2, as the number of stochastic representations increases, the changes of ECE and NLL are limited.  ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster