AutoBalance: Optimized Loss Functions for Imbalanced Data
Authors: Mingchen Li
Xuechen Zhang
Christos Thrampoulidis
Jiasi Chen
Samet Oymak
Venue: NeurIPS 2021 Poster (2021)
Keywords: Label imbalance
fairness
bilevel optimization
personalization

Abstract: Imbalanced datasets are commonplace in modern machine learning problems. The presence of under-represented classes or groups with sensitive attributes results in concerns about generalization and fairness. Such concerns are further exacerbated by the fact that large capacity deep nets can perfectly fit the training data and appear to achieve perfect accuracy and fairness during training, but perform poorly during test. To address these challenges, we propose AutoBalance, a bi-level optimization framework that automatically designs a training loss function to optimize a blend of accuracy and fairness-seeking objectives. Specifically, a lower-level problem trains the model weights, and an upper-level problem tunes the loss function by monitoring and optimizing the desired objective over the validation data. Our loss design enables personalized treatment for classes/groups by employing a parametric cross-entropy loss and individualized data augmentation schemes. We evaluate the benefits and performance of our approach for the application scenarios of imbalanced and group-sensitive classification. Extensive empirical evaluations demonstrate the benefits of AutoBalance over state-of-the-art approaches. Our experimental findings are complemented with theoretical insights on loss function design and the benefits of the train-validation split. All code is available open-source.

AutoBalance: Optimized Loss Functions for Imbalanced Data
Mingchen Li
Xuechen Zhang University of California. Riverside (m1i176, x2han394]@ucr. edu
Christos Thrampoulidis University of British Columbia cthrampo@ece. ubc. edu, ca
Jiasi Chen
Samet Oymak
University of California, Riverside jiasi@es fasi@css.o ucr.edu
University of California, Riverside oymakßece ucr.edu
Abstract
Imbalanced datasets are commonplace in modern machine learning problems. The presence under-represented classes or groups with sensitive attributes results in concerns about generalization and fairness. Such concerns are further exacerbated by the fact that large capacity deep nets can perfectly fit the training data and appear to achieve perfect accuracy and faimess during training, but perform poorly during test. To address these challenges, we propose AutoBalance, a bi-level optimization framework that automatically designs training loss function to optimize a blend accuracy and beinnss--seeking objectives. Specifically, lower-level problem trains the model weights, and upper-level problem tunes the loss function by monitoring and optimizing the desired objective over the validation data. Our loss design enables personalized treatment for classes/groups by employing a parametric cross-entropy loss and individualized data augmentation schemes. We evaluate the benefits and performance of our approach for the application scenarios imbalanced and group-sensitive classification. Extensive empirical evaluations demonstrate the benefits of AutoBalance over state-of- the-an art approaches. Our experimental findings are complemented with theoretical insights on loss function design and the benefits of train-validation split. All code is available open-source.
Introduction
Recently, deep learning, large datasets, and the evolution of computing power have led to unprece- dented success in computer vision, and natural language processing [15, 39, 62]. This success is partially driven by the availability of high-quality datasets, built by carefully collecting a sufficient number of samples for each class. In practice, real-world datasets are frequently imbalanced and exhibit ooig-tailed behavior, necessitating a careful treatment of the minorities [20, 53, 23J. Indeed, modern classification tasks can involve thousands of classes, so is perhaps intuitive that some classes should be weer/under-representee compared to others. Besides class imbalance, minorities can also appear the feature-level: for instance, the specific values of the features of an example can vary depending on that example's membership in certain sensitive or protected groups, e.g. race, gender, disabilities (see also Figure la). scenarios where imbalances are induced by heterogeneous client datasets (e.g., in the context of federated learning), addressing these imbalances can help ensure that machine learning model works well for all clients, rather than just those that generate the majority of the training data. This rich set of applications motivate the careful treatment of imbalanced datasets.
In the imbalanced classification literature, the recurring theme is maximizing fairness-seeking objective, such as balanced accuracy. Unlike standard accuracy, which can be dominated by the majorities, fairness-seeking objective seeks to promote examples from minorities, and downweigh
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Train Datreet
Train Cotteet
Consect
224 wastecn Sy
GOW Tisliing Si
Water binds
Nelwork
Hyper- peramiters miny 4aal (fe)
Network
Vetwork ming Cala 1/
fue L/ n*)
Lard bink
Witer backg sand
Optiral pprarsorrmmtterreeo
Optimal parameter 8
fa-
Search phase
Retraining phase
Evaluation phase
(a)
(b)
Figure 1: (a) Example group imbalance on the Waterbirds dataset [63, 73J. Groups correspond to the distinct background types, while classes are distinct bird types. (b) Framework overview. The search phase conducts a bilevel optimization to design the optimal training loss function parameterized by a by minimizing the validation loss, using train-validation split (c.g. 80%-20%). The retrain phase uses the original training data and ex" to obtain the optimal model parameters 0". The evaluation phase predicts the test data using 0*
examples from majorities. Here, note that there is a distinction between the test and training objectives. While the overall goal is typically to maximize non-differentiable objective such as balanced accuracy on the test set, during training, we use a differentiable proxy for this, such as weighted cross-entropy. Thus, the fundamental question of interest is:
How to design training loss to maximize a fairness-seeking objective on the test set?
classical answer to this question is to use a Bayes-consistent loss functions. For instance, weighted cross-entropy (e.g., each class gets a different weight, see Sec. 2) is traditionally good choice for optimizing weighted accuracy objectives. Unfortunately, this intuition starts to break down when the training problem is overparameterizeed which is a common practice in deep learning: in essence, for large capacity deep nets, the training process can perfectly fit to the data, and training loss is no longer indicative of test error. In fact, recent works [8, 38] show that weighted cross- has minimal benefit to balanced accuracy, and instead alternative methods based on margin adjustment be effective (namely, by ensuring that minority classes further away from decision boundary). These ideas led to the development of a parametric cross-entropy function E(y, fx)) wy log (1 +1+
which allows for personalized treatment of the individual classes via the design parameters (wa, lt, As) term whereas la and A are additive and multiplicative logit adjustments. However, despite these developments, it is unclear how such parametric cross-entropy functions can be tuned for use for different fairness objectives, for example to tackle class or group imbalances. The works by [8, 53] provide aheoretically-motivtted motivated choices for (Wy,44 44), while [38] argues that (Wk, 4x) is not as effective as Ak in the interpolating regime of zero training error and proposes the simultancous use of all three different parameter types. However, these works do not provide an optimized loss function that can be systematically tailored for different faimess objectives, such as balanced accuracy common in class imbalanced scenarios, or equal opportunity [23, 17] which is relevant in group-sensitive settings.
Exty ela-ty. e e (c))
K [8. 38, 53, 35, 67J. Here, Wk is the classical weighting
In this work, we address these shortcomings by designing the loss function within the optimization in a principled fashion, to handle different fairness-seeking objectives. Our main idea to use bi-level optimization, where the model weights are optimized over the training data, and the loss function is automatically tuned by monitoring the validation loss. Our core intuition that unlike training data, the validation data is difficult to fit and will provide consistent estimator of the test objective.
Contributions. Based on this high-level idea, this paper takes step towards systematic treatment of imbalanced learning problems with contributions along several fronts: state-of-the-ar performance, data augmentation, applications to different imbalance types, and theoretical intuitions. Specifically:
We introduce AutoBalance -a bilevel optimization framework- that designs a fairness-secking loss function by jointly training the model and the loss function hyperparameters in systematic way (Figure 1b, Section 2). We introduce novel strategies that narrow down the search space to improve convergence and avoid overfitting. To further improve the performance, our design also incorporates data augmentation policies personalized to subpopulations (classes or groups). We demonstrate the benefits of AutoBalance when optimizing various fairness- seerness-seekinggg objectives over the wate-of-the-are such as logit-adjustment (I.A) [53] and abbel-distibbutionavn margin (I.DAM) [8] losses. The code is available online [42].

---

Algorithm 1: AutoBalance via Bilevel Optimization
Input: Model Jo with weights 0, dataset S = =S U Sy. step sizes 1/- & njo, # iterations > 4
Initialize a with ((ir(()) = faain (15 ox)
// Consistent initialization
2 Train 0 for & iterations (a is fixed)
// Search Phase: Starts with warm-up
3 for i 4- { to i2 do
Sample training batch By from STi
By - A(BT)
1/ Apply class-porsonalized augmentation
0 0 no V V (So:
Sample validation batch By from Sy; Compute hyper- syper-gradieet V Va (fo)
// via Approx. Implicit Differentiation
a + a 1/a V L (So)
// //Update loss function parameeer
10 end Set a. a. ST f S, reset weights 0
// //Retraining Phase: Use all data and
Train 0 for le iterations using a. Result: The final model 0. + 0 and hyper-parameters O.,
Extensive experiments provide several takeaways (Section 3). First, AutoBalance discovers loss functions from scratch that are consistent with theory and intuition: hyperparameters of the minority classes evolve upweight the training loss of minority promote them. Second, the impact of individual design parameters in the loss function is revealed, with the additive adjustment 4 and multúplicative adjustment A synergistically improving the fairness objective. Third, personalized data augmentation can further improve the performance over a single generic augmentation policy.
Beyond class imbalance, we consider applications of loss function design to the group-sensitive setting (Section 4). Our experiments show that AutoBalance consistently outperforms various baselines, leading to a more efficient Pareto-frontic of securacy-fairnes tradcoffs.
1.1 Problem Setup for Class Imbalance
We first focus on the label-imbalance problem. The extension to the group-imbalanced setting (approach, algorithms, evaluations) is deferred to Section 4. Let [K] denote the set {1..... K}. Suppose we have a dataset S = (2); Mi )=> sampled i.i.d. from distribution D with input space X' and K classes. For training example (ær, y),
X' is the input feature and y [K] is the output label. Let f X -> IRK be a model that outputs a distribution over classes and let (x))) arg [ fxz). The standard classification error is denoted by E(f) = PDly A őy (x)]. For loss function E 9) (e.g. cross-entropy), similarly denote
Population risk: L() = Ep( (a))) and Empirical risk: ( 1 Σ ( D1 (2:)).
V Setting: Imbalanced classes. Define the frequency of the k ''th class via π. = P (m ( (y = k). Label/class-imbalance occurs when the class frequencies differ substantially, i.e., max:e[k] क, minge[K] π., Let us introduce
K
Balanced risk: (f) = K
Eca0 and Class-conditional risk: La(S) = ED, [C (x)))
1m
Similarly, let Ef) be the class-conditional classification error and E (J) :- =1/K) &
be the balanced error. In this setting, rather than the standard test error E(f), our goal is to the minimize balanced error. At a high-level, we propose to do this by designing an imbalance-aware training loss that maximizes balanced validation accuracy.
2 Methods: Loss Functions, Search Space Design, and Bilevel Optimization
Our main goal in this paper is automatically designing loss functions to optimize target objectives for imbalanced learning (e.g., Settings A and B). We will employ parametrizable family of loss functions that can be tailored to the needs of different classes or groups. Cross-entropy variations have been proposed by [44, 35, 16] to optimize balanced objectives. Our design space will utilize recent works which introduce aabel-distributionaaaar margin (LDAMM) [8], Logit-adjustment (LAA [53], Class-dependent temperatures (CDT) [67], Vector scaling (VS) [38] losses. Specifically,
3

---

build on the following parametric loss function controlled by three vectors w. I. A E RK; Σe e (Asill)))A///lt)))
ely, fx)) wy log (1
(2.1)
A
Here, Wy enables conventional weighted CE and Ly. and Ay are additive and multiplicative adjust- ments to the logits. This choice is same as the VS-loss introduced [38], which borrows the A term from [67] and I term from [8, 53]. [53] makes the observation that we can use I rather than w while ensuring Fisher consistency in balanced error. We make the following complementary observation.
Lemma 1
Parametric loss function (2.1) is not consistent for standard or balanced errors if there are distinct multiplicative adjustments i.e. A + for some i.j E [K]]
While consistency is a desirable property, it is intuitively more critical during the earlier phase of the training where the training risk is more indicative of the test risk. In the interpolating regime of zero-training error, [38] shows that w, I can be ineffective and multiplicative A-adjustment can be more favorable. Our algorithm will be initialized with a consistent weighted-CE: however, we will allow the algorithm to automatically adapt to the interpolating regime by tuning and A.
Proposed training loss function. For our algorithm, we will augment (2.1) with data augmentation that can be personalized distinct classes. Let us denote the data augmentation policies by A (Ay) where each A, stochastically augments an input example with label y. Additionally, we
clamp A; with the sigmoid function a to limit its range to (0,1) to ensure non-negativity. To this end, our loss function for the lower-level optimization (over training data) is as follows:
(A) e E (A )
Cuain x,, f: -A wy log
(2.2)
Here, a is the set of hyperparameters of the loss function that we wish to optimize, specifically = |w, 1, A. param(A)]. param(A) is the paramcterization of the augmentation policies (Ag) (E] Personalized data augmentation (PDA). Remark-
Equal Augmentatinn Lounnieee gauraag
Minorrky-avvored Augmentation busslaystit seile
able benefits of data augmentation techniques provide natural motivation to investigate whether one can benefit from learning class-persone plass-personalized aug- mentation policies. The PDA idea relates to SMOTE [9]. where the minority class is over-sampled by creating synthetic examples. To formalize the benefits of PDA, consider a spherical augmentation strategy where (z) samples vector uniformly from an balball of radius Sy around æ. As visualized in Figure for linear classifier, if the augmentation strengths of both classes are equal, the max max-margin classifier not affected by the application of the data augmentation and remains identical. Thus, augmentation has no benefit. However by applying stronger augmentation on minority, the decision boundary is shifted to protect minority which can provably benefit the balanced accuracy [38]. The following intuitive observation links the PDA to parametric loss 12.
Figure 2:
Data augmentation can shift the de- cision boundary to benefit the minority class by providing a larger margin. Lemma 2 establishes an equivalence hetween spherical data augmentation and parametric cross-entropy loss.
Lemma Consider a binary classification task with labels 0 and 1 and a linearly separable truining dataset. For any parametric loss (2.1) choices of (4, A., w.) w)) there exists spherical augmentation strengths for minority/majority classes so that, without regularization, optimizing the logistic loss with personalized augmentations returns the same classifier as optimizing (2.1).
This lemma similar in flavor to [31], which considers a larger uncertainty set around the minority class. But, as discussed in the appendix. Lemma 2 is relevant in the overparameterized regime whereas the approach of [31] is ineffective for separable data [51]. Algorithmically, the augmentations that we consider are much more flexible than the balbslls of [31] and our experiments showcase the value of our approach in state-of-the-ae art multiclass settings. Besides, note that the (theoretical) benefits of PDA can go well-beyon Lemma 2 by leveraging the invariances [10, 14] (via rotation, translation).
2.1 Proposed Bilevel Optimization Method
We formulate the loss function design as a bilevel optimization over hyperparameters a and a hypothesis set F. We split the datasct S into training ST and validation Sy sets with ny and ny

---

Niske of sampless
240
Hy yor Inraaitttts valle
Da Conlacery Can Boaae
pwr.rpparmttttee value
Hyper Hyper-parane
Ityper Alype-puu
100
Sorted class
Epoch
Epoch
Epoch
(a) train-vaidaation sizes (b) Evolution of when (c) Evolution of 1 when (d) Evolution of A and I only training l
for CIFARI00-LT clusters only training A
when optimizing jointly
Figure 3: (a) Visualizing class clustering and valiaaaaiittionn split, (b), (c), (d) Evolution of loss function parameters I, A over epochs for CIFARI00-LT where solid curves and dashed curves corresponds to A and I respectively. We display average value of 20 classes for better visualization. Based on theory, the minority classes should be assigned larger margin. During the initial 120 epochs, we use weighted cross-entropy training and AutoBalance kicks in after epoch 120. Observe that, AutoBalance does indeed learn larger parameters (yy, Ay) for minority class clusters (cach containing 20 classes) consistent with theoretical intuition. In all Figures (b), (c), (d), by the end of training. the colors are ordered according to the class frequency. However, when Ay is trained jointly with ly (Fig d), the training is more stable compared to training A, alone (Fig b). Thus, besides its accuracy benefits in Table l, also seems have optimization benefits.
examples respectively. Let Etair be the desired test-error objective. When Etarr is not differentiable, we use weighted cross-entropy (CH) loss function (yr ú) chosen to be consistent with Etair- For instance, Etair could be a superposition of standard and balanced classification errors, i.e. & = (1- + AEhal. Then, we simply choose fair = (1 A)CE + CEpat. The hyperparameter a aims to minimize the loss fair over the validation set Sy and the hypothesis f E F aims to minimize the training loss (2.2) as follows:
min CSy (fa)
1 WHERE fa = arg min min Cin (f: == TLT
(vain F; f()..
(2.3)
JeF
Here, Liair/ S are the test/validation risks associated with ffair. e.g. Laii Ep [fairl as in Section 1.1. Algorithm summarizes our approach and highlights the key components. The training loss (minn:; a)) is also initialized to be consistent with Enir (e.g., same fair). In line with the literature on bilevel optimization, we will refer to the two minimizations of the validation and training losses in (2.3) upper and lower level optimizations, respectively.
Implicit Differentiation and Warm-up training, For a loss function parameter at the hyper-gradient can be written via the chain-rule )Lar((") 0Ca + HCnn ae- aa
20" da [46]. Here, 0 is the solution of the lower-level problem. We note that dLtar/Oa = 0 since a does not appear within the upper- level loss. Also observe that (0*))
dex can be directly computed by taking the gradient. To compute a0 /dox we follow the recent work [46] and employ the Implicit Function Theorem (IFT). = 0 and regularity conditions are satisfied, then around ax" there exists function 0(a)) such that O(a*) 0' and we also 80 have De = -
If there exists a fixed point (0' ,ox)) that satisfies ô Iuaiin (0', (²,'a'))8 (aa,
d² Cumi.
a0da However, directly computing inverse Hessian @ > is usually 30
time consuming or even impossible for modern neural networks which have millions of parameters. To compute the pperrggaddent while avoiding extensive computation, we approximate the inverse Hessian via the Neumann series, which is widely used for inverse Hessian estimation [43, 46). Finally, the warm-up phase of our method (Line 2 of Algo. 1) is essential to guarantee that the IFT assumption 00mml0 ""'""
0 is approximately satisfied.
Why Bilevel Optimization? We choose differentiable optimization over alternative hyperparameter tuning methods because our hyperparameter space is continuous and potentially large (e.g. in the order of K = 8, 142 for iNaturalist. In our experiments the runtime of our method was typically 4-55 times that of standard training (with known hyperparameters). Intuitively, the runtime is at least twice due to our use of separate search and retraining phases. In the appendix, we also compare against alternative approaches (specifically SMAC of [26]) and found that our approach is faster and more accurate on CIFARIO-LT,
2.2 Reducing the Hyperparameter Search Space and the Benefits of Validation Set
Suppose we wish to optimize the hyperparameter a (wy, A of the parametric loss 12. An important challenge is the dimensionality of a,, which is proportional to the number of classes K, as need triplet (wy, Lg. Ay) for each class. For instance, ImageNet has K 1, 000
5

---

whereas iNaturalist has K = =8, 142 classes resulting in high- -ige--imensional hyperparameters. In our experiments, we found that directly optimizing over such large spaces leads convergence issues likely because of the difficulty of hypergradient estimation. Additionally, with large number of hyperparameters there is increased concern for validation overfitting. This is especially so for the tail classes (e.g. the smallest class in CIFAR 100-LT has only 1 validation example with an 80-20% split), On the other hand, it is well-known in AutoML literature (e.g. neural architecture search [45], AutoAugment [12]) that designing good search space is critical for attaining faster convergence and good validation accuracy. To this end, we propose subspace-based search spaces for hyperparameters (wy: ly. Ay)- To explain the idea, consider the logit-adjustment parameters 1 [11 ... [x] and A1 Ak]. We propose representing these K dimensional vectors via K' < K dimensional embeddings V' A' follows
A - DAA' and D2,' where A' E ERK
Here, DT E RR is frequency aware dictionary matrix that we design, and the range space of Dr becomes the hyperparameter search space. our algorithm, we cluster the classes in terms of their frequency and assign the same hyperparameter to classes with similar frequencies. To be concrete, if each cluster has size C, then K" [K/C]. For CIFARIO-LT; CIFAR ImageNet- LT and iNaturalist, we use C (1, 10, 20, 40) respectively. In this scheme, each column d of the matrix Dr is the indicator function one of the K' clusters, i.e. d4 = ith class is within the cluster and 0 otherwise. Clusters are pictorially illustrated in Fig. 3a. Finally, we remark that the specific hyperparameter choices of CDT [67], LA [53], and VS [38] losses in the corresponding papers, can be viewed as specific instances of the above search space design. For instance, LA-loss chooses a scalar T and sets 4, = T (or, This corresponds to a dictionary containing a single column d E RK with entries d: log( ..).
Why is train- valiataaioo split critical? Figure we plot the balanced errors of training/validationnesst datasets each epoch of the search phase. The train- ing data is fixed whereas we evaluate different val- idation set sizes. The first finding is that training loss always overfits (dotted) until zero error whereas validation loss mildly overfits (dashed vs solid). Sec- ondly, larger validation does help improve test ac- curacy (compare solid lines). The training behavior is in line with the fact that large capacity networks can perfectly fit and achieve 100% training accuracy [18, 56, 29]. This also means that different accuracy metrics or fairness constraints can be perfectly satis- fied. To truly find model that lies the Pareto-front of the (accuracy, fairness) tradeoff, the optimization procedure should (approximately) evaluate on the population loss. Thus, as in Figure 4, the validation phase provides this crucial test-proxy in the overpa- rameterized setting where training eтror is vacuous. Following the model selection literature [32, 33]. the intuition that, as the dimensionality of the hyper-parameter a typically smaller than the validation size ny, validation loss will not overfit and will be indicative of the test even if the training loss zero. Our search space design in Section 2.2 also helps to this end by increasing the over- sampling ratio ny /dim(a) via class clustering. In the appendix, we formalize these intuitions for multi-objective problems (e.g., accuracy + fairness), Under mild assumptions, we show that a small amount of validation data is sufficient to ensure that the Pareto front of the validation risk uniformly approximates that of the test risk. Concretely, for two objectives (L1; L2), uniformly over all X, the hyperparameter a minimizing the validation risk S (f) = (1 5 + >CS in (2.3) also approximately minimizes the test risk Lair (f).
100
# val # train= 1/40
# vall/ # train-1/8 1/8
80
di val / # train=1/4
60
Solid: Test error Dashed:Validate Validation err Dotted: Training error
Error
40
20
0
50
100
150
200
250
300
Epoch
Figure 4: Ttain/Validation/Tee errors during CIFARIO-LT search phase with different vali- dation sizes and fixed training size.
3 Evaluations for Imbalanced Classes
In this section, we present our experiments on various datasets (CIFAR- 10, CIFAR-100, iNaturalist- 2018 and ImageNet) when the classes are imbalanced. The goal is to understand whether our bilevel optimization can design effective loss functions that improve balanced error Enm on the test set. The setup is as follows. Ebal is the test objective. The validation loss Ltaie is the balanced cross-entropy

---

Method
CIFARIO-LT
CIFARIO0-LT
ImageNet-LT
iNaturalist
Cross-Entropy LDAM loss [8]
30.45
62.69
55.47
39.72
26.37
59.47
54.21
35.63
LA loss (T [53]
23.13
58.96
52.46
34.06
CDT loss [67]
20.73
57.26
53,47
34.46
AutoBalance: of LA loss
21.82
58.68
52.39
34.19
AutoBalance: I
23.02
58.71
52.60
34.35
AutoBalance: A
22.59
58.40
53.02
34.37
AutoBalance: A&
21.39
56.84
51.74
33.41
AutoBalance: & LA init
21.15
56.70
50.91
33.25
Table 1: Evaluations of balanced accuracy on
data. Algo. with A&T design space and LA initialization (bottom row) outperforms most of the baselines, across various datasets.
CB We consider various designs for frain such as individually tuning w. L, and augmentation. We report the average result of 3 random experiments under this setup.
Datasets. We follow previous works [53, 13, 8] to construct long-tailed versions of the datasets. Specifically, for a K-class dataset, we create a long-tailed dataset by reducing the number of examples per class according to the exponential function n n,je', where n, the original number of examples for class i, ni is the new number of examples per class, and < is a scaling factor. Then, we define the imbalance factor p ngg/n/k which is the ratio of the number of examples the largest class (n() to the smallest class (n'x). ). For the CIFARIO-LT and SIFAR100-LE dataset, we construct long-tailed versions of the datasets with imbalance factor p = 100. ImageNet-LT contains 115,846 training examples and 1,000 classes, with imbalance factor = 256. iNaturalist-2011 contains 435,713 images from 8, 142 classes, and the imbalance factor is p = 500. These choices follow that of [53]. For all datasets, we split the long-tailed training set into 80% training and 20% validation during the search phase (Figure lb).
Implementation. In both CIFAR datasets, the lower-level optimization trains a ResNet-32 model with standard mini-batch stochastic gradient decent (SGD) using learning rate 0.1, momentum 0.9, and weight decay 1e -4. 4, over 300 epochs. The learning rate decays epochs 220 and 260 with factor 0.1. The upper-level hyper-parameter optimization computes the hyper-gradient via implicit differentiation. Because the hyper-gradient is mostly meaningful when the network achieves near zero loss (Thm of [46]), we start the validation optimization after 120 epochs of the training optimization, using SGD with initial learning rate 0.05, momentum 0.9, and weight decay 1e we follow the same learning rate decay at epoch 220 and 260. For CIFAR LO-LT, 20 hyper-parameters are trained, corresponding to ly and A, of each 10 classes. For CIFARIOO-LT, ImageNet-LT, and iNaturalist, we reduce the search space with cluster sizes of 10, 20, and 40 as visualized in Figure 3(a) and as described in Section 2.2. For ImageNet-LT and iNaturalist, following previous work [53], we use ResNet-50 and SGD for the lower and upper optimizations, For the learning rate scheduling, we use cosine scheduling starting with learning rate 0.05, and batch size 128. In searching phase, we conduct 150 epoch training with 40 epoch warm-up before the loss function design starts. For the retraining phase, we train for 90 epochs, which is the same as [53] but only due to the lack of training resources we change the batch size to 128 and adjust initial learning rate accordingly as suggested by [22].
Personalized Data Augmentation (PDA). For PDA, we utilize the AutoAuggent [12] policy space and apply bilevel search for the augmentation policy. Our approach follows existing differentiable augmentation strategies (c.g. [24]); however, we train separate policies for each class cluster to ensure that the resulting policies can adjust class frequencies. Due space limitations, please see supplementary materials for further details.
Results and discussion. We compared our methods with the state-of-the-art long-tail learning methods. Table I shows the results of our experiments where the design space is parametric CE (2.1). In the first part of the table, we conduct experiments for three baseline methods: normal CE, 1 [53]. The latter choice guarantees Fisher consistency. In the second part of the Table 1, we study Algo. I with design spaces 1, A, and L& A. The first version of Algo. in Table tunes the LA loss parameter where l is parameterized by a single scalar as ly = T log(TT). The next three versions of Algo. consider tuning I, A, L& A respectively (Figure 3b-d shows the evolution of the and A parameters during the optimization). Finally, in last version of Algo. 1, the loss design is initialized with LA loss with T = 1 (rather than balanced CE). The takeaway from these results is that our approach consistently leads to
LDAM [8] and Logit Adjustment loss with temperature parameter
7

---

a superior balanced accuracy objective. That said, tuning the LA loss alone is highly competitive with optimizing A and I alone (in fact, strictly better for CIFARIO-LT, indicating Algo. does not always converge to the optimal design). Importantly, when combining && A, our algorithm is able to design a better loss function and outperform all rows across all benchmarks. Finally, when the algorithm further is initialized with LA loss, the performance further improves accuracy, demonstrating that warm-starting with good designs improves performance.
In Table 2, we study the benefits of data augmentation, following our intu- itions from Lemma 2. We compare to
Method
CIFARIO-LT
CIFARI00-LT
ImugeNet-LT
MADAO [24]
24.39
59.10
55.31
AutoBalance: PDA
22.53
58.55
54.47
the differentiable augmentation base- line of MADAO [24] which trains a single policy for the full dataset. PDA is a personalized variation of MADAO and leads to noticeable improvement across all benchmarks (most noticeably in CIFARIO-LT). More importantly, the last two lines of the table demonstrates that PDA can be synergistically combined with the parametric CE (2.1) which leads to further improvements, however, we observe that most of the improvement can be attributed to (2.1).
AutoRalance: A&d
21.39
56.84
51.74
AutoBalance: PDA, A&l
20.76
56.49
51.50
Table 2: The evaluations on personalized data optimization.
Approaches and Evaluations for Imbalanced Groups
While Section focuses on the fundamental challenge of balanced error minimization, a more ambitious goal is optimizing generie fairness- objectives. In this section, we study accuracy- fairness tradeoffs by examining the mroup-imbalancee setting.
V Setting: Imbalanced groups. For the setting with G' groups, dataset is given by S (a,, yo 9: )-> where 9i E [G] is the group wrou--membership. In the fairness literature, groups represent sensitive protected attributes. For (æ, V. D, define the group and (class, group) frequencies as
ボイ Pplg = j), and Tkj = PD k. -) for (k.j) E [K] x [G].
The group-imbalance occurs when group or (class, group) frequencies differ, i.e.. ma म > je] ;, or maX(k.j) This > min(k.)) Tk.j. A typical goal is ensuring that the prediction of the model independent of these attributes. While many fairness metrics exist, in this work, we focus on the Difference of Equal Opportunity (DEO) [23, 17]. Our evaluations also focus on binary classification (with labels denoted via +) and two groups (K = = 2). With this setup, the DEÒ risk is defined as Leco (f) = ( L+2((f). Here (f. is the (class, group)- risk evaluated on the conditional distribution of "Class & Group j". When both classes are equally relevant (rather than y +1 implying semantically positive outcome), we use the symmetric DEO:
(f) = (f) - 22,2(f) + |C (f) - 2-f(
(4.1)
We will study the pareto-frontiers of the DEO (4.1), group-balanced error, and standard error. Here, group-balanced risk is defined as 4 (J) = Ke
1 Σx Eÿ= (f). Note that this definition treats each (class, group) pair as its own (sub)group. Throughout, we explicitly set the validation loss to cross-entropy for clarity, thus we use CE, CT CE to refer to L, C Leeo-
Validation (upper-level) loss function. In Algo. 1, we set Ltair
-1- CE
Aval CEdco for varying O S Aval S 1. The parameter Aval enables a trade-off between accuracy and fairness. Within Aval, we use the subscript "val" to highlight the fact that we regularize the validation objective rather than the training objective.
Group-sensitive training loss design. As first proposed in [38], the parametric cross-entropy (CE) can be extended to (class, group) imbalance by extending typer-parameee a to [K] x [G] variables w, L, A E [[]]x[G] generalizing (2.1), (2.2). This leads us to the following parametric loss function for group-sensitive classification
((A)n))y(()))
Erain (y,9, 9, f(x); = Wyy log
(4.2)
EkE(K] (o(Aan)
Here, Wag applies weighted CE, while Ayg and lug are logit adjustments for different (class, groups). This loss function is used throughout the imbalanced groups experiments.
TTis is generalization multiple classes of the proposal in [38] for binary group sensisensstive classificaiton.
8

---

Loss function
Balanced Error
Worst (class, group) error
DEO
Cross entropy (CE)
23.38
43.25
33.75
C
20.83
36,67
20.25
Group-LA loss
22.83
40,50
29.33
Cd
19.29
35.17
25.25
0.1 CE + 0.9 CEdeo (A = 0.1)
20.06
31.67
26.25
DRO [58]
16.47
32.67
6.91
AutoBalance: Liir with Xval = 0.1
15.13
30.33
4.25
Table 3: Comparison of fairness metries for group-imbalanie experiments. The first six rows are different training loss choices, where CE Group-LA, CEdco- and DRO promote group fairness. The last row is Algo. 1, which designs training loss for the validation loss choice of 0.1 CEE + Ce We note that, DEO can be trivially minimized by always predicting the same class. To avoid this, we use mild amount of CE loss with Aval 0.1 in Algo. 1.
Baselines. We will compare Algo. 1 with training loss functions parameterized via (1-A))CE+ Creg. Here Legg is a fairness- promoting regularization. Specifically, as displayed in Table 3 and Figure 5, we will Lreg be CEE Ce and Group LA. "Group LA" is natural generalization of the LA loss to group-sensitive setting; it chooses weights Wig = 1 st to balance group frequencies and then applies logit-adjustment with
over the classes conditioned the group-membership.
Datasets. We experiment with the modified Waterbird dataset [58]. The goal is correctly classify the bird type despite the spurious correlations due to the image background. The distribution of the original data as follows. The binary classes k {-++} correspond (waterbird, landbird), and the groups [G] {1.2} correspond to (land background, water background), The fraction of data in each (class, group) pair is - 0.22, π -T 0.012, 4 0.038, and + 0.73. The landbird on the water background ({+,2)) and the waterbird on the land background (f-, 1}) are minority sub-groups within their respective classes. The test set, following [58], has equally allocated bird types on different backgrounds, i.e., Ti. 0.25. As the test dataset is balanced, the standard classification error {f) defined to be the weighted error {f) = π (f))
Implementation. We follow the feature extraction method from [58], where æ; are 512-dimensional ResNet1 8 features. When using Algo. 1, we split the original training data into 50% training and 50% validation. The search phase uses 150 epochs of warm up followed by 350 epochs of bilevel optimization. The remaining implementation details are similar to Section 3. Results and discussion. We consider var- ious fairness-related metries, including the worst (class, group) error, DEO (f) and the balanced error Eo (f).We seek to under- stand whether AutoBalance algorithm can im- prove performance
Bolamedd ctos
22.5
20
20.0
37.5
DEO
the test compared to the baseline training loss functions of the form (1 A) CE + X Log.. In Figure 5, show the influence of the parameter where Lreg is chosen to be CF CF. Group-L.A (each point on the plot represents a different A value). As we sweep across values of X, there arises a tradeoff between standard clas- sification error E() and the fairness metrics. We observe that Algo. 1 significantly Pareto- dominates alternative approaches, for example achieving lower DEO or balanced error for the same standard error. This demonstrates the value of automatic loss function design for rich class of faimess faimesss-sccing objectives.
15.0
Grop
Clas
GroupLA
12.5
Ago 3:C0ac
10.0
Standard classiicction ernir
Standand classiiictiion
Figure 5: Waterbirds fairness-accuracy tradeoff's for parametrized loss designs (1 (1-A) CE + A Ereg for different Lay choices. Group-balanced error (f( (left) and DEO (f) (right) are plotted
function of the misclassification error E(f). Algo. exhibits a noticeably better tradeoff' curve as it uses a DEO-based validation objective to design an optimized training loss function.
Next, Table 3, we solely focus on optimizing the fairness objectives (rather than standard error).
Thus, we compare against CE CE Group-LA as the baseline approaches as well as blending CE with x = 0.1. We also compare to the DRO approach of [58]. Finally, we display the outcome of Algo. 1 with Aval 0.1. While DRO is competitive, similar Figure 5, our approach outperforms all baselines for all metrics. The improvement is particularly significant when it comes to DEO. Finally, we remark that [38] further proposed combining DRO with the group-adjusted VS-Ioss for improved performance, We leave the evaluation of AutoBalance for such combinations to future.

---

5
Related Work
Our work relates to imbalanced classification, fairness, bilevel optimization, and data augmentation. Below we focus the former three and defer the extended discussion to the supplementary.
Long-tailed learning. Learning with tongttilled data has received substantial interest historically, with classical methods focusing on designing sampling strategies, such as over- or under-sampling [40, 60, 9, 41, 1, 71, 57, 68, 5, 50]. Several loss re-weighting schemes [54, 52, 25, 13, 6, 34] have been proposed to adjust weights of different classes or samples during training. Another line of work [70, 37, 53] focuses on post-hoc correction. More recently, several works [44, 35, 16, 34, 8, 13, 53, 67] develop more refined class-balanced loss functions (e.g. (2.1)) that better adapt to the training data. In addition, several works [30, 74] point out that separating the representation learning and class balancing can lead to improvements. In this work, our approach is in the vein of class-balanced losses; however, rather than fixing balanced loss function (e.g. based on the class probabilities in the training dataset), we employ our Algorithm 1 to automatically guide the loss design.
Group-sensitive and Fair Learning. Group-sensitive sensitive learning aims to ensure faimess in the presence of represeepee groups (e.g., gender, race). [7, 23, 69, 64] propose several fairness metrics as well as insightful methodologies. A line of research [4, 19] optimize the worst-case loss over the test distribution and further applications motivate (label, group) metrics such as equality of opportunity [23, 17] (also recall DEO (4.1)). [58] discusses group-sensitive learning in an over- wer-parameetrized regime and proposes that strong regularization ensures fairness. Closer to our work, [58, 38] also study Waterbirds dataset. Compared to the regularization-bait based approach of [58], we explore parametric loss design (inspired by [38] to optimize fairness-risk over validation. [17] proposes methods and statistical guarantees for fair empirical risk minimization. A key observation of our work is that, such guarantees based on training-onie -onll optimization can be vacuous in the overparameterized regime. Thus, using validvaaittiion split (e.g. our Algo 1) is critical for optimizing fairness metrics more reliably. This is verified by the effectiveness of our approach in the evaluations of Section 4. Bilevel Optimization. Classical approaches [61] for hyper-parameter optimization are typically based on derivative- frree schemes, including random search [65] and reinforcement learning [76, 3, 66, 72]. Recently, a growing line of works focus on differentiable algorithms that are often faster and can scale up to millions of parameters [46, 49, 59, 28, 48]. These techniques [45, 36. 75, 47] have shown significant success in neural architecture search, learning rate scheduling, regularization, etc. They are bilevel optimization problem: the upper and lower optimizations minimize the validation and training losses, respectively. Some theoretical guarantees (albeit restrictive) are also available [11, 21, 2. 55]. Different from these, our work focuses on principled design of training loss function to optimize mairess--sekingg objectives for imbalanced data. Here, a key algorithmic distinction (e.g. compared to architecture search) is that, our loss function design is only used during optimization and not during inference. This leads to a more sophisticated aeper-gradient and necessitates additional measures to ensure stability of our approach (see Algo 1).
typically formulated
Conclusions and Future Directions
This work provides an optimization- based approach to automatically design loss functions to address imbalanced learning problems. Our algorithm consistently outperforms, is at least competitive with, the state-of-the-art approaches for optimizing balanced accuracy. Importantly, our approach is not restricted to imbalanced classes or specific objectives, and can achieve good tradeoffss between (accuracy, fairness) on the Pareto frontier. We also provide theoretical insights on certain algorithmic aspects including loss function design, data augmentation, and train-validation split.
Potential I. Limitations, Negative Societal Impacts, & Precautions: Our algorithmic approach can be considered within the realm of automated machine learning literature ( (AutoMI.) [27]. AutoMI. algorithms often optimize the model performance, thus reducing the need for engineering expertise at the expense of increased computational cost and increased carbon footprint. For instance, our proce- dure is computationally more intensive compared to the theory-inspired loss function prescriptions of [53, 8). A related limitation is that Algo. 1 can be brittle in extremely imbalanced scenarios with very few samples per class. We took the several steps to help mitigate such issues: first, our algorithm is initialized with a Bayes consistent loss function to provide a warm-start (such as the proposal of [53]). Second, to improve generalization and avoid overfitting to validation, we reduce the hyperparameter search space by grouping the classes with similar frequencies. Finally, evaluations show that the designed loss functions are interpretable (Fig. dd,, and are consistent with theoretical intuitions.
10

---

Acknowledments
This work is supported in part by the National Science Foundation under grants CCF-20468 16, CNS-1932254, CCF-2009030, HDR-1934641, CNS-1942700 and by the Army Research Office under grant 1911N-221---011.
References
[1] Shin Ando and Chun Yuan Huang. Deep over-sampling framework for classifying imbalanced data. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pages 770-785. Springer, 2017.
[2] Yu Bai, Minshuo Chen, Pan Zhou, Tuo Zhao, Jason D Lee, Sham Kakade, Huan Wang, and Caiming Xiong. How important is the rrii-validation split in meta-learning? arXiv preprint arXiv:2010.05843, 2020.
[3] Bowen Baker, Otkrist Gupta, Nikhil Naik, and Ramesh Raskar. Designing neural network architectures using reinforcement learning. arXiv preprint 1rXiv:1611.02266 2016.
[4] Aharon Ben- Tal, Dick den Hertog, Anja De Waegenaere, Bertrand Melenberg, and Gijs Rennen. Robust solutions of optimization problems affected by uncertain probabilities. Management Science, 392) 2013.
[5] Mateusz Buda, Atsuto Maki, and Maciej A Mazurowski. systematic study of the class imbalance problem in convolutional neural networks. Neural Networks, 106:249-2593 2018.
[6] Jonathon Byrd and Zachary Lipton. What the effect of importance weighting in deep learning? In International Conference on Machine Learning, pages 872-881. 1-L-881.........M- PMLR, 2019.
[7] Toon Calders, Faisal Kamiran, and Mykola Pechenizkiy. Building classifiers with independency constraints. 2009 IEEE International Conference on Data Mining Workshops, pages 13-18.
IEEE, 2009.
[8] Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning imbalanced datasets with label-distributionnaaaar margin loss. arXiv preprint arXiv: 1906.00400074115 2019.
[9] Nitesh V Chawla, Kevin W Bowyer, Lawrence Hall, and Philip Kegelmeyer. Smote: synthetic minority over-sampling technique. Journal of artificial intelligence research, 16:321- 357, 2002.
[10] Shuxiao Chen, Edgar Dobriban, and Jane Lee. group-theoretic framework for data augmentation. Journal of Machine Learning Research, 21(245):1-71, 2020.
[11] Nicolas Couellan and Wenjuan Wang. On the convergence of stochastic bi-level gradient methods. Optimization, 2016.
[12] ED Cubuk, В Zoph, D Mane, V Vasudevan, and QV Le. Autoaugment: Learning augmentation policies from data. arxiv 2018. arXiv preprint 1805 09501.
[13] Yin Cui, Menglin Jia, Tsung- Yi Lin, Yang Song, and Serge Belongie. Class-balanced loss based on effective number of samples. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 9268 9277, 2019.
[14] Tri Dao, Albert Gu, Alexander Ratner, Virginia Smith, Chris De Sa, and Christopher Ré. A kemeel theory of modern data augmentation. In International Conference Machine Learning. pages 1528-1537. PMLR, 2019.
[15] Jia Deng, Wei Dong, Richard Socher. Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. 2009 IEEE conference on computer vision and pattern recognition, pages 245-255. leee, 2009.
[16] Qi Dong, Shaogang Gong, and Xiatian Zhu. Imbalanced deep learning by minority class incremental rectification. IEFE transactions on pattern analysis and machine intelligence, -366311367 2018.

---

[17] Michele Donini, Luca Oneto, Shai Ben-David, John Shawe-Taylor, and Massimiliano Pontil. Empirical risk minimization under fairness constraints. arXiv preprint 0rXiv:1802.08626 2018.
[18) Simon Du, Jason Lee, Haochuan Li, Liwei Wang, and Xiyu Zhai. Gradient descent finds global minima of deep neural networks. In International Conference on Machine Learning, pages 1675-1685. PMLR, 2019.
[19] John Duchi, Peter W Glynn, and Hongseok Namkoong. Statisties of robust optimization: generalized empirical likelihood approach. Mathematics of Opeeratinns Research, 2021.
[20] Vitaly Feldman. Does learning require memorization? a short tale about a long tail. In Proceedings of the 52nd Annual ACM SIGACT Symposium Theory of Computing, pages 954-959, 2020.
[21] Luca Franceschi, Paolo Frasconi, Saverio Salzo, Riccardo Grazzi, and Massimiliano Pontil. Bilevel programming for hyperparameter optimization and meta-learning. In International Conference Machine Learning, pages 1568-1577 PMLR, 2018.
[22] Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangging Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in I hour. arXiv preprint 1706.0770,, 2017
[23] Moritz Hardt, Eric Price, and Nathan Srebro. Equality of opportunity in supervised learning. arXiv preprint arXiv: 2016.
[24] Ryuichiro Hataya, Jan Zdenek, Kazuki Yoshizoe, and Hidcki Nakayama. Meta approach to data augmentation optimization. arXiv preprint arXiv:2006.0790 2020.
[25] Chen Huang, Yining Li, Chen Change Loy, and Xiaoou Tang. Learning deep representation for imbalanced classification. In Proceedings of the IEEE conference computer vision and pattern recognition, pages
2016.
[26] Frank Hutter, Holger H Hoos, and Kevin Leyton-Brown. Sequential model-based optimization for general algorithm configuration. International conference on learning and intelligent optimization, pages 507-523. Springer, 2011.
[27] Frank Hutter, Lars Kotthoff, and Joaquin Vanschoren. Automated machine learning: methods, systems, challenges. Springer Nature, 2019.
[28] Simon Jenni and Paolo Favaro. Deep bilevel learning. In Proceedings of the European conference computer vision (ECCV), pages 618 08-33,, 2018.
[29] Ziwei Ji and Matus Telgarsky. Polylogarithmic width suffices for gradient descent to achieve arbitrarily small test error with shallow relu networks. arXiv preprint arXiv: 1909. 12292, 2019.
[30] Bingyi Kang, Saining Xie, Marcus Rohrbach, Zhicheng Yan, Albert Gordo, Jiashi Feng, and Yannis Kalantidis. Decoupling representation and classifier for long-tailed recognition, arXiv preprint
2019.
[31] Shuichi Katsumata and Akiko Takeda. Robust cost sensitive support vector machine. In Artificial intelligence and statistics, pages 434 443, PMLR, 2015.
[32] Michael Kearns. A bound on the error of cross validation using the approximation and estimation rates, with consequences for the training-test split. Advances in Neural Information Processing Systems, pages 183-189, 1996.
[33] Michael Kearns and Dana Ron. Algorithmic stability and sanity- bounds for leave-one-out cross-validation. Neural computation, (1(6):1427-1)) 1999.
[34] Salman Khan, Munawar Hayat, Syed Waqas Zamir, Jianbing Shen, and Ling Shao. Striking the right balance with uncertainty. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 103-112, 2019.
12

---

[35] Salman H Khan, Munawar Hayat, Mohammed Bennamoun, Ferdous A Sohel, and Roberto Togneri. Cost-sensitive learning deep feature representations from imbalanced data. IEEE Iransactions on neural networks and learning systems, 29()):353333887 2017.
[36] Mikhail Khodak, Liam Li, Maria-Florina Balcan, and Ameet Talwalkar. On weight-sharing and bilevel optimization in architecture search. 2019.
[37] Byungju Kim and Junmo Kim. Adjusting decision boundary for class imbalanced learning. IEEE Access, 8:81674- 674-616688, 2020.
[38] Ganesh Ramachandra Kini, Orestis Paraskevas, Samet Oymak, and Christos Thrampoulidis. Label-imbalanced and group-sensitive classification under overparameterization. accepted to the Thirty-fitth Conference on Neural Information Processing Systems (NeurIPS), 2021.
[39] Alex Krizhevsky, lyy Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems, pages 1097-1105, 2012.
[40] Miroslav Kubat, Stan Matwin, et al. Addressing the curse of imbalanced training sets: one-sided selection. In Icml, volume 97, pages 179-186. Citeseer, 1997.
[41] Hansang Lee, Minseok Park, and Jummo Kim. Plankton classification on imbalanced large scale database via convolutional neural networks with transfer learning. In 2016 IEEE international conference on image processing (ICIP), pages 3713-3717. IEEE, 2016.
[42] Mingchen Li, Xuechen Zhang, Christos Thrampoulidis, Jiasi Chen, and Samet Oymak. Auto- balance source code. https //t:://gttttt //tgbtttubb.... com/ucr--eee http://ccccottm//ttaaaaaaaa /utt//uttBaaaaaaa ance, 2021.
[43] Renjie Liao, Yuwen Xiong, Ethan Fetaya, Lisa Zhang, KiJung Yoon, Xaq Pitkow, Raquel Urta- sun, and Richard Zemel. Reviving and improving recurrent back- soeek-roppggtion. In International Conference Machine Learning, pages 3082-3091, PMLR, 2018.
[44] Tsung-Y Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pages 2980-2988, 2017.
[45] Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. arXiv preprint 1 2018.
[46] Jonathan Lorraine, Paul Vicol, and David Duvenaud. Optimizing millions of hyperparameters by implicit differentiation. In International Conference on Artificial Intelligence and Statistics, pages 1540-1552. PMLR, 2020.
[47] Jelena Luketina, Mathias Berglund, Klaus Greff, and Tapani Raiko. Scalable gradient- -based tuning of continuous regularization hyperparameters. In International conference on machine learning, pages 2952-2960 PMLR, 2016.
[48] Matthew MacKay, Paul Vicol, Jon Lorraine, David Duvenaud, and Roger Grosse. Self-tuning networks: Bilevel optimization of hyperparameters using structured best-response functions. arXiv preprint arXiv: /903. 1903.03088, 2019.
[49] Dougal Maclaurin, David Duvenaud, and Ryan Adams. Gradient- hyperparameter opti- mization through reversible learning. In International conference on machine learning, pages 2112-2122. PMLR, 2015.
[50] Dhruv Mahajan, Ross Girshick. Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe, and Laurens Van Der Maaten. Exploring the limits of weakly supervised pretraining. In Proceedings of the European Conference on Computer Vision (ECCV), pages 181-196, 2018
[51] Hamed Masnadi-Shine Shirazii Nuno Vasconcelos, and Arya Iranmehr. Cost-sensitive support vector machines. arXiv preprint arXiv: 2012.
13

---

[52] Aditya Menon, Harikrishna Narasimhan, Shivani Agarwal, and Sanjay Chawla. consistency of algorithms for binary classification under class imbalance. In International Conference Machine Learning, pages 603-611. PMIR, 2013.
On the statistical
[53] Aditya Krishna Menon, Sadeep Jayasumana, Ankit Singh Rawat, Himanshu Jain, Andreas Veit, and Sanjiv Kumar. Long-tail learning via logit adjustment. arXiv preprint arXiv:2007.0753 2020.
[54] Katharina Morik, Peter Brockhausen, and Thorsten Joachims. Combining statistical learning with knowledge-basee approach: case study in intensive care monitoring. Technical report, Technical Report, 1999.
[55] Samet Oymak, Mingchen Li, and Mahdi Soltanolkotabi. Generalization guarantees for neural architecture search with train-validation split. International Conference on Machine Learning. 2021.
[56] Samet Oymak and Mahdi Soltanolkotabi. Towards moderate overparameterization. global convergence guarantees for training shallow neural networks. IEEE Journal on Selected Areas in Information Theory, 2020.
[57] Samira Pouyanfar, Yudong Tao, Anup Mohan, Haiman Tian, Ahmed Kaseb, Kent Gauen, Ryan Dailey, Sarah Aghajanzadeh, Yung-Hsiang Lu, Shu-Ching Chen, et al. Dynamic sampling in convolutional neural networks for imbalanced data classification. In 2018 IEEE conference on multimedia information processing and retrieval (MIPR), pages 112-117. IEEE, 2018.
[58] Shiori Sagawa, Pang Wei Koh, Tatsunori Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. arXiv preprint arXiv:1911.088 2019.
[59] Amirreza Shaban, Ching-An Cheng, Nathan Hatch, and Byron Boots. Truncated back- propagation for bilevel optimization. In The 22nd International Conference on Artificial Intelligence and Statistccs, pages 1723-1732. PMLR, 2019.
[60] Li Shen, Zhouchen Lin, and Qingming Huang. Relay backpropagation for effective learning of deep convolutional neural networks. In Europeun conference on computer vision, pages 467-483 Springer, 2016.
[61] Ankur Sinha, Pekka Malo, and Kalyanmoy Deb. A review on bilevel optimization: from classical to evolutionary approaches and applications. IEEE Transactions on Evolutionary Computation, 22(2):276-295, 2017.
[62] Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International Conference Machine Learning, pages 6105-61 6114. PMLR, 2019.
[63] Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-uesd birds-200-2011 dataset. 2011.
[64] Robert Williamson and Aditya Menon. Fairness risk measures. In International Conference on Machine Learning, pages 6786-6797. PMLR, 2019,
[65] Sirui Xie, Hehui Zheng, Chunxiao Liu, and Liang Lin. Snas: stochastic neural architecture search. arXiv preprint arXiv:1812.09926, 2018.
[66] Zhen Xu, Andrew M Dai, Jonas Kemp, and Luke Metz. Learning an adaptive learning rate schedule. arXiv preprint (XXx:1999.097111 2019.
[67] Han-Jia Ye. Hong- You Chen, De-Chuan Zhan, and Wei-Lun Chao. Identifying and compen- sating for feature deviation in imbalanced deep learning. arXiv preprint arXiv:2001 01388,, 2020.
[68] Xi Yin, Xiang Yu, Kihyuk Sohn, Xiaoming Liu, and Manmohan Chandraker. Feature transfer learning for deep face recognition with under-represented data, arXiv preprint 1803:v80000000 2018.
14

---

[69] Muhammad Bilal Zafar, Isabel Valera, Manuel Gomez Rodriguez. and Krishna P Gummadi. Fairness beyond disparate treatment & disparate impact: Learning classification without dis- parate mistreatment. In Proceedings of the 26th international conference on world wide web, pages 1171-1180, 2017.
[70] Junjie Zhang, Lingqiao Liu, Peng Wang, and Chunhua Shen. To balance or not to balance: A simple-yet--ffectiie approach for learning with long-tailed distributions. arXiv preprint arXiv:1912.04446, 2019.
[71] Yuan Zhao, Jiasi Chen, and Samet Oymak. On the role of dataset quality and heterogeneity in model confidence. ICML Workshop on Uncertainty & Robustness in Deep Leurning, 2020.
[72] Zhao Zhong, Zichen Yang, Boyang Deng, Junjie Yan, Wei Wu, Jing Shao, and Cheng-Lin Liu. Blockqnn: Efficient block-wise neural network architecture generation. IEEE transactions on pattern analysis and machine intelligence, 2020.
[73] Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: 10 million image database for scene recognition. IEEE transactions on pattern analysis and machine intelligence, 40(6):1452-144, 2017.
[74] Boyan Zhou, Quan Cui, Xiu-Sher Wei, and Zhao-Min Chen. Bbn: Bilateral-bramn network with cumulative learning for long-tailed visual recognition. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9719-97222 2020.
[75] Pan Zhou, Caiming Xiong, Richard Socher, and Steven CH Hoi. Theory-inspired path- regularized differential network architecture search. arXiv preprint arXiv:2006.16537, 2020.
[76] Barret Zoph and Quoc V Le. Neural architecture search with reinforcement learning. arXiv preprini 16 1.01578, 2016.
15


Reviews:
[{"review_id": "gTynGKNWSEy", "reviewer": "Reviewer_mSPX", "summary": "This paper proposes a method for long-tailed classification. The key idea is to use a generalized loss function parameterized by hyper-parameters \\alpha and to (automatically) tune \\alpha on a validation set. Results are presented on CIFAR-10-LT, CIFAR-100-LT, ImageNet-LT and iNaturalist datasets.", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n-------------\n(1) Experimental results are good.\n(2) The paper is written well and the key technical details are sufficiently described.\n\nWeaknesses\n-----------------\n(1) Unfortunately, there is limited scientific novelty in this work in my opinion. The main loss function is a straightforward generalization of the loss functions developed in previous works such as [8], [50] and others. The authors then optimize the hyper-parameters of this function on a validation dataset using the method proposed in [43]. Therefore, the overall method proposed in this paper is a highly-engineered solution with limited scientific novelty/interest.\n\n(2) The idea of fitting hyper-parameters to a validation set depends on the premise that (as stated in line 62): \"unlike training data, the validation data is difficult to fit and will provide a consistent estimator of the test objective\". However, this assumption is flawed. With millions of parameters in the actual model and a handful of hyper-parameters, it is easy to overfit to the validation set as well, specially when the validation set is small. For example, in CIFAR-100-LT, the validation set authors use has only 1 example for the tail-most class (= 500 / 100 * 20%). It is not reasonable to assume that it is difficult to overfit to such a small dataset when automatically optimizing over hyper-parameters using a gradient descent procedure.\n\n(3) Since the loss function itself (Eq. 4.2) is not novel, one could argue that the key contribution of this paper is automatically tuning hyper-parameters on the validation set. But then the method should be compared to other automatic hyper-parameter tuning methods such as Bayesian Optimization methods.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Authors have adequately addressed the limitations and potential negative societal impact.", "review_text": "", "extra_scores": {"ethical_concerns": "No ethical concerns.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3 hours"}, "extra_text": {}}, {"review_id": "YrFwmgjgUAr", "reviewer": "Reviewer_GFgA", "summary": "This work introduces a novel algorithm (AutoBalance) to address the imbalance and group sensitive classification problems and maximise the fairness-seeking objectives. The algorithm uses a bi-level optimization framework to optimize the model weights over the training data while automatically fine-tuning the loss function by monitoring the validation data. In addition, AutoBalance uses data augmentation policies personalised to subpopulations in the data.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\\+ means strength, \\- means weakness\n\n**Originality**\n\n\\+ *The paper presents a novel algorithm.* The algorithms applies ideas from bilevel optimization [43, 46, 55, 26, 45] to fairness and class imbalance learning. Unlike previous work [41, 33, 16, 32, 8, 13, 50, 63], which employs fixed class balancing loss functions based on some training data statistics, the proposed method automatically guides the loss function design and learns a more optimal set of parameters for the balanced loss.  \n\n**Quality**\n\n\\+ *Claims are well supported through experimental evidence.* \n\n\\+ *The authors are careful and honest about reporting the paper's limitations.*\n\n\\+ *The paper compares its method to recent state-of-the-art [8,50] and beats them.* \n\n\\- *Lack of more baselines.* The paper compares performance to only two baselines [8,50], and it is not clear to me why more baselines are not evaluated (e.g. [41, 33, 16, 32, 13, 63]).\n\n\\- *No error bars in the results.* There are no error bars in the result tables, making it more challenging to assess the significance of the improvements.\n\n**Clarity**\n\n\\+ *The paper is written very well, and it is easy to follow.* On the most part, the paper adequately informs the reader. \n\n\\- *Some clarifications about the experiments are needed.* Table 1 does not have any $PDA$ component in Algo. 1, does this mean that no $PDA$ was applied in those experiments? (if $PDA$ was used, did the authors also apply it to the baselines?) Similarly, did experiments in Table 2 apply $LA\\~\\text{init}$? And lastly, does Table 3 show results for \"Algo 1. $ \\alpha \\leftarrow \\Delta \\\\& l, LA~\\text{init}, PDA$\"? \n\n\\- *A possible inconsistency in the results* but I may also be misunderstanding something (see above for my confusion). Specifically, if I am correct in understanding, the diagonal results in Table 3 show the performance for \"Algo 1. $ \\alpha \\leftarrow \\Delta \\\\& l, LA\\~\\text{init}, PDA$\" which have a higher error rate than both \"Algo 1. $ \\alpha \\leftarrow PDA, \\Delta \\\\& l $\" in Table 2 and \"Algo 1. $ \\alpha \\leftarrow \\Delta \\\\& l, LA\\~\\text{init}$ \" in Table 1. This suggests that, in fact, $PDA$ and $LA\\~\\text{init}$ degrade performance when applied together. If my understanding is correct, this undermines the second contribution point (lines 73-77) and should be explicitly clarified in the paper. If my understanding is wrong, why are the results in the diagonal in Table 3 different from either the bottom row of Table 1 or Table 2? \n\n**Significance**\n\n\\+ *This work aims to address a timely problem of fairness-seeking optimization, and the paper will be relevant to the community.*\n\n___\n### Post rebuttal update\n___\nI have read other reviewers' comments and the authors' rebuttals, and I am inclined to keep my original score of 7 with increased confidence of 4, condioned on the additional baselines being included in Table 1 in the main paper. The rebuttal addresses the clarity issues, and the additional results show that the method is at least as good as other baselines [28][63][41] in the Cifar-10-LT setting. The performance advantage over [28][63] in the ImageNet-LT dataset is much more obvious. Although results for [41] in the ImageNet-LT setting were not provided, likely the method would also be beaten like in the Cifar-10-LT setting. I think this work is good, addressing a timely problem and it will be relevant for the NeurIPS conference. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The paper adequately and fairly outlines its limitations in a dedicated section. ", "review_text": "", "extra_scores": {"ethical_concerns": "n/a", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "MQOeoYQQ2vE", "reviewer": "Reviewer_FVKf", "summary": "The paper proposes and empirically validates a way to use - and tune - high-dimensional hyperarameters affecting training loss, which are used for fairness-seeking test objectives.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "To the best of my knowledge, the proposed method is novel. My experience is primarily in high-dimensional bilevel optimization, and the related work looks adequately cited there.  I am less able to comment on related work long-tailed learning or group-sensitive and fair learning.\n\nThe paper's bilevel optimization framework seems technically sound.  The gradient estimates seem reasonable, and I’m particularly impressed by the authors results with the final re-training phase with fixed optimal hyperparameters.  \n\nA contribution the authors did not emphasize, but I think is present is the following. It is common to neglect this training phase, and many models fail to re-train to reasonable performance with the final hyperparameters due to hysteric optimization effects.  The hysteric effects are particularly strong when tuning the loss.  However, the authors circumvent this with their loss parameterization. It would be interesting to try to generalize these techniques to more general setups than fairness, or understand this phenomena more abstractly. For example, by building on Lemma 2.\n\nI think the paper suffers from various issues with clarity.\n\nThe text from 207-219 is unclear and the notation seems inconsistent.  Is \\theta^* a function or a point?  Where are derivatives evaluated at?  What do you mean by “Also, observe …. Computed by taking the gradient”?\n\nYou discuss the importance of a train-val split in 220-233, but I do not see a discussion of how you tuned the split for this problem?  You also discuss the effects of overparameterization, but its not clear to me what kind of sizes of alpha, theta, train size, val size, etc. are actually used in practice.\n\nI find it difficult to read objects like table 1, table 4 due to the dense notations.  I would appreciate a notation table summarizing it. \n\nThe paragraph from line 39-58 is long and dense.  Perhaps the ideas should be modularized.  The results and discussion from 273-299 is similar.\n\nMany components of line 252-267 could probably be moved to the appendix.\n\nI believe the results are reasonably significant because they propose a useful way to introduce high-dimensional hyperparameters controlling the loss that still work even after retraining.  The results seem useful for fairness seeking objectives too. Practitioners are likely to use the method, because it is simple to understand, does not look difficult to implement, does not introduce excessive computational burden, and seems to work reasonably well. The paper also claims to advance the state-of-the-art, which seems supported by the experiments to the best of my knowledge.\n\nMy score is 7, because I think the paper makes good contributions to learning losses with fairness seeking objectives. My confidence is a 3, because I am not an expert on fairness seeking objectives. The score would be improved to an 8 if the authors largely improved the clarity.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have adequately addressed the potential negative societal impacts of their work.\n\nThe authors present reasonable limitations.", "review_text": "", "extra_scores": {"ethical_concerns": "I do not see ethical issues with the paper.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "7"}, "extra_text": {}}, {"review_id": "5CVVpR4zxTg", "reviewer": "Reviewer_oNxf", "summary": "This paper studies learning in settings with imbalanced data or under-represented subgroups in the data, where the target objective is fairness-based. E.g. in the setting of imbalanced data, the target objective is to maximize the balanced test accuracy, and in the setting with minority subgroups, target objectives studied are balanced (w.r.t. subgroups) test accuracy, and Difference of Equal Opportunity (DEO) risk. \n\nThe paper is motivated by the fact that many existing methods for imbalanced datasets rely on training theoretically-motivated modifications to the cross-entropy loss, where there are multiplicative and additive adjustments applied to the predicted logits for each class. This paper proposes a bi-level optimization procedure to directly learn these modifications, by directly learning the parameters governing the multiplicative and additive adjustments to the class logits. The bi-level optimization optimizes a classifier on a loss defined with specific instantiations of the adjustment parameters in an inner loop, and the outer loop optimizes some desired test objective (e.g., class balanced loss) on a separate holdout validation set. The paper also proposes learning class-specific data augmentation, with the motivation that minority classes are regularized more strongly than majority classes. \n\nThe empirical results evaluate the algorithm on imbalanced class and imbalanced subgroup settings.n They show that their algorithms can outperform existing methods in several evaluation metrics.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "In summary, this is a strong paper with clear writing, a clean and well-motivated algorithm, and impressive empirical performance. More details below:\n\nOriginality: The methods in this paper are novel and related work is well-cited.\n\nQuality: Claims are well-supported by strong empirical results. \n\nClarity: The paper is generally clearly written and easy to follow. \n\nSignificance: The results in this paper serve as a nice way to encompass some of the more theoretically-motivated prior work which needed to mathematically derive and justify different modifications to the cross entropy loss. In contrast, this paper observes that directly learning these modifications is possible, given the low dimensionality of the parameter space for cross entropy loss adjustments. This is a nice idea and a neat application of AutoML to the imbalanced learning setting, and could hopefully motivate to other applications of AutoML in related scenarios. \n\nSome questions/concerns for the authors:\n\n-- The main concern is that the experiments for the imbalanced group setting are only performed for a single dataset -- it would be nice to see experimental results for additional datasets. \n-- What is the comparison of runtime for the proposed algorithm v.s. prior work? Also, what is the schedule for retraining the parameters after the search phase has concluded (e.g, learning rate, # of epochs, etc.)\n-- How sensitive are the results to the exact choice of parameters $\\alpha$ found by the search phase?\n-- For the imbalanced group setting, is there any extension of DEO to the case with multiple groups in a class? ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes. Authors address limitations and societal impact in the conclusion section of their paper. ", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "098iIviRZNR", "reviewer": "Reviewer_WkxY", "summary": "The paper demonstrates how to jointly optimize parametrized cross entropy and model parameters and shows improvement on datasets with class imbalance.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The central theme of paper is to optimize the model parameters on training data and parametrized cross entropy on validation set. The model itself can be optimized using simple gradient based approaches, more sophisticated approaches from literature are used for computing parametrized cross entropy on the validation set. Additionally data augmentation for minority samples is proposed to increase classifier margin from low population samples. Authors perform extensive experimentation on the cifar, imagenet and inaturalist imbalanced datasets.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "1. Paper contributions state that it \"automatically designs a training loss function\". However the actual training loss is cross entropy only with multiplicative and additive parameters. Since the actual loss is a variation of bias+weighted cross entropy this needs to be mentioned in both abstract and contributions in clear terms. There are other works out there which use different kind of loss functions to optimize for class imbalance [1]- they should be cited.\n2. The papers is written in a complicated way and is difficult to read and understand. Simple statements have been made into complex math jargon, and a lot of places terms are first used in equation and then defined later.\n3. Imbalanced data of only images is considered - other modalities should be considered\n4. There are many small and large datasets which are not modified to create LT datasets - for example cubs and flowers [1]. The paper would greatly benefit by analyzing performance on such datasets. \n5. \"Hyper\" gradient computation on L_up is critical to implementation of paper - this needs to appear in main text not just supplementary. Its also not completely clear how to implement it.\n6. Paper uses data augmentation (previously proposed epsilon ball perturbations - applied to low population classes). Upon first inspection the data augmentation seems to create a bigger impact than the primary algorithm. This needs to be highlighted properly.\n\n\n[1] Dubey, A., Gupta, O., Guo, P., Raskar, R., Farrell, R., & Naik, N. (2018). Pairwise confusion for fine-grained visual classification. In Proceedings of the European conference on computer vision (ECCV) (pp. 70-86).", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a bi-level optimization framework to design a training loss for long-tail and group-fair learning. The reviews attracted a a lot of back-and-forth discussion with the authors, and I appreciate the authors for providing very detailed responses and additional experimental results.

While there were concerns raised about the proposed approach being prone to overfitting to the validation sample, I think the authors have satisfactorily explained how they take precautions to avoid it. I think Reviewer mSPX's concern about there being limited novelty does have some merit, but given that this is one of the first few works to successfully engineer an elegant loss-tuning procedure for long-tail learning and that the experimental results are significant, I would recommend an accept. 

Having said this, I strongly urge the authors to use the feedback provided to improve their manuscript, and in particular include the following promised additions to the final version:
- a detailed discussion on the risk of over-fitting and the precautions taken to avoid it (do report the validation errors, perhaps in the appendix)
- comparison to other Bayesian optimization approaches
- comparison to the additional baselines mentioned by Reviewer GFgA

Finally, here are some additional references on prior work on learning loss functions for specialized tasks, which may be of some relevance:

https://arxiv.org/pdf/1905.10108.pdf

https://arxiv.org/pdf/1803.09050.pdf
Decision: Accept (Poster)