Distributionally Robust Post-hoc Classifiers under Prior Shifts
Authors: Jiaheng Wei
Harikrishna Narasimhan
Ehsan Amid
Wen-Sheng Chu
Yang Liu
Abhishek Kumar
Venue: ICLR 2023 poster (2023)
Keywords: Distributional robustness
post-hoc scaling
group robustness
class imbalance
spurious correlations

Abstract: The generalization ability of machine learning models degrades significantly when the test distribution shifts away from the training distribution. We investigate the problem of training models that are robust to shifts caused by changes in the distribution of class-priors or group-priors. The presence of skewed training priors can often lead to the models overfitting to spurious features. Unlike existing methods, which optimize for either the worst or the average performance over classes or groups, our work is motivated by the need for finer control over the robustness properties of the model. We present an extremely lightweight post-hoc approach that performs scaling adjustments to predictions from a pre-trained model, with the goal of minimizing a distributionally robust loss around a chosen target distribution. These adjustments are computed by solving a constrained optimization problem on a validation set and applied to the model during test time. Our constrained optimization objective is inspired from a natural notion of robustness to controlled distribution shifts. Our method comes with provable guarantees and empirically makes a strong case for distributional robust post-hoc classifiers. An empirical implementation is available at https://github.com/weijiaheng/Drops.


Published as a conference paper at ICLR 2023
DISTRIBUTIONALLY ROBUST POST-HOC CLASSIFIERS UNDER PRIOR SHIFTS
Jiaheng W UC Santa Cruz
Harikrishna Narasimhan Google Research
Ehsan Amid Google Research
Wen-Sheng Chu Google Research
Yang Liu UC Santa Cruz
Abhishek Kumar Google Research
ABSTRACT
The generalization ability of machine learning models degrades significantly when the test distribution shifts away from the training distribution. We investigate the problem of training models that are robust to shifts caused by changes in group-priors The presence of skewed train-
the distribution of class-priors ing priors can often lead to the models overfitting to spurious features. Un- like existing methods, which optimize for either the worst or the average per- formance over classes or groups, our work is motivated by the need for finer control over the robustness properties of the model. We present an extremely lightweight post-hoe approach that performs scaling adjustments to predictions from pre-trained model, with the goal of minimizing distributionally robust loss around chosen target distribution. These adjustments are computed by solving constrained optimization problem on validation and applied the model during test time. Our constrained optimization objective is inspired from a natural notion of robustness to controlled distribution shifts. Our method comes with provable guarantees and empirically makes a strong case for distri- butional robust post-hoc classifiers. An empirical implementation is available https: / g thub. com we i ji Daheng/Drops
INTRODUCTION
Distribution shift, a problem characterized by the shift of test distribution away from the training distribution, deteriorates the generalizability of machine learning models and is major challenge for successfully deploying these models in the wild. We are specifically interested in distribution shifts resulting from changes in marginal class priors or group priors from training to test. This is often caused by a skewed distribution of classes or groups in the training data, and vanilla empirical risk minimization (ERM) can lead to models overfitting to spurious features. These spurious features seem to be predictive on the training data but do not generalize well to the test set. For example, the background can act spurious feature for predicting the object of interest in images, e.g., camels in desert background, water-birds in water background (Sagawa et al. 2020).
Distributionally robust optimization (DRO) (Ben-Tal et al.) 2013 Duchi et al.) 2016 Duchi & Namkoong 2018 (aaawa et al. 2020) is a popular framework to address this problem which formu- lates a robust optimization problem over class- group-specific losses. The common metrics of in- terest in the DRO methods are either the average accuracy or the worst accuracy over classes/groups (Menon al. [02T; Jitkrittum al. 2022 Rosenfeld et al. 2022 Piratla et al. 2022; Sagawa al.) 2020 Zhai et al. [021]] Xu et 2020; Kirichenko et al.) (022). However, these metrics only cover two ends of the full spectrum of distribution shifts in the priors. We are instead motivated by the need to measure the robustness of the model at various points on the spectrum of distribution shifts.
To this end, we consider applications where we are provided target prior distribution (that could either come from practitioner default to the uniform distribution), and would like to train
Work done during an internship at Google Research, Brain Team.
Correspondence Jiaheng Wei <jiahengweii edu,, Abhishek Kumar <abhishk @ google.com>.

---

Published as a conference paper at ICLR 2023
model that is robust to varying distribution shifts around this prior. Instead of taking the conven- tional approach of optimizing for either the average accuracy or the worst-case accuracy, we seek to maximize the minimum accuracy within ó-radius ball around the specified target distribution. This strategy allows us to encourage generalization on a spectrum of controlled distribution shifts governed by the parameter 6. When ò 0, our objective is simply the average accuracy for the spec- ified target priors, and when ò + oo, it reduces to the model's worst-case accuracy, thus providing a natural way to interpolate between the two extreme goals of average and worst-case optimization.
To train a classifier that performs well on the prescribed distributionally robust objective, we propose a fast and extremely lightweight post-hoc method that learns scaling adjustments to predictions from a pre-trained model. These adjustments are computed by solving a constrained optimization problem on validation set, and then applied to the model during evaluation time. A key advantage of our method is that it is able to reuse the same pretrained model for different robustness requirements by simply scaling the model predictions. This is contrast to several existing DRO methods that train all model parameters using the robust optimization loss (Sagawa et al., 2020; Piratla et al. 2022 which requires group annotations for the training data and requires careful regularization to make it work with overparameterized models (Sagawa et al. 2020). On the other hand, our approach only needs group annotations for a smaller held-out set and works by only scaling the model predictions of pre-trained model test time. Our method also comes with provable convergence guarantees. We apply our method on standard benchmarks for class imbalance and group DRO, and show that it compares favorably the existing methods when evaluated on range of distribution shifts away from the target prior distribution.
BACKGROUND
We are primarily interested in two specific prior shifts for distributional robustness of classifiers. In this section, we briefly introduce the problem setting of the two prior shifts and set the notation.
Class-Level Prior Shifts. We are interested in a multi-class classification problem with instance space X' and output space [m]] {1....,m) m. Let DD denote the underlying data distribution over X x [m]. the random variables of instance X and label Y satisfy that (X, ~ D We define the conditional-class probability my, (z) = IP(Y X = x) and the class priors Ty P(Y v). note that Ty = E [v, (x)) We use = -
to denote the uniform prior over classes.
Our goal is then to learn a multi-class classifier h : X' - [m]] that maps an instance x E X to one of m classes. We will do so by first learning scoring functiio f X Am that estimates the conditional-clas probability for a given instance, and construct the classifier by predicting the class with the highest score: h(r) = arg maxje[mi (z)). We measure the performance of scoring function using loss function l [m]] x A.. + IR+ and measure the per-class loss using (f) E [e(y, [e(v, f(x)) y = i].
Let [ y:) Y be set of training data samples. The empirical estimate of training set prior is n; == 1, Jen [ i) where 1(-) is the indicator function. In class prior shift, the class prior probabilities at test time shift away from f. A special case of such class- prior shifts includes lass-imbalanced learning (Lin et al.] 2017 Cui et al. 2019; Cao et af.] 2019 Ren et al., 2020) Menon al. (02I)) where is a long-tailed distribution while the class prior at test time is usually chosen to be the uniform distribution. Regular ERM tends to focus more on the majority classes the expense of ignoring the loss of the minority classes. Recent work (Menon al. 202T) uses senperature-scalle logit adjustment with training class priors to adapt the model for average class accuracy. Our method also applies post-hoc adjustments to model probabilities, but our goal differs from (Menon et al., 2021) as we care for varying distribution shifts around the uniform prior and the scaling adjustments are learned using a held-out set to optimize for a constrained robust loss.
Group-Level Prior Shifts. The notion of groups arises when each data point (z, y) is associated with some attribute a E A that is spuriously correlated with the label. This is used to form m |A] x |Y| groups the cross-product of |A| attributes and YY| classes. The data distribution D is taken to be a mixture of m groups with mixture prior probabilities π, and each group- condiciont distribution given by Dj. j E [m]. In this scenario, we have training samples [ y/) ) drawn i.i.d. from D. with empirical group prior probabilities f. For skewed group-prior probabilities म. regular ERM is vulnerable to spurious correlations between the attributes and labels, and the accuracy degrades
2

---

Published as a conference paper at ICLR 2023
when the test data comes from shifted group prior (e.g.. balanced groups). Domain-aware methods typically assume that the attributes are available for the training examples and optimize for the worst or average group loss (Sagawa al. 2020 Piratla et al. 2022). However, recent work (Rosenfeld et al. 2022 Kirichenko et al. 2022) has observed that ERM on skewed group priors is able to learn core features (in addition spurious features), and training linear classifier top of ERM learned features with a small balanced held-out set works quite well. Our proposed method is aligned with this recent line of work in assuming access only small held-out set with group annotations. However, we differ in two aspects: (i) Our method is more lightweight and works by only scaling the model predictions post-hoc during test time, (ii) The scaling adjustments are learned to allow more control over the desired robustness properties than implicitly targeting the worst or average accuracies as Kirichenko et al. 2022).
Evaluation Metries under Prior Shifts. Typical evaluation metrics used under prior shifts are:
Mean: This evaluation metric assigns uniform weights for each class or group, measuring the average class- group-level test accuracy.
Worst: This evaluation metric measures the performance on the worst class or group, and fails to examine the effectiveness of proposed methods on the remaining classes.
Stratified: To overcome the issue of worst-case metric, stratified metrics are sometimes used that divide the classes into three strata and report average accuracy in each stratum, specifically (i) Head: average accuracy on subsets of classes that contain more than a specified number (e.g.. 100) of training samples; (ii) Torso: average accuracy on classes that contain, e.g., 20 to 100 samples; (iii) Tail: average accuracy on tail classes.
The mean and worst metrics are limited in that they probe the model robustness only on the two ends of the full spectrum. The stratified metric looks into three strata of classes, but the design of strata more heuristic and doesn't 't allow for principled interpolation between mean and worst metrics.
DISTRIBUTIONAL ROBUSTNESS OBJECTIVE UNDER PRIOR SHIFTS
We assume that we are given a target prior distribution (that could either come from a practitioner or default to the uniform distribution), and our goal is to train model that is robust to varying distribution shifts around this prior. To this end, we consider an objective that naturally captures this goal and is based on the weighted sum of class-level (or group-level) performances on the test data, ie., Lie(m) g:Ace, where Acc; indicates the accuracy of the class/group
8-worst accuracy:
min gEAm
Σ g:Acc,, s.t. r) sE/m]
(1)
Here Am denotes the (m -iddmenniional probability simplex and r E Am is a specified reference (or target) distribution. The d-worst accuracy seeks the worst-case g-weighted performance with the weights constrained to lie within the 8-radius ball (defined by the divergence D Am x A. + R) around the target distribution For uniform distribution reduces to the mean accuracy for 5 = O and the worst accuracy for 5 - xo. The objective interpolates between these two extremes for other values of 5 and captures our goal of optimizing for variations around target priors instead of more conventional objectives of optimizing for either the average accuracy at the target prior the worst-case accuracy. The divergence constraint in the 5-worst objective is convex in g for several divergence functions (e.g., f-divergence, Bregman divergence). and allows for efficient measurement as well (if the per-class accuracies are known) using standard packages such as CVXPY (Diamond & Boyd 2016).
and any choice of divergence D,
4 ROBUST POST-HOC CLASSIFIERS UNDER CLASS PRIOR SHIFTS
In this section, we propose a Distributionally RObust PoSt-hoc method, DROPS, which enables the reuse of pre-trained model for different robustness requirements by simply scaling the model predictions. In alignment with the earlier 5-worst objective, we aim to learn a classifier h with scoring function f as follows:
Goal:
min
max Σ 9: (XX)= #Y), s.t. D(g.r) 56
(2)
Am DE
3E(m]
3

---

Published as a conference paper at ICLR 2023
4.1 BAYES-OPTIMAL SCORER
Recall that we measure the performance of a scoring function using loss function é [im] x Am + R+. and the per-class loss using (4()) E [())) y ij. To facilitate theoretical analysis, we consider the scenario where the target distribution r is the uniform prior u m. .... 1. We
then interested in minimizing the following robust objective optimization:
min DRL(f:8) fmm
min
91 4()) (3)
f.X
geo(8)
(g Am Díg. u) S 8} for some and divergence function D Am Am R.. We first derive the Bayes-optimal solution to the learning problem in equation
where (9
Theorem (Bayes-Optimal scorer). Suppose f(y, z) is a proper loss that is convex in its second argument and ((g.)) convex in 9- Let 5 be such that G(ō) is non-emply. The optimal solution to equation takes the following form for some g" E G6))
fg (2)
m.y (2)..
Ty
The proof is provided in Appendix A.1 and would use in its intermediate step, the following standard result for cost-sensitive learning:
Lemma 2. Suppose E(y, 2) a proper loss and (,()) = E [é(y, f(x)) = For any fixed g E
Am, the following is the minimizer the objective >i 9s E,() over all measurable functions x
Ami fy (x) x 2z
Rv (1)()).
4.2 POST-HOC APPROACH
Given a pre-trained model f) : X' Am that estimates the conditional-class probability function we seek to approximate the Bayes-optimal classifier. To do so, we write the Lagrangian form of equation as: L(f.g, A) Em 9: (f)(((()((()))))))))))))) u) -)) We next show the optimization w.r.t. the Lagrangian form has the following equivalent unconstrained problem, specified in Proposition
min max min ()) f:X f:X-+Am
g A20
Proposition 1. The equivalence of two optimization tasks:
min
D)L(F:8)
min
max min C(f.g,A)
f:x+Aam
f:X
geAm X20
To understand why the above two optimization tasks are equivalent, we point out that if the constraint violated, then the minimizer over would yield an unbounded objective. The maximizer over g. as result, will always choose a that satisfies the constraint.
Build upon Proposition ! one can then solve the equivalent min-max problem by alternating be- tween gradient-descent update on X, an Exponentiated Gradient (EG)-ascent update (Kivinen & Warmuth 1997) on g, and a full minimization over f:: miny.x D 9; (f), the optimal solu- tion for which given from Lemma 2 by (ar) o 8x (z)) Thus, the pre-trained model f), we propose to make post-hoc adjustments over 7 to make the classifiers more robust to prior shifts. We construct the "post-shifted" classifier by predicting the class with the "post-shifted" highest score for each .r:
DROPS: h(x) E arg max (x()). 2E(m]
Intuitively, model prediction class is up-scaled the class is assigned with a large weight gis is of a small prior カ.
An Empirical Alternative. Given sample x, the "post-shifted" classifier takes the form h(z) e argmax;e|mm gi (x). The softmax activation function then maps the per-class mode prediction
4

---

Published as a conference paper at ICLR 2023
logit, denoted by Logit,(2)) into the corresponding probability. Specifically, for i E [mm] we have exp(Logit (x))
(x) = E expit (m))' Thus, an empirical alternative the proposed DROPS logit-
adjustment approach where, given a sample æ, for fixed class weights * we have:
DROPS: h(x) E argmax yE mi Ty
fv
h(x)) E argmaxye(m) Logit, + log( 9V Mg (2)
4.3
CONVERGENCE OF POST-HIOC CLASSIFIER
In practice, we only have access to an empirical estimate of the Lagrangian form computed using validation set {(11, 91)..... (Zn. }n
g(f.9,)) Σ 9: &(f) X(D)) u)
-8), -
(5)
where 2, = D (e.)) fx)) and म, = (z. E : = = We can then approx-
imately solve the saddle-point problem in equation 4 by repeating the following three steps for T number of iterations:
Step 1: updating A Given the step size > O, we perform gradient updates on À through:
(++1) ( nA VL(( (0), (() 1 11 D)g(),
For KL divergence, the updated (++1) amounts to ( + ( - n0 - E. log( Pi )). We
can clip if the update falls out of the desired range: (++1) cl (A nd Dgg u))
for some R >
Step 2: updating (0) Given the step size ng
> 0, we perform gradient updates on through:
(++1) < 96 (1) exp ng VgL((f ,0))
9
To obtain the updates for g, we consider the fixed (0) E R2 and adopt a simplified version as (++1) exp ((((11) empirically. Pi
Step 3: scaling the predictions. In this step, we apply the post-hoe shifts on the classifier's prediction through ((++)(())
d. (x),, in which we use the pre-trained estimate r of the
conditional-class probabilities.
The final scorer returned is an average of scorers from each iteration: f(x) ET= (z).
We now provide a convergence guarantee for this procedure. For a fixed pre-trained model f) X Am, we denote the class of post-hoc adjusted functions derived from it by
F=
B (x) En
(m) (r)
/:00
1,3 Am
and we let [F] denote measure of complexity of this function class.
Theorem 3. Fix 8 > O. Suppose l(y, z) be proper loss that is convex and L Lipischhit in its second argument, and Ely, 2) < for some B, > 0. Suppose D(g, u) S BD. some BD > 0, and D is convex and -ppLipschitz in its first argument. Furthermore, suppose max 5, Z, for some
2 > 0. Then when n 2 87 log(2n (2m//2//))))) 3/), and we set T = (n), R = 2B, 7/ 5, dΛ BDV= and
log(m)) fg 28,L+RLD n
we have with probability at least 1 a over draw of validation sample D", the classifier returned [(x)) = DT ((0 satisfies:
(v /lgg(m(Fl// (
DRL(F;0) < min DRL(F;0) + +0 -
+ E. [1](()) - v(=))ll))
5

---

Published as a conference paper at ICLR 2023
The proof is provided in Appendix A.3 The complexity measure [F] can be further bounded using, for example, standard covering number arguments (Shalev- Shalav-S & Ben-Davis 2014). Notice that the convergence to the optimal classifier is bounded by two terms: the first a sample complex- ity term that goes to 0 as the number of validation samples n - oo; the second term measures how well the pre-trained model rj is calibrated, i.e., how well its scores match the underlying conditional- class probabilities.
4.4 ROBUST POST-IIO CLASSIFIERS UNDER GROUP PRIOR SHIFTS
We now introduce the variant of our approach to address group prior shifts. We are interested in the performance of the trained classifiers with additional attribute information available in a held-out validation set. Note that the settings of class and group prior shifts differ in the knowledge of group information at validation and test time. Thus, the first variant of DROPS under group prior shifts is the same as the class prior setting: DROPS completely ignores the per-example group information and post-shifts the model predictions using only class labels. We consider another variant where we have access to the attribute information during both validation and test. This scenario can actually arise practice when the attribute information is readily available for test examples, such as device or sensor type in the case of data coming from different devices/sensors. In this case, the attribute- specific class priors take the form of na. = P(y = ila). DROPS can be naturally adapted to this setting by learning multiple sets of scaling adjustments, one for each attribute type, and using the scaling adjustment corresponding to the attribute of the test example at prediction time. We provide the discussion, and the form of the Bayes-optimal classifier for this setting in Appendix B
5 RELATED WORK
laasanemaee Learning. Most work in class-imbalanced learning is typically interested in generalizing a uniform class prior when the training data has skewed or imbalanced class prior. Existing solutions to the class imbalanced learning problem could be categorized into three major lines: (1) Information augmentation methods, which make use of additional information such as open set data (Wei et al.] (021). adopt a transfer learning approach to enrich the representation on the tail classes (Liu al. 2020; Yin et al.) 2019). use advanced data-augmentation tech- niques (Perez& & Wang. 2017]] 7; Shorten & Khoshgoftaar 2019); (2) Module improvement methods, i.e., the decoupled training on the head/tail classes (Kang et al. 2019 Chu et al. 2020 Zhong
let al. (222). or through an ensemble way to make use of multiple networks with different exper- cise/concentration (Zhou et al. 2020 Guo & Wang 2021 Wang et al. 2020 (3) The most related to our work are the class re-balancing based methods, which mitigate the impact of class glass-imbalanced data by adjusting the logits using the class prior (Menon et al., (02T). or align the distributions of the model prediction and set of balanced validation set (Zhang et al. (021b). or modify the loss
values by referring to the label frequency (Ren et al.] 2020), sample influence (Park et al. 202T),
among many other robust loss designs (Amid et al. | 2019 Wei & Liu| 2021 Zhu et al. 2021) and
re-weighting schemes (Kumar & Amid| [021]] Cheng et al. 2021; Wei et al. 2022). Label shift is related problem setting where the training class prior is not imbalanced but the class prior shifts during test time with p(x) y) staying the same, and the goal is to mitigate the effect of this shift (Lipton et al. 22018] Azizzadenesheli al. 2019 Alexandari et 2020). However, our goal differs from these methods as we are primarily interested in generalization at worst case variations around the target distribution.
Group Distributional Robustness. It has been observed that classifiers learned with regular ERM are vulnerable to spurious correlations between the attributes and labels, and tend to perform worse when the test data comes from shifted group prior. Most methods for group distributional robustness are interested in the average or worst group performance. Several prior works utilize the group information at the training time (Sagawa et al. 2020: Piratla et al.] 2022). Recent works consider a more practical setting where the classifier does not have access to the group information at the training time, i.e... data re-balancing (Idrissi et al. 2022) or re-weighting high loss examples
(Liu et al. 2021). }. or logit-correction (Liu et al. 2023). vision transformer (VIT) models (Ghosal et al. 2022). It was also observed recently that ERM is able to learn features that can be reused for group robustness by training a linear classifier using a balanced held-out set (Kirichenko et al, [022] Rosenfeld et al. 2022). Most relevant to us is AdyShift (Zhang al. 2021a), which mitigates the

---

Published as a conference paper at ICLR 2023
impact of label shifts by optimizing distributionally robust objective function with respect to the model parameters. Out work is in similar vein in assuming access to only a held-out set with group annotations, however method is more lightweight and allows to optimize for varying worst- case perturbations around the target distribution of interest using only post-hoc scaling of model predictions. Other related work includes CGD (Piratla et al. (022) that proposes learning paradigm for optimizing the average group accuracy, and Invariant risk minimization (IRM) (Arjovsky et al.) 2019 Ahuja al., 2020) that aims to learn core features using data from multiple environments for mitigating the spurious correlations.
EXPERIMENTS
In this section, we empirically demonstrate the effectiveness of our proposed method DROPS, for the tasks of class-imbalanced learning and group distributional robustness.
6.1 EXPERIMENTS ON CLASS-IMBALANCED LEARNING
We consider the dass-imbalanceed task to illustrate the robustness of DROPS under class prior shifts. For CIFAR-10, and CIFAR- 100 datasets, we down-sample the number of samples for each class to simulate the class-imbalance as done in earlier works (Cui et al., 2019 Cao et al., (2019). We define
the imbalance ratio as p min [m] My
mAX (mm) Fy. and experiment with p E [10, 50, 100].
To demonstrate the effectiveness of DROPS, we compare the performance of our proposed method with several popular class-imbalanced learning approaches, including: Cross- pooss-Entropy (CE) loss, (CB) loss (Cui et al.) 2019). LDAM (Cao al. 2019), (Menon et al. (022). and AdvShift (Zhagg 202 on the original test data. All methods are trained with the same architecture (PreAct- ResNet 18 (He et al. 2016)) with 5 random seeds, same data augmentation techniques, the same SGD optimizer with momentum of 0.9 with Nesterov acceleration. All methods share the same initial learning rate of 0.1 and a piece-wise constant learning rate schedule of [10-2 10-3, 10041 at [30, 80, 110j epochs, respectively. We use batch size of 128 for all methods and train the model for 140 epochs. A balanced held-out validation set is utilized for hyper-pane parameter tuning. All baseline models are picked by referring to the 5 1.0-worst case performances on the validation set, which is made up of the last 10% of the original CIFAR training dataset. DROPS obtains the optimal post-hoc shifts under variety of Suain parameters, which the perturbation hyper- paremmerreeter & in We take the divergence D to be the KL.-divergence.
Focal Loss (Lin et al. 2017).
Balanced- Softmax (Ren et al. 2020),
ct al.
the Lagrangian of equation 4
Performances Comparisons on the Generic 5-Worst Case Accuracy. Experiment results in Table T demonstrate that DROPS can not only give promising performance under the reported 5-worst case accuracy (for& 8 1.0), it also outperforms other methods in the worst case accuracy, and remains competitive the mean accuracy as well (performs best the CIFAR-10 setting).
To further investigate the robustness of each method under different level of prior shifts, we visualize the performance of each method under list of ó within the 8-worst case accuracy. For the uniform distribution r = u, the 5 values recover both the mean accuracy (with = 0) and the worst accuracy (for large enough 8), and interpolates between these two metrics for other values of 8. For DROPS, we train using ōunin E [0.5, 1.0, 1.5, 2.0] (CIFAR-10) and 8yain E [0.5, 1.0, 2.0, 3.0, 4.0) (CIFAR- 100) with KL-divergence while evaluating for the aforementioned range of &eval values. We also evaluate using Reverse-KL divergence to examine the behavior under a different divergence function from that used in learning the scaling adjustments with DROPS. Figure ! jillustrates the robustness and effectiveness of our proposed DROPS method: specifically, in each sub-figure of Figure [ the r-axis indicates the value of 8, and the y-axis denotes the corresponding d-worse case aceuracy. For experiment results on CIFAR-10 10 datasets (Ist row), the curves of DROPS under different 8pain are consistently higher than the other baselines, indicating that with the increasing of perturbation level of the distribution shifts (from left to right in each sub-figure). DROPS is more robust to the distribution shift, As for CIFAR- 100 dataset, we do observe that optimizing for the controlled worst case performance may lead to trade-off if in the worst case accuracy and averaged test accuracy, i.e., Logit-Ad) Adj is more competitive than DROPS in the measure of the averaged accuracy, while DROPS still suffers less from the increase of perturbation level.
7

---

Published as a conference paper at ICLR 2023
Performance comparisons on class-imbalanced CIFAR datasets: mean + std of averaged class accuracy, 8 = 1.0-worst case accuracy, and worst class accuracy of runs are re- ported. Best performed methods (corresponds to the averaged accuracy of 5 runs) in each setting are highlighted in purple. And we defer the paired student t-lest between each baseline method and DROPS in Appendix (Table to demonstrate the robustness of DROPS.
Table 1:
CIFAR-IO (Averaged)
CIFAR-100 (Averaged)
Method
p=
p=
p= 100
p=
p= 50
p
Cross Entropy
86.5120.17
75.7420.97
69.28:0.94
55.5910.50
43.39:0.76
37,96=027
Foral
75.3414.65
58.5543.34
46,23:2.11
51.5241.93
32.68:0.755
26.53-0.74
LDAM
86.63-0..33
76.00+0.77
68.81+1001
55.4240.67
43.0340..8
38.35=0063
Balanced- Balanced-Softmax
87.18+0.24
78.76+0.44
35.55++..35
56.37+0.54
45.10++.51
41.21-0.23
AdvShift
86.33+0.22
74.74:1..54
69.4140.83
55.38-1047
42.71+00.6
38.44=0.58
Logit- Logit-Adjust
0..0
82.60+10.22
77.49+10.1
59.18+0.30
47.19+1.33
43.03=00.4
Logit-Adi (post-hoc)
88.98=0.2
82.48:0.75
77,9440.50
59.33+0.51
47.77=1449
43.68=1.08
DROPS (6 0.9)
89.17=0.24
83.12:00.45
80.15+00.50
59.69:10.9
47.83+00880 80
43.20=0.78
CIFAR-1O (6
L.0worrt case acc)
CIFAR-100 (6
1.0-worst case acc)
Method
p=10 10
p=
p= 100
p=
p= 50
p=
Cross Entropy
79.9820.13
19.4811775
43.3043.63
27.0610.53
11,24:0.97
6.52:10.18
Focal
64.9625..67
43.041..05
30.46:1.113
23.62:#.119
6.32+10.57
3.42::0.24
LDAM
79.68:0.611
59.5843.06
42.944242
1..44
11.26:1060
6.62:10.71
Balanced-Softman
80.72=0.51
68.3410.55
10.0033.33
30,36:0.81
17.06:0:47
12.00=0.52
AdvShift
80.70+0.44
60.36++0.90
48. 18
29.37-+0.94
14.6540.62
10.71
Logit-Adjust
82.18:0.46
13.50++1009
63.62:+1.70
18.02:11.8
1.
11.66=1.44 1.40
Logit-Adj (post-hoc)
82.52+0.47
13.8411.69
64.54
33.46:11.26
1.09
04,34-0.53
DROPS (6 0.9)
86,20:0.34
79 40-
75.46:10.44
44.56:00.52
30. 30.12:0066
25.58
CIPAR-IO
CIFAR-IOO (Worst)
Method
p= 10
p 50
100
p
p 50
p 100
Cross Entropy
78.29=0088
55.32++3.58
36.00++6.11 11
9,79+2.57
1.38-0.91
0.00+10.00
Focal
63.6925.99
40.73:14.98
28. 18.16433.94 94
8.65+3.58
0,00100.00
0.00:10.00
LDAM
76.87+00.9
56.57433118
35.99422.7
10.5742.65
1.82+:1.16
0,27-+0.50
Balanced- Balanced-Softtaas Softmax
78.86=00.84
67.11=10.98
22.63++0.29
16.17++3.85
6.05++2.24
2.58:11442
AdvShift
79.60=0.933
17.71
44.70:12.49
13.38:2..36
3.18:1.29
2.05++1.32
Logit-Adjust
8..54=1.11
71.24:1.722
61.1842077
15.9943.52
5.441131
187
Logit-Ad (post-hoc hoc)
8141+1224
72.00+1..2
63.164+2.33
16,21+3.93
4.47+1.71
3.39+1.46
DROPS (6 0.9)
12.2221.30
76.33.0.89
71.6211.4
25.98133955
11.65
8.61
Cress Entropy
Cress Entropy Sall-Mae LogitAAi
Worst Caze Ace
Logi-Adi
AdgitAdilPosthhoo hoc)
Logi: Adj hoc!
DROP5 (Srin 0.51
DROPS (Eris 0.51
Worrs
DRDPS (6re
(dhsss 0.7)
DROPS (dran 0.7)
DROPS (Bua. 0.91
DRDPS (huu. - 0.91
20
gencel
(ReceeseeR)) divergence)
2033
p-131
Cross opy
Cross Entropy
Balanced Sort Max
Balanced Max
Logi-Ac
(Worr) Cu02 Acc
Logit-Ag
DROPS
DROPS
DRCPS 16... +0.9)
DRGPS (8... 0.9)
DRCPS (0eum 1..0
DRCPS (6ean 1.00
DROPS
DROPS 1.11
XL divergencet
((Revarse-Kt -l. divergerce)
Figure 1: PerformanceRRousttnes of methods under different perturbation 8-worst case accuracy. (1st row: CIFAR-I0 with imbalance ratio = 100, adopt the KL-divergence measure (left) and Reverse-KL. divergence (right) for the 8-worst case accuracy calculation; 2nd row: CIFAR-100 with imbalance ratio = 100, adopt the -dvveerreeee measure (left) and Kl (right) for the 5-worst case accuracy calculation.)
6.2 EXPERIMENTS ON GROUP ROBUSTNESS TASKS
We consider the group robustness tasks to show the robustness of DROPS under group prior shifts. For waterbirds and CelebA datasets, we obtain the datasets described in (Sagawa et al. 2020).
We compare the performance of our proposed method with several popular group-Dro approaches, ie., Just-train-twice (JTT) (Liu al. 2021), Group Distributional Robustness (G-DRO) (Sagawal
8

---

Published as a conference paper at ICLR 2023
et al. 2020 data balancing strategies (SUBG) (ldiiss et al. 2021), last-layer re-training (DFR) Kirichenko et al. 2022). Among the baseline methods, ERM. JT. G-DRO, and SUBG are trained with the same architecture (ResNet 50 (He et al. 2016)) for 5 runs, the base model for DFR and our approach DROPS also used the same ERM pre-trained model. Note that the knowledge require- ment over group information and validation set differs among the reported methods, clarify the differences in the column "Group Info" of TaTlle 2
As for the performance of DROPS in the group DRO datasets, since there are only two classes within the two datasets, we define a single post-hoc scalar as w, which is considered to be hyper- parameter by scaling the classifier's prediction on class Note that the spurious features (minority groups among each class) tend to have a significant impact on the ERM performances by referring to the imbalanced group distribution. We use DROPS to learn post-doc corrections for both ERM trained model and for the DFR (Kirichenko et al. 2022) model that additionally retrains the last layer of the ERM-trained model with a balanced held-out set. We refer to the post-hoc corrected DFR as DROPS* in the result tables.
Performance Comparisons on various 6-Worst Accuracies. Post-hoc scaling on the ERM model (referred as DROPS in Table 2 improves the robustness of the model as measured at various d values (including = and - 20). DROPS also outperforms JTT on the CelebA dataset. When applying post-hoc scaling to better pre-trained model, i.c., with DFR (Kirichenko al. 2022) that re-trains the last layer on the validation set, DROPSR outperforms all baseline methods in most settings, with the performance improvement especially clear the CelebA dataset.
Table 2: Performance comparisons on Waterbirds and CelebA: averaged accuracy with training prior weight and uniform weight, 6-worst accuracy for several ô perturbations, and worst group-level test accuracy are reported. The Group Info column indicates whether the group labels are available to the methods on train/validation sets. The symbol / means the method also re-trains the last layer on the validation set (w/o needing the group information for training).
Method
Group Info Train
(Train prioe) Averaged
Waterbirds 5= 0.0.--Worst
uniform prior)
Val
Averaged
== 0.05-Worst
5 6= 0.2-Worst
8 = 0.5-Worst
Worst
ERM
X
98.08=0.20
88.09220.90 9.0
84.31=1.24 244
82.71=1.41
80,51=1.68
76.65:2.25 2.2
70.65=3.32
JTT
93.05-0.36
89 89.56.10.69
88.5910.74
88.24100.75
87.81
87.13.0.855
86.1811.08
DROPS
97.95=0.16
88.55:11.15
15.50:11.51
84.33=1.64
82.1==179
80.41:2.00
77.14=2.15 2.
G-DRO
93.03=0.34
91:67:0:22
91.23-+0.33
91.06=0.34
90:3==0:0
90:44:::22
09.85=0.73
SUBG
91.97-0.50
90.05-+0.44
89.4640.40
89.24+0.39
88.98-0.40
88 59-+0.49
124-0.66
DFR
95.83
93.45.10.49
92.77.10.48
10.51
92.16.10.55
91.58.10.7
90.72.0.91
DFR
93.17=1.80 1.30
93.29:10.80
02.9810.84
92.66-0.88
92.70:10.87
92.42:10.88
92.01:0.855
DROPS
93.42.10.61
93.0810.81
92.9810.90
92.76.11022
92.45.1123
9.,9911.56
Method
Train
Group Inlo
Val
(Train price) Averaged
Averaged
0. D.0G-Worst
CelebA
0. 1-I-Wttst
unitorm prior)
8 0.2- DorsWorst
8 0.5-Worst
Worst
ERM
95.33:00:1 12
81. 31:88::1600
73.78-52.36 36
20,53=2.60
65.97=3315 15
$7.8243.90 98
44,89:55.5 30
JTT
87.78-0.73
85.04++-1.0 .05
83.34-+11 1.35
82,89 1.49
82.35+1.71
81.51+2.12
79.71+22.85
DROPS
90.67.00.76
89.05.10888
86.80.11.33
1.55
8..67.11888
82.5912.500
39.441...58
G-DRO
92.59=0.87
90.95:10.52
90.18:10:46
89.8 89.8920.43
89.53=0037
88.97=00226
88.21=0017
SUBG
91.09=0.62
88.75:+0.34
87 07.69+0.311
87,27=0.41
86.72-0..73
85 80:10.0000
84.45=0028
DFR
90.36-0.64
89.25-+0.73
87 97.30-00.77
86.53+ 1.09
85.50+ 1.26
83.77+1.63
81.22+2..11
DFR
/V
90.90=0.79 79
91. 13++0.40
90.32++0.54
90.02=0.58
89.64=0065
89.05-+0:77
88.25= 1.03
DROPS*
WV
95.69=0041
93.59:20.36
92.64220446
92.28:0.51
91.8220.58
91.10:10.6
90.1920.81
7 CONCLUSIONS
We study the problem of improving the the distributional robustness of a pre-trained model under controlled distribution shifts. We propose DROPS, fast and lightweight post-hoe method that leams scaling adjustments to predictions from a pre-trained model. DROPS learns the adjustments by solving a constrained optimization problem on a held-out validation set, and then applies these adjustments to the model predictions during evaluation time. DROPS is able to reuse the same pre- trained model for different robustness requirements by simply scaling the model predictions. For group robustness tasks, our approach only needs group annotations for smaller held-out set. We also showed provable convergence guarantees for our method. Experimental results on standard benchmarks for class imbalance (CIFAR-10. CIFAR-100) and group DRO (Waterbirds, CelebA) demonstrate the effectiveness and robustness of DROPS when evaluated on a range of distribution shifts away from the target prior distribution.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGEMENT
The work is done during JW's internship at Google Research, Brain Team. JW and YL are partially supported by the National Science Foundation (NSF) under grants IIS-2007951, IIS-2143895, and the Office of Naval Research under grant NO0014 1-0014-2---2 JW is also partially supported by the Center for Research in Open Source Software UC Santa Cruz, which is funded by donation from Sage Weil and industry memberships. We are thankful to Kevin Murphy for providing several helpful comments on the manuscript.
REFERENCES
Kartik Ahuja, Karthikeyan Shanmugam, Kush Varshney, and Amit Dhurandhar. Invariant risk min- imization games. In International Conference on Machine Learning, pp. 145-155. PMLR, 2020.
Amr Alexandari, Anshul Kundaje, and Avanti Shrikumar. Maximum likelihood with bias-corrected calibration is hard-to-beat at label shift adaptation. In International Conference on Machine PMLR, 2020.
Learning, pp.
Ehsan Amid, Manfred KK Warmuth, Rohan Anil, and Tomer Koren. Robust bi-tempered logistic loss based on bregman divergences. Advances in Neural Information Processing Systems, 32, 2019.
Martin Arjovsky, Léon Bottou, Ishaan Guirajani, and David Lopez- Invariant risk minimization. arXiv preprini urXiv:/907.028800 2019.
Kamyar Azizzadenesheli, Anqi Liu, Fanny Yang, and Animashree Anandkumar. Regularized learn- ing for domain adaptation under label shifts. In International Conference on Learning Represen- tations, 2019.
Aharon Ben-Tal, Dick Den Hertog, Anja De Waegenaere, Bertrand Melenberg, and Gijs Rennen. Robust solutions of optimization problems affected by uncertain probabilities. Management Sci- ence, 59(2):341-333 2013.
Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning imbalanced datasets with label-distribution--aarr margin loss. Advances in neural information processing systems, 32, 2019.
Hao Cheng, Zhaowei Zhu, Xingyu Li, Yifei Gong, Xing Sun, and Yang Liu. Learning with instance- dependent label noise: A sample sieve approach. In International Conference Learning Rep- resentations, 2021. URL
net/ fett://frruml1=22yyyyyym.m
Peng Chu, Xiao Bian, Shaopeng Liu, and Haibin Ling. Feature space augmentation for long-tailed data. In European Conference on Computer Vision, pp. 694-710. Springer, 2020.
Yin Cui, Menglin Jia, Tsung- Yi Lin, Yang Song, and Serge Belongie. Class- bass-balanced loss based on effective number of samples. In Proceedings of the IEEF/CVF conference computer vision and pattern recognition, pp. 9268-9277, 2019.
Steven Diamond and Stephen Boyd. CVXPY: A Python-embeddded modeling language for convex optimization. Journal of Machine Learning Research, 17(83):1-5, 2016.
John Duchi and Hongseok Namkoong. Learning models with uniform performance via distribution- ally robust optimization. arXiv preprint arXiv:1810.08750, 2018.
John Duchi, Peter Glymn, and Hongseok Namkoong. Statistics of robust optimization: A generalized empirical likelihood approach. arXiv preprint 1rX1v.16100003225, 2016.
Soumya Suvra Ghosal, Yifei Ming, and Yixuan Li. Are vision transformers robust to spurious correlations? arXiv preprint arXhv:2003.09125, 2022.
Hao Guo and Song Wang. Long-tailed multi-label visual recognition by collaborative training uniform and re- ra-baeane samplings. In Proceedings of the IEEE/CVÉ Conference on Computer Vision and Pattern Recognition, pp. 15089-15098, 2021.
10

---

Published as a conference paper at ICLR 2023
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Badr Youbi Idrissi, Martin Arjovsky, Mohammad Pezeshki, and David Lopez-Paz. Simple data balancing achieves competitive worst-grop--acerracy arXiv preprint arXiv:2110. 2021.
Badr Youbi Idrissi, Martin Arjovsky, Mohammad Pezeshki, and David Lopez-Paz. Simple data balancing achieves comperitive worst-group-accuracyy In Conference on Causal Learning and Reasoning, pp. 336-351. PMLR, 2022.
Wittawat Jitkrittum, Aditya Krishna Menon, Ankit Singh Rawat, and Sanjiv Kumar. Elm: Embed- ding and logit margins for long-tail learning. arXiv preprint arXiv:2204. 13208, 2022.
Bingyi Kang, Saining Xie, Marcus Rohrbach, Zhicheng Yan, Albert Gordo, Jiashi Feng, and Yannis Kalantidis. Decoupling representation and classifier for long-tailed recognition. arXiv preprint arXiv:1910.092117 2019.
Polina Kirichenko, Pavel Izmailov, and Andrew Gordon Wilson. Last layer re-training is sufficient for robustness to spurious correlations. arXiv preprint arXiv:2204.0295 2022.
Jyrki Kivinen and Manfred K Warmuth. Exponentiated gradient versus gradient descent for linear predictors. information and computation, 132(1):1-63, 1997.
Abhishek Kumar and Ehsan Amid. Constrained instance and class reweighting for robust learning under label noise. arXiv preprint arXiv:2111.05425 2021.
Tsung- Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE international conference on scomputer vision, pp. 2980-2988, 2017.
Zachary Lipton, Yu-Xiang Wang, and Alexander Smola. Detecting and correcting for label shift with black box predictors. In International conference on machine learning, pp. 3122-3130. PMLR, 2018.
Evan Z Liu, Behzad Haghgoo, Annie S Chen, Aditi Raghunathan, Pang Wei Koh, Shiori Sagawa, Percy Liang, and Chelsea Finn. Just train twice: Improving group robustness without training group information. In International Conference on Machine Learning, pp. 6781-6792. PMLR, 2021.
Jialun Liu, Yifan Sun, Chuchu Han, Zhaopeng Dou, and Wenhui Li. Deep representation learning on long-tailed data: A learnable embedding augmentation perspective. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 2970-2979, 2020.
Sheng Liu, Xu Zhang, Nitesh Sekhar, Yue Wu, Prateek Singhal, and Carlos Fernandez- Avoiding spurious correlations via logit correction. In International Conference on Learning Representations, 2023. (tRLhttss::// http:::::///eee.eeiieeee net/ wwr.mmiiiiooo wiw.ruiiii-aaaaCSSSSSSS 5BaqCFV55LL
Aditya Krishna Menon. Sadeep Jayasumana, Ankit Singh Rawat, Himanshu Jain, Andreas Veit, and Sanjiv Kumar. Long-tail learning via logit adjustment. In International Conference on Learning Representations, 2021. URL (tRlstttsst.. wpw.e. net. forum? o= wot.orrrmmmnnnooeeeeoe 3
Harikrishna Narasimhan, Andrew Cotter, and Maya Gupta. Optimizing generalized rate metrics with three players. In Advances in Neural Information Processing Systems, 2019,
Seulki Park, Jongin Lim, Younghan Jeon, and Jin Young Choi. Influence-balanced loss for im- balanced visual classification. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 735-744, 2021.
Luis Perez and Jason Wang. The effectiveness of data augmentation in image classification using deep learning. arXiv preprint arXiv: 2017.
Vihari Piratla, Praneeth Netrapalli, and Sunita Sarawagi. Focus on the common good: Group dis- tributional robustness follows. In International Conference on Learning Representations, 2022.
URL htttps:://pparevviieeeee www.ee/wwwwweeeeeeoooooeeeeeeee FARV 2VF

---

Published as a conference paper at ICLR 2023
Jiawei Ren, Cunjun Yu, Xiao Ma, Haiyu Zhao, Shuai Yi, et al. Balanced meta-softmax for long- tailed visual recognition. Advances in neural information processing systems, 33:4175-4186 2020.
Elan Rosenfeld, Pradeep Ravikumar, and Andrej Risteski. Domain-adjusted regression or: Erm may already learn features sufficient for duttof-distribution generalization. arXiv preprint arXiv:2202.06886, 2022.
Shiori Sagawa, Pang Wei Koh, Tatsunori Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generaliza- tion. In International Conference on Learning Representations, 2020.
S. Shalev-Shwartz and S. Ben-David. Understanding Machine Learning: From Theory to Algo- rithms. Cambridge University Press, New York, NY, USA, 2014.
Shai Shalev-Shwartz et al. Online learning and online convex optimization. Foundations and trends in Machine Learning, 4(2):107-194, 2011.
Connor Shorten and Taghi M Khoshgoftaar. A survey on image data augmentation for deep learning. Journal of big data, 6(1):1-48, 2019.
Maurice Sion. On general minimax theorems. Pacific Journal of Mathematics, 8(1):171-176, 1958.
Serena Wang. Harikrishna Narasimhan, Yichen Zhou, Sara Hooker, Michal Lukasik, and Aditya Kr- ishna Menon. Robust distillation for worst-class performance, 2022.
Tao Wang, Yu Li, Bingyi Kang, Junnan Li, Junhao Liew, Sheng Tang, Steven Hoi, and Jiashi Feng- The devil is in classification: simple framework for long-tail instance segmentation. In Euro- pean conference on computer vision, pp. 728-744. Springer. 2020.
Hongxin Wei, Lue Tao, Renchunzi Xie, and Bo An. Open-set label noise can improve robustness against inherent label noise. Advances in Neural Information Processing Systems, 34:7978-7992, 2021.
Jiaheng Wei and Yang Liu. When optimizing $fS-divergence is robust with label noise. In Interna- tional Conference on Learning Representations, 2021. URL htttps://ppereviiii wwt.nrrreeeweeeeeeeeeeeeee
Forum?id=West 1 CoRV015
Jiaheng Wei, Hangyu Liu, Tongliang Liu, Gang Niu, Masashi Sugiyama, and Yang Liu. To smooth or not? when label smoothing meets noisy labels. In International Conference on Machine Learn- ing. pp. 23589-23611 PMLR, 2022.
Ziyu Xu, Chen Dan, Justin Khim, and Pradeep Ravikumar. Class- weighte classification: Trade- offs and robust approaches. In International Conference on Machine Learning, pp. 10544-10554. PMLR, 2020.
Xi Yin, Xiang Yu, Kihyuk Sohn, Xiaoming Liu, and Manmohan Chandraker. Feature transfer learn- ing for face recognition with under-represented data. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 5704-5713, 2019.
Runtian Zhai, Chen Dan, Zico Kolter, and Pradeep Ravikumar. Doro: Distributional and outlier ro- bust optimization. In International Conference Machine Learning, pp. 12345-12355. PMLR, 2021.
Jingzhao Zhang, Aditya Krishna Menon, Andreas Veit, Srinadh Bhojanapalli, Sanjiv Kumar, and Suvrit Sra. Coping with label shift via distributionally robust optimisation. In International Con- ference Learning Representations, 2021a. URL https:: 1d=Bt
https://openzeviertt http//ppeeeeeeeeeeeeeeeeeeeeeeee net forum?
Songyang Zhang, Zeming Li, Shipeng Yan, Xuming He, and Jian Sun. Distribution alignment: A unified framework for long-tail visual recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 2361-2370, 2021b.
12

---

Published as a conference paper at ICLR 2023
Zhisheng Zhong, Jiequan Cui, Shu Liu, and Jiaya Jia. Improving calibration for long- taigttiledd recog- nition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 16489-16498, 2021.
Boyan Zhou, Quan Cui, Xiu-Shen Wei, and Zhao-Min Chen. Bbn: Bilateral-branch network with cumulative learning for long-tailed visual recognition. In Proceedings of the IEEE/CVE confer- ence on computer vision and pattern recognition, pp. 9719-9728. 2020.
Zhaowei Zhu, Tongliang Liu, and Yang Liu. A second-order approach to learning with instance- dependent label noise. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 10113-10123, 2021.
13

---

Published as a conference paper at ICLR 2023
A PROOFS
A.1 PROOF OF THEOREM 1
The proof is adaptation of similar result in Wang (022). We first prove Lemma
Proof of Lemma
2
We first expand the objective:
Σ 91 (0-4(()
Σ
9. Euly=i (e(i, f(x))
Σ
9: E. (x)
L(i, f())))
1=1
1=1
i=1
Given that is proper loss, we have that the minimizer of this objective takes the form:
21 17p (1)
f((x)
91 1= 11 (x)
We now ready to prove Theorem
I
Proof of Theorem
The min-max problem in equation
can be expanded as:
min DRL(F;6) fx
min f-X-AA (EG(S)
max
E (x)
E(y, f()))
(6)
Ty
w(g.f)
Note that the objective ff is clearly linear in g (for fixed f), and with é chosen to be convex in fxx (for fixed g). i.e., w(g,kff Kfi + (1 - x)f2) S kw(g, +1) k)w(g, f2), Vfi, J2 X + Am, & E [0, 1]. Furthermore, given that divergence Dg..-) is convex in g. we have that G(8) is convex compact set, while the domain of f is convex. follows from Sion's minimax theorem Sion 1958) that:
min max w(g.f) f:a-aa 9EB(8)
max min w(g,f). SEG(A) f:X-am
Let (g', fr) be such that:
S E argmin (RL(J;8) argmin max w(g, f); fiX' -m gEG(8)
g" E argmax min w(g. f). SEG(8) f..-A
Such an f exists because DRL(J;5) takes bounded value when of DRL(J;6) yields value below that; because (RL(/;8) 2 0 and convex in f. such minimizer exits. Similarly, also exists because G(o) is compact set, and for any fixed g. miny.x-Am. w(g, f) is bounded above (owing to the existence of a minimizer from Lemma 2
n. and any minimizer
We now show that (g", ()) is saddle-point for equation 6 i.c.,
w") max w(g.f") gEG(A)
min fa--Am
w(g",11
(8)
To see this, notice that:
w(g",5))
max w(g. SEG(8)
min max w(g. f) = max min
w(g.f)
f.x f-X+Aa gEG(8)
gEG(8) fix+Am
min w(g", f) < wlg", .f),
+
where able swap the min and max in the second step using equation 7
14

---

Published as a conference paper at ICLR 2023
We thus from equation 8 that f is minimizer of (g",, f), i...
Lú .E (mn((x) C(v, f()))
f E argmin f:X+Amm
Ty
Following Lemma
we further have that for any I X:
(x) x g ny()))
Ty
A.2 PROOF OF PROPOSITION
Proof. Now we prove the equivalence of two optimization tasks:
min f=X+Am
(RI.(5;))
min max min L(S. A).
f=X+Am gEAm A20
Remember that
min DRL(f:8)
max fim-AAm gEG(E)
min
95 4(f).
f.X-Am
1=1
firstly show that any (f* g") given by (L.H.S) yields the optimum of the equation (R.H.S), for some X² Note that for any f.g E G()), for any such that (gg,r) S 6, we have:
Σ
9.((f) N Σ gl(())
min A&8
max
(ED
sE(m)
RE(m)
Plugging f",g" into the R.H.S, we then have:
= Σ ((D6))))
tE(m]
Since D(g", r) S 6, 3X E R+ such that the optimization task X = arg min L(f', g", A). To show (f ,9"" X²) returns the optimum of the R.H.S, we prove by contradiction and assume that there exists f' g' such that: C(f', ,¹, < C(f g", X*). This indicates that:
Σ SALF") - Eotc(os) - (D(g - X (D(
-deer)~ E0te(r) <0.
(Due to Complementary Slackness Condition)
This contradicts with the fact that E. 2 E.fe(()) Thus, (f",g", ("'"","") mum of R.H.S.
(x)) returns the opti-
We next prove that any (f",g* .', given by R.H.S yields the optimum in L.H.S. Due to Comple- mentary Slackness Condition, we have: X (D(g", -) O. Note that X R+. we then have D(g", r) - 8 = 0 and the constraint L.H.S is satisfied. Thus, we have: R.H.S = Σ; g (f*). Again, if there exists g', f' such that the L.H.S is minimized, where g' + g"", + f*, then have (g', f', X' 0) which satisfies: L(f', g', X) < L(f", ,g", X*), which contradicts with the argmin
pairs (J ( A*).
Thus, we finished the proof.
A.3 PROOF OF THEOREM 3
The proof builds prior convergence results in constrained and robust optimization (Narasimhan
et al. 2019 Wang et al. 2022). We will find it useful to define the following quantities:
(l Σ gi l(f))s
L((f.))= = - ge i.(f);;
15

---

Published as a conference paper at ICLR 2023
C2(g. - u) 8).
We further define averages of different iterates Ā 7 7 X00 and 9 = E g(0)
We would also find the following lemmas useful. The first is bound on our estimate of the class priors.
Lemma 4. Under the assumptions in Theorem 1 with probability at least 1 a/2 over draw of validation sample S ~ D
1 fy
Vy [m].
27
Proof. The proof follows from a direct application of Chernoff's bound (along with union bound over all classes), noting that min Fy 2 Z and 2 8Z log(2m/a).
Throughout the proof, we will assume that the statement in the above lemma holds with probability least 1 a//2.
Our second lemma shows that the equivalence between the saddle-point optimization in equation and the original constrained optimization problem in equation 3 still holds when minimize the Lagrange multiplier only over bounded set:
Lemma 5, Under the assumptions in Theorem we have any f X - Am:
min max x) max x geAm.
91 -((f).
g gEG(d)
Proof. Let X E argmin max L( g. A) be the λ-minimizer over all non-negative R. Such A20 g
minimizer exists for the following reason. Owing to the continuity of D we know there exits at least one g' Am for which D(g, u) = 6 and therefore we have that the minimization objective is bounded:
max ,(
= Σ 4400) S ZB.
gEAm
remains to be shown that < To do this end,
g" E argmax
Σ 91 E,())
gEAm D(g.u)
We note that:
m
97 ((()) min max C(f', .) max g())) max Σ ge
-))
geAm.
ge
Choose ÿ such that D u) 8/2, which exits thanks to the continuity of D. Upper bounding the max on RHS in the above equality by substituting G. we get:
Σ of 4(F) S
Σ
9: 4(F) - ('(((g, - 5) Σ 445) X-8/2.
which gives us:
2 m
X
91 4()))
Σ of ((()))
< ZB: R.
as desired.
L
The lemmas below follow from Lemma and standard results in online convex optimization Shaley-Shwart et al. 2011].
16

---

Published as a conference paper at ICLR 2023
Lemma 6. Under the assumptions in Theorem and fnr nl 2R, logiml with probabil-
ity at least a.2 over draw of S ~ DD,, the sequence of iterates
g(T) satisfies:
Σ L 9, 1 T
1
Σ
T
L , ,0, 2(2B, RLD)
log(m)
max gEAm
T
T
Proof. The proof follows from standard convergence result for the aroonettiaed-ggradiee descent algorithm noting that
|V,L(f
max +
,
Br max 1 + RID S 27 + RLD:
where we use the bound on the class prior estimates in Lemma The last step holds with probability
least 1 a/2.
Lemma 7. Under the assumptions in Theorem and for пл
R BDVT
with probability at least
1 a/2 over draw of validation sample S D the sequence of iterates 20) ........(7... satisfies:
X
T
T
RBD VT
Σ (,(g(),,10))
min AE(ORR
Σ (2la
T
T
Proof. The proof follows from standard convergence result for the online gradient descent algorithm noting that (2 X) < (") u)) - 8 < Bn and A| < R.
The following lemma provides generalization bound for the empirical I Lagrangian.
Lemma 8. Under the assumptions Theorem with probability at least a over draw of validation sample S ~ DD, for all & E (T):
(C (0) (0, 2 "0 (0)) o ( log(m| (F/(m))///)
71
Proof. For any E [T]. we first bound the left-hand side by:
m
(1)) - "0) <
Σ
(
(40) (40)-(())
max (4f)) 2() (9) sc(m]
9
Further define i,
T. My Cly f())).
We then can bound the above difference
(() - 4()) in the above bound using:
(C0) &(C)
< (((()) (6(()) &QNI
[(((i,(()) 1(y= =)] i)]
1
Σ
((i, f(;)) 1 = i)
+
n
e
((1,(
n
# n
1 Σ
B
E
{i,f((;;)
1(15 =1)
m Azl
17

---

Published as a conference paper at ICLR 2023
We know S z/- Further, applying Lemma 4 we can bound #.. We therefore have with probability at least 1 - -a/2 over draw of S D":
|4()) - RCDI < 2 B
1 = =i]
Σ Mi, (i,S(z,))
10y = 1) + 222 T At|
An application Hoeffding's inequality to both the above terms, noting that the loss lyy =) is bonded, together with union bound over all I and class E [m]] gives us with probability at least 1 - a/2 over draw of S ~ D", for all f E JF and i E [m]]
(a(()) - &00 &00 so so (V
(V
[og(m|F1/a)
Plugging back into equation 9 and taking a union bound over both the high probability statements completes the proof.
We will additionally use the following regret bound for the f f-minmiization step:
Lemma 9. Under the assumptions in Theorem 7 for fixed g Am, with probability least ak over draw of S ~ D",
(1( min () s BeZ E, [1((()) m((]l1a] + o [lo(mm/a
(V
fEF
Proof. We first expand L1 in terms of the conditional-clas. probability function v)z):
f(ff.) Σ 9: (a(t)
Σ 9i E. f(t,))
We know from Lemma 2 that the minimizer of Li(f .) over all f takes the form f*(x
my (x))
We also know from Lemma that ((0) is the minimizer of similar objective where is replaced by the pre-trained model f/:
((
gi Σ
E. ((.
n
We would like to bound:
L1 (9) 9) fr(",, .9) (1(10) :0) (1(00 .9) + E1('00 .9) (2(5" .9)
< () .g) - (f(t) .g) + Ci .g) - ()(
E
Σg 9i
n
(x(2)
(1(2))
lli, (x))
+ E, Σ gi
[T((z)
f((x))
eli, (())
T
#
tr:
1=1
[x(x))
(x(x))
Eg
gi
(f(i. (10 - fl (x)))
#
i=1
m (x( 1 (x))
< E,
max 9: 18(i, (()) Eli, (*)(
(x(x))
(x)
< B, E,
f
(xx))
(x)
(x)
(x)
B, EE
#
18

---

Published as a conference paper at ICLR 2023
m
(x)
(x(
B₁ max sc[m] T
Fe [1|](x) (x)))] + E
R3
1
- B,Z Es [1lm(()) - (x)1l + ER [I]
ll max velm
B&Z E2 [11:(() (z)]) (1) max rE[m] n
#sl
where in the second step, we use the fact that f()) minimizes L1; in the second- last step, apply Holder's 's inequality; in the last step, we use the fact that gi E [0. 1], n; S ž and Ely. 2) S By.
We know गi S 1. Further, applying Lemma 4 we can bound म.. We have with probability at least 1 a/2 over draw of S D"
Li g) Li( 9) s B,Z E. [lb(()) n [x) + 2z max |T: -
iE[m]
An application of Hoeffding's inequality to the last term completes the proof.
We are now ready to prove Theorem 3
(V log(ml F Fl/o)
Proof of Theorem Let Kin = O
+ F2 [1((()) - m(())lli) We start by combin-
ing Lemma 6 with the generalization bound in Lemma least 1 - a over draw of validation sample S S~77 D",
8 from which we have with probability at
1 T
Σ
C(F() (LL'),g,A'))
C(Fe) 9 (2)) o
log(m)
log(m|Fl/a)
max gEAm
T
T
1=1
t=1
log(m)
log(mL FF//a)
Σ
21 ())
Σ
") ,((t))
o
T 1=1
11
T
log(m|F1/a
π Σ (,(r0)g))))
- Σ
")
((00))
72
(10)
where we have used the fact that T = O(n).
Applying Lemma
7 to the right-hand side of equation 10 with T = O(n), we have with probability
at least 1 - a.
T Σ
T (1((0)
T
(Vogg(mr/F//0
max DE T
L
+
min xE
L2 A) + 0
T
T
1=1
1=1
1=1
min
1 Σ
L ())
min xE
Σ () A)
Km
T
T
t=1
t=1
min 7K-3a
(i(
+ min XE
C2 A) + Kns
(11)
where in the pen-ultimate step, we apply Lemma and the last step uses the fact that Li(f. g) is linear in y and applies Jensen's inequality to [2 g) noting that is concave in y (as result of D(g. u) being concave g).
Applying Jensen's inequality again to the LHS of equation 11
noting that 4 =
E [E(y, f(x) y = i] and as a result L, (f. is convex in f (owing to E(y, being convex in 2) and additionally using the fact that L2(g, X) is linear in X, we further have:
max C(F < A
min
L(f.g,)) + Kn.
AE
19

---

Published as a conference paper at ICLR 2023
Lower bounding the LHS by min over À E [0, R] (noting that À E [0, Rl), and the RHS by max over E Ams we have:
min 0 gEam..
L(f,9,)) S
min
(() Kn-
ge f:a Ae.
Exchanging the min's 's and max's 's using min-max theorem,
max min C(f.g.A) gedm Ae
min max min ((f.g,2) + Knn fal ge XE 0.
In other words for any 1 X Am.
max min C(f.g.)) & max min C(f", 9) + kn- ge X
gEAm XE(ORR)
An application of Lemma to both the LHS and RHS gives for any f : X Am
max gEG(6)
9: e(f) <
Σ gi l(ft* + Kns g
max
which completes the proof.
A.4 EG-UPDATE FOR (t)
To obtain the updates for g. we consider the fixed (() E R, and adopt a EG-style computation. Taking the KI. divergence for illustration, we provide the closed form of g(++1) in Proposition 2
Proposition (Un-normalized) EG-updates for g under DxL is given by:
(1+1) 9.
(2) 93 exp(ng&((1) Ang log(u,)))
r=ing
Regarding the EG-updates for (0),, it straightforward from Proposition that classes with larger loss higher target distribution weight tend to receive a larger weight.
Proof. In this proof, we consider the generic target distribution r which covers the uniform prior as special case. obtain gi
(1+1) for KI. divergence, where (g., Ee 9i log gi
we need:
Pt
0f (-1 1 E. 91 log
E.gel(i - > E Ii log
2 - 0))
g,
0
Ogi
(t)
91
1
9
9i
ri
log
9:
+ +.() A log
Ags
=
77g
9
17
log
91
1
+
(f)
A log
9i
A = 0
1lg
9:
1/a
Ti
-40
1
X X Aog
gi
+
log
9i
ri
fl
gi
-lif)
1
X + Alog(r:)
+
1
log log(o;" (t)
Alog (gi)
log (9:)
77
179
7y
= - + +1) = (.(f)gg - - Ang + XTg log(r:) + +log(g.") (2)
4(f)ng - Ay + Xny log(r:) + log(g]") (1)
-9; = exp
Xm +1
Log(g) (1) + (.ff)ggg + Ang log(ri)
-13
- exp
1 Aria
20

---

Published as a conference paper at ICLR 2023
Remove the constant, then have:
(1)) + )
((()) + Ang log(r:)
91 exp
Ang
91 (9. exp(Tgee(f)+ An log(r:)})
B EXTENSION TO GROUP-PRIOR SHIFTS
We now show how our theoretical results extend to the group-prior shift setting. Suppose each instance X' is associated with group conditional- probability to be (, a) = P(y i|æ, and the group-specific class priors Te,i Py ila). In this case, we wish to learn a scoring function
A,
[Y] x |A]. We define the group-specific
X A + Am that takes both the instance r and group into account. We use (J) = E. [e(v. f(x, a)) aj to denote the class-i loss conditioned on group g. Our goal is to minimize the following group-specifie DRE objective:
DRL(Fi0) = min max f-X-Am WED(B)
Σga. go.i - (a)(f),
(12)
a,
where G()) - {gg E Am D(g,r) 8) for some 5 > 0, divergence function Am x Am + I, and target distribution E Am.
Theorem 10 (Bayes-optimal scorer for group-prior shift). Suppose fly is a proper loss that is convex in its second argument and D(g.-) -) is convex in g. Let 5 > 0 be such that G(8) is non-empty. For some 9 G()), then the optimal solution equation takes the form:
Pax
fg (2r, a) x
(2, a).
Ta,y
The proof follows the same steps as Theorem except that it uses the following lemma instead of Lemma
Lemma 11. Suppose E(y, 2) is a proper loss. For any fixed g E kx the following is minimizer to the objective Ea,y Ja.y lay(S) ()) over all measurable functions J functiion
x A - Am:
fz a) x
Ja. (,(xxa)) a).
To.9
Proof. We first expand the objective:
Eoss (f,(f))
Σ Ja.i Exjayy (0l, f(x,,
Σ Ja.i
E. (2) ((x
Ta,i
a.
Given that is proper loss, we have that the minimizer of this objective takes the form:
"h (Ir, a) Da.i
(x,
Fu.d
Σ,
Ras
C ADDITIONAL EXPERIMENT DETAILS AND RESULTS
In this section. we introduce omitted experiment details and additional experiment results of our proposed methods.
21

---

Published as a conference paper at ICLR 2023
C.I ABLATION STUDY OF DROPS ON CLASS-MBBALANCED CIFAR
We offer the ablation study of DROPS in Table 3 Suppose the designer is interested in Seval = 1.0 robustness. setting mmin E [0.5, 1.0] frequentl reaches best three performances in mean/(Oeval w.0)-worst/worst by referring to the performance of averaged 5 runs, which indicates that DROPS is less sensitive the parameter òuain- Setting ouain to coarse estimate the Seval should be able to appropriately improve the model robustness under prior shifts.
Table 3: Ablation study of DROPS class-imbalanced CIFAR-100 100 dataset: mean + std of averaged class accuracy, 8 = 1.0-wors: case accuracy, and worst class accuracy of runs are ported. The best three performed for in each setting are highlighted.
CIFAR-100 (Averaged)
Method
10
p 50
100
DROPS (5 0.1)
59.08-0.38
48.24+0.63
43.37-00.22
DROPS (5 0.2)
59.12-0.21
47.90+0.52
-3.61
DROPS ( 0.3)
58.62+00.56
48.17:0.91
42.85=0.86
DROPS 0.4)
58.98-10.37
47.6411088
42.60=1.115
DROPS 0.5)
58.79110211
47.84.10.5
42.52-1043
DROPS (6 0.6)
59.35-0.227
47.44+0.65
42.47-0.84
DROPS (5 0.7)
59.42+0..30
48.04+0.47
43.20-0.92
DROPS (8 0.8)
59.63-10.10
47.78+00.6
43.44=0.60
DROPS 0.9)
09.69-0.33
47.83:10.00
43.20=00778
DROPS 1.0)
59.60-0..3
47.96.10.86
43.23=1.45
DROPS
59.00
48.31110.46
43.72-0055
DROPS
59.36-0.37
18.02
43.11=0.35
DROPS 1.3)
59.54-10.75
1.08
42.80=0.67
CIFAR-100 (Ō 1.0-worst case acc)
Method
10
p 50
100
DROPS (5 0.1)
42.98-10.39
29.62:10.52
24.26=0.68
DROPS 0.2)
43.96-10.18
1..98:11.00
05.46
DROPS (5 0.3)
43.54-0043
30.18+1.02
05.14
DROPS (5 0.4)
44.16-00.76
30.50+0.96
24.90-0.88
DROPS (8 0.5)
44.18-00.34
31.10+0..5
25.08
DROPS 0.6)
44.64:10.35
30.12:10.73
2..60=0.64
DROPS 0.7)
44.34:10.90
30.30:10.41
25.30=0085
DROPS 0.8)
44.16-10.43
30.32:10.68
25.48-0.55
DROPS 0.9)
44.96-0.52
30.12:10.66
25.58_0.50
DROPS
44.86+1.05
31.1410.774
26.24=1.88
DROPS 1.1)
1..74+1.33
31.18:10.56
25 25.84=0.55
DROPS 1.2)
44.92-1.69
31.28:11.27
25 0.55
DROPS 1.3)
45.04-10.84
31.22210.98
25.22-0064
CIFAR-T00 (Worst)
Method
10
p 50
100
DROPS (5 0.1)
20.61+3.24
8.25+2.46
6.02+1.500
DROPS (8 0.2)
22.51+3.10
9.58+3.34
7.40+1.04
DROPS (8 0.3)
23.32+4.399
10.90++3.90
1.99
DROPS 0.4)
22.51
10.79:12.07
8.28
DROPS 0.5)
25.13=3.055
12.52:12.57
6.70-12.9
DROPS 0.6)
24.86.14.18
10.33
7.43+2.78
DROPS 0.7)
21.74-14.97
10.8911.007
9.03-2.67
DROPS 0.8)
24.78-14.84
11.17:1.54
8.21+2.06
DROPS 0.9)
25.98-33995
11.65:13.05
8.61+2.22
DROPS 1.0)
24.80+12.85
11.86:12.85
7.52:12.11
DROPS 1.1)
24.22-14.83
12.8112.15
9.42-11.99
DROPS 1.2)
25.77-4.58
12.89.22.66
7.27-2.32
DROPS 1.3)
26.49-3.66
12.54
8.56+1102
C.2 HYPOTUESIS TESTING OF PERFORMANCE COMPARISONS ON CLASS-IMBALANCED CIFAR
We provide the statistical testing of results in Table 1 in Table we included the paired student t-test results between each baseline method and DROPS, for each dataset and each metric (mean/ō- worst/worst accuracy), and the inputs of samples of each method for testing the test accuracies
22

---

Published as a conference paper at ICLR 2023
of 5 runs 3 imbalance ratio settings). And each cell indicates the (f-statistics and p-value). It is quite obvious that for most results, there exists negative statistics, meaning that the given baseline method is significantly (if p-value is small enough, i.e,, p < 0.05) worse than DROPS.
Table 4: Paired student t-test of the performance comparisons between each baseline method and DROPS: cells in right 6 columns denote (statístics, p-value) of the hypothesis testing results between each baseline method and DROPS, the scenario where negative statistics and p-value less than 0.05 indicates that DROPS statistically significant better than the corresponding baseline method.
CIFAR-I0
CIFAR-I0O
Method V.S. DROPS
Mean
8 1.0-worst
Worst
Mean
1.0-orst
Worst
Cross Entropy
11.3, 1.96
-.6.
1..7,1.4e
-7.9, 1.5e
3.7
-.3.,,3.3
Focal
2.9,2.
-15.1,4.e
-21 4.8e
-9.4,36e 7
5.6, 8.8.8.8e 5
-40,5.0e
LDAM
-9.1,2.8
-9,8, 1.2e
-9.2,2.6r-
-7.4.3.1e
-6.6, 1.2e-
-7.4.3.2-
Halanced-
Silanced-Softmax Softmax
1.11
-3.9, 1.6e
-3.6,3.2-
-5.5.8.4e-5
-5.7,5.4e-
-..1,26e
Logit-Adjust
1.9,0.08
-4.6, 4.de
-5.9,3.9e
33.5,
-3.4,47e
-.,1,
Logit-Adj (post-hoc)
0.3,
-6.1,266e
-5.7,5.6e
-3.4,4.3e
-3.1,7.26
-3.4,4.1e"
C.3 COULD DROPS OUTPERFORM FINE-TUNING?
We adopt two kinds of fine-tuning strategies by using the pre-trained model Cross-Entropy loss in Table I and re-train another 2000 iterations with a fixed learning rate (1e-3, 1e-4, le-5, 1e-6), including the following two options in T
Fine-tuning the whole networks: named the row "Cross-Entropy (le-3)", empirically we ob- served that the learning rate 1e-3 consistently better than the others and is reported here.
Fine-tuning the last layer (DFR): where learning rate 1e-3 is consistently better than others.
All other settings and model selection criterion remains the same as the vanilla training procedure, except for replacing the training data by the whole validation set. Although these two fine-tuning strategies are beneficial improving the model performance across each metric and setting over Cross-Entropy loss in Table [ they still fall largely behind DROPS in most scenarios and require re-training the model on the additional validation set, while DROPS only needs the information of per-class accuracy decide on the parameters of post-hoc scaling, as also utilized for the model selection of all other baseline methods appeared in Table I
Table 5: Performance comparisons on class-imbalanced CIFAR datasets: mean + std of averaged class accuracy, 5 = 1 1-0-worst case accuracy, and worst class accuracy of 5 runs are re- ported: for all three baseline methods, we perform fine-tune of the pre-trained model appeared in Table [ on the hold out balanced validation set (last 10% of the original balanced training set). Best performed methods each setting are highlighted. While DROPS only makes use of the validation set to calculate per-class accuracy, without needing to training samples in the validation set.
CIFAR-10 (Averaged)
CIFAR-100 (Averaged)
Method
p- 10
p- 50
0
100
0 10
p- 50
0 100
Cross Entropy (no line-tune)
86.51_0.17
75.7410.97
69.2
55.5910.50
43.3910.76
37.9621.27
Cross Entropy (le-3)
87.80 0.30
82.66 0.224
19.68-1.66
57.24+0.40
48.22++0.40
44.80=0062
DFR
87.58-0.25
80.32-1.94
75.84=10995
57.08:00.52
46.68=0.28
42.9620.69
DROPS (6 0.9)
89.17 0.24
83.12
80.30
19,6910,39
47.8310.80
10.20
CIFAR-10 (6
1.0-worst case acc)
CIFAR-100
(8
1.0-worst case acc)
Method
p= 10
p= 50
p= 100
n= 10
p == 50
100
Cross Entropy (no fine-tune)
79.98-0.13
59.48:1.75
43.3013.63
27.06:10.33
11.2410.97
6,52:10.18
Cross Entropy (Io-3)
79.14=1.02
72.48=0.47
6..02-+1.31
34.38=0.55
25.04=0.72
21.20=#1.24
Vil
79.80
70.24
65. 65.10.00.91
34.1410.91
22.78
18. 18.86.10 10..3
DROPS (Ō 0.9)
86,20 0.34
79.4010.57
75.46
44.96.0.52
30. 30.12.00.66
25,58
CIFAR-1O (Worst)
CIFAR- 100 (Worst)
Method
p= 10
50
| 100
#= 10
p 50
100
Cross Entropy (no fine-tune)
78.29-0.86
55.3243.58
36.00-6.11
9.79+2.57
1.98
0.00=E0.00
Cross Entropy (Io-3)
75.48-1.53
68.32+1.13
65.06=22.04
18.60+1..2
10.10=1.00 1.02
7.40+22.58
DFR
1.27
66.30.12.24
60.96.11.23
20.60.1162
11.00 1.41
6.80.12.93
DROPS (Ō 0.9)
82,22
76.3310.89
7..62.1.33
25.9813.95
11.65
8.6112.22
23


Reviews:
[{"review_id": "Z-Ydb2ASMN", "reviewer": "Reviewer_b7xp", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work proposes to apply a post-hoc learning process for correcting label (class-prior) shifts between training and test distribution. It utilizes a labelled validation set that has the same label prior distribution as the test data and optimizes a delta-constrained objective (Eq.(1) or (2)). The delta quantifies how much the prior can change from the test/validation class distribution. Using the Lagrangian method, the algorithm alternates between different components. Convergence analysis is provided and experiments on common benchmarks show the superiority of the proposed method, Distributionally RObust PoSt-hoc method (DROPS).", "summary_of_the_review": "The paper proposes a simple method with theoretical convergence justification. Experiments on common benchmark datasets show that the proposed method can work better than alternative methods. Despite some missing details in the experiments, the results are mostly convincing.", "strength_and_weaknesses": "Strengths\n- Writing is clear\n- A simple method that is easy to implement\n- Theoretical justification\n\nWeaknesses\n- Some experiment details are missing\n\nDetail comments & questions regarding the experiments:\n\n1. What are the sizes of the validation sets? What happens if we fine-tune the model on the validation set?\n\n2. It is mentioned in the tables’ captions that the best performed methods in each setting are highlighted. They are best in terms of what criteria? Did you perform any statistical testing?\n\n3. Figure 1 is concerning. For CIFAR10, why delta_{train}=1 is so good across different eval deltas? If the model is trained based on delta=2, then we should expect it to work best when the evaluation delta is also 2. However, it seems that delta_{train}=1 works best even for evaluation delta = 2. What’s so special about delta_train=1?\n\n4. What’s the performance of DFR for Table 1?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "UZP6hheJz2", "reviewer": "Reviewer_xUWa", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a classifier that performs well under the shift of class proportions. The approach involves a distributionally robust formulation (a min-max problem) where the weight vector is constrained in a $f$-divergence ball of radius $\\delta$. The optimal post-hoc classifier is shown to be a reweighed classifier. The paper contains a gradient descent-ascent type algorithm to find the optimal post-hoc classifier from an initialization.", "summary_of_the_review": "There is not enough evidence about the novelty and the significance of the results.", "strength_and_weaknesses": "I experience certain difficulties in assessing the quality of this paper. I believe that the authors did not clarify well the contributions of the paper.\n\nI am relatively familiar with adversarial training and minimax formulation of equation (3). Theorem 1 of this paper is simply an implication of the Nash equilibrium results, which are abundant in the field of adversarial training and robust learning. For $f$-divergence type set $\\mathbb{G}(\\delta)$, the results of Duchi and Namkoong (2018) and BenTal et al. (2013) have provided much information about the reweighing schemes and how to solve the problem. The gradient descent-ascent algorithm is also widely used to solve min-max problems. Finally, Theorem 3 provides the guarantee only for the ergodic average $\\bar f$, but not for the terminal $f^{(T)}(x)$, thus it is of limited usefulness in practice.\n\nFrom the perspective of a researcher with moderate experience in adversarial learning, I do not feel that I have learned something new with this paper. I hope that the authors can provide strong justifications for their contributions.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "QeNX57JI7oS", "reviewer": "Reviewer_P4Hw", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studied the problem of distributional shifts in classification models and proposed a distributional robust formulation to tackle this problem. The authors showed that the Bayes optimal solution is a post-hoc scaling of the conditional class probabilities, where the scales can be learned via solving a saddle-point problem. They showed the convergence of this post-hoc classifier, and experimentally demonstrated the superior performance of the proposed algorithm on several datasets compared to baselines. ", "summary_of_the_review": "Overall I think this paper is of good quality, and the resulting post-hoc classifier could be potentially valuable to practitioners who are faced with imbalanced classification problem. I like the succinctness of the results regardless of the fact that it was derived from a complicated min-max formulation. The experimental results also look promising. One thing that confuses me is the authors' claim about how their approach differs from DRO (or worst case optimization), which I expect the authors to further clarify. I'd also like to understand in practice, how should we pick the parameter \\delta to ensure an appropriate amount of robustness?", "strength_and_weaknesses": "Strength: \n\n1. The paper is well written. Problem is clearly set up and described. The algorithm is well justified both theoretically and empirically. \n\n2. Starting from a complicated distributional robust formulation, the resulting post-hoc classifier is yet in a very simple form, which would facilitate its use in many practical problems. \n\n3. The proposed method does not require knowledge of the class distribution in the test set, which adds to its applicability compared with a class of methods that uses undersampling to match the distribution between train and test.\n\nWeakness: \n\n1. Even though the authors claimed that their formulation is essentially different from the worst-case optimization (as done in DRO, for example), from what I read, their approach is not very different from DRO. It is still trying to minimize the worst-case loss among a set of distributions centered around some nominal distribution (Eq. (2) and (3)). The conservativeness of the formulation is controlled by some parameter \\delta, which is the same with DRO. Could you please clarify what is essentially the difference between your approach and DRO, if any?\n\n2. In Proposition 1, the RHS, should we remove the constraints w.r.t g under max, since we already penalize the constraint violation in the Lagrangian?\n\n3. In Theorem 1, \\eta is the ground truth conditional class probabilities, right? Do we assume there is no shift in \\eta between train and test? Do we require the estimated \\eta to have good properties for your method to work well? I think explanations on the \\eta component needs to be added. \n\n4. In the experiments section under label shifts, can you also compare with the DRO solutions that handle class label shifts? Many DRO models reduce to adding a regularizer to the loss function. I am curious about how your method performs compared with those traditional DRO methods.\n\n5. In practice, how should we pick the parameter \\delta to ensure an appropriate amount of robustness?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster