Modeling the Second Player in Distributionally Robust Optimization
Authors: Paul Michel
Tatsunori Hashimoto
Graham Neubig
Venue: ICLR 2021 Poster (2021)
Keywords: distributionally robust optimization
deep learning
robustness
adversarial learning

Abstract: Distributionally robust optimization (DRO) provides a framework for training machine learning models that are able to perform well on a collection of related data distributions (the "uncertainty set"). This is done by solving a min-max game: the model is trained to minimize its maximum expected loss among all distributions in the uncertainty set. While careful design of the uncertainty set is critical to the success of the DRO procedure, previous work has been limited to relatively simple alternatives that keep the min-max optimization problem exactly tractable, such as $f$-divergence balls. In this paper, we argue instead for the use of neural generative models to characterize the worst-case distribution, allowing for more flexible and problem-specific selection of the uncertainty set. However, while simple conceptually, this approach poses a number of implementation and optimization challenges. To circumvent these issues, we propose a relaxation of the KL-constrained inner maximization objective that makes the DRO problem more amenable to gradient-based optimization of large scale generative models, and develop model selection heuristics to guide hyper-parameter search. On both toy settings and realistic NLP tasks, we find that the proposed approach yields models that are more robust than comparable baselines.

Published as a conference paper at ICLR 2021
MODELING THE SECOND PLAYER IN DISTRIBUTIONALLY ROBUST OPTIMIZATION
Paul Michel
Tatsumori Hashimoto
Graham Neubig
School Computer Science Carnegie Mellon University pmi mmichell 1 cs. cmu. edu
Computer Science Department Stanford University Mtashim@stanforet edu
School of Computer Science Camegie Mellon University gneubi gess cmu. eddu
ABSTRACT
Distributionally robust optimization (DRO) provides framework for training ma- chine learning models that are able to perform well on a collection of related data distributions (the "uncertainty set"). This is done by solving min-max game: the model is trained to minimize its maximum expected loss among all distributions in the uncertainty set. While careful design of the uncertainty set is critical the success of the DRO procedure, previous work has been limited to relatively simple alternatives that keep the min-max optimization problem exactly tractable, such as f-divergence balls. In this paper, we argue instead for the use of neural generative models to characterize the worst-case distribution, allowing for more flexible and problem-specific selection of the uncertainty set. However, while simple conceptually, this approach poses number of implementation and opti- mization challenges. To circumvent these issues, we propose relaxation of the KL-constrained inner maximization objective that makes the DRO problem more amenable to wradient-based optimization of large scale generative models, and develop model selection heuristics to guide hyper-paramett search. On both toy settings and realistic NLP tasks, find that the proposed approach yields models that are more robust than comparable baselines
INTRODUCTION
Machine learning models trained with empiriial risk minimization (ERM) are able to achieve high aggregate performance data sampled from their training distribution. However, they often exhibit drops in accuracy when confronted with data from domains that are under-representted in their train- ing data, such those of different topic (Gururangan et al. 22000), sociolect (Blodgett al. 2016), accent (Amodei et al., 2016) or writer age (Hovy & Søgaard (2015) in language processing tasks, or skin color (Grother al. 2019) or lighting (Georghiades al.] (000]) in image processing tasks. This is particularly egregious issue in applications where higher eтror rates can have far reach- ing negative implications, such as the silencing of underrepresented minorities in toxicity detection systems (Dixon et al. (2088) or disparity amplifying feedback loops in credit rating models (Fuster cct al. 2018).
This behaviour often arises from the objective function of ERM where the parameters 0 of the model are learned by minimizing the expectation of loss function & under data distribution p (or, specifically in practice, associated empirical data distribution P)
(0) = (=,,))= y,
When the model encounters data sampled from a different distribution Dea A performance can suffer significantly. Distributionally robust optimization (DRO) (Ben-Tal et al. (013b) provides a natural solution to this issue by replacing the expected risk under single distribution p with the worst expected risk over pre-determined family of distributions Q (the "uncertainty set")
LDRO (f) = max ((z.)) ((x, 0)..
(2)
Code reproduce experiments can be found at https::/ggthhbb.co che che133111511 3051:mm111111111111111110101

---

Published as a conference paper at ICLR 2021
If Q contains Qest. the DRO objective upper bounds the expected risk under Teat However, priori knowledge of possible test distributions is not always available or easy to acquire. For example, training a model to be robust to some demographic attributes (Q Addeoographic 1 Ademographic 2 requires collecting and annotating data with the necessary information, an expensive and ethically fraught endeavour. In the absence of such information, set analytically, drawing on onc's s intuition of what constitutes possible test distribution given the observed training distribution, such as using moment constraints (Delage & Ye 2010 Nguyen al. 2020), f-divergence (Ben-lal Tal et al., 2013a Hu & Hong 2013 Faury et 20200, Wasserstein/IPN
has to resort defining the uncertainty
(Sinha et al. 2018; Husaiin| 2020) balls, or coarse- grained mixture models (Oren et al. 2019] Hu et al. 2018). However, the need for keeping the inner supremum in Eq. tractable limits the possible choices.
saarse-raiind
In this paper, we propose that the uncertainty set be instead defined family of parametric gen- erative models. The resulting DRO objective ( is differentiable game with two players: the original model (xx y; ()) and model of its worst-case distribution (z, y). the titular "second player" which we hereafter refer to as the adversary. Using this formulation - which we call Para- metric DRO (P-DROJ- allows for more flexibility in the choice of the adversary's 's architecture (and so the uncertainty set). Unfortunately, finding a solution of this game via direct application of simultaneous gradient descent (Singh et al.) 20000 is difficult (Balduzzi et al., 2018). In particular, direct gradient descent on the uncertainty set suffers from instability due to the large variance of the gradients (Greensmith et al. 2004), and cyyeer-paramete selection not straightforward.
To address these challenges, we make two main contributions (33): first, we propose new relaxation of the [RO game's ' inner maximization problem (with KL constraints). The resulting objective is more amenable to simultaneous gradient update than the original zero-sum game and significantly improves training stability, while still yielding useful adversaries. Second, we develop a principled we leverage the learned adversaries to decide which of any
approach for selecting two given models trained with P-DRO is more robust than the other.
We do an in-depth set of experiments analyzing the effect of our proposed changes both toy task as well as a more realistic, yet still synthetic sentiment classification task ($4) Finally, we show that in the more realistic setting of toxicity detection, (ctDRt yields models that are more robust to changes in demographic groups, even though these groups are unknown at training time, opening up applications in combatting dataset bias
2 PARAMETERIZING THE UNCERTAINTY SET
Consider a model parameterized by 0 E [R Minimizing the DRO objective described in Eq. over the uncertainty set Q turns the optimization problem into the min-max (or zero-sum) game
2
min max B(x ax, y, B)
(3)
DER qEQ
The first player controls the parameters 6, whilst the second player controls the worst-case distribu- tion q. In the absence of explicit information on groups of interest (such as demographics, domain, etc.), an adequate choice of the uncertainty set Q is critical to the success of DRO This is in fact very much an active area research (Sinha et (2019), see Rahimian & Mchrotra (2019) for a survey). @ must be sufficiently large to contain test distributions of interest, but if it too large it may contain "adversarial" distributions on which no model can perform well. Moreover, the design of Q is also circumscribed by the necessity of keep- ing the min-max problem tractable, particularly in the context of stochastic optimization. In Hu & Hong (2013) and Duchi et al. (2016) for example, the choice of /-divergence balls allows the use of duality arguments to reformulate (3) as a more manageable min-min problem. Others, like Hu et al.) (2018) or Oren et al. (2019), propose using mixture models, the simplicity of which enables them to solve the inner maximization problem efficiently.
20188: Duchi & Namkoong (2018) 8); Oren et al.
Instead, we propose to explicitly model the second player in the DRO game as parametric model qu of the data. Of course, not all parameterizations 1e E dew of given generative model represent useful distributions, and we require that the adversary stay "close" to the underlying true data distri- bution As a measure of distance between is and p. we choose the KL (Kullback & Leibler 195I) divergence due to its wide acceptance in the machine learning community, well its appealing
2

---

Published as a conference paper at ICLR 2021
moodel
Bp qv (2r,
((2, yr 0)
Model
94% (a,, y)
((2,3: 0)
(a, v)
Exp
log qol (2,
Adversary
Zr
(ar, y)
Lust
VilCadv
Figure 1: Summary of P-DRO: every step of training, (z,, y) pairs are sampled from the data distribution and fed to both the model 0 and the adversary th. For every sample, the model produces loss values (x, y; 0) and the adversary produces densities qu (z, y). Both are combined into m and Ladv, which used update the and 1b respectively, via simultaneous gradient updates.
properties in the context of DRO The KL upper bound, is left parameter to be decided by experimenter. We refer to the resulting DRO formulation as Parametric DRO
min max 1 kL (k
(( ((x,
(aDRo((,0)
3 OPTIMIZING - P-DRO
The min-max problem in Eq. belongs to a class of games called "diffcrentiable games" (another famous representative being generative adversarial networks (Goodfellow al., (014)). We can search for a solution of this game with simultaneous gradient descent (Singh et al. 2000), i.e. by simultaneously updating 0 and with - 4ED and V 4EDRO respectively. Unfortunately, in general, there is no theoretical guarantee that simultaneous gradient descent will converge to a Nash equilibriumf) Balluzzi et al. 2018). nor that any such equilibrium even exists if the objective is non- convex in (or non-concave in w)). The success of GANs and the follow-up literature (Wang et al., 2019) serves as an encouraging example that gradient based methods can yield useful solutions despite the pessimistic theoretical results. In this section, we discuss difficulties that arise when optimizing & and b jointly, and propose modifications of the objective address them.
3.1 TRAINING THE MODEL
We could train the model 0 by taking negative gradient steps on [a E(x, 0). This gra- dient can be estimated by sampling examples from qu and averaging the gradient their losses. Unfortunately, this objective requires that qu is well-behaved source of supervision for A. If qu is initialized incorrectly or begins producing unrealistic (æ. y), it begins learn predictor
all iterations, as it is the only
the quality of & degrades As an alternative, we opt to compute the gradients for with importance sampling, iee. rewriting
invalid training examples from de-
Reloy)) CI-DRO as (x.g)><p pla.y] f(z, y; 0), which ensures that all (z, y) samples will be derived from the training itself. Unfortunately, the true density p unknown to us. As an approximation, (n.w) with the likelihood ratio between qu and the maximum likelihood estimate p, replace p(a.))
Qvo arg max, B(z.))= log (x, u). This changes the min-max problem to
To (2, 8) Qy' (1t, y)
min max
(x.y)
(xx y, 0).
(5)
KL(g=llp)<M
Lunaa
Forr instance: (ll(glp) < +oo implies that g stays within the support of Nash equilibria (Osoorne & Rubinstein, 1994) can be thought of the game theoretic analog of global minima in optimization.
3

---

Published as a conference paper at ICLR 2021
This becomes simple expected loss objective, which can estimate by sampling from the empir- ical distribution p. In experiments, we find that with this formulation we are able to train robust even when qe is only mediocre generative model (see Appendix C.22) To further stabilize training at the beginning of the optimization process, we initialize vú with 1oo making the objective exactly the same as HRM for the first gradient step.
3.2 TRAINING THE ADVERSARY V
pla.)) (xx y, 0) within a KL ball According to Eq. the adversary & must maximize E( 4oo (2:3)
of fixed radius. This is challenging for several reasons: first, enforcing the bound is intractable for complex familics of adversaries where e.g. projecting onto the KL ball is another difficult optimiza- tion problem of its own. Second, maximizing the expectation with respect to the parameters of the distribution Zø is prone to instability due to large gradient variance (Greensmith et al. 2004).
Lagrangian Relaxation To address the first difficulty, loosen the strict KL constraint and instead consider the Lagrangian relaxation L
p(x, v) Two (I, v)
( 7) ((z,))=
[xx (, 0) - (KL(gelIV) - -k).
(6)
We fix the Lagrangian multiplier T > treat it "temperature" hyper-parameter. With some reorganization (which we develop in Appendix A.I we can show that
L(v,)) = - (gwllll + C.
piz.vl ((mugro))
and C is a constant in vb. In other words, maximizing L in & is equivalent to minimizing the KL divergence between qv and 97 One difficulty with this objective is that 27,9 depends upon the unknown probability density p(x, y). We avoid this problem by treating the density ratio p(x.w) (sr,)) as a constant, which is closely related to assumptions that have been used successfully in past formulations of DRO (Oren et al. 2019). Empirically, we find that incorporating Iva as a surrogate for p is a serviceable approximation, as demonstrated in Section 4
Where Fh 0 p(æ; y) Regg (P-2)
Reversing the KL Minimizing the KL divergence in this direction is difficult for several reasons. First, it entails optimizing an expectation in ge over vú, which is difficult due to the large variance of the gradients (Greensmith al., 2004). Second, computing this KL necessitates access to the true theoretical density p(z, 3) in order to compute g (z, y) in the argument of the expectation, but this quantity is unknown in practice To sidestep these issues, direction (L(g", (1 instead. Due to the KL divergence being non-symmetric, this is a rather crude approximatiorf/ the implications of which we find that this approach dramatically stabilizes the gradient dynamics while still yielding good adversaries, observed empirically in Section 4.4 Discarding the entropy term (constant in 1)), the resulting problem is equivalent to minimizing
elect minimize the reverse
discussed in Norouzi et al. (2016). However,
((m.v:))
Luav (1/, (1b,T)
T)
Ep e
log qot (2,
Z
in vi where Zr,0 by substituting the empirical distribution ý for in the expectation.
E, (:
is the normalizer of q², In this case, we can estimate this expectation
Computing the Normalizer Approximating the inverse normalizer Z.,o in a minibatch yields biased estimator. On the other hand, computing Zr.9 over the entire training data at each step is prohibitive since it requires computing the loss of every single example. As a middle ground, keep running normalizer Zy computed from the average of the normalizers over fixed number
"Note that substituting the empirical distribution ş for p poses issues here, hecause To not absolutely contin- uous with respect to p.
For instance, the optimum of the reverse KI. doesn't necessarily match that of the forward KI. within the parametric confusion Q
4

---

Published as a conference paper at ICLR 2021
k of consecutive minibatches. In other words, if B, and 0, denote the minibatch and adversary parameters at step i respectively, the normalizer at step will be
2/ = Σ List- B
Σ Σ
Best-RR,ye R.
If too low, there is risk of under-estimating the normalizer, especially if the distribution of weights contains infrequent high weight samples. On the other hand, if l: is too high there is a risk of using "stale" weights in the normalizer. In experiments, we treat hyper-parameter.
3.3 OPTIMAL STOPPING
When should one stop training model with WiDOO In ERM is customary to stop training after the empirical risk - periodically evaluated on held out validation dataset - stops decreasing. This particularly important to prevent over-fitting to the training data. However, it is not appropriate criterion for [-DRO] since the model not trained to minimize empirical risk in the first place. A more pertinent choice is compare the robust validation losses
(x( ((x, y; 0)
Crobust, valid (0)
max Ev
Σ
(10)
Dvalid
Is0 (z,z)
Draie
(9.
However, finding the inner supremum for each of the T evaluation checkpoints .1... 0r is expensive as requires solving T independent optimization problems. Instead, we leverage the existence of adversaries v associated with each model O,, well as the initial adversary the and take the maximum over the T + 1 adversaries {slo..... v need weed out adversaries which might violate it. Specifically,
Since our relaxation of the P-DRO objective
loosens the KL constraint, estimate the KL [p) = Ep go/plog /p on the validation set, using 9 as a stand-in for /p/p, and reject all adversaries for which the result is greater than threshold, which set log 10 based preliminary experiments detailed in Appendix C.1 We refer to this stopping criterion as Minmax.
Computing the full min max necessitates keeping track of T' models and T adversaries, which is ponderous when the model is large. As a solution, we propose an approximation, Greedy-Minmax, in which only keep best model 8". At each evaluation step T, compare Or 0", and update 0 to whichever achieves lower robust validation loss over the T + adversaries voo, WT.
By keeping track of only one additional model, and using the weights Te, (as.y))
dsg (z..2)) of individual exam- ples in Dvalid as sufficient statistics for computing the loss against each adversary. Greedy Minddxx can be achieved with space complexity 2dmmodel T) Deatal, which is much more efficient than the T(dmodel + dadv) of Minmax.
3.4 HYPER- PAPEE-PAR SELECTION
Our proposed P-DRO method relies on 3 different hyper-parameters (in addition the model's hyper-parameters)) the adversary learning rate X, the temperature т and the size of the re- normalizing window As consequence, configurations is better. This model comparison bears many similarities with the stopping prob- lem described above. Therefore, we resort to a similar solution: given two models fr, 02 trained with PDO and their respective adversaries {8%).... ). f w (for instance, the adver- saries associated with 0, and 02 at periodic checkpoints during training), we select the best model following
need a reliable criterion for deciding which of two
0"
max Be (E(6,,0, 00. de))ee(( (E(v) .
arg min
(O 1)).
(11)
"To simplify notation, this additional constraint is implicit the of this section.
5

---

Published as a conference paper at ICLR 2021
4 EXPERIMENTAL ANALYSIS OF P-DRO
Before moving on to a real world scenario in Section 5 we first demonstrate that P-DRO is able to learn robust models in synthetic Natural Language Processing (NLP) task, and perform ablation studies examine the importance the various modifications described in Section
4.1 EXPERIMENTAL SETTING
For analysis purposes, we design simple NLP task amenable to DRO We specifically choose NLP as a domain due the striking success of language models as generative models of textual data (Sundermeyer et al., 2012, Radford et al., 2018), which can be used to model the uncertainty set. We base our task off of the binary version of the Stanford Sentiment Treebank dataset (SST-2; Socher et al. (2013)) which we modify to introduce spurious correlation. Specifically, we introduce a distractor token to some sentences. The distractor we use consists of prepending "so " to the sentence ("i hated this movie" + "so. hated this movie"), which doesn't 't change the underlying sentiment. The resulting samples can be categorized in 4 "groups" depending on their label (positive negative) and the presence absence of the distractor. In particular, 95% of the negative reviews and 5% of the positive reviews in the training and validation set, so that the presence of the distractor strongly correlates with negative sentiment (a similar construction is proposed in (Utama et al. ). In the test data, we modify 50% of all sentences for each class equitably ensure that there enough data in each group, but we report "average" test accuracy by re-weighting the group accuracies to mimick the training distribution. We call this modified task BiasedSST.
add this distractor to
For the classifier, we train simple one layer BiLSTM model with embedding/hidden dimension 300. For the adversary, we adopt an auto-regressive transformer model based on the successful GPT-2 language model architecture but with 6 layers, dimension of 512 and 8 attention heads (we experiment with smaller, LSTT based adversary Appendix C.2) In order to model the input output pair (z, y). we pre-pend a special label-specific token to sentences before running them through the language model. We train the model with Adam (Kingma & Ba 2014) and the adversary with vanilla stochastic gradient descent (which we found more stable in experiments). We refer to Appendix B for specific details of the experimental setting.
4.2
P- DRO CAN LEARN ROBUST MODELS
We train 7 models with P-DRO on BiasedSST using dif- ferent hyper-parameters for the adversary. We start from configuration À 10 we run a configuration with a smaller and higher value, keeping all other hyper-parameter the same. We train for 50 epochs and select the best model using the strategies described in Section
Table 1: Average and robust accura-
0.01, k 5, and for
cies on BiasedSST. Underlining indi- cates statistically significant difference
each
compared to ERM (p 0.05)
Robust
Average
We also compare three other approaches. First, to appreci- how well the model could perform if the groups were known at training time, we train with Group-DRO on the oracle groups using an exponentiated-gradient based on- line algorithm (Oracle DRO; Sagawa et al.) (2020)) ). Sec-
ERM
2.15 097
95.09
Topic CVaR
5.18 + 1.46
95.00 o.10
NonParam P-DRO
28. 11 2.16
92.45 1.55
34.98 9.99
84.21 2.111
Oracle DRO
67.71 3.03
77.91 4.4.9
ond, we implement Topic CVaR (Oren al. (019). a
method for [RO] on NLP where the uncertainty set is determined by mixtures of a topic model. Finally, we compare to non non-parametric DRO with Kullback-Leeibee (KI.) constrained uncertainty set (Hu & Hong 2013 Hu et al. 2018), which we adapt to fit online mini-batch training setting (NonParam). We refer to Appendix B.3 for details and paramteas of the basclines.
We report the worst-case ("robust") accuracy over all groups on the test set, as well the average accuracy in Ta we report the mean and standard deviation over 5 runs). We find that both Topic- CVaR, NonParam and P-DRO are more robust than ERM but the latter outperforms the former two close to 30 and points respectively, achieving 52% of Oracle DRO's robust accuracy, while not leveraging any information the oracle groups

---

Published as a conference paper at ICLR 2021
Table 2: Effect of different optimal stopping and syper-parametee selection strategies on robust validation accuracy.
(a) Optimal stopping
(b) Hyper-paramecte selection
Criterion
Robust accuracy
Criterion
Robust accuracy
Average
0.00 0.00
Average
31.03 a 12.16
Minmax
25.22 1: 13.01
Minmax
28.62 1 12.37
+KL constraint
31.30 10.07
+KL constraint
35.65 1 11.47
+Greedy-Minmax
32.17 + + 11.20
Oracle
38.26 4: 13.01
Oracle
50.95 5.01
4.3 OPTIMAL STOPPING AND
SELECTION ABLATION
To understand the importance of the optimal stopping and hyper-parameter selection strategy de- scribed in Section 3.3 we perform an ablation on the BiasedSST dataset comparing strategies:
Average: models are selected based on their average zero-one loss (i.e. error rate) on the unmodified validation set. This is the baseline stopping criterion.
Minmax: selection based on the adversaries (as described in Section 3.3). with and without the KL. constraint, well as its variant Greedy-Minmax for stopping.
Oracle: in this setting the groups are known (in the validation set), and models are selected based on their error rate on the worst performing group. This is the optimal criterion for the group-DRO setting we are considering.
To compare stopping criterions experiments, we only consider one set of hyper-parameters: À = 10-1, k 5 and 0.01. From the robust validation accuracies reported in Table 2a we first observe that Average stopping results in robust accuracy of O, highlighting the necessity for suitable stopping criterion. We find that Minmax, especially with a KL constraint, ter strategy, recovering 60% of the performance achievable with Oracle stopping. Notably, the Greedy-Minmax variant which we use practice reaches very close results (< 1 point difference) despite its requiring to keep track of only 2 out of the 50 model checkpoints at any time.
much bet-
To understand the effectiveness of the Minmax strategy for selecting hyper-parameters We take the models trained in Section 4.1 but select the best hyper-parameters using the different strategies described above. Results, shown in Table 2b) confirm that Minmax (with the KL constraint) is a better choice than Average for selecting syper-parameters, though the improvement is not as striking as for stopping.
4.4 IMPORTANCE OF CADY
Finally, we investigate the importance of modifying the adversary's ob- jective as described Section 3.2 For this experiment, we devise a sim- pler toy task on which directly training the constrained [RR] objective possible. Specifically, we consider the two-dimensional binary clas- sification problem pictured in Figure 2 The training data consists of 10,000 points partitioned two normally distributed "domains" " with 1:50 sampling ratio and different classification boundaries. We train logistic regression model, which cannot perfectly fit the training data and must trade-off between accuracy on each domain. For the sake of simplic- ity, we only model the input variables a as isotropic normal distributions with fixed variance: the adversaries' parameter vé g² represents the location of the Gaussian (we fix the variance to the empirical variance of the data).
Figure 2: A toy classifi-
cation task.
II other words, we set qo (z, y) = ply (x), where ply x), is the true conditional which will be canceled out in the ratio gelz.)) 90g (e:0]]

---

Published as a conference paper at ICLR 2021
We compare 3 different versions P-DDO first, naive si-
Table 3: Ablation of to train the
multaneous gradient descent on the zero-sum game, with- out any constraint on the adversary (bare P-DRO), then the same, but with an approximation of the explicit KL. constraint between de and Qwo (+KL. constraint; see Ap- pendix A.2 for more details). Finally we report results us-
linear model on the toy task. We report
accuracy on both domains, as well as robust accuracy.
Average
Robust
ing our relaxation and the KL reversal described in Section 3.2 (+Ladr). For each setting, we report the average and robust accuracy with mean and standard deviation over 10 runs. For the KL constraint and the relaxation, we report
ERM
84.66 0.10
49.75 0.05
bare P-DRO
49.97
49.85 0.10
+kI constraint
76.63 1. 993
58.41
a 9.25
76.97 0.43
64.32 1,31
the best results among values of the KI. bound and the temperature T respectively.
In Table 3 we observe that bare P-DROO is too unstable and systematically diverges. The addition of a KI. constraint mitigates this behaviour, but the zero-sum objective still unstable, as evidenced by the high standard deviations. Finally, we find that the addition of Liev stabilizes the training process greatly, leading to consistently high robust accuracy.
P-DRO IN PRACTICE: CASE STUDY OF TOXICITY DETECTION
In this section, we demonstrate the effectiveness of P-DRO in the more realistic setting of toxicity detection, the task of recognizing various forms of toxic language (eg. hate speech or offensive language). Identifying online abuse on the internet is a crucial challenge, and has garnered much interest in the NLP community (Schmidt & Wiegand 2017 Fortuna & Nunes 2018). However, recent work (Sap et al. 2019) has shown that there is strong correlation between toxic labels and the presence of certain markers of dialects of English spoken by minority groups. This correlation is in turn amplified by hate speech classifiers trained on such data, leading to biased prediction.
Our results on BiasedSST suggest that P-DRO can provide one solution preventing models from absorbing spurious correlations present in their training data, even in the absence of protected at- tributes (such as language variety here).
5.1 EXPERIMENTAL SETTING
Following Sap et al. (0119) and Xia et al. (2020), we perform experiments on two datasets: DWMW17 (Davidson et al. 2017). a corpus of 25K tweets classified in three categories: hate speech (6%). offensive (76%) and neither (18%), and FDCL18 (Founta et al. 2018). 100k sized dataset, also collected from Twitter and annotated with an additional spam label, with the following breakdown by categories: hateful (5%), abusive (27%), normal (54%) and spam (14%).
The released version of these datasets does not contain information on the dialect of each user. In order to be able to evaluate our models, and to train Oracle DRO baseline, we follow Sap et al. 2019) and use annotations provided by the dialect classifier described in Blodgett et al. (2016) to label each example as one of four English varieties: White-aligned, African American, Hispanic, and Other. Note that, as these are automatically obtained labels, the groups may not exactly correspond to the actual racial sociolects, however (Sap al. (0019) does report that they correlate highly with self-reported race, and they serve as a useful proxy in the absence of manual annotation.
We formulate the group-DRO problem by separating each dataset into independent groups identified by both language variety and label, for total of 12 and 16 groups for DWMW 17 and FDCL18 18 re- spectively. Some of these groups are severely under-represented in the test set. In order to make our robust accuracy results reliable yet still representative of the under-represented groups, we combine groups that contain less than 100 samples into a single group to compute robust test accuracies.
On DWMW 7, we train the same BiLSTM model as described in Section 4.3 To illustrate the ap- plicability of P-DRO to other model architectures, we pick BERT (Devlin et al. 2018). a large seale pre-trained model as classifier on FDCL18. In both cases, we adopt the Transformer architecture described in Section 4.3 as the adversary. We train the adversary with a temperature of T = 0.01 and normalizing window k = 10. To demonstrate the efficacy of automatic hyper-parameter selec- tion in the training 3 models with X E (10-5, 10-1, 10-3) and selecting the best using the Minmax criterion
setting, we delegate the choice of the adversary's 's learning rate À to geid-search,
8

---

Published as a conference paper at ICLR 2021
Table 4: Robust test accuracy on the DWMW17 and FDCL18 toxicity detection tasks.
(a) No group information.
DWMW17
FDCL18
Robust
Average
Robust
Average
ERM
53.19 + 1.70
69.44 0.53
19.57 = 7.00
81.56
0.26
Topic CVaRR
45.26
1: 3.47
61.68 5.02
16.48 1: 5.46
80.49 1 0.49
NonParam
54.13
70.54
17.54
81.20 1 D.11
P-DRO
69.06
+ 1.70
69.69 = 2.50
30.25 = 10.13
79.91 + 1.41
Oracle DRO
74.50 +: 1.74
65.79 0.76
55.23 3.97
72.43 2.61
With group information on the validation set.
Robust
Average
Robust
Average
ERM
53.15 + 0.87
69.64
+: 1.11
34.07
= 3.20
78.78 0.38
Topic CVaR
52.02 1.26
69.11
0.49
34.82
3.73
79.59
+: 0.85
NonParam
49.41
5.60
58.53
5.71
43.13
6.7
69.51
3.07
P-DRO
63.05
4.25
63.07
+: 3.92
47.61
+ 4.53
74.82
1.90
Oracle DRO
74.50
+ 1.74
65.79
+: 0.76
55,23
+ 3.97
72.43 2.61
We also report numbers for Oracle DRO and Topic CVaR. Results averaged over 5 runs, each with a different random seed.
described in Section 3.4
5.2 PRODDUCE
PRODUCE MORE ROBUST MODELS?
Table repports the robust test accuracies of all models both tasks. Importantly, except for Oracle DRO, none of the methods compared here necessitate any knowledge of the groups, neither in the training nor validation data. We observe that both settings P-DRO is able to achieve higher robust accuracy than ERM Topic-CVaR and NonParam.
This suggests P-DRO as useful option in case no group information whatsoever is available. However, in practice, it may be feasible to annotate at least small amount of data with group information. To emulate this scenario, we perform the same experiment, but assume that group annotations are available the validation data, which we use to determine optimal stopping and hyper-parameters Results for this setting are reported Table We find that, while the use of ro- bust validation accuracy yields more robust models even for ERM (especially still the best alternative that doesn't 't require group annotation on the training data.
FDCL18), P-DRO
6 IMPLICATIONS AND OUTLOOK
We have shown that there is promise in using parametric families of neural generative models for defining the uncertainty set in distributionally robust optimization. While we only perform experi- ments on NLPPttsks this approach can, in theory, be applied in any modality and in future work we hope to pursue this direction. such cases where good quality generative models are unavailable, or such model cannot produce densities efficiently, an interesting direction would be to model the like- lihood ratio qu directly. This alternative formulation poses different implementation challenges, and we leave it as promising avenue for future research.
ACKNOWLEDGEMENTS
The authors would like to thank the anonymous reviewers for their insightful feedback which helped improve the paper to its current version. In addition, this paper greatly benefited from discussion and feedback from various colleagues CMU, in particular Chunting Zhou, Haohan Wang, Zachary Lipton and Zico Kolter. This work was supported by a Facebook Sponsored Research Award and by the DARPA GAILA project (award HR001 I 1990063). The views and conclusions contained in this document are those the authors and should not be interpreted as representing the official policies, either expressed implied, of the sponsors.

---

Published as a conference paper at ICLR 2021
REFERENCES
Dario Amodei, Sundaram Ananthanarayanan, Rishita Anubhai, Jingliang Bai, Eric Battenberg, Carl Case, Jared Casper, Bryan Catanzaro, Qiang Cheng, Guoliang Chen, Jie Chen, Jingdong Chen, Zhijie Chen, Mike Chrzanowski, Adam Coates, Greg Diamos, Ke Ding, Niandong Du, Erich Elsen, Jesse Engel, Weiwei Fang, Linxi Fan, Christopher Fougner, Liang Gao, Caixia Gong, Awni Hannun, Tony Han, Lappi Johannes, Bing Jiang, Cai Ju, Billy Jun, Patrick LeGresley, Libby Lin, Junjie Liu, Yang Liu, Weigao Li, Xiangang Li, Dongpeng Ma, Sharan Narang, Andrew Ng. Sherjil Ozair, Yiping Peng. Ryan Prenger, Sheng Qian. Zongfeng Quan, Jonathan Raiman, Vinay Rao, Sanjeev Satheesh, David Seetapun, Shubho Sengupta. Kavya Srinet. Amuroop Sriram, Haiyuan Tang, Liliang Tang. Chong Wang, Jidong Wang. Kaifu Wang, Yi Wang. Zhijian Wang. Zhiqian Wang. Shuang Wu, Likai Wei, Bo Xiao, Wen Xie, Yan Xie, Dani Yogatama, Bin Yuan, Jun Zhan, and Zhenyao Zhu. Deep speech : End-to-end speech recognition in english and mandarin. In Proceedings of the 33rd International Conference on Machine Learning (ICML), pp. 173-182, 2016. URL http::/prrroeeednggs mll. wwwpprrssssvv... / http:///4//mmme1111111 amodeil6 htmml
David Balduzzi, Sebastien Racaniere, James Martens, Jakob Foerster, Karl Tuyls, and Thore Grae- pel. The mechanics of n-player differentiable games. In Proceedings of the 35th International Conference on Machine Learning (ICML), pp. 354 363, 2018. WtR.htttp//proceeeeee mlr. press / v htt.00/aalllzzza / 1.a...pa.pppp.................
Aharon Tal, Dick Den Hertog, Anja De Waegenaere, Bertrand Melenberg, and Gijs Rennen. Robust solutions of optimization problems affected by uncertain probabilities. Management Sci- ence, 59()):3412333 1-75,, 2013a.
Aharon Ben-Tal, Dick Den Hertog, Anja De Waegenaere, Bertrand Melenberg, and Gijs Rennen. Robust solutions of optimization problems affected by uncertain probabilities. Management Sci- ence, 59(2):341-357 2013b.
Su Lin Blodgett, Lisa Green, and Brendan O'Connor. Demographic dialectal variation in social media: A case study of African-American English. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (ACL), pp. 1119-1130, 2016. URL https:
77 www. ac ac www...bbrr. or.. httg///hhhooooyyyyooool
Thomas Davidson, Dana Warmsley, Michael Macy, and Ingmar Weber. Automated hate speech detection and the problem of offensive language. In Proceedings of the 11th International AAAI Conference on Weblogs and Social Media (ICWSM), 2017.
Erick Delage and Yinyu Ye. Distributionally robust optimization under moment uncertainty with application to data-driven problems. Operations research, 5811)2595-61,, 2010.
Jacob Devlin, Ming- Wei Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT)), 2018
Lucas Dixon, John Li, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. Measuring and miti- gating unintended bias in text classification. In Proceedings of the 2018 AAAI/ACM Conference on Al. Ethics, and Sociery, pp. 67-73, 2018.
John Duchi and Hongseok Namkoong. Learning models with uniform performance via distribu- tionally robust optimization. arXiv preprint 181.v:1810.08750, 2018. URL BRtpptss:///rxii
prq/pdf/1810.
John Duchi, Peter Glynn, and Hongseok Namkoong. Statistics of robust optimization: A generalized empirical likelihood approach. arXiv preprint arXiv:1610.03442, 2016.
Louis Faury, Ugo Tanielian, Elvis Dohmatob, Flena Smirnova, and Flavian Vasile. Distributionally robust counterfactual risk minimization. In Proceedings of the 34th Meeting of the Association for Advancement of Artificial Intelligence (AAAI), volume 34, pp. 3850-3857, 2020.
Paula Fortuna and Sérgio Nunes. A survey on automatic detection of hate speech in text. ACM Computing Surveys (CSUR), (1(4):1-33, 2018.
10

---

Published as a conference paper at ICLR 2021
Sarigoni-Mariae Founta, Constantinos Djouvas, Despoina Chatzakou, Ilias Leontiadis, Jeremy Blackburn, Gianluca Stringhini, Athena Vakali, Michael Sirivianos, and Nicolas Kourtellis. Large scale crowdsoureing and characterization of twitter abusive behavior. In Proceedings of the 12th International AAAI Conference on Weblogs and Social Media (ICWSM), 2018.
Andreas Fuster, Paul Goldsmith- Boldsmith-PPikkham, Tarun Ramadorai, and Ansgar Walther. Predictably un- equal? the effects of machine learning on credit markets. The Effects of Machine Learning on Credit Markets (November 6, 2018), 2018.
Athinodoros S. Georghiades, Peter N. Belhumeur, and David J. Kriegman. From few to many: Illumination cone models for face recognition under variable lighting and pose. IEEE transactions on paltern analysis and machine intelligence, 23(6):643 660, 2001.
lan Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sher- jil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Proceedings of the 28th Annual Conference on Neural Information Processing Sys- tems (NIPS), pp. 2672-26888 2014. URL http://papers: niw.. cc.pserrr/ 5423 wer--generrtiiee adversarial Senneratvv-addveessraal netsspet......
Evan Greensmith, Peter L Bartlett, and Jonathan Baxter. Variance reduction techniques for gradient estimates reinforcement learning. Journal of Machine Learning Research, 54Nov1)147 2004.
Patrick Grother, Mei Ngan, and Kayee Hanaoka. Face Recognition Vendor Test (FVRT): Part 3, Demographic Effects. National Institute of Standards and Technology, 2019.
Suchin Gururangan, Ana Marasović, Swabha Swayamdipta. Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. Don't stop pretraining: Adapt language models domains and tasks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguisties, pp. 8342-8360, Online, July 2020. Association for Computational Linguistics. doi: 18.18653/v1/ 2020.acl- monddd-main.400.. URL hww.sttppp..... www. ac web wwwpor//ntttooooo. wacpthhhoogyyyyyoo ogy 2020 acl mair..
740
Hershey and P. Olsen. Approximating the kullback leibler divergence between gaussian mixture models. Proceedings the International Conference on Acoustics, Speech, and Signal Processing 4ICV3SS) 2007.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735-1780, 1997.
Dirk Hovy and Anders Søgaard. Tagging performance correlates with author age. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics (ACL), pp. 483-488, 2015. (RLhsttss:
www. ac web. org/ anthology/PPi 2079.
Weihua Hu, Gang Niu, Issei Sato, and Masashi Sugiyama. Does distributionally robust supervised learning give robust classifiers? Proceedings of the 35th International Conference on Machine Learning (ICML), pp. 2029-2037, 2018. URL WtRLhttp:///// htttt:://prooeedings. ml htt.ceeeddnng.mmrrrrrrssssss//e pr hulBa/hulBa.pon
Zhaolin Hu and L Jeff Hong. Kullback-leibler divergence constrained distributionally robust opti- mization. Available at Optimization Online, 2013.
Hisham Husain. Distributional robustness with ipms and links to regularization and gans. In Pro- ceedings of the 34th Annual Conference on Neural Information Processing Systems (NIPS), 2020.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Proceedings of the International Conference on Learning Representations (ICLR), 2014.
Solomon Kullback and Richard A Leibler. On information and sufficiency. The annals of mathe- matical statistics, 22(1):79-86, 1951.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. Proceedings of the International Conference Learning Representations (ICI.),, 2017.

---

Published as a conference paper at ICLR 2021
Viet Anh Nguyen, Nian Si, and Jose Blanchet. Robust bayesian classification using an optimistic score ratio. In Proceedings of the 37th International Conference on Machine Learning (ICML), 2020.
Mohammad Norouzi, Samy Bengio, zhifeng Chen, Navdeep Jaitly, Mike Schuster, Yonghui Wu, and Dale Schuurmans. Reward augmented maximum likelihood for neural structured prediction. In Proceedings of the 30th Annual Conference on Neural Information Processing Systems (NIPS),
pp. 1723-1731. 2016.
Yonatan Oren, Shiori Sagawa, Tatsunori Hashimoto, and Percy Liang. Distributionally robust language modeling. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 4227-4237, 2019. URL https: / www. ac wwwweb..wbb.................... web. org/
anthol.ogy/DI--143 1432
Martin J. Osborne and Ariel Rubinstein. A Course in Game Theory, The MIT Press, 1994, ISBN 0262150417.
Alec Radford, Karthik Narasimhan, Time Salimans, and Ilya Sutskever. Improving language under- standing with unsupervised learning. Technical report, Technical report, OpenAI, 2018.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAl Blog, 1:8, 2019.
Hamed Rahimian and Sanjay Mehrotra. Distributionally Robust Optimization: A Review. arXiv preprint arXiv:1908.00 05659, 2019.
Shiori Sagawa, Pang Wei Koh, Tatsunori B Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generaliza- tion. In Proceedings of the International Conference on Learning Representations (ICLR), 2020. URL
wwtpo////p///to//////////tt//ee/ http:/pff/191111 11t.1/9911..0.........
Maarten Sap. Dallas Card, Saadia Gabriel, Yejin Choi, and Noah A. Smith. The risk of racial bias in hate speech detection. In Proceedings of the 57th Anmual Meeting of the Association for Computational Linguistics (ACL),, pp. 1668-1678, 2019. URLhttps: // /www acwweb..b........... org/
lanthology/II9-163
Anna Schmidt and Michael Wiegand. A survey on hate speech detection using natural language processing. In Proceedings of the 5th International Workshop on Natural Language Process- ing for Social Media (SocialNIPP), pp. 1-10, 2017. URL https: / www. ac web . org/
enthology/w17-1111
Satinder P. Singh, Michael J. Kearns, and Yishay Mansour. Nash convergence of gradient dynamics in general-sum games. In Proceedings of the 16th Conference on Uncertainty in Artificial Intelli- gence, UAI '00, pp. 541-548, San Francisco, CA, USA, 2000. Morgan Kaufmann Publishers Inc ISBN 1558607099.
Aman Sinha, Hongseok Namkoong, and John Duchi. Certifying some distributional robustness with principled adversarial training. In Proceedings of the International Conference Learning Representations (ICLR), 2018
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng. and Christopher Potts. Recursive deep models for semantic compositionality over sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1631-1642, 2013.
Martin Sundermeyer, Ralf Schlüter, and Hermann Ney. Lstm neural networks for language model- ing. In Proceedings of the 13th Annual Conference of the International Speech Communication Association (InterSpeech), 2012.
Prasetya Ajie Utama, N. Moosavi, and Iryna Gurevych. Towards debiasing nlu models from un- known biases. arXiv preprint arXiv:2009. 12303, 2020.
Zhengwei Wang, Qi She, and Tomas E Ward. Generative adversarial networks in computer vision: A survey and taxonomy, arXiv preprint arXiv: arXiv:1966.011 1906.0152, 2019.
12

---

Published as a conference paper at ICLR 2021
Mengzhou Xia, Anjalie Field, and Yulia Tsvetkov. Demoting racial bias in hate speech detection. In Proceedings of the 9th International Workshop on Natural Language Processing for Social Me- dia (SocialNLP), pp. 7-14, 2020. BRLhatt:::// //ww::://wwww// www.wawwwwb.ooom http://wwww.........oet//tttttoa
p-.1pp1110
A DERIVATIONS
A.1 REORGANIZING THE LAGRANGIAN L(
Let us write the I Lagrangian I. explicitly:
p(x,3) 3)
L(b,r) r) [(x,y)~
((2, y, - [KL(gw x)
(12)
aNo (I,, v)
p(u, 1)
To 20 1)
f( y, 0)
(a, log
+ TK
(13)
Woo (r, y)
p(x, y)
mlm aca.w..)
p(x, ( "0o (0.0) (ais))
-T log
+ TR
(14)
())
p(r.s) (lo....))
= T((K - - (44 ) + log
Biz.ginp e 90 (R.0)
(15)
This last step requires that the log moment generating function of é under p exist for т. In most scenarios we consider, l is typically the negative log likelihood a neural network model, which is generally bounded. Therefore the moment generating function defined everywhere.
Note that the KL term is the only one dependent on ah, therefore maximizing IL for &b is equivalent maximizing (L (e)),, in other words minimizing (L(ggll9",,)
A.2 ENFORCING THE KL CONSTRAINT THE ToY SETTING
Even in this simplest setting, the exact KL between Tu (a gaussian) and (a mixture of gaussians) does not have an analytical expression (Hershey & Olsen 2007). Instead, we fall back on enforcing the KL constraint between de and lvo both isotropic gaussians with the same standard deviation. Let and μi E I2 denote their respective mean, and >> 0 their standard deviation. In this context, their KI, divergence reduces to:
1
KL (1 KL(g (4 =
|| moll2
20
In other words, the KL. divergence equivalent to the euclidean distance between the distributions' means. We use this fact to project vi (in the KL. sense) onto B, = {vb KL < k):
proj (+b) arg min (L(g+119g))
BEB,
V2no
rho + | voll (v voo)
B EXPERIMENTAL DETAILS
We describe in more details some of the experimental settings for our NLP experiments. More details can found our code release: /tt.ts:://gtthhh cwmm pmi che 31 31415/P-DRO
B.1 MODEL SETTINGS
In all experiments, split the text into sub-word tokens using the tokenizer described in (Devlin et al. 2018). During training, we sample minibatches that contain at most 64 sentences or 2500 tokens, whichever is greater, in order to prevent GPU memory overflow in case of long sentences.
13

---

Published as a conference paper at ICLR 2021
We train all models with Adam (Kingma & Ba 2014) with an initial learning rate of x 10 which we decay linearly at each step until the end of training. We validate the models every epoch. For BERT, we start from the bert pert-basee -bwse--ucasseeeee checkpoint.
B.2 ADVERSARY SETTINGS
In all experiments, we use a Transformer model based on the GPT-2 architecture (Radford et al., 2019) to serve as the adversary. order to initialize the adversary (to obtain v/0). we first pre-train the model 2017). We also use a batch size of 64 samples or 2500 tokens, and train with Adam for 10 epochs, with a fixed learning rate of 10- Then, fine-tune this model each dataset, this time minimizing the negative log-likelihood of the (x, y) pair (by introducing the special "[label]" token described in Section B), using the same hyper-parameters but smaller learning rate (10-5). We find that, due to the small to medium size of the datasets under consideration, this LM pretraining step helped achieve lower error on the generative modeling task.
generic, relatively large language modeling dataset, WikiText-103 (Merity et al.
B.3 BASELINE SETTINGS
B.3.1 TOPIC CVAR
To train the topic model for Topic CVaR, we first pre-process the text by removing all punctuation, urls and user mentions (for twitter data). Importantly, we remove stop-words for our toxicity exper- iments but nol for our BiasedSST experiment. This is because the distractor token we use ("so") belongs most English stop words lists, and removing it would completely prevent the topic model from picking up on the groups of interest. We then estimate the parameters of the model with Gen- sim and use similar settings Oren al. (2019) (a 0.1, 3 1.0), setting the number of topics to 10.
For both Oracle-DRO and Jopic-CVake we use the algorithm proposed in Sagawa al. (2020) es- timate the worst-case group (either oracle group or topic in (Va)) online during training. We perform grid-search over {1, 0.1, 0.01) to find the best learning rate for the group weights update. For Oracle [RO] the best model is simply selected by robust validation accuracy. For Topic CVaR, unless specified otherwise, we select the model with the lowest worst-case error over all topics.
B.3.2 NONPARAM
In the Kl-cconsrrd non-paran con-parametric setting, the min-max problem reads
min 0
max S.s. kl(g
((x,))/((I, 0)
(16)
Here, & is the desired radius of the KL ball, and is treated as Shyper-parameter. The solution of the
((m.g.))
inner maximum has analytical solution of the form q Zo Z p(x, y)e
(see Hu & Hong
Ala.yio and T" such that 2013); : Hu et al. (2018) for details) with Zo, Epe
((xx)))) log Zo.r K.
(L(g"llp) Ep Zo,
7
Note that both computing Z and KI. [lp) require taking expectations over p. In our setting, where f(z, y: 0) is the output of a large neural network, cannot afford to take this expectation over the entire training data at each step. Instead, we fall back to taking the average over each mini- batch. We find x" with binary search in log space within the [10 10 10"0] interval and clip the lowest or highest value should the result lie outside the search interval.
htspss//raiinrehurek.ccomm
14

---

Published as a conference paper at ICLR 2021
Robust validation accuracy
A= 10
40
10
A 10
20
2 dol logi
log 19020
log1 00 logso
log1 000
log10000
KL threshold
Figure 3: Evolution of the robust validation accuracy of the model selected by Greedy Minmax as function of the KL threshold Kvalid
In all experiments, we try 4 different values for 6: 0.01, 0.1, and 10. Unless indicated otherwise, we perform early stopping and hyper-parameter selection using our Minmax criterion using the non-parametric weights as adversaries on the validation data.
C ADDITIONAL EXPERIMENTS
C. MINMAX VALIDATION KL THRESHOLD
The
Monte-Carlo estimate of KL (Ip)
on the validation set
[Pauul Ex.eEDan To (m.y)) p(x.a) log 4o (a..)
Similarly to Section
3
approximate the (unknown)
p(a.y/)
likelihood ratio (2,2)) with (w.,))
p(z.y)
(2.9)
We want to reject all adversaries where this approximated KL is greater than some threshold, Kvalids but how do we choose good value for Kvali? Consider an adversary which selects a fraction of the validation data of size a Dvanil for some a (0, 1]. In such case, the likelihood ratio is 1/a on this subset and everywhere else, and the resulting KL estimate will be log a. In other words, choosing threshold of Kvali means allowing the adversary to potentially select any subset of size at least 1/e" of the original data. Our heuristic choice, log 10, corresponds to allowing subsets of size least 10% of Dvatia/-
Of course, this is only a heuristic because the adversary can reweight the validation set non- uniformly. To assess the effect of Kiaaldd Greedy-Minmax, idation error of the selected model across runs for different values of the adversary's 's learning rate. Results BiasedSST, depicted in Figure show that adversaries with higher learning rate more sensitive to the choice of threshold, but all values of Kvalid between log 5 and log 20 seem to work for these settings.
compute the average robust val-
C.2 P-DRO EXPERIMENTS WITH AN LSTM ADVERSARY
We replicate the experiments BiasedSST experiments in Section but this time using a smaller generative model, which unlikely to generate good samples. Specifically, we use a one layer LSTM model (Hochreiter & Schmidhuber, 1997) with embedding and hidden dimension 256. We only perform geircsearch over À [10=>
31 and select the best with Minmax.
Once pre-trained on the BiasedSST dataset, this model achieves perplexity of 227.0, .0, more than 4 times worse than the transformer model we use in other experiments (49.8). However, as evidenced by its robust accuracy displayed in Table 5 PPDRO is still able to learn robust model. We take this as evidence that the re- reigeighting introduced in Section helps stabilize training even when gu is not perfect model of the data.
15

---

Published as a conference paper at ICLR 2021
Table 5: Average and robust accuracies BiasedSST when P-DRO is trained with an LSTM
adversary. Underlining indicates statistically significant difference compared to ERM (p < 0.05)
Robust
Average
ERM
2.15 1. 0.97
95.09 1: 0.16
Topic CVaR
5.18 1.46
95.00 0.10
NonParam
28.1
2.16
92.45
1: 1.55
P-DRO
43.68
1: 4,93
86.58
1: 1.77
Oracle DRO
67.71 4: 3.03
77.91 4: 4.49
Table 6: Effect of hyper-paramettrss robust validation accuracy BiasedSST
Robust accuracy
Minmax stopping
Oracle stopping
À = 10 10-6
28.62 12.37
45.10 4.50
A = 10
44. + 3.24
50.43 5.05
À 10-3
25.57
38.70 2.97
T 0.1
39.72 5.55
50.00 4.98
7 0.01
44.74 3.24
50.43 5.05
T 0.001 k = 1
44.74 3.24
50.87 = 5.09
41.98 1.4.8
49.60 1: 5.39
k 5
44.74 3.24
50.43 5.05
k 10
32.17 + 11:20
50.95 = 5.01
C.3 INFLUENCE OF HYPER-PARAMEN ON P-DRO
We study the influence of the hyper-paramedie т (temperature), k (size the renormalization window) and À (learning rate of the adversary) the performance of P-DRO All experiments are run on the BiasedSST dataset, and the analysis proceeds as follows: starting from configuration 0.01, k and 10-4 and vary each of the prper-parameters independently. We report two numbers for each configuration: robust accuracy of the best model using Greedy-Minmax stopping and using Oracle stopping. The latter is useful to disentangle the effect of the stopping criterion.
As seen in the results shown in Table 6 we find that r has the least effect on robust accuracies. While the renormalization window parameter has some effect on optimal stopping, the best robust accuracy achieved by the model (with oracle stopping) varies little. We observe the adversary's learning rate be the most sensitive hyper-parameter, which is why restrict our grid-search to A in Section 5
16


Reviews:
[{"review_id": "r0JSBbD5a6P", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Good points\n----\n- The objective of the paper is sound: fight distributional shift in systems whose predictions\nmight have life-changing consequences (e.g data bias toxicity prediction models, etc.).\n- The paper is well-written and easy to follow.\n\nBad points\n----\n- I don't see just how this model is \"parametric\". In statistics, \"parametric\" the adversarial\ndistribution is modeled as a gaussian, etc. with sought-for parameters (mean, covariance, etc.).\nIn the absence of that, I would have expected \"parametric\" to mean parametrizing the adversarial\ndistribution as the (softmax) output of a neural network. Neither of the above is the case in \nthis paper. So, what are the \"parameters\" in the proposed DRO adversary ? All I can see is that\nthe authors do  a full search over all distributions, subject to a KL constraint (see sections\n2 and 3.2).\nThere is nothing \"parametric\" about this.\n- The authors say \"In particular, direct gradient descent on the uncertainty set suffers from\ninstability due to the large variance of the gradients (Greensmith et al., 2004), and\nhyper-parameter selection is not straightforward.\" I'm not sure about this claim (which\nis one of the main premises of the manuscript. What do the authors make of this paper\nfor example Faury et al. (AAAI 2020) \"Distributionally Robust Counterfactual Risk Minimization\" ?\nThe authors of that paper demonstrate how to efficiently formulate and solve KL-based DRO\nproblems. That paper also contains both theoretical and practical insights.\n- The technical contribution of the paper is negligible (if any).\n- The arguments in the paper very heuristic.\n- Since the paper is supposed to be empirical (see previous points), I would have expected\nexperiments on real datasets.\n\n\nErrors\n---\n- Change \"solve the inner-max efficient\" to \"solve the inner maximization problem efficiently\"\n- Change \"$x, y ~$ \" to \"$(x,y) ~ $\" all through the manuscript\n- Eqn (5): why not take $p$ and $q_{\\psi_0}$ to equal the empirical distribution (as is usually\ndone) in DRO ?\n- In eqn defining $q_{\\psi_0}$, replace $\\arg\\max_{q_\\psi}$ with $\\arg\\max_\\psi$", "extra_scores": {}, "extra_text": {}}, {"review_id": "KRBnn7t612t", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes to define the uncertainty set in the DRO problem as a family of parametric generative models, which is to allow more flexibility in the choice of the uncertainty set architecture. To realize this idea, the paper first proposes a new relaxation of the DRO game's inner maximization problem (with KL constraints) so as to improve the training stability. It then develops a principled approach to select the hyper-parameters of the proposed method.\n\nStrengths:\n+ The paper is well-written.\n+ The proposed method is novel and important for the DRO community.\n+ Experiments with real-world problems are conducted to evaluate the effectiveness of the proposed method. I particularly like the experimental analysis the authors conducted to understand the behavior of their proposed method. \n\nWeaknesses:\n- The experiments are only on NLP tasks.\n\nI have few questions to the authors:\n1) How good the adversary model needs to be for the proposed method to perform well? In the experiments, an auto-regressive transformer model based on the GPT-2 language model is employed. What is the accuracy of this model on the train dataset of the DRO problem? Will the proposed method performance be too sensitive to the accuracy of the adversary model?\n2) In the experiment (last paragraph of Section 5.1), the temperature \\tau and the normalizing window k are fixed whilst the adversary learning rate \\lambda is searched by grid-search. So how \\tau and k are selected in practice? What is the performance of the proposed method when \\tau and k vary? \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "HOSukVouhlc", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper considers distributionally robust optimization (DRO) and uses the neural generative models to characterize the uncertainty sets. To tackle the optimization challenges, several implementation tricks are incorporated to solve the minimax problem. The proposed robust method is validated on NLP tasks. \n\nThis paper is well-written and of a good structure. Although the main idea is simple, the authors make several modifications to the algorithm to make it tractable and with performance guaranteed heuristically. To summarize, the main contribution of this paper is a new algorithm that combines standard techniques, such as Lagrangian relaxation and KL reverse, into the DRO problem with KL uncertainty sets. And this algorithm was shown to perform well under synthetic and real-data NLP tasks. Since there is no novel techniques proposed in this paper and there is no performance guarantee for the proposed framework, overall, I think this is a borderline paper due to its limitations in theoretical development and technical novelty. \n\nMoreover, if the main focus of this paper is on developing a new computational framework that can lead to more robust results, then the authors should compare with more benchmark methods, while I only see the comparison with ERM, Topic-CVaR, etc. For example, I am wondering is it applicable to compare with Wasserstein DRO or Huber's classical work of Total variation based DRO, or some other DRO works in the literature, so that it will be more convincing on the performance of the proposed method. \n\nA minor typo in the paper: in section 6, there is a duplicated \"produce\" in the sentence: \"In such cases where good quality generative models are unavailable, or such model cannot produce produce densities efficiently\".\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "-6XcaSqdJt5", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "TL;DR: The paper makes an interesting contribution from a practical point of view, but two important theoretical concerns need to be addressed in the rebuttal for acceptance.\n\nThe paper proposes the use of ideas taken from the literature on distributionally robust optimization within a parametric framework. More precisely, the main idea is to consider only a (parameterised) subset of the traditional KLD-uncertainty sets. As this avoids the need for elegant analytic solutions (at the expense of a more brute force computation), it has the flavour of a more ‘black box’ approach towards the deployment of DRO.  Overall, I really enjoyed the way this paper was written. Purpose and use of the contributions are clear throughout, and the reader is drawn in.  I also liked the contribution and believe that the paper demonstrated its ideas to be useful. There are however two points of major concern from a more theoretical side. In my mind, these are rather substantial, and I will list them below. To recommend that the paper be accepted, these points will have to be addressed in a future version of the paper:\n\n(1) How do you ensure that the KLD between $q_{\\psi}$ and $p$ is finite? p clearly is the empirical measure (as is emphasised e.g. just above eq. (5)), but $q_{\\psi}$ will be continuous. This means that the KLD between the two distributions is not defined/infinity for any value of \\psi (Mismatch of support problem). These kind of problems are the precise reasons why other quasi-distances (like the Wasserstein distances) have become increasingly interesting for ML. As far as I can tell, this problem is not elaborated upon anywhere in the paper. \n\n(2) It is totally unclear to me why it should be viable to suddenly flip the direction of the KLD. The KLD is not symmetric and in general will not even have the same minimum. In fact, generally speaking the only time the minimum will be the same in either direction is when the KLD’s global minimum is such that $q_{\\psi} = q_{\\tau, \\theta}$ (i.e. we can drop the KLD term for the loss in (7) completely, so that it simply equals $C$). Given the definition of $q_{\\tau, \\theta}$, it is unreasonable to assume that this global minimum is attained. This makes the flipping of the KLD’s direction questionable at best. Calling the outcome an ‘approximation’ is then grossly inaccurate.  (See e.g. the visualisations here: https://wiseodd.github.io/techblog/2016/12/21/forward-reverse-kl/)\n\nLastly, since the chief concern of the paper is the construction of new uncertainty sets, I would have liked to see two additional recent references discussed which have produced uncertainty sets purely based on moments (https://arxiv.org/abs/2007.04458, ICML 2020) as well as on general IPMs (https://arxiv.org/abs/2006.04349, NeurIPs 2020). Both these types of uncertainty sets do *not* suffer from the mismatch of support problem, and—like the famous f-divergence based uncertainty sets—have elegant dual forms. \n\n\nPOST-DISCUSSION: The authors promised to clarify the two issues I pointed out in ways that are satisfactory for a paper whose main concern is practicality (as opposed to theoretical rigour). I will thus raise my score to a weak accept.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: # Paper Summary

This paper considers the problem of distributionally robust optimization (DRO), in which one is attempting to minimize a loss on the worst of all distributions that are some distance (here, measured in terms of KL divergence) from the training set. The main novelty here is that this adversarial distribution is represented as a model, with parameters that are learned jointly with the primary model.

This is an intuitive idea, but as the authors explain, attempting to implement it leads to a number of complications. One of these is that it is challenging to constrain the adversarial distribution model to be a certain KL divergence away from the training set. To address this, they write down the Lagrangian, but do not actually optimize over the Lagrange multiplier resulting from this constraint: instead, they keep it at a fixed constant value (a hyperparameter). A second, and potentially more worrisome, issue is that it is difficult to optimize the KL divergence as written--instead, they swap the two parameters, which is of course incorrect but they claim leads to much nicer convergence behavior.

They also propose a stopping condition, which terminates optimization once the robust validation loss (i.e. the validation loss w.r.t. the worst permissible distribution) stops decreasing. Normally, this would require a search for the worst such distribution at every iteration, which would be prohibitively expensive, so they propose instead only checking the distributions that have been found by the adversary during the course of optimization.

They close with a set of experiments that is nicely designed to narrow in on and explore particular details of their approach (e.g. they have an experiment that validates their stopping criterion), and have a realistic experiment on two NLP datasets.

# Pros

1. Reviewers agreed that it was very well-written, well-organized, and comprehensive
1. Good discussion of background material. The paper is very accessible
1. Intuitive idea, although the details of the approach become somewhat complex
1. Aside from the "realistic" experiment, each is designed to explore a particular facet of their approach

# Cons

1. Some reviewers were concerned that the baselines were insufficient. In response the authors added the new Hu et al. baseline (NonParam), which seemed to be satisfactory
1. While the approach is more general, one reviewer noted that the experiments only consider NLP problems. This is a minor negative point, in my view
1. One reviewer was concerned that the results were "too good", and encouraged the authors to double-check their results. My belief is that, at least on the non-"realistic" experiments (which were mostly intended to drill down into specific attributes of their approach, rather than demonstrate its overall performance), this is because the problem was constructed to perform especially poorly with a non-DRO approach
1. One reviewer was unsatisfied with the idea of swapping the parameters to the KL divergence (I share this concern). The authors clarified, both in the response and in the paper, that swapping the parameters is indeed incorrect, and may in fact be a very bad approximation to the true quantity of interest, but that the performance difference was so dramatic that it couldn't be undone. This seemed to partially satisfy the reviewer

# Conclusion

All four reviewers ultimately recommended acceptance. The major concerns were (i) that the baselines weren't good enough (which the authors addressed by adding a new baseline), and (ii) that swapping the parameters to the KL divergence results in a very poor approximation to the original KL divergence (which the authors now explicitly acknowledge in the paper, with an explanation for why they feel it is necessary). Overall, this is a nice idea, and while bringing it into practice may require more hand-waving than would be ideal (which is the main reason I suggested a poster acceptance instead of a spotlight or oral), it seems to work well experimentally, and the experiments are overall very careful and well thought-out. Additionally, the writing quality is excellent, as is the organization and presentation of background material.
Decision: Accept (Poster)