SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Once we have  $w^t$ , a new classifier  $h^t$  is computed to minimize the weighted loss relative to  $w^t$ , and added to the ensemble. This update is called the Best Response (BR) update. Learning  $h^t$  in this way helps us fix past classifiers' mistakes, eventually leading to an ensemble with good performance.
Algorithm 1 Game play algorithm for solving Equation (1)
Input: Training data  $\{(x_i,y_i)\}_{i = 1}^n$  , loss function  $\ell$  , constraint set  $W_{n}$  , hypothesis set  $H$  , strongly concave regularizer  $R$  over  $W_{n}$  , learning rates  $\{\eta^t\}_{t = 1}^T$    
1: for  $t\gets 1$  to  $T$  do   
2: FTRL:  $w^{t}\gets \mathrm{argmax}_{w\in W_{n}}\sum_{s = 1}^{t - 1}\mathbb{E}_{w}\ell (h^{s}(x),y) + \eta^{t - 1}\mathrm{Reg}(w)$    
3: BR:  $h^t\gets \mathrm{argmin}_{h\in H}\mathbb{E}_{w^t}\ell (h(x),y)$    
4: end for   
5: return  $P^{T} = \frac{1}{T}\sum_{t = 1}^{T}w^{t},QT = \mathrm{Unif}\{h^{1},\ldots h^{T}\}$
Greedy. We now take an optimization theoretic viewpoint to design algorithms for Equation (1). Let  $L(Q)$  denote the inner maximization problem of (1):  $L(Q) \coloneqq \max_{w \in W_n} \mathbb{E}_{h \sim Q} \mathbb{E}_w \ell(h(x), y)$ . When  $L(Q)$  is smooth (this is the case when  $W_n$  is a strongly convex set), one could use Frank-Wolfe (FW) to minimize it. The updates of this algorithm are given by
$$
Q ^ {t} \leftarrow (1 - \alpha^ {t}) Q ^ {t - 1} + \alpha^ {t} G, \quad \text {w h e r e} G = \operatorname * {a r g m i n} _ {Q} \left\langle Q, \nabla_ {Q} L (Q ^ {t - 1}) \right\rangle .
$$
Here,  $\nabla_Q L(Q^{t - 1}) = \mathrm{argmax}_{w\in W_n}\mathbb{E}_{h\sim Q^{t - 1}}\mathbb{E}_w\ell (h(x),y)$ . This algorithm is known to converge to a minimizer of  $L(Q)$  at  $O(1 / t)$  rate [Jaggi, 2013]. When  $L(Q)$  is non-smooth, we first need to smooth the objective before performing FW. In this work we perform Moreau smoothing [Parikh et al., 2014], which is given by
$$
L _ {\eta} (Q) = \max  _ {w \in W _ {n}} \mathbb {E} _ {h \sim Q} \mathbb {E} _ {w} \ell (h (x), y) + \eta \operatorname {R e g} (w). \tag {3}
$$
Here  $\operatorname{Reg}(\cdot)$  is a strongly concave regularizer. If  $\operatorname{Reg}(\cdot)$  is 1-strongly concave, it is well known that  $L_{\eta}(Q)$  is  $O(1 / \eta)$  smooth. Once we have the smoothed objective, we perform FW to find its optimizer (see Algorithm 2 for pseudocode).
Relaxing the simplex constraint. We now derive a slightly different algorithm by relaxing the simplex constraint on  $Q$ . Using Lagrangian duality we can rewrite  $\min_{Q \in \Delta_H} L_\eta(Q)$  as the following problem for some  $\lambda \in \mathbb{R}$
$$
\min _ {Q \succeq 0} L _ {\eta} (Q) + \lambda \sum_ {h \in H} Q (h).
$$
One interesting observation is that when  $W_{n}$  is the entire simplex and when  $\lambda = -1 / 2$ , we recover the AdaBoost algorithm. Given the practical success of AdaBoost, we extend it to general  $W_{n}$ . In particular, we set  $\lambda = -1 / 2$  and solve the resulting objective using greedy coordinate-descent. The updates of this algorithm are given in Algorithm 2.
Remark 6 Algorithm 2 takes the step sizes  $\{\alpha^t\}_{t=1}^T$  as input. In practice, one could use line search to figure out the optimal step-sizes, for better performance.
Algorithm 2 Greedy algorithms for solving Equation (1)
Input: Training data  $\{(x_i,y_i)\}_{i = 1}^n$ , loss function  $\ell$ , constraint set  $W_{n}$ , hypothesis set  $H$ , strongly concave regularizer  $R$  over  $W_{n}$ , regularization strength  $\eta$ , step sizes  $\{\alpha^t\}_{t = 1}^T$
1: for  $t \gets 1$  to  $T$  do
2:  $G^{t} = \operatorname{argmin}_{Q}\left\langle Q,\nabla_{Q}L_{\eta}(Q^{t - 1})\right\rangle$
3:  $\mathbf{FW}$ :  $Q^t \gets (1 - \alpha^t) Q^{t-1} + \alpha^t G^t$  / Gen-AdaBoost:  $Q^t \gets Q^{t-1} + \alpha^t G^t$
4: end for
5: return  $Q^T$
We provide convergence rates for the algorithms below:
Proposition 7 (Convergence Rates) Let  $l(h(x),y)\in [0,1]\forall h\in H,(x,y)\in D$  and  $Reg:\Delta_n\to \mathbb{R}$  be a 1-strongly concave function w.r.t norm  $\| .\| _1$ . Let  $Q^T$  be the output returned from running Algorithm 1 or 2 for  $T$  iterations. Let  $D_{R}$  be a constant S.T.  $D_R^2 = \max_{x,y\in W_n}|Reg(x) - Reg(y)|$ .
1. (Gameplay) If  $\eta^t = \eta$ , then  $Q^T$  satisfies  $L(Q^T) \leq \min_Q L(Q) + \frac{\eta D_R^2}{T} + \mathcal{O}\left(\frac{1}{\eta}\right)$ .
2. (Greedy) If line-search is performed for  $\alpha^t$ , then  $Q^T$  (FW or the Gen-AdaBoost update) satisfies  $L(Q^T) \leq \min_Q L(Q) + \eta D_R^2 + \mathcal{O}\left(\frac{1}{\eta T}\right)$ .
We refer the reader to Appendix F.1 for a simple proof using existing theory on online convex optimization [McMahan, 2011, Jaggi, 2013]. Another useful insight is that Algorithms 1 and 2 are related to each other under special settings as shown by Appendix H.1.
Corollary 8 Consider  $\operatorname{Reg}(w) = -\sum_{i=1}^{n} w_i \log w_i$  and  $l$  as the zero-one loss. Then, Algorithm 1 and Algorithm 2 (line-search) achieve  $\epsilon$ -approximate NE with  $\epsilon$  as  $\mathcal{O}\left(\sqrt{\frac{\log(n)}{T}}\right)$ .
Weak Learning Conditions It might not be practical for  $H$ -player to play BR (Step 3: Algorithm 1) or correspondingly, to find the best possible classifier at every round (Step 2: Algorithm 2). Under weak learning conditions, we can indeed achieve (approximate) convergence when we only solve these problems approximately. See Appendix H.2 for more details.
6 Generalization Guarantees
In this section, we study the population RAI risk and present generalization bounds which quantify the rates at which empirical RAI risk converges to its population counterpart.
6.1 Population RAI Games
Recall, the empirical RAI risk optimizes over all sample re-weightings  $w \in W_{n}$  that lie within the probability simplex  $\Delta_{n}$ . Thus it's population counterpart optimizes over distributions  $P$  that are absolutely continuous with respect to the data distribution  $P_{\mathrm{data}}$ :
$$
R_{W}(h) = \sup_{P:P\ll P_{\mathrm{data}},\frac{dP}{dP_{\mathrm{data}}}\in W}\mathbb{E}_{P}[\ell (h(x),y)].
$$
Following [Shapiro et al., 2021], we can rewrite this as follows. Suppose we use  $Z = (X, Y) \in \mathcal{Z} := \mathcal{X} \times \mathcal{Y}$ , so that  $P, P_{\mathrm{data}}$  are distributions over  $Z$ . We then define  $\ell_h: \mathcal{Z} \mapsto \mathbb{R}$  as  $\ell_h(z) = \ell(h(x), y)$ . We can then write the population RAI risk as (see Appendix G for a proof):
$$
R _ {W} (h) = \sup  _ {r: \mathcal {Z} \mapsto \mathbb {R} _ {+}, \int r (z) d P _ {\mathrm {d a t a}} (z) = 1, r \in W} \mathbb {E} _ {P _ {\mathrm {d a t a}}} [ r (z) \ell_ {h} (z) ]. \tag {4}
$$
For classification, we define the RAI-Bayes optimal classifier as:  $Q_W^* = \arg \min_Q R_W(Q)$ . Here, the minimum is w.r.t the set of all measurable classifiers (both deterministic and random). This is the "target" classifier we wish to learn given finite samples. Note that this might not be the same as the vanilla Bayes optimal classifier:  $Q^* = \arg \min_Q \mathbb{E}[\widehat{R}(Q)]$ , which only minimizes the expected loss, and hence may not satisfactorily address RAI considerations.
We now try to characterize the RAI-Bayes optimal classifier. However, doing this requires a bit more structure on  $W$ . So, in the sequel, we consider constraint sets of the following form:
$$
W = \left\{r: \mathcal {Z} \mapsto \mathbb {R} _ {+}: \int g _ {i} (r (z)) d P _ {\mathrm {d a t a}} (z) \leq c _ {i}, i \in [ m ] \right\}, \tag {5}
$$
where we assume that  $g_{i}:\mathbb{R}_{+}\mapsto \mathbb{R}, i\in [m]$  are convex. Note that this choice of  $W$  encompasses a broad range of RAI games including DRO with  $f$ -divergence, CVaR, soft-margin uncertainty sets. Perhaps surprisingly, the following proposition shows that the minimizer of population RAI risk is nothing but the vanilla Bayes optimal classifier.
Proposition 9 (Bayes optimal classifier) Consider the problem of binary classification where  $\mathcal{Y} = \{-1, + 1\}$ . Suppose  $\ell (h(x),y) = \phi (yh(x))$  for some  $\phi :\mathbb{R}\to [0,\infty)$  which is either the  $0 / 1$  loss, or a convex loss function that is differentiable at 0 with  $\phi^{\prime}(0) < 0$ . Suppose the uncertainty set  $W$  is as specified in Equation (5). Moreover, suppose  $\{g_i\}_{i = 1\dots m}$  are convex and differentiable functions. Then, the vanilla Bayes optimal classifier is also a RAI-Bayes optimal classifier.
Remark 10 In the special case of  $m = 1$  in Equation (5), we recover the result of [Hu et al., 2018]. However, our proof is much more elegant than the proof of [Hu et al., 2018], and relies on the dual representation of the population RAI risk.
One perspective of the above result is that the vanilla Bayes optimal classifier is also "responsible" as specified by the RAI game. This is actually reasonable in many practical prediction problems where the label annotations are actually derived from humans, who presumably are also responsible. Why then might we be interested in the RAI risk? One advantage of the RAI risks is in finite sample settings where the equivalence no longer holds, and the RAI risk could be construed as encoding prior knowledge about properties of the Bayes optimal classifier. We also note that the above equivalence is specific for binary classification.
6.2 Generalization Guarantees
Our generalization bounds rely on the following dual characterization of the RAI population risk.
Proposition 11 Suppose the uncertainty set  $W$  is as specified in Equation (5). Then for any hypothesis  $h$ , the population RAI risk can be equivalently written as
$$
R _ {W} (h) = \inf  _ {\lambda \geq \mathbf {0}, \tau} \mathbb {E} _ {P _ {d a t a}} G _ {\lambda} ^ {*} (\ell_ {h} (z) - \tau) + \sum_ {i = 1} ^ {m} \lambda_ {i} c _ {i} + \tau , \tag {6}
$$
where  $G_{\lambda}^{*}$  is the Fenchel conjugate of  $G_{\lambda}(t) = \sum_{i=1}^{m} \lambda_i g_i(t)$ .
We utilize the above expression for  $R_W(h)$  to derive the following deviation bound for  $\widehat{R}_{W_n}(h)$ .
Theorem 12 Consider the setting of Proposition 11. Suppose  $\{g_i\}_{i = 1\dots m}$  are convex and differentiable functions. Suppose  $\ell_h(z)\in [0,B]$  for all  $h\in H$ $z\in \mathcal{Z}$ . Suppose, for any distribution  $P_{data}$ , the minimizers  $(\lambda^{*},\tau^{*})$  of Equation (6) lie in the following set:  $\mathcal{E} = \{(\lambda ,\tau):\| \lambda^{*}\|_{\infty}\leq \bar{\Lambda},|\tau^{*}|\leq T\}$ . Moreover, let's suppose the optimal  $\lambda^*$  for  $P_{data}$  is bounded away from 0 and satisfies  $\min_i\lambda_i^*\geq \underline{\Lambda}$ . Let  $G,L$  be the range and Lipschitz constants of  $G_{\lambda}^{*}$ :
$$
G := \sup  _ {(\lambda , \tau) \in \mathcal {E}} G _ {\lambda} ^ {*} (B - \tau) - G _ {\lambda} ^ {*} (- \tau), \quad L := \sup  _ {x \in [ 0, B ], (\lambda , \tau) \in \mathcal {E}, \lambda : \min  _ {i} \lambda_ {i} \geq \underline {{\Lambda}}} \left| \left| \frac {\partial G _ {\lambda} ^ {*} (x - \tau)}{\partial (\lambda , \tau)} \right| \right| _ {2}.
$$
For any fixed  $h \in H$ , with probability at least  $1 - 2e^{-t}$
$$
\left| R _ {W} (h) - \widehat {R} _ {W _ {n}} (h) \right| \leq 1 0 n ^ {- 1 / 2} G (\sqrt {t + m \log (n L)}).
$$
Given Theorem 12, one can take a union bound over the hypothesis class  $H$  to derive the following uniform convergence bounds.
Corollary 13 Let  $N(H, \epsilon, \| \cdot \|_{L^{\infty}(\mathcal{Z})})$  be the covering number of  $H$  in the sup-norm which is defined as  $\| h \|_{L^{\infty}(\mathcal{Z})} = \sup_{z \in \mathcal{Z}} |h(z)|$ . Then with probability at least  $1 - N(H, \epsilon_n, \| \cdot \|_{L^{\infty}(\mathcal{Z})}) e^{-t}$ , the following holds for any  $h \in H$ :  $|R_W(h) - \widehat{R}_{W_n}(h)| \leq 30n^{-1/2} G(\sqrt{t + m\log(nL)})$ . Here  $\epsilon_n = n^{-1/2} G\sqrt{t + m\log(nL)}$ .
The above bound depends on parameters  $(\lambda^{*},\tau^{*},G,L)$  which specific to the constraint set  $W$ . To instantiate it for any  $W$  one needs to bound these parameters. We note that our generalization guarantees become sub-optimal as  $\underline{\Lambda}\rightarrow 0$ . This is because the Lipschitz constant  $L$  could potentially get larger as  $\lambda$  approaches the boundary. Improving these bounds is an interesting future direction.
Remark 14 We note that aforementioned results result from relatively stringent assumptions. Exploring the impact of relaxing these assumptions is an interesting direction for future works.
In this section, we demonstrate the generality of proposed RAI methods by studying one of the most well-studied problems in RAI i.e. the case of subpopulation shift. Given a large number of possible  $W$ , we acknowledge that this is not a complete analysis, even with respect to the problems that live within the minimax framework. Instead, we aim to display convergence, plug-and-play generality, and superior performance over some seminal baselines of this task. We conduct experiments on both synthetic and real-world datasets. Please refer to Appendix for details on synthetic experiments. We consider a number of responsible AI settings, including subpopulation shift, in the domain-oblivious (DO) setting where we do not know the sub-populations [Hashimoto et al., 2018, Lahoti et al., 2020, Zhai et al., 2021a], the domain-aware (DA) setting where we do [Sagawa et al., 2019], and the partially domain-aware (PDA) setting where only some might be known.
Datasets & Domain Definition. We use the following datasets: COMPAS [Angwin et al., 2016], CIFAR-10 (original, and with a class imbalanced split [Jin et al., 2021, Qi et al., 2021]) and CIFAR-100. See the Appendix for more details on our datasets. For COMPAS, we consider race (White vs Other) and biological gender (Male vs Female) as our sensitive attributes. This forms four disjoint subgroups defined by these attributes. In the PDA setting, we partition only across the attribute race while training, but still run tests for all four subgroups. On CIFAR-10, class labels define our 10 subpopulations. Similarly as above, for the PDA setting, we make 5 super-groups of two classes each. On CIFAR-100, class labels define our 100 subpopulations. For the PDA setting, we make 20 super-groups, each consisting of five classes.
Baselines. We compare our method against the following baselines: (a) Deterministic classifiers trained on empirical risk (ERM) and DRO risks, particularly the quasi-online algorithm for Group DRO [Sagawa et al., 2019] (Online GDRO), and an ITLM-inspired SGD algorithm [Zhai et al., 2021b, Shen and Sanghavi, 2018] for  $\chi^2$  DRO ( $\mathrm{SGD}(\chi^2)$ ) (b) Ensemble models AdaBoost [Schapire, 1999]. Note that the purpose of our experiments is to show that we can match baselines for a specific single desideratum (e.g. worst-case sub-population) while allowing for learning models that can solve multiple responsible AI desiderata at the same time, for which we have no existing baselines.
Proposed Methods. We focus on Algorithm 2 and refer to FW and Gen-AdaBoost updates as RAI-FW and RAI-GA, respectively. Moreover, our implementations include the following alterations: - We track the unregularized objective value from Equation 1 for the validation set, and whenever it increases we double the regularization factor  $\eta$ , which we find can improve generalization. - We also use this objective w.r.t the normalized  $Q^t$  to perform a line search for the step size  $\alpha$ . For the FW update, our search space is a ball around  $\frac{1}{t}$  at round  $t$ , while for GA, we search within  $(0,1)$ .
Base Learners & Training. Training time scales linearly with the number of base learners. Inference, though, can be parallelized if need be. We usually find training on 3-5 learners is good enough on all scenarios explored in the paper. We defer further details of our base learners and hyperparameter choices to the Appendix.
Constraint sets  $\mathbf{W}_{\mathbf{n}}$ . For RAI algorithms, we use the following constraint sets: • Domain Oblivious (DO): We use the  $\chi^2$ -DRO constraint set to control for worst-case subpopulations. • Domain Aware (DA): We use the Group DRO constraint set as the domain definitions are known. • Partially Domain-Aware (PDA): We use a novel set  $W_{n}$  which is the intersection over Group DRO constraints over the known domains and  $\chi^2$  constraints to control for unknown group performance. For baselines, we use AdaBoost and  $\mathrm{SGD}(\chi^2)$  for the DO setting. Online GDRO serves as our baseline for both DA and PDA settings, where the algorithm uses whatever domain definitions are available.
Table 2: (Table 1 in the paper) Mean and worst-case expected loss for baselines, RAI-GA and RAI-FW. (Complex) indicates the use of larger models. Constraint sets  $W_{n}$  are indicated in (.). Each experiment is carried out over three random seeds and confidence intervals are reported.
Setting	Algorithm	COMPAS	CIFAR-10 (Imbalanced)	CIFAR10	CIFAR100
Average	Worst Group	Average	Worst Class	Average	Worst Class	Average	Worst Class
DO (Complex)	ERM	31.3 ±0.2	31.7 ±0.1	12.1 ±0.3	30.4 ±0.2	8.3 ±0.2	21.3 ±0.5	25.2 ±0.2	64.0 ±0.7
RAI-GA (χ2)	31.3 ±0.2	31.2 ±0.2	11.7 ±0.4	29.0 ±0.3	8.2 ±0.1	19.0 ±0.1	25.6 ±0.4	56.8 ±0.8
RAI-FW (χ2)	31.2 ±0.1	31.4 ±0.3	11.9 ±0.1	29.1 ±0.2	8.0 ±0.3	15.4 ±0.4	25.4 ±0.2	58.0 ±1.1
DO	ERM	32.1 ±0.3	34.6 ±0.4	14.2 ±0.1	33.6 ±0.3	11.4 ±0.4	27.0 ±0.1	27.1 ±0.3	66.0 ±1.1
AdaBoost	31.8 ±0.4	32.6 ±0.3	15.2 ±0.2	40.6 ±0.2	12.0 ±0.1	28.7 ±0.3	28.1 ±0.2	72.2 ±1.2
SGD (χ2)	32.0 ±0.2	33.7 ±0.2	13.3 ±0.3	31.7 ±0.4	11.3 ±0.3	24.7 ±0.1	27.4 ±0.1	65.9 ±1.2
RAI-GA (χ2)	31.5 ±0.2	33.2 ±0.3	14.0 ±0.1	32.2 ±0.2	10.8 ±0.4	25.0 ±0.2	27.4 ±0.4	65.0 ±0.8
RAI-FW (χ2)	31.6 ±0.1	32.5 ±0.5	13.9 ±0.1	32.6 ±0.3	10.9 ±0.4	23.4 ±0.2	27.5 ±0.1	63.8 ±0.6
DA	Online GDRO	31.7 ±0.2	32.2 ±0.3	13.1 ±0.2	26.6 ±0.2	11.2 ±0.1	21.7 ±0.3	27.3 ±0.1	57.0 ±0.5
RAI-GA (Group)	32.0 ±0.1	32.7 ±0.1	13.0 ±0.3	27.3 ±0.4	11.5 ±0.1	22.4 ±0.2	27.4 ±0.2	56.6 ±1.1
RAI-FW (Group)	32.1 ±0.2	32.3 ±0.2	13.0 ±0.2	26.0 ±0.1	11.4 ±0.3	20.3 ±0.1	27.9 ±0.2	52.9 ±0.9
PDA	Online GDRO	31.5 ±0.1	32.7 ±0.2	13.4 ±0.1	32.2 ±0.2	11.3 ±0.2	25.2 ±0.1	27.7 ±0.2	64.0 ±0.8
RAI-GA (Group∩χ2)	31.4 ±0.4	32.9 ±0.2	13.0 ±0.3	30.1 ±0.1	10.8 ±0.2	23.7 ±0.2	27.5 ±0.1	62.5 ±0.6
RAI-FW (Group∩χ2)	31.8 ±0.2	32.3 ±0.1	13.5 ±0.3	29.4 ±0.3	11.2 ±0.4	24.0 ±0.2	27.9 ±0.3	58.9 ±0.7
Results and Discussion. We run our methods and baselines under the settings described above and report the results in Table 2. As such, we can make the following observations:
1. RAI-FW and RAI-GA methods significantly improve the worst-case performance with only a few base learners across all datasets in all three settings, while maintaining average case performance. Moreover, For seemingly harder tasks i.e. a large gap between average and worst-case performance, the algorithms are able to improve significantly over the baselines. For example, we observe a  $5\%$  improvement in performance in the case of CIFAR-100.
2. The plug-and-play framework allows for several different  $W_{n}$  to enhance various responsible AI qualities at once. We demonstrate this with the partial domain aware setting (PDA), where the performance lead widens, indicating that RAI is able to jointly optimize effectively for both known and unknown subpopulations while Online GDRO suffers from some of the group information being unknown. In practice, one can construct many more novel sets  $W_{n}$ .
3. Although bigger (complex) models exhibit stronger performance than RAI ensembles, there are several caveats to this observation. Firstly, these models are  $\sim 10 - 15$  times larger than our base models. This limits their use w.r.t both training & inference compute required. However, RAI ensembles utilize a small number of much smaller models which can be individually trained quite easily. Even with these large models as base learners, constructing ensembles exhibits a performance boost, indicating that our framework is able to "boost" models of varying complexities.
Under the umbrella of "responsible AI", an emerging line of work has attempted to formalize desiderata ranging over ethics, fairness, robustness, and safety, among others. Many of these settings (Table 1) can be written as min-max problems involving optimizing some worst-case loss under a set of predefined distributions. For all the problems that can be framed as above, we introduce and study a general framework, which we refer to as Responsible AI (RAI) games. Our framework extends to classical boosting scenarios, offering boosting-based algorithms for RAI games alongside proven convergence guarantees. We propose practical algorithms to solve these games, as well as statistical analyses of solutions of these games. We find that RAI can guarantee multiple responsible AI aspects under appropriate choices of uncertainty sets.
We acknowledge the support of DARPA via HR00112020006, and NSF via IIS-1909816.
Microsoft. Microsoft. https://www.microsoft.com/en-us/ai/responsible-ai, 2021. Accessed: Date Accessed.
Google. Google. https://ai.google/responsibility/responsible-ai-practices/, 2020. Accessed: Date Accessed.
Hongseok Namkoong and John C Duchi. Variance-based regularization with convex objectives. Advances in neural information processing systems, 30, 2017.
John Duchi and Hongseok Namkoong. Learning models with uniform performance via distributionally robust optimization. arXiv preprint arXiv:1810.08750, 2018.
Shiori Sagawa, Pang Wei Koh, Tatsunori B Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. arXiv preprint arXiv:1911.08731, 2019.
Runtian Zhai, Chen Dan, Arun Suggala, J Zico Kolter, and Pradeep Ravikumar. Boosted cvar classification. Advances in Neural Information Processing Systems, 34:21860-21871, 2021a.
Tatsunori B. Hashimoto, Megha Srivastava, Hongseok Namkoong, and Percy Liang. Fairness without demographics in repeated loss minimization. International Conference On Machine Learning, 2018.
Runtian Zhai, Chen Dan, Zico Kolter, and Pradeep Ravikumar. Doro: Distributional and outlier robust optimization. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 12345-12355. PMLR, 18-24 Jul 2021b. URL https://proceedings.mlr.press/v139/zhai21a.html.
Yuji Roh, Kangwook Lee, Steven Euijong Whang, and Changho Suh. Fr-train: A mutual information-based approach to fair and robust training. arXiv preprint arXiv: Arxiv-2002.10234, 2020.
Sanjeev Arora, Elad Hazan, and Satyen Kale. The multiplicative weights update method: a metaalgorithm and applications. Theory of computing, 8(1):121-164, 2012.
Brendan McMahan. Follow-the-regularized-leader and mirror descent: Equivalence theorems and 11 regularization. In Geoffrey Gordon, David Dunson, and Miroslav Dudík, editors, Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, volume 15 of Proceedings of Machine Learning Research, pages 525–533, Fort Lauderdale, FL, USA, 11–13 Apr 2011. PMLR.
Sebastien Bubeck. Introduction to online optimization. Lecture notes, 2:1-86, 2011.
Weihua Hu, Gang Niu, Issei Sato, and Masashi Sugiyama. Does distributionally robust supervised learning give robust classifiers? In International Conference on Machine Learning, pages 2029-2037. PMLR, 2018.
Peter Buhlmann. Invariance, causality and robustness. arXiv preprint arXiv: 1812.08233, 2018.
Soroosh Shafieezadeh-Abadeh, Peyman Mohajerin Esfahani, and D. Kuhn. Distributionally robust logistic regression. Neural Information Processing Systems, 2015.
Rui Gao and Anton Kleywegt. Distributionally robust stochastic optimization with Wasserstein distance. Mathematics of Operations Research, 2022.
Hongseok Namkoong and John C. Duchi. Stochastic gradient methods for distributionally robust optimization with f-divergences. In Neural Information Processing Systems, 2016. URL https://api-semanticscholar.org/CorpusID:7481496.