SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yash Gupta
yashgup2@cs.cmu.edu
Arun Suggala
Runtian Zhai
rzhai@cs.cmu.edu
pradeepr@cs.cmu.edu
Several recent works have studied the societal effects of AI; these include issues such as fairness, robustness, and safety. In many of these objectives, a learner seeks to minimize its worst-case loss over a set of predefined distributions (known as uncertainty sets), with usual examples being perturbed versions of the empirical distribution. In other words, aforementioned problems can be written as min-max problems over these uncertainty sets. In this work, we provide a general framework for studying these problems, which we refer to as Responsible AI (RAI) games. We provide two classes of algorithms for solving these games: (a) game-play based algorithms, and (b) greedy stagewise estimation algorithms. The former class is motivated by online learning and game theory, whereas the latter class is motivated by the classical statistical literature on boosting, and regression. We empirically demonstrate the applicability and competitive performance of our techniques for solving several RAI problems, particularly around subpopulation shift.
In recent years, AI is increasingly being used in high-stakes decision-making contexts such as hiring, criminal justice, and healthcare. Given the impact these decisions can have on people's lives, it is important to ensure these AI systems have beneficial social effects. An emerging line of work has attempted to formalize such desiderata ranging over ethics, fairness, train-time robustness, test-time or adversarial robustness, and safety, among others. Each of these forms rich sub-fields with disparate desiderata, which are sometimes collated under the umbrella of "responsible AI". Many organizations are increasingly advocating the use of responsible AI models [Microsoft, 2021, Google, 2020].
But how do we do so when the majority of recent work around these problems is fragmented and usually focuses on optimizing one of these aspects at a time (DRO [Namkoong and Duchi, 2017, Duchi and Namkoong, 2018], GDRO [Sagawa et al., 2019], CVaR [Zhai et al., 2021a], Distribution Shift [Hashimoto et al., 2018, Zhai et al., 2021b])? Indeed optimizing for just one of these aspects has even been shown to exhibit adverse effects on the other aspects [Roh et al., 2020]. To address this, we study a general framework that is broadly applicable across many of the settings above, and which we refer to as Responsible AI (RAI) games. Our starting point is the recent understanding of a unifying theme in many of these disparate problems, that a learner seeks to minimize its worst-case loss over a set of predefined distributions. For example, in fairness, we seek to perform well on all sub-groups in the data. In robustness, we aim to design models that are robust to perturbations of the training data or the test distribution. This allows us to set up a zero-sum game between a learner that aims to learn a responsible model and an adversary that aims to prevent the learner from doing so. In
The relevant code for this work can be found at https://github.com/yashgupta-7/rai-games
the general RAI game setting, this is a computationally intractable game that need not even have a Nash equilibrium. To address this computational issue, we study a relaxation of the single predictor RAI game, which we term the ensemble RAI game, which can also be motivated as a linearization of the original RAI game.
We note that our framework encompasses not only the responsible AI settings but also the setting of classical boosting. Drawing upon the insights from boosting, we provide boosting-based algorithms for solving responsible AI games. We provide convergence guarantees of our algorithms by relying on the connections between boosting and online convex optimization, two-player gameplay [Arora et al., 2012, McMahan, 2011, Bubeck, 2011]. We also conduct empirical analyses to demonstrate the convergence and utility of our proposed algorithms. Interestingly, the algorithms allow for plug-and-play convenience, with changes in the RAI settings requiring only simple changes to the algorithms. More importantly, we could consider intersections of different responsible AI considerations, which in turn can simply be incorporated into our algorithms. Finally, we also study the population risks of our algorithms in certain important settings. We show a surprising result that for the case of binary classification with the  $0/1$  loss, the optimal predictor for a large class of RAI games is the same as the Bayes optimal predictor, thus generalizing an emerging line of results demonstrating this for certain specific games [Hu et al., 2018]. Under such settings, solving the RAI game could nonetheless be helpful in finite sample settings (as also demonstrated in our experiments) since the RAI game serves to encode desiderata satisfied by the Bayes optimal classifier.
2 Problem Setup and Background
We consider the standard supervised prediction setting, with input random variable  $X \in \mathcal{X} \subseteq \mathbb{R}^d$ , output random variable  $Y \in \mathcal{Y}$ , and samples  $S = \{(x_i, y_i)\}_{i=1}^n$  drawn from a distribution  $P_{\mathrm{data}}$  over  $\mathcal{X} \times \mathcal{Y}$ . Let  $\widehat{P}_{\mathrm{data}}$  denote the empirical distribution over the samples. We also have a set  $H$  of hypothesis functions  $h: \mathcal{X} \mapsto \mathcal{Y}$  from which we wish to learn the best predictor. We evaluate the goodness of a predictor via a loss function  $\ell: \mathcal{Y} \times \mathcal{Y} \mapsto \mathbb{R}$ , which yields the empirical risk:  $\widehat{R}(h) = \mathbb{E}_{\widehat{P}_{\mathrm{data}}} \ell(h(x), y)$  where  $\mathbb{E}_{\widehat{P}_{\mathrm{data}}} (f(x, y)) = \frac{1}{n} \sum_{i=1}^{n} f(x_i, y_i)$ . Apart from having low expected risk, most settings require  $h$  to have certain properties, for example, robustness to distribution shift, fairness w.r.t subpopulations, superior tail performance, resistance to adversarial attacks, robustness in the presence of outliers, etc. We cast all these subproblems into an umbrella term "Responsible AI". Each of these properties has been studied extensively in recent works, albeit individually. In this work, we attempt to provide a general framework to study these problems.
We draw our unified framework from seminal works over the past decade by responsible AI researchers on devising non-expected risk objectives, particularly min-max problems, to ensure ML models are responsible. These have resulted in a multitude of different objectives (even for a single responsible AI desideratum such as fairness), and also multiple different sub-communities (so that fairness and multiple disparate robustness communities are relatively fractured), many (if not all) of which we combine within a single umbrella. There is emerging work on relating worst-case performance to invariance [Buhlmann, 2018]; in other words, we might be able to get approximate group invariance via minimizing an appropriately constructed worst-group risk and vice-versa.
RAI aspects as constraints. Many prior works have enforced robustness as a constrained optimization [Shafieezadeh-Abadeh et al., 2015, Gao and Kleywegt, 2022, Namkoong and Duchi, 2016, Ben-Tal et al., 2011]. There have also been few prior works enforcing fairness constraints [Mandal et al., 2020]. To the best of our knowledge, there exists minimal prior work focusing on multiple desiderata at once in this regard.
Multi Objective Optimization. Several works have considered a multi-objective view of ensuring fairness in classifiers [Martinez et al., 2020, Oneto et al., 2018]. If used for multiple RAI objectives, there is usually overhead in choosing a model that achieves a good trade-off between various losses. Also, it is difficult to guarantee that the solution is robust to any of the involved aspects. Our framework guarantees a certain level of performance on each of the RAI aspects under consideration.
Distribution shift. [Koh et al., 2021] classifies distribution shift problems into two categories: Domain generalization, and subpopulation shift. In this work, we focus on the subpopulation shift problem, where the target distribution is absolutely continuous to the source distribution. It has two
main applications: fairness [Hashimoto et al., 2018, Hu et al., 2018, Sagawa et al., 2019, Zhai et al., 2021b] and long-tail learning (i.e. learning on class-imbalanced datasets) [Cao et al., 2019, Menon et al., 2021, Kini et al., 2021].
Distributionally Robust Optimization (DRO). In DRO one aims to study classifiers that are robust to deviations of the data distribution. DRO has been studied under various uncertainty sets including  $f$ -divergence based uncertainty sets [Namkoong and Duchi, 2017, Duchi and Namkoong, 2018, Sagawa et al., 2019], Wasserstein uncertainty sets [Sinha et al., 2017, Gao et al., 2022], Maximum Mean Discrepancy uncertainty sets [Staib and Jegelka, 2019], more general uncertainty sets in the RKHS space [Zhu et al., 2020]. [Li et al., 2021a] evaluate model performance under worst-case subpopulations. Owing to its importance, several recent works have provided efficient algorithms for solving the DRO objective [Namkoong and Duchi, 2016, Qi et al., 2020, Kumar et al., 2023, Jin et al., 2021]. However, a lot of these techniques are specific to particular perturbation sets and are not directly applicable to the more general framework we consider in our work. Furthermore, in our work, we aim to learn an ensemble of models instead of a single model.
Boosting. Classical boosting aims to improve the performance of a weak learner by combining multiple weak classifiers to produce a strong classifier [Breiman, 1999, Friedman et al., 2000, Friedman, 2001, Freund and Schapire, 1995, Freund et al., 1996, Mason et al., 2000]. Over the years, a number of practical algorithms have been introduced such as AdaBoost [Schapire, 1999], LPBoost [Demiriz et al., 2002], gradient boosting [Mason et al., 1999], XGBoost [Chen and Guestrin, 2016], boosting for adversarial robustness [Zhang et al., 2022], [Meunier et al., 2021], [Balcan et al., 2023], and holistic robustness [Bennouna and Parys, 2022]. The algorithms we develop for RAI games are inspired by these algorithms.
Fairness. There are a number of fairness notions for algorithmic fairness, ranging from individual fairness [Dwork et al., 2012, Zemel et al., 2013], group fairness [Hardt et al., 2016a, Zafar et al., 2017], counterfactual fairness [Kusner et al., 2017], Rawlsian max-min fairness [Rawls, 2020, Hashimoto et al., 2018] and others [Barocas et al., 2017, Chouldechova and Roth, 2018, Mehrabi et al., 2021]. Our framework includes the popular notion of minimax group fairness. It doesn't capture other notions of group fairness such as Demographic Parity, Equality of Odds, Equality of Opportunity.
Population RAI Risks. Several recent works have studied properties of the population risks arising in various responsible AI scenarios. Hu et al. [2018] showed that the minimizer of population DRO risk (under general  $f$ -divergences) is the classical Bayes optimal classifier. Li et al. [2021b], Duchi and Namkoong [2018], Sinha et al. [2017] provided generalization guarantees for DRO risk under various divergence families ranging from  $f$ -divergences to Wasserstein perturbations.
3 RAI Games
In many cases, we do not wish to compute an unweighted average over training samples; due to reasons of noise, tail risk, robustness, and fairness, among many other "responsible AI" considerations.
Definition 1 (RAI Risks) Given a set of samples  $\{(x_i, y_i)\}_{i=1}^n$ , we define the class of empirical RAI risks (for Responsible AI risks) as:  $\widehat{R}_{W_n}(h) = \sup_{w \in W_n} \mathbb{E}_w(h(x), y)$ , where  $W_n \subseteq \Delta_n$ , is some set of sample weights (a.k.a uncertainty set), and  $\mathbb{E}_w(f(x, y)) = \sum_{i=1}^n w_i f(x_i, y_i)$ .
Various choices of  $W_{n}$  give rise to various RAI risks. Table 1 presents examples of RAI risks that are popular in ML. Interestingly, classical problems such as boosting are special cases of RAI risks. In this work, we rely on this connection to design boosting-inspired algorithms for minimizing RAI risks. More choices for  $W_{n}$  can be obtained by combining the one's specified in Table 1 using union, intersection, convex-combination operations. For example, if one wants models that are fair to certain pre-specified groups, and at the same-time achieve good tail-risk, then one could choose  $W_{n}$  to be the intersection of Group-DRO and  $\alpha$ -CVaR uncertainty sets.
Given the empirical RAI risk  $\widehat{R}_{W_n}(h)$  of a hypothesis, and set of hypotheses  $H$ , we naturally wish to obtain the hypothesis that minimizes the empirical RAI risk:  $\min_{h\in H}\widehat{R}_{W_n}(h)$ . This can be seen as solving a zero-sum game.
Definition 2 (RAI Games) Given a set of hypothesis  $H$ , and a RAI sample weight set  $W_{n}$ , the class of RAI games is given as:  $\min_{h\in H}\max_{w\in W_n}\mathbb{E}_w(h(x),y)$ .
We thus study RAI Games for the special cases above and for an arbitrary constraint set  $W_{n}$ .
Name	\(W_{n}\)	Description
Empirical Risk Minimization	\(\{\widehat{P}_{\text{data}}\}\)	object of focus in most of ML/AI
Worst Case Margin	\(\Delta_{n}\), entire probability simplex	used for designing margin-boosting algorithms [Warmuth et al., 2006, Bartlett et al., 1998]
Soft Margin	\(\{w:KL(w||\widehat{P}_{\text{data}})\leq\rho_{n}\}\)	used in the design of AdaBoost [Freund and Schapire, 1995]
α-Conditional Value at Risk (CVaR)	\(\{w:w\in\Delta_{n},w\preceq\frac{1}{\alpha n}\}\)	used in fairness [Zhai et al., 2021a, Sagawa et al., 2019]
Distributionsally Robust Optimization (DRO)	\(\{w:D(w||\widehat{P}_{\text{data}})\leq\rho_{n}\}\)	various choices for D have been studied f-divergence [Duchi and Namkoong, 2018]
Group DRO	\(\{\widehat{P}_{\text{data}}(G_1),\widehat{P}_{\text{data}}(G_2),\ldots,\widehat{P}_{\text{data}}(G_K)\}\) \(\widehat{P}_{\text{data}}(G_i)\) is dist. ofithgroup	used in group fairness, agnostic federated learning [Mohri et al., 2019]
Table 1: Various ML/AI problems that fall under the umbrella of RAI risks.
4 Ensemble RAI Games
In this section, we begin our discussion about ensembles. In general, a statistical caveat with Definition 2 is that good worst-case performance over the sample weight set  $W_{n}$  is generally harder, and for a simpler set of hypotheses  $H$ , there may not exist  $h \in H$  that can achieve such good worst-case performance. Thus it is natural to consider deterministic ensemble models over  $H$ , which effectively gives us more powerful hypothesis classes. Let us first define RAI risk for such classifiers.
Definition 3 (Deterministic Ensemble) Consider the problem of classification, where  $\mathcal{V}$  is a discrete set. Given a hypothesis class  $H$ , a deterministic ensemble is specified by some distribution  $Q \in \Delta_H$ , and is given by:  $h_{\text{det};Q}(x) = \arg \max_{y \in \mathcal{Y}} \mathbb{E}_{h \sim Q} \mathbb{I}[h(x) = y]$ . Correspondingly, we can write the deterministic ensemble RAI risk as  $\widehat{R}_{W_n}(h_{\text{det};Q}(x)) = \max_{w \in W_n} \mathbb{E}_w \ell(h_{\text{det};Q}(x), y)$ .
We discuss alternative definitions of deterministic ensembles in the Appendix. This admits a class of deterministic RAI games:
Definition 4 (Deterministic Ensemble RAI Games) Given a set of hypothesis  $H$ , a RAI sample weight set  $W_{n}$ , the class of RAI games for deterministic ensembles over  $H$  is given as:
$$
\min_{Q\in \Delta_{H}}\max_{w\in W_{n}}\mathbb{E}_{w}\ell (h_{det;Q}(x),y).
$$
However, the aforementioned game is computationally less amenable because of the non-smooth nature of de-randomized predictions. Moreover, there are some broader challenges with RAI games given by Definitions 2 and 4. Firstly, they need not have a Nash Equilibrium (NE), and in general, their min-max and max-min game values need not coincide. This poses challenges in solving the games efficiently. Next, in some cases, directly optimizing over the worst-case performance might not even be useful. For instance, [Hu et al., 2016, Zhai et al., 2021a] show the pessimistic result that for classification tasks where when models are evaluated by the zero-one loss, ERM achieves the lowest possible DRO loss defined by some  $f$ -divergence or the  $\alpha$ -CVaR loss, given that the model is deterministic. To this end, we consider the following randomized ensemble:
Definition 5 (Randomized Ensemble) Given a hypothesis class  $H$ , a randomized ensemble is specified by some distribution  $Q \in \Delta_H$ , and is given by:  $\mathbb{P}[h_{\text{rand};Q}(x) = y] = \mathbb{E}_{h \sim Q} \mathbb{I}[h(x) = y]$ . Similarly, we can define its corresponding randomized ensemble RAI risk:  $\widehat{R}_{\text{rand};W_n}(Q) = \max_{w \in W_n} \mathbb{E}_{h \sim Q} \mathbb{E}_w \ell(h(x), y)$ .
We can then also define the class of ensemble RAI games:
Definition 6 (Randomized Ensemble RAI Games) Given a set of hypothesis  $H$ , a RAI sample weight set  $W_{n}$ , the class of mixed RAI games is given as:
$$
\min  _ {Q \in \Delta_ {H}} \max  _ {w \in W _ {n}} \mathbb {E} _ {h \sim Q} \mathbb {E} _ {w} \ell (h (x), y). \tag {1}
$$
This is a much better class of zero-sum games: it is linear in both the hypothesis distribution  $P$ , as well as the sample weights  $w$ , and if the sample weight set  $W_{n}$  is convex, is a convex-concave game. As shown below, under some mild conditions, this game has a Nash equilibrium which can be well approximated via efficient algorithms.
Proposition 1 Let  $H$  be parameterized by  $\theta \in \Theta \subseteq \mathbb{R}^p$ , for convex, compact set  $\Theta$  and let  $W_{n}$  be a convex, compact set. Then  $\min_{Q\in \Delta_H}\max_{w\in W_n}\mathbb{E}_{h\sim Q}\mathbb{E}_w\ell (h(x),y) = \max_{w\in W_n}\min_{Q\in \Delta_H}\mathbb{E}_{h\sim Q}\mathbb{E}_w\ell (h(x),y)$
The proposition follows as a direct consequence of well known minimax theorems (Appendix D.3).
4.1 Going from Deterministic to Randomized Ensembles
To begin, we point out that what we want is a deterministic ensemble rather than a randomized ensemble. In fact, it can be seen that the deterministic ensemble in Definition 3 is a specific de-randomization of the randomized ensemble. It is such deterministic ensembles that we usually simply refer to as ensemble predictors. But the RAI risk for the ensemble predictor is NOT equal to the ensemble RAI risk minimized by our desired game in Equation 1 above for randomized ensembles. Thus, the ensemble RAI game might not in general capture the ideal deterministic ensemble. In this section, we study why and when might solving for a random ensemble is meaningful.
Binary Classification. Interestingly, for the very specific case of binary classification, we can provide simple relationships between the risks of the randomized and deterministic ensemble.
Proposition 2 Consider the setting with  $\mathcal{Y} = \{-1,1\}$ , the zero-one loss  $\ell$ , and  $W_{n} = \Delta_{n}$ . Then,
$$
\widehat {R} _ {W _ {n}} \left(h _ {\text {d e t}; Q}\right) = \mathbb {I} \left[ \widehat {R} _ {W _ {n}} \left(h _ {\text {r a n d}; Q}\right) \geq 1 / 2 \right].
$$
See Appendix E.2 for a simple proof. In this case, we can also relate the existence of a perfect deterministic ensemble ("boostability") to a weak learning condition on the set of hypotheses. Specifically, suppose  $H$  is boostable iff there exists  $Q \in \Delta_H$  s.t.  $\widehat{R}_{W_n}(h_{\mathrm{det};\mathbb{Q}}) = 0$ . From the above proposition this is equivalent to requiring that  $\widehat{R}_{W_n}(h_{\mathrm{rand};\mathbb{Q}}) < 1/2$ . We thus obtain:
$$
\inf  _ {Q \in \Delta_ {H}} \sup  _ {w \in W _ {n}} \mathbb {E} _ {w, Q} \ell (h (x), y) <   1 / 2 \iff \sup  _ {w \in W _ {n}} \inf  _ {h \in H} \mathbb {E} _ {w} \ell (h (x), y) <   1 / 2
$$
where the equivalence follows from the min-max theorem and the linearity of the objective in  $P$ . The last statement says that for any sample weights  $w \in W_{n}$ , there exists a hypothesis  $h \in H$  that has  $w$ -weighted loss at most  $1/2$ . We can state this as a "weak-learning" condition on individual hypotheses in  $H$ . The above thus shows that for the specific case of  $\mathcal{Y} = \{-1,1\}$ , the zero-one loss  $\ell(y, y') = \mathbb{I}[y \neq y']$ , and  $W_{n} = \Delta_{n}$ , we can relate boostability of  $H$  to a weak learning condition on hypothesis within  $H$ .
General Classification But in general, we do not have simple connections between  $\widehat{R}_{W_n}(h_{\mathrm{det};\mathrm{Q}})$  and  $\widehat{R}_{W_n}(h_{\mathrm{rand};\mathrm{Q}})$ . All we can guarantee is the following upper bound:
Proposition 3 Let  $\gamma_{Q} = 1 / \min_{i\in [n]}\max_{y\in \mathcal{Y}}\mathbb{P}_{Q}[h(x_{i}) = y]$ . Then,
$$
\widehat {R} _ {W _ {n}} \left(h _ {\text {d e t}; Q}\right) \leq \gamma_ {Q} \widehat {R} _ {W _ {n}} \left(h _ {\text {r a n d}; Q}\right).
$$
See Appendix E.2 for a simple proof.
Corollary 4 For binary classification, we have  $\gamma_{P} \leq 2$  and thus, we recover the well known bound  $\widehat{R}_{W_n}(h_{\text{det};Q}) \leq 2\widehat{R}_{W_n}(h_{\text{rand};Q})$
Remark 5 These bounds might be loose in practice. Specifically, for the binary case, if  $\widehat{R}_{W_n}(h_{rand;Q}) \leq \frac{1}{2}$  then we have  $\widehat{R}_{W_n}(h_{det;Q}) = 0$ . To this end, prior work [Lacasse et al., 2006, Germain et al., 2015, Masegosa et al., 2020] have developed tighter bounds using second-order inequalities. We leave the analyses of these second-order RAI games to future work.
As such, we can cast minimizing randomized RAI risk as minimizing an upper bound on the deterministic ensemble RAI risk. Thus, the corresponding randomized RAI game can be cast as a relaxation of the deterministic RAI game. In the sequel, we thus focus on this randomized ensemble RAI game, which we will then use to obtain a deterministic ensemble. Following the bounds above, the corresponding deterministic ensemble risk will be bounded by the randomized ensemble RAI risk
5 Algorithms
In this section, we present two algorithms for solving the RAI game in Equation (1). Our first algorithm is motivated from online learning algorithms and the second algorithm is motivated from
greedy stepwise algorithms that have been popular for solving many statistical problems such as regression. For simplicity of presentation, we assume  $H$  is a finite set. However, our results in the section extend to uncountable sets.
Game-play. In game play based algorithms, both the min and the max players are engaged in a repeated game against each other. Both players rely on no-regret algorithms to decide their next action. It is well known that such a procedure converges to a mixed NE of the game Cesa-Bianchi and Lugosi [2006]. In this work, we follow a similar strategy to solve the game in Equation (1) (see Algorithm 1 for the pseudocode). In the  $t^{th}$  round of our algorithm, the following distribution  $w^{t} \in W$  is computed over the training data points
$$
w ^ {t} \leftarrow \operatorname {a r g m a x} _ {w \in W _ {n}} \sum_ {s = 1} ^ {t - 1} \mathbb {E} _ {w} \ell \left(h ^ {s} (x), y\right) + \eta^ {t - 1} \operatorname {R e g} (w) \tag {2}
$$
This update is called the Follow-The-Regularized-Leader (FTRL) update. Here,  $\mathrm{Reg}(\cdot)$  is a strongly concave regularizer and  $\eta^{t-1}$  is the regularization strength. One popular choice for  $\mathrm{Reg}(\cdot)$  is the negative entropy which is given by  $-\sum_{i} w_i \log w_i$ . This regularizer is also used by AdaBoost, which is a popular boosting algorithm. In Appendix F.2, we provide analytical expressions for  $w^t$  for various choices of  $W_n$ ,  $\mathrm{Reg}(\cdot)$ . We note that the regularizer in the FTRL update ensures the stability of the updates; i.e., it ensures consecutive iterates do not vary too much. This stability is naturally guaranteed when  $W_n$  is a strongly convex set (an example of a strongly convex set is the level set of a strongly convex function. See Appendix for a formal definition and more details). Consequently, the regularization strength  $\eta^{t-1}$  could be set to 0 in this case, and the algorithm still converges to a NE [Huang et al., 2017].