SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Parikshit Gopalan  
Apple, Inc.
Michael P. Kim  
UC Berkeley
Omer Reingold
Stanford University
We introduce and study Swap Agnostic Learning. The problem can be phrased as a game between a predictor and an adversary: first, the predictor selects a hypothesis  $h$ ; then, the adversary plays in response, and for each level set of the predictor  $\{x \in \mathcal{X} : h(x) = v\}$  selects a loss-minimizing hypothesis  $c_v \in \mathcal{C}$ ; the predictor wins if  $p$  competes with the adaptive adversary's loss. Despite the strength of the adversary, our main result demonstrates the feasibility Swap Agnostic Learning for any convex loss. Somewhat surprisingly, the result follows by proving an equivalence between Swap Agnostic Learning and swap variants of the recent notions Omniprediction [15] and Multicalibration [20]. Beyond this equivalence, we establish further connections to the literature on Outcome Indistinguishability [6, 14], revealing a unified notion of OI that captures all existing notions of omniprediction and multicalibration.
Since its inception as an extension to Valiant's PAC framework [30, 19, 23], Agnostic Learning has been the central problem of supervised learning theory. Agnostic learning frames the task of supervised learning through loss minimization: given a loss function  $\ell$ , a hypothesis class  $\mathcal{C}$ , and  $\varepsilon \geq 0$ , a predictor  $h$  is an agnostic learner if it achieves loss that competes with the minimal achievable within the hypothesis class.
$$
\mathbf {E} [ \ell (\mathbf {y}, h (\mathbf {x})) ] \leq \min  _ {c \in \mathcal {C}} \mathbf {E} [ \ell (\mathbf {y}, c (\mathbf {x})) ] + \varepsilon \tag {1}
$$
While the agnostic learning paradigm has been remarkably successful, in recent years, researchers have investigated alternative learning paradigms to address concerns of the modern prediction pipeline, including fairness and robustness. The work of [20] introduced multicalibration as a new paradigm for learning fair predictors. Multicalibration asserts fairness as a first-order goal, requiring that predictions appear calibrated even conditioned on membership in one of a potentially-huge collection of subgroups  $\mathcal{C}$  of the domain. As a solution concept, multicalibration can be achieved efficiently using a weak learner for  $\mathcal{C}$  to identify subgroups where predictions are miscalibrated. In contrast to agnostic learning, multicalibration does not make reference to minimizing any loss.
Yet, it turns out that there are surprising connections between multicalibration and agnostic learning. This connection was first discovered in the work of [15], who introduced the notion of omniprediction as a new solution concept in supervised learning. Intuitively, an omnipredictor is a single predictor that provides the agnostic learning guarantee for many losses simultaneously. More formally, for a collection of loss functions  $\mathcal{L}$  and a hypothesis class  $\mathcal{C}$ , a predictor is an omnipredictor if for any loss in the collection  $\ell \in \mathcal{L}$ , the predictions achieve loss that competes with the minimal achievable
This collection of subgroups is suggestively denoted by  $\mathcal{C}$ , indicating that (in correspondence with a hypothesis class) subgroup membership can be computed using models of bounded capacity, such as small decision trees, halfspaces, or neural networks.
within the hypothesis class. $^2$  The main result of [15] demonstrates that multicalibrated predictors are omnipredictors for  $\mathcal{L}_{\mathrm{cvx}}$  the class of all convex loss functions. In other words, the multicalibration framework is capable of guaranteeing agnostic learning in a very strong sense: learning a single multicalibrated predictor gives an agnostic learner, for every target convex loss  $\ell \in \mathcal{L}_{\mathrm{cvx}}$ . The results of [15] stand in contrast to the conventional wisdom that optimizing predictions for different loss functions requires a separate training procedure for each loss.
On the surface, multicalibration seems like a fundamentally different approach to supervised learning than agnostic learning. The implication of omniprediction from multicalibration, however, suggests a deeper connection between the notions. In fact, follow-up work of [14] gave new constructions of omnipredictors (for different loss classes  $\mathcal{L}$ ), similarly deriving the guarantees from variants of multicalibration. To summarize the state of the art, we have constructions of omnipredictors of various flavors, and all such constructions rely on some variant of multicalibration. While multicalibration suffices to guarantee omniprediction, a glaring question remains in the development of this theory: Is multicalibration necessary for omniprediction? We investigate this question, exploring the connections between agnostic learning, notions of omniprediction, and multicalibration.
Our Contributions. In this work, we provide a new perspective that unifies these seemingly incomparable paradigms for supervised learning. Key to this perspective, we introduce a new learning task, which we call Swap Agnostic Learning. This swap variant of agnostic learning is inspired by the notion of swap regret in the online learning literature [10, 2], where the learner must achieve vanishing regret, not simply overall, but even conditioned on their decisions. While swap agnostic learning is a natural extension, at first glance, it is a considerably stronger goal than standard agnostic learning. Nevertheless, our work demonstrates an efficient algorithm for swap agnostic learning for any convex loss function, leveraging only a weak agnostic learner for the hypothesis class. The algorithm follows by discovering a surprising connection: swap agnostic learning and swap variants of omniprediction and multicalibration are actually equivalent. In other words, once we move to swap variants, multicalibration is necessary for omniprediction, and even for agnostic learning. We show how the original multicalibration algorithm of [20] actually guarantees the stronger goal of "swap multicalibration" (and thus, swap omniprediction and swap agnostic learning). Our results provide an exact characterization of these swap learning notions, as well as relationships between other learning desiderata explored in recent works [6, 14].
Our motivation for introducing Swap Agnostic Learning comes from trying to understand the relationship between (standard) multicalibration and omniprediction. Prior work shows that multicalibration implies (convex) omniprediction, but leaves open the question of whether omniprediction implies multicalibration. The present work sheds new light on this question, suggesting that the answer is no: standard omniprediction does not imply multicalibration. The answer to this question comes by introducing the idea of Swap Learning, in the agnostic learning/omniprediction setting, as well as in multicalibration. Importantly, for multicalibration, Swap multicalibration and Standard multicalibration are essentially the same notion. Claim 2.9, as well as our analysis of prior algorithms for multicalibration, shows that standard multicalibration is already strong enough to capture the swap variant. The distinction (definitionally and algorithmically) between swap and standard multicalibration is minimal: our results suggest that one should really think of them as a single notion of multicalibration. For agnostic learning and omniprediction, however, Swap Learning is much stronger than Standard Learning. We prove separations between swap and standard omniprediction in Appendix B. The separations are summarized in figure 1. This tells us that (swap) multicalibration is equivalent to the stronger notion of swap omniprediction, which is provably stronger than standard omniprediction.
Organization. We continue the manuscript with formal setup and definitions of Swap Agnostic Learning, Swap Omniprediction, and Swap Multicalibration. Then, in our main result, we prove the equivalence of these notions. This equivalence suggests an efficient algorithm for Swap Agnostic Learning. We conclude with an overview of our other results, as well as related work and discussion. Throughout the manuscript, we include formal definitions of key notions and proofs of essential claims. We introduce the notion of swap loss OI which generalizes omniprediction in Section . In
${}^{2}$  On a technical level,the omniprediction guarantee is made possible by post-processing the predictions after the loss is revealed. Importantly, the post-processing is a data-free univariate optimization that can be performed efficiently, based only on the loss function and not the data distribution.
Section B, we discuss the relation and show separations between various notions of omniprediction. All omitted proofs of formal claims are included in the Appendix.
2 Formal Setup: Swap Notions of Supervised Learning
We work in the agnostic learning setting, where we assume a data distribution  $(\mathbf{x},\mathbf{y})\sim \mathcal{D}$  supported on  $\mathcal{X}\times \{0,1\}$ . The objects of study in agnostic learning are real-valued hypotheses  $h:\mathcal{X}\to \mathbb{R}$ . Omniprediction and multicalibration study the special case of predictors  $\tilde{p}:\mathcal{X}\rightarrow [0,1]$  that map domain elements to probabilities. We denote the Bayes optimal predictor as  $p^* (x) = \operatorname *{Pr}[\mathbf{y} = 1|\mathbf{x} = x]$ . As is standard in agnostic learning, we make no assumptions about the complexity of  $p^*:\mathcal{X}\to [0,1]$ . While, in principle, predictors and hypotheses may take on continuous values in  $\mathbb{R}$ , we restrict our attention to functions supported on finitely-many values. We let  $\mathrm{Im}(\tilde{p})\subseteq [0,1]$  denote the set of values taken on by  $\tilde{p} (x)$ .
Given a hypothesis  $h: \mathcal{X} \to \mathbb{R}$ , it will be useful to imagine drawing samples from  $\mathcal{D}$  in two steps: first, we sample a prediction  $\mathbf{v} \in \mathbb{R}$  according to the distribution of  $h(\mathbf{x})$  under  $\mathcal{D}$ ; then, we draw the random variables  $(\mathbf{x}, \mathbf{y})$  according to the conditional distribution  $\mathcal{D}|h(\mathbf{x}) = \mathbf{v}$ . Accordingly, let  $\mathcal{D}_h$  denote the distribution of  $h(\mathbf{x})$  where  $\mathbf{x} \sim \mathcal{D}$ .
2.1 Swap Agnostic Learning
Swap agnostic learning is defined with respect to a loss function  $\ell : \{0,1\} \times \mathbb{R} \to \mathbb{R}$  and a hypothesis class  $\mathcal{C} \subseteq \{c : \mathcal{X} \to \mathbb{R}\}$  and can be viewed as a game with two participants. The predictor plays first and selects a hypothesis  $h : \mathcal{X} \to \mathbb{R}$ . The adversary plays in response: for each level set  $\{x \in \mathcal{X} : h(x) = v\}$ , the adversary may choose a separate loss-minimizing hypothesis  $c_v \in \mathcal{C}$ .
Definition 2.1 (Swap Agnostic Learning). For a loss function  $\ell$ , hypothesis class  $\mathcal{C}$ , and error  $\varepsilon \geq 0$ , a hypothesis  $h$  is a  $(\ell, \mathcal{C}, \varepsilon)$ -swap agnostic learner if
$$
\mathbf {E} [ \ell (\mathbf {y}, h (\mathbf {x})) ] \leq \underset {\mathbf {v} \sim \mathcal {D} _ {h}} {\mathbf {E}} \left[ \min  _ {c _ {v} \in \mathcal {C}} \mathbf {E} [ \ell (\mathbf {y}, c _ {v} (\mathbf {x}) \mid h (\mathbf {x}) = \mathbf {v} ] \right] + \varepsilon . \tag {2}
$$
We borrow nomenclature from online learning: a predictor using a swap agnostic learner  $h$  has no incentive to "swap" any of their fixed predictions  $h(\mathbf{x}) = v$  to predict according to  $c_v \in \mathcal{C}$ . Contrasting the requirement in (2) to that of agnostic learning in (1), we have switched the order of quantifiers, such that the minimization is taken after the expectation over the choice of  $h(\mathbf{x}) = \mathbf{v}$ . Swap agnostic learning strengthens standard agnostic learning, where the predictor only competes against the single best hypothesis  $c \in \mathcal{C}$ .
Indeed, swap agnostic learning seems to be a much more stringent condition. Provided the class  $\mathcal{C}$  contains all constant functions, the adversary can simply imitate the predictor when  $h(x) = v$  by choosing  $c_v(x) = v$ . For such  $\mathcal{C}$ , (2) implies that imitation is the best strategy for the adversary.
2.2 Swap Omniprediction
As in [15, 14], we observe that for any loss function  $\ell$  and known distribution on outcomes  $\mathbf{y} \sim \mathrm{Ber}(p)$  for  $p \in [0,1]$ , there is an optimal action  $k_{\ell}(p)$  that minimizes the expected loss  $\ell$ . Formally, we define the optimal post-processing of predictions by the function  $k_{\ell}: [0,1] \to \mathbb{R}$ , which specifies the action that minimizes the expected loss.
$$
k _ {\ell} (p) = \underset {t \in \mathbb {R}} {\arg \min } \underset {\mathbf {y} \sim \operatorname {B e r} (p)} {\mathbf {E}} [ \ell (\mathbf {y}, t) ]. \tag {3}
$$
This observation is particularly powerful because the function  $k_{\ell}$  is given by a simple, univariate optimization that can be used as a data-free post-processing procedure on top of a predictor  $\tilde{p}$ .
We recall the original notion of omniprediction proposed by [15], which requires a predictor to yield an agnostic learner simultaneously for every loss in a collection  $\mathcal{L}$ . Concretely, omniprediction requires that the function  $h_\ell = k_\ell \circ \tilde{p}$  is an  $(\ell, \mathcal{C}, \varepsilon)$ -agnostic learner for every  $\ell \in \mathcal{L}$ .
3If there are multiple optima, we break ties arbitrarily.
Definition 2.2 (Omnipredictor, [15]). For a collection of loss functions  $\mathcal{L}$ , a hypothesis class  $\mathcal{C}$ , and error  $\varepsilon > 0$ , a predictor  $\tilde{p} : \mathcal{X} \to [0,1]$  is an  $(\mathcal{L}, \mathcal{C}, \varepsilon)$ -omnipredictor if for every  $\ell \in \mathcal{L}$ ,
$$
\mathbf {E} [ \ell (\mathbf {y}, k _ {\ell} (\tilde {p} (\mathbf {x})) ] \leq \min  _ {c \in \mathcal {C}} \mathbf {E} [ \ell (\mathbf {y}, c (\mathbf {x})) ] + \varepsilon . \tag {4}
$$
We propose a strengthened notion of Swap Omniprediction, where the adversary may choose both the loss  $\ell_v\in \mathcal{L}$  and hypothesis  $c_{v}\in \mathcal{C}$  based on the prediction  $\tilde{p} (x) = v$
Definition 2.3 (Swap Omnipredictor). For a collection of loss functions  $\mathcal{L}$ , a hypothesis class  $\mathcal{C}$ , and error  $\varepsilon > 0$ , a predictor  $\tilde{p}: \mathcal{X} \to [0,1]$  is a  $(\mathcal{L}, \mathcal{C}, \varepsilon)$ -swap omnipredictor if for any assignment of loss functions  $\{\ell_v \in \mathcal{L}\}_{v \in \mathrm{Im}(\tilde{p})}$ ,
$$
\mathbf {E} _ {\mathbf {v} \sim \mathcal {D} _ {\tilde {p}}} \left[ \mathbf {E} \left[ \ell_ {\mathbf {v}} \left(\mathbf {y}, k _ {\ell_ {\mathbf {v}}} (v)\right) \mid \tilde {p} (\mathbf {x}) = \mathbf {v} \right] \right] \leq \mathbf {E} _ {\mathbf {v} \sim \mathcal {D} _ {\tilde {p}}} \left[ \min  _ {c _ {v} \in \mathcal {C}} \mathbf {E} \left[ \ell_ {\mathbf {v}} \left(\mathbf {y}, c _ {v} (\mathbf {x})\right) \mid \tilde {p} (\mathbf {x}) = \mathbf {v} \right] \right] + \varepsilon . \tag {5}
$$
Swap omniprediction gives the adversary considerable power. For instance, the special case where we restrict the adversary's choice of losses to be constant,  $\ell_v = \ell$ , realizes swap agnostic learning for  $\ell$ .
Claim 2.4. If  $\tilde{p}$  is a  $(\mathcal{L},\mathcal{C},\varepsilon)$ -swap omnipredictor, it is a  $(\ell ,\mathcal{C},\varepsilon)$ -swap agnostic learner for every  $\ell \in \mathcal{L}$ , and hence a  $(\mathcal{L},\mathcal{C},\varepsilon)$ -omnipredictor.
Analogous to the standard notions, swap omniprediction implies swap agnostic learning for every  $\ell$ ; however, swap omniprediction gives an even stronger guarantee since the adversary's loss  $\ell_v$  may be chosen in response to the prediction  $\tilde{p}(\mathbf{x}) = v$ .
2.3 Swap Multicalibration
Multicalibration was introduced in the work of [20] as a notion of algorithmic fairness following [26], but has since seen broad application across many learning applications (e.g., [6, 15, 25, 18]). Informally, multicalibration requires predictions to appear calibrated, not simply overall, but also when we restrict our attention to subgroups within some broad collection  $\mathcal{C}$ . The formulation below appears in [14].
Definition 2.5 (Multicalibration, [20]). For a hypothesis class  $\mathcal{C}$  and  $\alpha \geq 0$ , a predictor  $\tilde{p}:\mathcal{X}\to [0,1]$  is  $(\mathcal{C},\alpha)$ -multicalibrated if
$$
\left. \max  _ {c \in \mathcal {C}} \mathbf {E} _ {\mathbf {v} \sim \mathcal {D} _ {\tilde {p}}} \left[ \left| \mathbf {E} [ c (\mathbf {x}) (\mathbf {y} - \mathbf {v}) \mid \tilde {p} (\mathbf {x}) = \mathbf {v} ] \right| \right] \leq \alpha . \right. \tag {6}
$$
When  $c: \mathcal{X} \to \{0,1\}$  is Boolean (and has sufficiently large measure), this definition says that conditioned on  $c(\mathbf{x}) = 1$ , the calibration violation is small, recovering the definition in [20].
Swap Multicalibration strengthens multicalibration, extending the pseudorandomness perspective on multicalibration developed in [6, 15]. Swap multicalibration requires that for the typical prediction  $\mathbf{v} \sim \mathcal{D}_{\tilde{p}}$ , no hypothesis in  $c_v \in \mathcal{C}$  achieves good correlation with the residual labels  $\mathbf{y} - \mathbf{v}$  over  $\mathcal{D}$  conditioned on  $\tilde{p}(\mathbf{x}) = \mathbf{v}$ .
Definition 2.6 (Swap Multicalibration). For a hypothesis class  $\mathcal{C}$  and  $\alpha \geq 0$ , a predictor  $\tilde{p}:\mathcal{X}\to [0,1]$  is  $(\mathcal{C},\alpha)$ -swap multicalibrated if
$$
\mathbf {E} _ {\mathbf {v} \sim \mathcal {D} _ {\tilde {p}}} \left[ \max  _ {c _ {v} \in \mathcal {C}} \left| \mathbf {E} \left[ c _ {v} (\mathbf {x}) (\mathbf {y} - \mathbf {v}) \mid \tilde {p} (\mathbf {x}) = \mathbf {v} \right] \right| \right] \leq \alpha . \tag {7}
$$
Again, the difference between swap and standard multicalibration is in the order of quantifiers. The standard definition requires that for every  $c \in \mathcal{C}$ , the correlation  $|\mathbf{E}_{\mathcal{D}|_{\mathbf{v}}}[c(\mathbf{x})(\mathbf{y} - \mathbf{v})]|$  achieved in small in expectation over  $\mathbf{v} \sim \mathcal{D}_{\tilde{p}}$ . Swap multicalibration considers the maximum of  $|\mathbf{E}_{\mathcal{D}|_{\mathbf{v}}}[c(\mathbf{x})(\mathbf{y} - \mathbf{v})]|$  over all  $c \in \mathcal{C}$  for each fixing of  $\mathbf{v} = v$  and requires this to be small in expectation over  $\mathbf{v} \sim \mathcal{D}_{\tilde{p}}$ . It follows that swap multicalibration implies standard multicalibration.
Claim 2.7. If  $\tilde{p}$  is  $(\mathcal{C},\alpha)$ -swap multicalibrated, it is  $(\mathcal{C},\alpha)$ -multicalibrated.
While swap multicalibration is nominally a stronger notion than standard multicalibration, the complexity of achieving swap multicalibration is essentially the same as multicalibration. We show that starting from a  $(\mathcal{C},\alpha)$ -multicalibrated predictor  $\tilde{p}$ , by suitably discretizing its values, we obtain a predictor that is close to  $\tilde{p}$  and  $(\mathcal{C},\alpha^{\prime})$ -swap multicalibrated, with some degradation in the value of  $\alpha^{\prime}$ .
Definition 2.8. Let  $\delta \in [0,1]$  so that  $m = 1 / \delta \in \mathbb{Z}$ . Define  $B_{j} = [(j - 1)\delta ,j\delta)$  for  $j\in [m - 1]$  and  $B_{m} = [1 - \delta ,1]$ . Define the predictor  $\bar{p}_{\delta}$  where for every  $x$  such that  $\tilde{p} (x)\in B_j$ ,  $\bar{p}_{\delta}(x) = j\delta$ .
Claim 2.9. Let  $\tilde{p}:\mathcal{X}\to [0,1]$  be a  $(\mathcal{C},\alpha)$ -multicalibrated. For any  $\delta \in [0,1]$  where  $1 / \delta \in \mathbb{Z}$ , the predictor  $\bar{p}_{\delta}$  is  $(\mathcal{C},2\sqrt{\alpha / \delta} +\delta)$ -swap multicalibrated, and  $\max_{x\in \mathcal{X}}|\tilde{p} (x) - \bar{p}_{\delta}(x)|\leq \delta$
While the bound of the theorem is valid for all  $\delta$ , the swap multicalibration guarantee is only meaningful when  $\delta \geq \Omega(\alpha)$ . Thus, by discretizing a  $(\mathcal{C}, \alpha)$ -multicalibrated predictor, we obtain a  $(\mathcal{C}, O(\alpha^{1/3}))$ -swap multicalibrated predictor. While this generic transformation suffers a polynomial loss in the accuracy parameter, such a loss may not be algorithmically necessary. As we argue in Lemma 3.8, known algorithms for achieving multicalibration [20, 15] actually guarantee swap multicalibration without any modification.
3 An equivalence: swap agnostic learning, omniprediction, multicalibration
Our main result is an equivalence between swap agnostic learning, swap omniprediction, and swap multicalibration. Concretely, this equivalence shows that swap agnostic learning for the squared error is sufficient to guarantee swap omniprediction for all (nice) convex loss functions. We begin with some preliminaries, then formally state and prove the equivalence. We conclude the section by showing how to use the existing framework for learning multicalibrated predictors to achieve swap agnostic learning for any convex loss.
Nice loss functions. For a loss function  $\ell : \{0,1\} \times \mathbb{R} \to \mathbb{R}$ , we extend  $\ell$  linearly to allow the first argument to take values in the range  $p \in [0,1]$  as:
$$
\ell (p, t) = \underset {\mathbf {y} \sim \operatorname {B e r} (p)} {\mathbf {E}} [ \ell (\mathbf {y}, t) ] = p \cdot \ell (1, t) + (1 - p) \cdot \ell (0, t).
$$
We say the loss function is convex if for  $y \in \{0,1\}$ ,  $\ell(y,t)$  is a convex function of  $t$ . By linearity, this convexity property holds for  $\ell(p,t)$  for all  $p$ .
As in [14], for a loss  $\ell$ , we define the partial difference function  $\partial \ell : \mathbb{R} \to \mathbb{R}$  as
$$
\partial \ell (t) = \ell (1, t) - \ell (0, t).
$$
We define a class of "nice" loss functions, which obey a minimal set of boundedness and Lipschitzness conditions.
Definition 3.1. For a constant  $B > 0$ , a loss function is  $B$ -nice if there exists an interval  $I_{\ell} \subseteq \mathbb{R}$  such that the following conditions hold:
1. (Optimality) If  $\Pi_{\ell}:\mathbb{R}\to I_{\ell}$  denotes projection onto the interval  $I_{\ell}$ , then  $\ell (p,\Pi_{\ell}(t))\leq \ell (p,t)$  for all  $t\in \mathbb{R}$  and  $p\in [0,1]$ .
2. (Lipschitzness) For  $y \in \{0,1\}$  and  $t \in I_{\ell}$ ,  $\ell(y,t)$  is 1-Lipschitz as a function of  $t$ .
3. (Bounded difference) For  $t \in I_{\ell}$ ,  $|\partial \ell(t)| \leq B$ .
The class of all  $B$ -nice loss functions by  $\mathcal{L}(B)$ . The subset of  $B$ -nice convex loss functions is denoted by  $\mathcal{L}_{\mathrm{cvx}}(B)$ .
Bounded loss functions generalize the idea of loss functions defined over a fixed interval of  $\mathbb{R}$  (by optimality) and of bounded output range (by bounded difference). By optimality of nice loss functions, we may assume  $k_{\ell}:[0,1]\to I_{\ell}$  since we do not increase the loss by projection onto  $I_{\ell}$ . Indeed, for convex losses  $\ell$ , the natural choice for  $I_{\ell}$  is to take the interval  $[k_{\ell}(0),k_{\ell}(1)]$ . The bounded difference condition implies that  $\ell$  is Lipschitz in its first argument, a property that will be useful.
Lemma 3.2. For every  $\ell \in \mathcal{L}(B)$  and  $t_0\in I_\ell$ , the function  $\ell (p,t_0)$  is  $B$ -Lipschitz as a function of  $p$ .
Concept classes. For a concept class of functions  $\mathcal{C}:\{c:\mathcal{X}\to \mathbb{R}\}$ , we assume that  $\mathcal{C}$  is closed under negation, and it contains the constant functions 0 and 1. Denoting  $\| \mathcal{C}\|_{\infty} = \max |c(x)|$  over  $c\in \mathcal{C},x\in \mathcal{X}$ , we say that  $\mathcal{C}$  is bounded if  $\| c\|_{\infty}\leq 1$ . For  $W\in \mathbb{R}^{+}$ , let  $\operatorname {Lin}(\mathcal{C},W)$  be all functions that can be expressed as a (W-sparse) linear combination of base concepts from  $\mathcal{C}$ ,
$$
c _ {w} (x) = \sum_ {c \in \mathcal {C}} w _ {c} \cdot c (x), \sum_ {c \in \mathcal {C}} | w _ {c} | \leq W.
$$