SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Characterizing the Optimal  $0 - 1$  Loss for Multi-class Classification with a Test-time Attacker
Sihui Dai $^{1*}$  Wenxin Ding $^{2*}$  Arjun Nitin Bhagoji $^{2}$  Daniel Cullina $^{3}$  Ben Y. Zhao $^{2}$  Haitao Zheng $^{2}$  Prateek Mittal $^{1}$ $^{1}$ Princeton University  $^{2}$ University of Chicago  $^{3}$ Pennsylvania State University {sihuid, pmittal}@princeton.edu {wenxind, abhagoji}@uchicago.edu {ravenben, htzheng}@cs.uchicago.edu cullina@psu.edu
Finding classifiers robust to adversarial examples is critical for their safe deployment. Determining the robustness of the best possible classifier under a given threat model for a fixed data distribution and comparing it to that achieved by state-of-the-art training methods is thus an important diagnostic tool. In this paper, we find achievable information-theoretic lower bounds on robust loss in the presence of a test-time attacker for multi-class classifiers on any discrete dataset. We provide a general framework for finding the optimal  $0 - 1$  loss that revolves around the construction of a conflict hypergraph from the data and adversarial constraints. The prohibitive cost of this formulation in practice leads us to formulate other variants of the attacker-classifier game that more efficiently determine the range of the optimal loss. Our valuation shows, for the first time, an analysis of the gap to optimal robustness for classifiers in the multi-class setting on benchmark datasets.
Developing a theoretical understanding of the vulnerability of classifiers to adversarial examples (27; 12; 8; 4) generated by a test-time attacker is critical to their safe deployment. Past work has largely taken one of two approaches. The first has focused on generalization bounds on learning in the presence of adversarial examples, by trying to determine the sample complexity of robust learning (10; 25; 21). The second has been to characterize the lowest possible loss achievable within a specific hypothesis class (22; 5; 6) for binary classification for a specified data distribution and attacker. The hypothesis class of choice is often the set of all possible classification functions. The optimal loss thus determined is a lower bound on robustness for classifiers used in practice, allowing practitioners to measure progress for defenses and step away from the attack-defense arms race (19; 31; 9).
In this paper, we take on the second approach and propose methods to find the lowest possible loss attainable by any measurable classifier in the presence of a test-time attacker in the multi-class setting. The loss thus obtained is referred to as the optimal loss and the corresponding classifier, the optimal classifier. We extend previous work (5; 6) which was restricted to the binary setting, allowing us to compare the robustness of multi-class classifiers used in practice to the optimal loss. Our first contribution is thus extending the conflict-graph framework for computing lower bounds on robustness to the multi-class setting. In this framework, given a dataset and attacker, we construct a conflict hypergraph which contains vertices representing training examples in the dataset, and hyperedges representing overlaps between adversarial neighborhoods around each training example. Using this hypergraph, we construct a linear program whose optimal value is a lower bound on the  $0 - 1$  loss for all classifiers and whose solution is the optimal classifier. The lower bound on robustness is thus achievable.
In practice, however, we find that the full multi-class formulation of the lower bound, although exact, can lead to prohibitively large optimization problems. Thus, we vary the information available to either the attacker or classifier to find alternative lower bounds that are quicker to compute. Our second contribution is the development and analysis of more efficient methods to determine the range of loss obtained by the optimal classifier. (see Table 1). We also interpret these methods as classifier-attacker games. In particular, we find lower bounds on the optimal loss by aggregating the set of all binary classifier-only lower bounds as well as by using truncated hypergraphs (hypergraphs with a restriction on the maximum hyperedge degree). We also upper bound the optimal loss with a generalization of the Caro-Wei bound (1) on a graph's independent set size. The gap between the lower and upper bounds on the optimal loss allows us to determine the range within which the optimal loss lies.
To analyze the performance of classifiers obtained from adversarial training (19; 31), we compare the loss obtained through adversarial training to that of the optimal classifier. We find a loss differential that is greatly exacerbated compared to the binary case (5; 22). In addition, we also determine the cases where, in practice, the bounds obtained from game variants are close to the optimal. We find that while the aggregation of binary classifier-only bounds leads to a very loose lower bound, the use of truncated hypergraphs can significantly speed up computation while achieving a loss value close to optimal. This is validated using the Caro-Wei upper bound, with the lower and upper bounds on the optimal loss closely matching for adversarial budgets used in practice. Thus, our final contribution is an extensive empirical analysis of the behavior of the optimal loss for a given attacker, along with its lower and upper bounds. This enables practitioners to utilize our methods even when the optimal loss is computationally challenging to determine.
The rest of the paper is organized as follows: §2 provides the characterization of the optimal loss; §3 proposes several upper and lower bounds on the optimal loss; §4 computes and compares the optimal loss to these bounds, as well as the performance of robustly trained classifiers and §5 concludes with a discussion of limitations and future work.
2 Characterizing Optimal 0-1 Loss
In this section, we characterize the optimal  $0 - 1$  loss for any discrete distribution (e.g. training set) in the presence of a test-time attacker. This loss can be computed as the solution of a linear program (LP), which is defined based on a hypergraph constructed from the classification problem and attacker constraint specification. The solution to the LP can be used to construct a classifier achieving the optimal loss, and lower bounds the loss attainable by any particular learned classifier.
Notation. We consider a classification problem where inputs are sampled from input space  $\mathcal{X}$ , and labels belong to  $K$  classes:  $y\in \mathcal{Y} = [K] = \{1,\dots,K\}$ . Let  $P$  be the joint probability over  $\mathcal{X}\times \mathcal{Y}$ . Let  $\mathcal{H}_{\mathrm{soft}}$  denote the space of all soft classifiers; i.e. specifically, for all  $h\in \mathcal{H}_{\mathrm{soft}}$  we have that  $h:\mathcal{X}\to [0,1]^K$  and  $\sum_{i = 1}^{K}h(x)_i = 1$  for all  $x\in \mathcal{X}$ . Here,  $h(x)_i$  represents the probability that the input  $x$  belongs to the  $i^{\mathrm{th}}$  class. We use the natural extension of  $0 - 1$  loss to soft classifiers as our loss function:  $\ell (h,(x,y)) = 1 - h(x)_y$ . This reduces to  $0 - 1$  loss when  $h(x)\in \{0,1\} ^K$ .
The adversarial classification game. We are interested in the setting where there exists a test-time attacker that can modify any data point  $x$  to generate an adversarial example  $\tilde{x}$  from the neighborhood  $N(x)$  of  $x$ . An instance of the game is specified by a discrete probability distribution  $P$ ,² hypothesis class  $\mathcal{H} \subseteq \mathcal{H}_{\mathrm{soft}}$ , and neighborhood function  $N$ . We require that for all  $x \in \mathcal{X}$ ,  $N(x)$  always contains  $x$ . The goal of the classifier player is to minimize expected classification loss and the goal of the
<sup>1</sup>A soft classifier can be interpreted as a randomized hard-decision classifier  $f$  with  $\operatorname*{Pr}[f(x) = y] = h(\tilde{x})_y$ , in which case  $\ell(h, (x, y)) = \operatorname*{Pr}[f(x) \neq y]$ , classification error probability of this randomized classifier.
Properly extending this game to more general data distributions is tricky due to issues with the measurability of the supremum of the loss. See Pydi and Jog for discussion of the multiple approaches taken in the literature and when various versions of the game are equivalent (23). These technicalities do not arise for discrete distributions, which are sufficient for our purposes.
attacker is to maximize it. The optimal loss is
$$
L ^ {*} (P, N, \mathcal {H}) = \inf  _ {h \in \mathcal {H}} \mathbb {E} _ {(x, y) \sim P} \left[ \sup  _ {\tilde {x} \in N (x)} \ell (h, (\tilde {x}, y)) \right] = 1 - \sup  _ {h \in \mathcal {H}} \mathbb {E} _ {(x, y) \sim P} \left[ \inf  _ {\tilde {x} \in N (x)} h (\tilde {x}) _ {y} \right]. \tag {1}
$$
Alternative hypothesis classes. In general, for  $\mathcal{H}' \subseteq \mathcal{H}$ , we have  $L^{*}(P,N,\mathcal{H}) \leq L^{*}(P,N,\mathcal{H}')$ . Two particular cases of this are relevant. First, the class of hard-decision classifiers is a subset of the class of soft classifiers  $(\mathcal{H}_{\mathrm{soft}})$ . Second, for any fixed model parameterization (i.e. fixed NN architecture), the class of functions represented by that parameterization is another subset. Thus, optimal loss over  $\mathcal{H}_{\mathrm{soft}}$  provides a lower bound on loss for these settings.
2.2 Optimal loss for distributions with finite support
Since we would like to compute the optimal loss for distributions  $P$  with finite support, we can rewrite the expectation in Equation 1 as an inner product. Let  $V$  be the support of  $P$ , i.e. the set of points  $(x,y) \in \mathcal{X} \times \mathcal{Y}$  that have positive probability in  $P$ . Let  $p \in [0,1]^V$  be the probability mass vector for  $P$ :  $p_v = P(\{v\})$ . For a soft classifier  $h$ , let  $q_N(h) \in \mathbb{R}^V$  be the vector of robustly correct classification probabilities for vertices  $v = (x,y) \in V$ , i.e.  $q_N(h)_v := \inf_{\tilde{x} \in N(x)} h(\tilde{x})_y$ . Rewriting (1) with our new notation, we have  $1 - L^{*}(P, \mathcal{H}_{\mathrm{soft}}, N) = \sup_{h \in \mathcal{H}_{\mathrm{soft}}} p^T q_N(h)$ . This is the maximization of a linear function over all possible vectors  $q_N(h)$ . In fact, the convex hull of all correct classification probability vectors is a polytope and this optimization problem is a linear program, as described next.
Definition 1. For a soft classifier  $h$ , the correct-classification probability achieved on an example  $v = (x, y)$  in the presence of an adversary with constraint  $N$  is  $q_{N}(h)_{v} = \inf_{\tilde{x} \in N(x)} h(\tilde{x})_{y}$ .
The space of achievable correct classification probabilities is  $\mathcal{P}_{\mathcal{V},N,\mathcal{H}}\subseteq [0,1]^{\mathcal{V}}$ , defined as
$$
\mathcal {P} _ {\mathcal {V}, N, \mathcal {H}} = \bigcup_ {h \in \mathcal {H}} \prod_ {v \in \mathcal {V}} [ 0, q _ {N} (h) _ {v} ]
$$
In other words we say that  $q' \in [0,1]^{\mathcal{V}}$  is achievable when there exists  $h \in \mathcal{H}$  such that  $q' \leq q_N(h)$ . The inequality appears because we will always take nonnegative linear combinations of correct classification probabilities.
Characterizing  $\mathcal{P}_{\mathcal{V},N,\mathcal{H}}$  allows the minimum adversarial loss achievable to be expressed as an optimization problem with a linear objective:
$$
1 - L ^ {*} (P, N, \mathcal {H} _ {\text {s o f t}}) = \sup  _ {h \in \mathcal {H} _ {\text {s o f t}}} \mathbb {E} _ {v \sim P} [ q _ {N} (h) _ {v} ] = \sup  _ {h \in \mathcal {H} _ {\text {s o f t}}} p ^ {T} q _ {N} (h) = \sup  _ {q \in \mathcal {P} _ {\mathcal {V}, N, \mathcal {H}}} p ^ {T} q. \tag {2}
$$
(6) characterized  $\mathcal{P}_{\mathcal{V},N,\mathcal{H}_{soft}}$  in the two-class case and demonstrated that this space can be captured by linear inequalities. We now demonstrate that this also holds for the multi-class setting.
2.3 Linear Program to obtain Optimal Loss
In order to characterize  $\mathcal{P}_{\mathcal{V},N,\mathcal{H}_{soft}}$ , we represent the structure of the classification problem with a conflict hypergraph  $\mathcal{G}_{\mathcal{V},N} = (\mathcal{V},\mathcal{E})$ , which records intersections between neighborhoods of points in  $\mathcal{X}$ . The set of vertices  $V$  of  $\mathcal{G}_{\mathcal{V},N}$  is the support of  $P$ .  $\mathcal{E}$  denotes the set of hyperedges of the graph. For a set  $S\subseteq \mathcal{V}$ ,  $S\in \mathcal{E}$  (i.e.  $S$  is a hyperedge in  $\mathcal{G}_{\mathcal{V},N}$ ) if all vertices in  $S$  belong to different classes and the neighborhoods of all vertices in  $S$  overlap:  $\bigcap_{(x,y)\in S}N(x)\neq \emptyset$ . Thus, the size of each hyperedge is at most  $K$ ,  $\mathcal{E}$  is downward-closed (meaning if  $e\in \mathcal{E}$  and  $e^{\prime}\subset e$ , then  $e^{\prime}\subset \mathcal{E}$ ), and every  $v\in \mathcal{V}$  is a degree 1 hyperedge.
Using the conflict hypergraph  $\mathcal{G}_{\mathcal{V},N}$ , we can now describe  $\mathcal{P}_{\mathcal{V},N,\mathcal{H}_{soft}}$ .
Theorem 1 (Feasible output probabilities (Adapted from (6))). The set of correct classification probability vectors for support points  $\mathcal{V}$ , adversarial constraint  $N$ , and hypothesis class  $\mathcal{H}_{soft}$  is
$$
\mathcal {P} _ {\mathcal {V}, N, \mathcal {H} _ {\text {s o f t}}} = \left\{q \in \mathbb {R} ^ {\mathcal {V}}: q \geq \mathbf {0}, B q \leq \mathbf {1} \right\} \tag {3}
$$
where  $B\in \mathbb{R}^{\mathcal{E}\times \mathcal{V}}$  is the hyperedge incidence matrix of the conflict hypergraph  $G_{\mathcal{V},N}$
For any loss function that is a decreasing function of  $h(\tilde{x})_y$ , the optimal loss can be specified as an optimization over  $\mathcal{P}_{\mathcal{V},N,\mathcal{H}}$ . In fact (6) focused on cross-entropy loss, which has this property.
Summary of Method	Location
Optimal 0-1 loss	LP on conflict hypergraph	§2.3
Lower bounds for optimal 0-1 loss	LP on truncated conflict hypergraph
Combining binary classification bounds	§3.1
§3.2
Upper bound for optimal 0-1 loss	Generalization of Caro-Wei bound	§3.5
Table 1: Summary of methods for computing the optimal  $0 - 1$  loss and efficient bounds on this value.
See Supplementary §A for proof. This characterization of  $\mathcal{P}_{\mathcal{V},N,\mathcal{H}_{\text{soft}}}$  allows us to express optimal loss as a linear program for any dataset and attacker using Eq. 2<sup>4</sup>.
Corollary 1 (Optimal loss as an LP). For any distribution  $P$  with finite support,
$$
1 - L ^ {*} (P, N, \mathcal {H} _ {\text {s o f t}}) = \max  _ {q} p ^ {T} q \quad s. t q \geq 0, B q \leq 1. \tag {4}
$$
Duality and adversarial strategies. The dual linear program is
$$
\min  _ {z} \mathbf {1} ^ {T} z \quad \text {s . t .} \quad z \geq 0, \quad B ^ {T} z \geq p.
$$
Feasible points in the dual linear program are fractional coverings of the vertices by the hyperedges (24). (See Section 3.4 and Supplementary §B for more discussion of fractional packings and coverings.) An adversarial strategy can be constructed from a feasible point  $z$  as follows. For each hyperedge  $e$ , choose an example  $\tilde{x}(e)$  such that  $\tilde{x}(e) \in N(x)$  for all  $(x, y) \in e$ . From the definition of the conflict hypergraph, a choice is always available. A randomized adversarial strategy consists of conditional distributions for the adversarial example  $\tilde{x}$  given the natural example  $x$ . When the adversary samples the natural example  $(x, y)$ , the adversary can select  $\tilde{x}(e)$  with probability at least  $\frac{B_{e,v}z_e}{p_v}$ . Note that  $B_{e,v}z_e$  is the amount of coverage of  $v$  coming from  $e$ . This is nonzero only for  $e$  that contain  $(x, y)$ . A conditional distribution satisfying these inequalities exists because  $\sum_{e} B_{e,v}z_e = (B^T z)_v \geq p_v$ .
Thus for a vertex  $v$  such that  $(B^T z)_v = p_v$ , there is only one choice for this distribution. For a vertex  $v$  that is over-covered, i.e.  $(B^T z)_v > p_v$ , the adversary has some flexibility. If  $v$  is over-covered in some minimal cost covering, by complementary slackness  $q_v = 0$  in every optimal  $q$ , so the optimal classifiers do not attempt to classify  $v$  correctly.
Three-class minimal examples. Corollary 1 demonstrates that the optimal loss for the multiclass problem is influenced by hyperedges between vertices which reflect higher order interactions between examples. Figure 1 shows an important distinction between two types of three-way interactions of examples.
We have  $1 - L^{*}(P,\mathcal{H}_{soft},N) = \max (p_{u},p_{v},p_{w},\frac{1}{2})$  while  $1 - L^{*}(P^{\prime},\mathcal{H}_{soft},N) = \max (p_u',p_v',p_w')$  The presence or absence of the size-three hyperedge affects the optimal loss if and only if the example probabilities are close to balanced, i.e. all at most  $\frac{1}{2}$ .
It is instructive to consider the optimal classifiers and adversarial strategies in the two cases. For  $\mathcal{V}$ , when  $\frac{1}{2} \leq p_u$ , the classifier  $h$  with  $q_N(h) =$
$(1,0,0)$  is optimal. One such classifier is the constant classifier  $h(x) = (1,0,0)$ . The optimal cover satisfies  $z_{\{u,v\}} + z_{\{u,w\}} = p_u$ ,  $z_{\{u,v\}} \geq p_v$ ,  $z_{\{u,w\}} \geq p_w$ ,  $z_{\{v,w\}} = 0$ . Thus when the adversary
Figure 1: Two possible conflict structures involving three examples, each from a different class. In the right case, all subsets of  $\mathcal{V}' = \{u', v', w'\}$  are hyperedges in the conflict hypergraph  $\mathcal{G}_{\mathcal{V}',N}$ . In the left case,  $\{u, v, w\}$  is not a hyperedge in  $\mathcal{G}_{\mathcal{V},N}$ , but all other subsets of  $\mathcal{V}$  are.
<sup>4</sup>We note that concurrent work by Trillos et al. (29) provides a general-purpose framework based on optimal transport to find lower bounds on robustness in the per-sample as well as distributional sense. They also independently construct the three-class minimal example from Figure 1.
samples  $v$  or  $w$ , it always produces an adversarial example that could be confused for  $u$ . When  $\max(p_u, p_v, p_w) \leq \frac{1}{2}$ , any classifier  $h$  with  $q_N(h) = \left(\frac{1}{2}, \frac{1}{2}, \frac{1}{2}\right)$  is optimal. To achieve these correct classification probabilities, we need  $h(\tilde{x}) = \left(\frac{1}{2}, \frac{1}{2}, 0\right)$  for  $\tilde{x} \in N(u) \cap N(v)$ ,  $h(\tilde{x}) = \left(\frac{1}{2}, 0, \frac{1}{2}\right)$  for  $\tilde{x} \in N(u) \cap N(w)$ , etc.. The cover  $z_{\{u,v\}} = p_u + p_v - \frac{1}{2}$ ,  $z_{\{u,w\}} = p_u + p_w - \frac{1}{2}$ , and  $z_{\{v,w\}} = p_v + p_w - \frac{1}{2}$  is optimal and has cost  $\frac{1}{2}$ . The adversary produces examples associated with all three edges.
For  $\mathcal{V}'$ , things are simpler. The cover  $z_{\{u, v, w\}} = \max(p_u, p_v, p_w)$  is always optimal. When  $p_u \geq \max(p_v, p_w)$ , the classifier that returns  $(1, 0, 0)$  everywhere is optimal.
3 Bounding the Optimal 0-1 Loss
While Corollary 1 characterizes the optimal loss, it may be computationally expensive to construct conflict hypergraphs in practice for a given dataset and to solve the linear program. Thus, we discuss several methods of bounding the optimal loss from the LP in Corollary 1, which are computationally faster in practice (§4.2).
3.1 Lower bounds on multiclass optimal loss via truncated hypergraphs
The edge set of the hypergraph  $\mathcal{G}$  can be very large: there are  $\prod_{i\in [K]}(1 + |V_i|)$  vertex sets that are potential hyperedges. Even when the size of the edge set is reasonable, it is not clear that higher order hyperedges can be computed from  $\nu$  efficiently. To work around these issues, we consider hypergraphs with bounded size hyperedges:  $\mathcal{G}^{\leq m} = (\mathcal{V},\mathcal{E}^{\leq m})$  where  $\mathcal{E}^{\leq m} = \{e\in \mathcal{E}:|e|\leq m\}$ . We refer to these hypergraphs as truncated hypergraphs. In the corresponding relaxation of (4),  $B$  is replaced by  $B^{\leq m}$ , the incidence matrix for  $\mathcal{E}^{\leq m}$ . Since  $\mathcal{E}^{\leq m}\subseteq \mathcal{E}$ , this relaxation provides a lower bound on  $L^{*}(P,N,\mathcal{H}_{\mathrm{soft}})$ .
Classification with side-information. This relaxation has an interpretation as the optimal loss in a variation of the classification game with side information.
Definition 2. In the example-dependent side information game with list length  $m$ , the adversary samples  $(x, y) \sim P$ , then selects  $\tilde{x}$  and  $C \subseteq \mathcal{V}$  such that  $y \in C$  and  $|C| = m$ . We call  $C$  the side information. The classifier receives both  $\tilde{x}$  and  $C$ , so the classifier is a function  $h: \mathcal{X} \times \binom{\mathcal{Y}}{m} \to [0,1]^K$ , where  $\binom{\mathcal{Y}}{m}$  is the set of  $m$ -element subsets of  $\mathcal{V}$ . Let
$$
L^{*}(m,P,\mathcal{H},N) = \inf_{h\in \mathcal{H}}\mathbb{E}_{(x,y)\sim P}\Big[\inf_{\tilde{x}\in N(x)}\min_{C\in \binom {\gamma}{m}:y\in C}(1 - h(\tilde{x},C)_{y})\Big]
$$
be the minimum loss in this game.
To illustrate this, consider the distribution  $P'$  from Figure 1 with  $m = 2$ . The adversary can select some  $\tilde{x} \in N(u') \cap N(v') \cap N(w')$ , but the classifier will use the side-information to eliminate one of the three classes. The classifier is in the same situation it would be if the distribution were  $P$  and the size-three hyperedge was absent.
Definition 3. For classifiers using class list side-information, the correct-classification probability is defined as follows:  $q_{m,N}(h)_{(x,y)} = \inf_{\tilde{x} \in N(x)} \min_{C \in \binom{[K]}{m}:y \in C} h(\tilde{x},C)_y$ . The set of achievable correct-classification probabilities is  $\mathcal{P}_{m,\mathcal{V},N,\mathcal{H}} = \bigcup_{h \in \mathcal{H}} \prod_{v \in \mathcal{V}} [0,q_{m,N}(h)_v]$ .
When  $m = K$ , the minimization over  $C$  is trivial and  $q_{m,N}(h)$  reduces to  $q_N(h)$ .
Theorem 2 (Feasible output probabilities in the side-information game). The set of correct classification probability vectors for side-information of size  $m$ , support points  $\mathcal{V}$ , adversarial constraint  $N$ , and hypothesis class  $\mathcal{H}_{\text{soft}}$  is
$$
\mathcal {P} _ {m, \mathcal {V}, N, \mathcal {H} _ {\text {s o f t}}} = \left\{q \in \mathbb {R} ^ {\mathcal {V}}: q \geq \mathbf {0}, B ^ {\leq m} q \leq \mathbf {1} \right\} \tag {5}
$$
where  $B^{\leq m} \in \mathbb{R}^{\mathcal{E} \times \mathcal{V}}$  is the hyperedge incidence matrix of the conflict hypergraph  $G_{\mathcal{V},N}^{<m}$ .
The proof can be found in Supplementary  $\S A$
Using the feasible correct classification probabilities in Theorem 2, we can now write the LP for obtaining the optimal loss for classification with side-information:
Corollary 2 (Optimal loss for classification with side information / truncation lower bound).
$$
1 - L ^ {*} (P, N, \mathcal {H} _ {\text {s o f t}}) = \max  _ {q} p ^ {T} q s. t q \geq 0, B ^ {\leq m} q \leq 1.
$$
3.2 Lower bounds on multiclass optimal loss via lower bounds for binary classification
For large training datasets and large perturbation sizes, it may still be computationally expensive to compute lower bounds via LP even when using truncated hypergraphs due to the large number of edge constraints. Prior works (5; 6) proposed methods of computing lower bounds for  $0 - 1$  loss for binary classification problems and demonstrate that their algorithm is more efficient than generic LP solvers. We now ask the question: Can we use lower bounds for binary classification problems to efficiently compute a lower bound for multi-class classification?
Consider the setting where we obtain the optimal  $0 - 1$  loss for all one-versus-one binary classification tasks. Specifically, for each  $C \in \binom{[K]}{2}$ , the binary classification task for that class pair uses example distribution  $P|Y \in C$  and the corresponding optimal loss is  $L^{*}((P|Y \in C), N, \mathcal{H}_{soft})$ . What can we say about  $L^{*}(P, N, \mathcal{H}_{soft})$  given these  $\binom{K}{2}$  numbers?
This question turns about to be related to another variation of classification with side information.