SlowGuess/NeurIPS_2023_OCR huggingface dataset:
[19] A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu. Towards deep learning models resistant to adversarial attacks. In ICLR, 2018.
[20] S. Mahloujifar, D. I. Diochnos, and M. Mahmooyd. The curse of concentration in robust learning: Evasion and poisoning attacks from concentration of measure. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 4536-4543, 2019.
[21] O. Montasser, S. Hanneke, and N. Srebro. Vc classes are adversarially robustly learnable, but only improperly. arXiv preprint arXiv:1902.04217, 2019.
[22] M. S. Pydi and V. Jog. Adversarial risk via optimal transport and optimal couplings. In Proceedings of the 37th International Conference on Machine Learning, pages 7814-7823, 2020.
[23] M. S. Pydi and V. Jog. The Many Faces of Adversarial Risk, Jan. 2022. URL http://arxiv.org/abs/2201.08956. arXiv:2201.08956 [cs, math, stat].
[24] E. R. Scheinerman and D. H. Ullman. Fractional Graph Theory: A Rational Approach to the Theory of Graphs. Wiley, Sept. 1997. ISBN 978-0-471-17864-4. Google-Books-ID: KujuAAAAMAAJ.
[25] L. Schmidt, S. Santurkar, D. Tsipras, K. Talwar, and A. Madry. Adversarily robust generalization requires more data. arXiv preprint arXiv:1804.11285, 2018.
[26] V. Sehwag, S. Mahloujifar, T. Handina, S. Dai, C. Xiang, M. Chiang, and P. Mittal. Robust learning meets generative models: Can proxy distributions improve adversarial robustness? In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022. URL https://openreview.net/forum?id=WVXONNVBBkV.
[27] C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. Goodfellow, and R. Fergus. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199, 2013.
[28] V. Tjeng, K. Y. Xiao, and R. Tedrake. Evaluating robustness of neural networks with mixed integer programming. In ICLR, 2019.
[29] N. G. Trillos, M. Jacobs, and J. Kim. The multimarginal optimal transport formulation of adversarial multiclass classification. Journal of Machine Learning Research, 24(45):1-56, 2023.
[30] Y.-Y. Yang, C. Rashtchian, Y. Wang, and K. Chaudhuri. Robustness for non-parametric classification: A generic attack and defense. In International Conference on Artificial Intelligence and Statistics, pages 941–951. PMLR, 2020.
[31] H. Zhang, Y. Yu, J. Jiao, E. P. Xing, L. E. Ghaoui, and M. I. Jordan. Theoretically principled trade-off between robustness and accuracy. arXiv preprint arXiv:1901.08573, 2019.
This Supplementary Material accompanies paper: Characterizing the Optimal 0 - 1 Loss for Multiclass Classification with a Test-time Attacker to NeurIPS 2023.
We first present proofs for theorems from the main body in  $\S A$ . We then explain fractional packing and covering in  $\S B$ . We present our algorithm for hyperedge finding in  $\S C$ .  $\S D$  contains further details about the experimental setup with all additional experimental results in  $\S E$ .
Proof of Theorem 1. This follows immediately from Theorem 2 with  $m = K$ .
Proof of Theorem 2. Recall that the definition of the correct-classification probabilities is
$$
q_{m,N}(h)_{(x,y)} = \inf_{\tilde{x}\in N(x)}\min_{C\in \binom {[K]}{m}:y\in C}h(\tilde{x},C)_{y}
$$
and the set of achievable correct-classification probabilities is
$$
\mathcal {P} _ {\mathcal {V}, N, \mathcal {H}} = \bigcup_ {h \in \mathcal {H}} \prod_ {v \in \mathcal {V}} [ 0, q _ {N} (h) _ {v} ].
$$
First, we will show  $\mathcal{P}_{m,\mathcal{V},N,\mathcal{H}_{soft}}\subseteq \{q\in \mathbb{R}^{\mathcal{V}}:q\geq \mathbf{0},B^{\leqq m}q\leq \mathbf{1}\}$ . The constraint  $h\in \mathcal{H}_{soft}$ ,  $\mathbf{0}\leq q_{m,N}(h)\leq \mathbf{1}$  holds because classification probabilities  $h(\tilde{x},C)_y$  must lie in the range [0, 1].
We will now demonstrate that the constraint  $B^{\leq m} q \leq 1$  must also hold. Let  $e = ((x_1, y_1), \ldots, (x_\ell, y_\ell))$  be a size- $\ell$  hyperedge in  $\mathcal{E}^{(\leq m)}$ . By construction of  $\mathcal{E}^{(\leq m)}$ , there exists some  $\tilde{x} \in \bigcap_{i=1}^\ell N(x_i)$ . Let  $S \in \binom{[K]}{m}$  be some superset of  $\{y_1, \ldots, y_\ell\}$ , which exists because  $\ell \leq m$ . From the definition of  $q_{m,N}(h)$ , we have that  $q_{m,N}(h)_{(x_i, y_i)} \leq h(\tilde{x}, S)_{y_i}$  for each  $1 \leq i \leq \ell$ . Thus,
$$
\sum_ {i = 1} ^ {\ell} q _ {m, N} (h) _ {(x _ {i}, y _ {i})} \leq \sum_ {i = 1} ^ {\ell} h (\tilde {x}, S) _ {y _ {i}} \leq \sum_ {j \in \mathcal {Y}} h (\tilde {x}, S) _ {j} = 1.
$$
This gives  $(B^{\leq m}q)_{e}\leq 1$
Now we will show  $\mathcal{P}_{m,\mathcal{V},N,\mathcal{H}_{soft}}\supseteq \{q\in \mathbb{R}^{\mathcal{V}}:q\geq \mathbf{0},B^{\leq m}q\leq \mathbf{1}\}$
For any vector  $q$  in the polytope, we have a classifier  $h: \mathcal{X} \times \binom{[K]}{m} \rightarrow \mathbb{R}^{[K]}$  that achieves at least those correct classification probabilities. This means that  $h$  has the following properties. First,  $h(\tilde{x}, L)_y \geq 0$  and  $\sum_{y \in [K]} h(\tilde{x}, L)_y = 1$ . Second, for all  $(x, y) \in \mathcal{V}$ , all  $\tilde{x} \in N(x)$ , and all  $L \in \binom{[K]}{m}$  such that  $y \in L$ , we have  $h(\tilde{x}, L)_y \geq q_{(x,y)}$ .
To get  $h$ , first define the function  $g: \mathcal{X} \times \binom{[K]}{m} \to \mathbb{R}^{[K]}$  so  $g(\tilde{x}, L)_y = 0$  for  $i \notin L$  and  $g(\tilde{x}, L)_y = \max(0, \sup \{q_{(x_y, y)} : x_y \in \mathcal{V}_y, \tilde{x} \in N(x_y)\})$ . Let  $L' \subseteq L$  be the set of indices where  $g(\tilde{x}, L)_y > 0$ . Then any list of vertices  $e = (x_y : y \in L', x_y \in \mathcal{V}_y, \tilde{x} \in N(x_y))$  forms a hyperedge of size  $|L'| \leq m$ . Thus
$$
\sum_ {y \in [ K ]} g (\tilde {x}, L) _ {y} = \sum_ {y \in L ^ {\prime}} g (\tilde {x}, L) _ {y} = \sup  _ {e} \sum_ {y \in L ^ {\prime}} q _ {(x _ {y}, y)} \leq \sup  _ {e} 1 = 1.
$$
To produce  $h$ , allocate the remaining probability  $(1 - \sum_y g(\tilde{x}, L)_y)$  to an arbitrary class.
Proof of Theorem 3. The first part of this proof applies for any side-information size  $m$ . The adversarial strategy for selecting  $C$  is a specified by a conditional p.m.f.  $p_{C|y}(C|y)$ . Thus  $p_{y|C}(y|C) = p_{C|y}(C|y)p_{\mathbf{y}}(y) / \sum_{y'}p_{C|y}(C|y')p_y(y')$ .
The optimal loss of the classifier against a particular adversarial strategy is just a mixture of the optimal losses for each class list:  $\sum_{C}p_{C|y}(C|y)\operatorname*{Pr}[p_y(y)L^* (P|(y\in \{i,j\}),N,\mathcal{H})]$
If  $p_{C|y}(C|y) = p_{C|y}(C|y')$  for all  $y, y' \in C$ , then  $p_{y|C}(y|C) = p_{\mathbf{y}}(y) / \sum_{y' \in C} p_{\mathbf{y}}(y')$  and the adversary has not provided the classifier with extra information beyond the fact that  $y \in C$ . Thus  $P_{x|y}P_{y|C} = P|(y \in C)$ .
Now we can specialize to the  $m = 2$  case. Any stochastic matrix  $s$  with zeros on the diagonal specifies an adversarial strategy for selecting  $C$  with  $p_{C|y}(\{i,j\}|i) = s_{i,j}$ . Furthermore, if  $s$  is also symmetric,  $p_{C|y}(\{i,j\}|i) = p_{C|y}(\{i,j\}|j)$  and  $p_{y|C}(i|\{i,j\}) = p_{y|C}(j|\{i,j\})$ . Then the optimal classifier for the side-information game uses the  $\binom{K}{2}$  optimal classifiers for the two-class games and incurs loss  $\sum_{i,j} Pr[Y = i]a_{i,j}s_{i,j}$  where  $a_{i,j} = L^{*}(P|(y \in \{i,j\}), \mathcal{H}, N)$ . Because the diagonal entries of  $a$  are all zero, there is always a maximizing choice of  $s$  with a zero diagonal. Thus it is not necessary to include that constraint on  $s$  when specifying the optimization.
Proof of Theorem 4. If  $w = 0$ , then the lower bound is zero and holds trivially. Otherwise,  $\frac{1}{\mathbf{1}^T w} w$  forms a probability distribution over the vertices. Let  $X \in \mathcal{V}^{\mathbb{N}}$  be a sequence of i.i.d. random vertices with this distribution. From this sequence, we define a random independent set as follows. Include  $v$  in the set if it appears in the sequence  $X$  before any of its neighbors in  $\mathcal{G}$ . If  $v$  and  $v'$  are adjacent, at most one of them can be included, so this procedure does in fact construct an independent set. The probability that  $X_i = v$  is  $\frac{w_i}{\mathbf{1}^T w}$  and the probability that  $X_i$  is  $v$  or is adjacent to  $v$  is  $\frac{((A + I)w)_v}{\mathbf{1}^T w}$ . The first time that the latter event occurs,  $v$  is either included in the set or ruled out. If  $w_i > 0$ , the probability that  $v$  is included in the set is  $\frac{w_v}{((A + I)w)_v}$  and otherwise it is zero. Thus the quantity  $P(S \text{ in Theorem 4})$  is the expected size of the random independent set and  $\mathcal{G}$  must contain some independent set at least that large.
B Fractional packing and covering
In this section, we record some standard definition in fractional graph and hypergraph theory (24).
Let  $\mathcal{G} = (\mathcal{V},\mathcal{E})$  be hypergraph and let  $B\in \mathbb{R}^{\mathcal{E}\times \mathcal{V}}$  be its edge-vertex incidence matrix. That is,  $B_{e,v} = 1$  if  $v\in e$  and  $B_{e,v} = 0$  otherwise. Let  $p\in \mathbb{R}^{\nu}$  be a vector of nonnegative vertex weights. Let  $\mathbf{0}$  be the vector of all zeros and let  $\mathbf{1}$  be the vector of all ones (of whichever dimensions are required).
A fractional vertex packing in  $\mathcal{G}$  is a vector  $q\in \mathbb{R}^{\nu}$  in the polytope
$$
q \geq \mathbf {0} \hskip 2 8. 4 5 2 7 5 6 p t B q \leq \mathbf {1}.
$$
The minimum weight fractional vertex packing linear program is
$$
\min  p ^ {T} q \quad \text {s . t .} \quad q \geq \mathbf {0} \qquad B q \leq \mathbf {1}.
$$
A fractional hyperedge covering in  $\mathcal{G}$  for weights  $p$  is a vector  $z\in \mathbb{R}^{\nu}$  in the polytope
$$
q \geq \mathbf {0} \quad B ^ {T} z \geq p.
$$
The minimum weight fractional hyperedge covering linear program is
These linear programs form a dual pair.
The independent set polytope for  $\mathcal{G}$  is the convex hull of the independent set indicator vectors.
Let  $\mathcal{G}'$  be a graph on the vertex set  $\mathcal{V}$ . A clique in  $\mathcal{G}'$  is a subset of vertices in which every pair is adjacent. Let  $\mathcal{C}$  be the set of cliques of  $\mathcal{G}'$  and let  $C \in \mathbb{R}^{C \times \mathcal{V}}$  be the clique vertex incidence matrix. (In fact this construction can be interpreted as another hypergraph on  $\mathcal{V}$ .) A fractional clique cover in  $\mathcal{G}'$  for vertex weights  $p$  is a vector  $z \in \mathbb{R}^C$  in the polytope
$$
q \geq \mathbf {0} \quad C ^ {T} z \geq p.
$$
Somewhat confusingly, the dual concept is called a fractional independent set in  $\mathcal{G}'$ . This is a vector  $q \in \mathbb{R}^{\nu}$  in the polytope
$$
q \geq \mathbf {0} \quad C q \leq \mathbf {1}.
$$
C Hyperedge Finding
One challenge in computing lower bounds for  $0 - 1$  loss in the multi-class setting is that we need to find hyperedges in the conflict hypergraph. In this section, we will consider an  $\ell_2$  adversary:  $N(x) = \{x' \in \mathcal{X} | ||x' - x||_2 \leq \epsilon\}$  and describe an algorithm for finding hyperedges within the conflict graph.
We first note that for an  $n$ -way hyperedge to exist between  $n$  inputs  $\{x_i\}_{i=1}^n$ ,  $\{x_i\}_{i=1}^n$  must all lie on the interior of an  $n-1$ -dimensional hypersphere of radius  $\epsilon$ .
Given input  $x_{1}, \ldots, x_{n}$  where  $x_{i} \in \mathbb{R}^{d}$ , we first show that distance between any two points in the affine subspace spanned by the inputs can be represented by a distance matrix whose entries are the squared distance between inputs. This allows us to compute the circumradius using the distance information only, not requiring a coordinate system in high dimension. Then we find the circumradius using the properties that the center of the circumsphere is in the affine subspace spanned by the inputs and has equal distance to all inputs.
We construct matrix  $X \in \mathbb{R}^{d \times n}$  whose  $i^{th}$  column is input  $x_{i}$ . Let  $D \in \mathbb{R}^{n \times n}$  be the matrix of squared distances between the inputs, i.e.,  $D_{i,j} = \| x_{i} - x_{j}\|^{2}$ .
We first notice that  $D$  can be represented by  $X$  and a vector in  $\mathbb{R}^n$  whose  $i^{th}$  entry is the squared norm of  $x_{i}$ . Let  $\Delta \in \mathbb{R}^n$  be such a vector such that  $\Delta_i = \| x_i\|^2 = (X^T X)_{i,i}$ . Then given that  $D_{i,j}$  is the squared distance between  $x_{i}$  and  $x_{j}$ , we have
$$
D _ {i, j} = \left\| x _ {i} \right\| ^ {2} + \left\| x _ {j} \right\| ^ {2} - 2 \langle x _ {i}, x _ {j} \rangle ,
$$
$$
D = \Delta \mathbf {1} ^ {T} + \mathbf {1} \Delta^ {T} - 2 X ^ {T} X.
$$
Let  $\alpha, \beta \in \mathbb{R}^n$  be vectors of affine weights:  $\mathbf{1}^T\alpha = \mathbf{1}^T\beta = 1$ . Then  $X\alpha$  and  $X\beta$  are two points in the affine subspace spanned by the columns of  $X$ . The distance between  $X\alpha$  and  $X\beta$  is  $\frac{-(\alpha - \beta)^T D(\alpha - \beta)}{2}$ , shown as below:
$$
\begin{array}{l} \frac {- (\alpha - \beta) ^ {T} D (\alpha - \beta)}{2} = \frac {- (\alpha - \beta) ^ {T} (\Delta \mathbf {1} ^ {T} + \mathbf {1} \Delta^ {T} - 2 X ^ {T} X) (\alpha - \beta)}{2} \\ = \frac {- (0 + 0 - 2 (\alpha - \beta) ^ {T} X ^ {T} X (\alpha - \beta))}{2} \\ = \| X \alpha - X \beta \| ^ {2}. \\ \end{array}
$$
Now we compute the circumradius using the squared distance matrix  $D$ . The circumcenter is in the affine subspace spanned by the inputs so we let  $X\alpha$  to be the circumcenter where  $\mathbf{1}^T\alpha = 1$ . Let  $e^{(i)} \in \mathbb{R}^n$  be the  $i^{th}$  standard basis vector. The distance between the circumcenter and  $x_i$  is  $\| X\alpha - Xe^{(i)}\|^2$ . From previous computation, we know that  $\| X\alpha - Xe^{(i)}\|^2 = \frac{-(\alpha - e^{(i)})^T D(\alpha - e^{(i)})}{2}$ . Since the circumcenter has equal distance to all inputs, we have
$$
\left(\alpha - e ^ {(1)}\right) ^ {T} D \left(\alpha - e ^ {(1)}\right) = \dots = \left(\alpha - e ^ {(n)}\right) ^ {T} D \left(\alpha - e ^ {(n)}\right). \tag {7}
$$
Note that the quadratic term in  $\alpha$  is identical in each of these expressions. In addition,  $e^{(i)T}De^{(i)} = 0$  for all  $i$ . So equation 7 simplifies to the linear system
$$
\begin{array}{l} e ^ {(i) T} D \alpha = c \Longrightarrow D \alpha = c \mathbf {1} \\ \Rightarrow \alpha = c D ^ {- 1} \mathbf {1} \\ \end{array}
$$
for some constant  $c$ . Since  $\mathbf{1}^T\alpha = 1$ , we have
$$
\begin{array}{l} 1 = \mathbf {1} ^ {T} \alpha = c \mathbf {1} ^ {T} D ^ {- 1} \mathbf {1} \\ \Longrightarrow \frac {1}{c} = \mathbf {1} ^ {T} D ^ {- 1} \mathbf {1} \\ \end{array}
$$
assuming that  $D$  is invertible. The square of the circumradius,  $r^2$ , which is the squared distance between the circumcenter and  $x_1$ , is
$$
\begin{array}{l} \left\| X \alpha - X e ^ {(1)} \right\| ^ {2} \\ = \frac {- (\alpha - e ^ {(1)}) ^ {T} D (\alpha - e ^ {(1)})}{2} \\ = e ^ {(1) T} D \alpha - \frac {\alpha^ {T} D \alpha}{2} \\ = c - \frac {c ^ {2} \mathbf {1} ^ {T} D ^ {- 1} \mathbf {1}}{2} \\ = \frac {c}{2} \\ = \frac {1}{2 \mathbf {1} ^ {T} D ^ {- 1} \mathbf {1}}. \\ \end{array}
$$
Therefore, assuming matrix  $D$  is invertible, the circumradius is  $\frac{1}{\sqrt{21^T D^{-1}} \mathbf{1}}$ .
The inverse of  $D$  can be computed as  $\frac{\operatorname*{det}D}{\operatorname*{adj}D}$ . Since  $\alpha = cD^{-1}\mathbf{1}$ , we have  $\alpha = c\frac{\operatorname*{det}D}{\operatorname*{adj}D}\mathbf{1}$ . As  $r^2 = \frac{c}{2}$ , constant  $c$  is non-negative. Therefore,  $\alpha \propto \frac{\operatorname*{det}D}{\operatorname*{adj}D}\mathbf{1}$ .
When all entries of  $\alpha$  are non-negative, the circumcenter is a convex combination of the all inputs and the circumsphere is the minimum sphere in  $\mathbb{R}^{n - 1}$  that contains all inputs. Otherwise, the circumsphere of  $\{x_{i}|\alpha_{i} > 0\}$  is the minimum sphere contains all inputs.
After finding the radius of the minimum sphere that contains all inputs, we compare the radius with the budget  $\epsilon$ . If the radius is no larger than  $\epsilon$ , then there is a hyperedge of degree  $n$  among the inputs.
D Experimental Setup
In this section, we describe our experimental setup. Our code for computing bounds is also available at https://github.com/inspire-group/multiclass_robust.lb.
Datasets: We compute lower bounds for MNIST (17), CIFAR-10, and CIFAR-100 (16). Since we do not know the true distribution of these datasets, we compute lower bounds based on the empirical distribution of the training set for each dataset.
Attacker: We will consider an  $\ell_2$  adversary:  $N(x) = \{x' \in \mathcal{X} | ||x' - x||_2 \leq \epsilon\}$ . This has been used in most prior work (5; 22; 29).
LP solver: For solving the LP in Equation (4), we primarily use the Mosek LP solver (3). When the Mosek solver did not converge, we default to using CVXOpt's LP solver (2).
Computing infrastructure. In order to compute lower bounds, we perform computations across 10 2.4 GHz Intel Broadwell CPUs. For adversarial training, we train on a single A100 GPU.
Training Details For MNIST, we use 40-step optimization to find adversarial examples during training and use step size  $\frac{\epsilon}{30}$  and train all models for 20 epochs. For CIFAR-10 and CIFAR-100, we use 10 step optimization to find adversarial examples and step size  $\frac{\epsilon}{7}$  and train models for 100 epochs. For MNIST TRADES training, we use  $\beta = 1$  and for CIFAR-10 and CIFAR-100, we use  $\beta = 6$ . Additionally, for CIFAR-10 and CIFAR-100, we optimize the model using SGD with learning rate and learning rate scheduling from (14). For MNIST, we use learning rate 0.01.
Architectures used: For CIFAR-10 and CIFAR-100, we report results from training a WRN-28-10 architecture. For MNIST, we train a small CNN architecture consisting of 2 convolutional layers, each followed by batch normalization, ReLU, and 2 by 2 max pooling. The first convolutional layer uses a 5 by 5 convolutional kernel and has 20 output channels. The second convolutional layer also uses a 5 by 5 kernel and has 50 output channels. After the set of 2 convolutional layers with batch normalization, ReLU, and pooling, the network has a fully connected layer with 500 output channels followed by a fully connected classifier (10 output channels). In §E.8, we consider the impact of architecture on closing the gap to the optimal loss.
In this Section, we provide additional experimental results to complement those from the main body of the paper. We organize it as follows:
- §E.1: We analyze a toy problem on 2D Gaussian data to get a better understanding of the impact of hyperedges on the optimal loss computation.
- §E.2: We investigate why higher degree hyperedges dp not have a large impact on lower bounds at lower values of  $\epsilon$ .
- §E.3: We show the runtime explosion at higher values of  $\epsilon$  that makes it challenging for us to report optimal loss values.
- §E.4: Classwise L2 distance statistics and heatmaps for pairwise losses used to compute class-only lower bounds in main paper.
- §E.5: We provide results with standard PGD-based adversarial training, and show it is outperformed by Trades.
- §E.6: We provide results on the CIFAR-100 dataset.
- §E.7: We show lower bounds for a different set of 3 classes than the one considered in the main body. Main takeaways remain the same.
- §E.8: We ablate across larger neural networks to check if increasing capacity reduces the gap to optimal.
- §E.9: We attempt dropping examples from the training set that even the optimal classifier cannot classify correctly in order to improve convergence.
- §E.10: We compute lower bounds on the test set for MNIST 3-class classification
E.1 Results for Gaussian data
Figure 4: A sample 3-class Gaussian problem (each color pertains to a class) and a corresponding classifier for this problem shown in black. The classifier classifies a sample incorrectly when it lies over the edge of the  $\epsilon$  margin (shown by the red lines) nearest the corresponding Gaussian center.
We begin with a 3-way classification problem on 2D Gaussian data. To generate our Gaussian dataset, we sample 1000 points per class from 3 spherical Gaussians with means at distance 3 away from the origin (a sample is shown in Figure 4). We compute multiclass lower bounds via the LP in Theorem 1 on robust accuracy at various  $\ell_2$  budget  $\epsilon$  and display these values in Figure 5 as  $L^{*}(3)$ . Additionally, we compare to a deterministic 3 way classifier. This classifier is the best performing out of the 2 strategies: 1) constantly predict a single class (thus achieving  $\frac{2}{3}$  loss) or 2) is the classifier in black in Figure 4 which classifies incorrectly when a sample lies over the edge of the nearest  $\epsilon$  margin of the classifier.
We observe that at smaller values of  $\epsilon$ , the loss achieved by the 3-way classifier matches optimal loss  $(L^{*}(3))$ ; however, after  $\epsilon = 2.5$  for  $\sigma^2 = 0.05$  and  $\epsilon = 2.3$  for  $\sigma^2 = 0.5$ , we find the classifier no longer achieves optimal loss. This suggests that there is a more optimal classification strategy at these larger values of  $\epsilon$ . In Figures 6 and 7, we visualize the distribution of correct classification probabilities obtained through solving the LP with and without considering hyperedges. These figures are generated by taking a fresh sample of 1000 points from each class and coloring the point based on the correct classification probability  $q_{v}$  assigned to its nearest neighbor that was used in the conflict
(a)  $\sigma^2 = 0.05$
(b)  $\sigma^2 = 0.5$
Figure 5: Lower bounds on error for the Gaussian 3-class problem ( $\sigma^2 = 0.05$  and  $\sigma^2 = 0.5$ ) computed using only constraints from edges ( $L^{*}(2)$ ) and up to degree 3 hyperedges ( $L^{*}(3)$ ) in comparison to the performance of the deterministic 3-way classifier depicted in Figure 4.
hypergraph when computing the lower bound. We observe from Figure 6, for all classes, the data are mostly assigned classification probabilities around 0.5. In Figure 7, we can see that when we consider hyperedges, we some of these 0.5 assignments are reassigned values close to  $\frac{2}{3}$  and  $\frac{1}{3}$ . Interestingly, we notice that when we do not consider hyperedges, our solver finds an asymmetric solution to the problem (strategies for class 0, 1, and 2 differ) while when considering hyperedges this solution becomes symmetric.
(a) Class 0
(b) Class 1
(c) Class 2
(a) Class 0
Figure 7: Distribution of optimal classification probabilities across samples from each class of the Gaussian obtained as a solution when computing  $L^{*}(3)$ .
Figure 6: Distribution of optimal classification probabilities across samples from each class of the Gaussian obtained as a solution when computing  $L^{*}(2)$ .
(c) Class 2
E.2 Impact of hyperedges
In Figure 8, we show the count of edges, degree 3 hyperedges, and degree 4 hyperedges found in the conflict hypergraphs of the MNIST, CIFAR-10, and CIFAR-100 train sets. We note that we did
not observe any increase in loss when considering degree 4 hyperedges at the  $\epsilon$  with a data point for number of degree 4 hyperedges in Figure 8. We find that the relative number of edges and hyperedges is not reflective of whether we expect to see an increase in loss after considering hyperedges. For example in CIFAR-10, at  $\epsilon = 4.0$ , we there are about 100 times more hyperedges than edges, but we see no noticeable increase in the  $0 - 1$  loss lower bound when incorporating these hyperedge constraints.
(b) CIFAR-10
Figure 8: Number of edges, degree 3 hyperedges, and degree 4 hyperedges found in the conflict hypergraphs of MNIST, CIFAR-10, and CIFAR-100 train sets. The red vertical line indicates the  $\epsilon$  at which we noticed an increase in the  $0 - 1$  loss lower bound when considering degree 3 hyperedges.
(c) CIFAR-100