SlowGuess/NeurIPS_2023_OCR huggingface dataset:
To understand why including more information about hyperedges does not influence the computed lower bound much, we examine the distribution of  $q_{v}$  obtained from solutions to the LP with  $(L^{*}(3))$  and without degree 3 hyperedges  $(L^{*}(2))$ . Fig. 9 contains a histogram of the distributions of  $q_{v}$  for MNIST. For small  $\epsilon$ , there is no change in the distribution of  $q_{v}$ , the distribution of  $q_{v}$  is almost identical between  $L^{*}(2)$  and  $L^{*}(3)$ . At larger values of  $\epsilon$ , in  $L^{*}(2)$ , a significant fraction of vertices are assigned  $q_{v}$  near 0.5. While these shift with the addition of hyperedges, very few of them were in triangles of  $\mathcal{G}^{\leq 2}$  that were replaced by a hyperedge in  $\mathcal{G}^{\leq 3}$ . This leads to the loss value changing minimally.
Similar to Figure 9, we plot the distribution of vertex weights  $q_{v}$  obtained through solving the LP for  $L^{*}(2)$  and  $L^{*}(3)$  for CIFAR-10 in Figure 10. Similar to trends for MNIST, we find that the gap between  $L^{*}(2)$  and  $L^{*}(3)$  only occurs when the frequency of 0.5 weights is higher.
(a)  $\epsilon = 2.5$
Figure 9: Distribution of optimal classification probabilities  $q$  obtained by solving the LP with up to degree 2 hyperedges ( $m = 2$ ) and up to degree 3 hyperedges ( $m = 3$ ) on the MNIST training set.
(b)  $\epsilon = 3.0$
(c)  $\epsilon = 3.5$
E.3 Computational complexity of computing lower bounds
Our experiments of  $L^{*}(3)$  and  $L^{*}(4)$  at higher  $\epsilon$  are limited due to computation constraints. Figure 11 we see that the time taken to compute the  $L^{*}(3)$  grows rapidly with  $\epsilon$ . We also report timings for all bounds for CIFAR-10 at  $\epsilon = 4$  in Table 2. In future work, we are seeking algorithmic optimization to achieve more results at high  $\epsilon$ .
(a)  $\epsilon = 3.5$
(b)  $\epsilon = 4.0$
(c)  $\epsilon = 4.5$
Figure 10: Distribution of optimal classification probabilities  $q$  obtained by solving the LP with up to degree 2 hyperedges  $(L^{*}(2))$  and up to degree 3 hyperedges  $(L^{*}(3))$  on the CIFAR-10 training set.
(b) CIFAR-10
Figure 11: Time taken to compute  $L^{*}(2)$  and  $L^{*}(3)$  for MNIST and CIFAR-10.
E.4 Classwise statistics and pairwise losses
In order to have a better understanding of the difficulty of the classification task under the presence of an  $\ell_2$  bounded adversary, we report the average  $\ell_2$  to the nearest neighbor of another class for each class in the MNIST and CIFAR-10 datasets in Table 3.
Another way of understanding the relative difficulty of classifying between classes is by computing the optimal loss for all 1v1 binary classification tasks. We note that these values are used in Section 4.2 to compute a lower bound on the optimal loss in the 10-class case from maximum weight coupling over optimal losses for 1v1 binary classification problems. In Figure 12, we show the heat maps for optimal losses for each pair of 1v1 classification problems for  $\epsilon = 3$  on MNIST and  $\epsilon = 4$  on CIFAR-10. We find that for both datasets only a few pairs of classes have high losses. Specifically, for MNIST, we find that the class pairs 4-9 and 7-9 have significantly higher loss than all other pairs of classes. For CIFAR-10, we find that 2-4 has the highest loss compared to other pairs, and 2-6 and 6-4 are also quite high.
E.5 Additional adversarial training results
In Figure 13, we also add the loss achieved by PGD adversarial training. We find that this approach generally performs worse on MNIST compared to TRADES and is also unable to fit to CIFAR-10 data at the  $\epsilon$  values tested.
E.6 Truncated hypergraph lower bounds for CIFAR-100
We provide results for truncated hypergraph lower bounds for the CIFAR-100 train set. We observe that similar to MNIST and CIFAR-10, including more hyperedge constraints does not influence the computed lower bound.
Loss bound	Runtime (s)
L*(2)	188.24
L*(3)	10413.91
L*(4)	>86400
Lco*(2)	327.92
LCW	192.27
Table 2: Runtimes for computing different bounds for CIFAR-10 dataset at  $\epsilon = 4$ . We note that the  $L_{co}^{*}(2)$  reports the time for computing all pairwise losses sequentially and can be sped up by running these computations in parallel. We also note that  $L^{*}(4)$  computation did not terminate within a day.
0	1	2	3	4	5	6	7	8	9
MNIST	7.07	4.33	6.94	6.29	5.72	6.29	6.42	5.52	6.29	5.15
CIFAR-10	8.96	10.84	8.48	9.93	8.22	10.04	8.72	10.05	9.23	10.91
Table 3: Average  ${\ell }_{2}$  distance to nearest neighbor in another class for each class in MNIST and CIFAR-10 datasets.
E.7 Computed bounds for a different set of 3-classes
In Figure 15, we plot 3-class lower bounds via truncated hypergraphs  $(L^{*}(2)$  and  $L^{*}(3))$  for a different set of 3 classes as shown in the main body. These classes generally have less similarity than the classes shown in the main body of the paper causing the loss bound to increase more slowly as epsilon increases. However, we find that patterns observed for the 3 classes present in the main body of the paper are also present here: the gap between  $L^{*}(2)$  and  $L^{*}(3)$  is only visible at large values of 0-1 loss (ie. loss of 0.4).
E.8 Impact of architecture size
Previously, we saw that adversarial training with larger values of  $\epsilon$  generally fails to converge (leading to losses matching random guessing across 3 classes). We now investigate whether increasing model capacity can resolve this convergence issue. In Figure 16, we plot the training losses of 3 WRN architectures commonly used in adversarial ML research across attack strength  $\epsilon$  for the 3-class CIFAR-10 (classes 0, 2, 8) problem. All models are trained with TRADES adversarial training. Interestingly, we find that the benefit of larger architecture size only appears for the smallest value of epsilon plotted  $(\epsilon = 1)$  at which the optimal classifier can theoretically obtain 0 loss. At larger values of epsilon, the improvement in using larger architecture generally disappears.
E.9 Impact of dropping "hard" examples
From our experiments involving adversarial training, we found that training with large values of  $\epsilon$  generally fails to converge. A potential way of trying to improve convergence using the results from our LP is to drop examples with optimal classification probability less than 1 and train with the remaining examples. Since even the optimal classifier cannot classify these examples correctly, these examples can be considered "hard" to learn. We find that in practice this does not lead to lower training loss; specifically, training without "hard" examples leads to a loss of 0.64 for CIFAR-10 with  $\epsilon = 3$  while training with "hard" examples leads to a loss 0.57. We note that this loss is computed over the entire training dataset (including "hard" examples) for both settings.
E.10 Lower bounds on test set
In the main text, we compute lower bounds on the train set as this would measure how well existing training algorithms are able to fit to the training data. In Table 4, we compute lower bounds on the test set (which contains 1000 samples per class) for MNIST 3-class classification between classes 1, 4, and 7. We find that the computed loss is similar to what is computed on a subset of the train set which contains the same number of samples per class.
(a) MNIST  $(\epsilon = 3)$
(b) CIFAR-10  $(\epsilon = 4)$
Figure 12: Heat maps for optimal loss for each pair of 1v1 classification problems.
(a) MNIST (1, 7, 9)
(b) CIFAR-10 (0, 2, 8)
Figure 13: Lower bounds on error for MNIST and CIFAR-10 3-class problems (1000 samples per class) computed using only constraints from edges  $(L^{*}(2))$  and up to degree 3 hyperedges  $(L^{*}(3))$  in comparison to TRADES adversarial training (TRADES-AT) and PGD adversarial training (PGD-AT) loss.
Figure 14: Lower bounds for optimal 0-1 loss the for CIFAR-100 train set
(a) MNIST (2, 5, 8)
Figure 15: Lower bounds on error for MNIST and CIFAR-10 3-class problems (1000 samples per class) computed using only constraints from edges  $(L^{*}(2))$  and up to degree 3 hyperedges  $(L^{*}(3))$
(b) CIFAR-10 (1, 4, 9)
Figure 16: Impact of architecture size on training loss for 3-class CIFAR (0, 2, 8) at different strengths  $\epsilon$  for models trained with TRADES adversarial training.
€	train set	train set
(1000 samples per class)	test set
2	0.0045	0.0020	0.0025
2.5	0.0260	0.0193	0.0149
3	0.1098	0.0877	0.0773
3.5	0.2587	0.2283	0.2181
4	-	0.4083	0.3987
Table 4: Optimal losses  ${L}^{ * }\left( 3\right)$  for MNIST 3 class problem (classes  $1,4,7$  ) computed across the train set,the first 1000 samples of each class in the train set,and computed across the test set.