SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\underset {\bar {c} \sim \mu_ {m} ^ {n}} {\mathbb {E}} [ h _ {\mathcal {U}} (\mathcal {A} ^ {s m a l l}; S) ] = \frac {1}{n} \mathbb {E} [ N _ {1} \cdot \mathbb {1} \{N _ {1} \geq n / m \} + N _ {2} \cdot \mathbb {1} \{N _ {1} <   n / m \} ].
$$
$$
\underset {\bar {c} \sim \mu_ {m} ^ {n}} {\mathbb {E}} [ h _ {\mathcal {U}} (\mathcal {A} ^ {s m a l l}; S) ] \geq \frac {1}{m} + \frac {1}{4} \sqrt {\frac {1}{m n}}
$$
by citing the following lemma:
Lemma 1 (Lemma 4 in [6].). Let  $n \geq m \geq 2$ . If  $(N_1, \ldots, N_m)$  is distributed according to a multinomial distribution with parameters  $(n; \frac{1}{m}, \ldots, \frac{1}{m})$ , then
$$
\mathbb {E} \left[ N _ {1} \cdot \mathbb {1} \{N _ {1} \geq n / m \} + N _ {2} \cdot \mathbb {1} \{N _ {1} <   n / m \} \right] \geq \frac {n}{m} + \frac {1}{4} \sqrt {\frac {n}{m}}.
$$
Case:  $k > 1$ : For a fixed  $S$ , and hence, a fixed  $\hat{f} = \mathcal{A}^{small}(S)$ , for each  $l \in \{1, \dots, m, \bot\}$ , let  $N_{i,l}^S$  be the number of examples in  $B_i$  s.t.  $\kappa_{\mathcal{U}}(\hat{f})(x) = l$ . Since whether  $\kappa_{\mathcal{U}}(\hat{f})(x)$  equals to  $\bot$  is only depends on the distribution of  $x$ , we have that  $(N_{i,1}^S, \dots, N_{i,m}^S, N_{i,\bot}^S)$  follows a multinomial distribution with parameters  $(|B_i|; \frac{C(S)}{m}, \dots, \frac{C(S)}{m}, 1 - C(S))$ , where  $C(S) = \mathbf{Pr}\left\{\kappa_{\mathcal{U}}(\hat{f})(x) \neq \bot\right\}$ .
Algorithm 2  $\mathcal{A}^{small}$  ( $k > 1$ )
1. Divide  $[n]$  into  $k - 1$  blocks  $B_{1},\ldots ,B_{k - 1}$  such that  $\frac{n}{k - 1}\leq |B_i|\leq \frac{n}{2(k - 1)}$  for each  $i$
2. For  $1 \leq i \leq k$ , define query  $f_{i}$  satisfying:
$$
f _ {i} (x _ {j}) = \left\{ \begin{array}{l l} 1, & \text {i f} j \in B _ {1} \cup B _ {2} \cup \dots \cup B _ {i - 1} \\ 2, & \text {o t h e r w i s e .} \end{array} \right.
$$
for  $x_{j}\in \mathcal{X}_{S}$  and
$$
f _ {i} (x) = f _ {i} \left(x _ {j ^ {\prime}}\right), \quad \text {w h e r e} j ^ {\prime} = \underset {j \in [ n ]} {\arg \min } d _ {p} (x, x _ {j})
$$
for  $x\notin \mathcal{X}_S$
3. Output  $\hat{f}$ , it predicts  $x_{j}$  by the following:
$$
\hat {f} (x _ {j}) = \left\{ \begin{array}{l l} 1 \text {f o r a l l} j \in B _ {i}, & \text {i f \operatorname {A c c} _ {\mathcal {U}} \left(f _ {i + 1} , S\right) \geq \operatorname {A c c} _ {\mathcal {U}} \left(f _ {i} ; S\right)} \\ 2 \text {f o r a l l} j \in B _ {i}, & \text {o t h e r w i s e .} \end{array} \right.
$$
and predicts the rest of  $x$  by  $\hat{f}(x) = \hat{f}(x_{j'})$ , where  $j' = \arg \min_{i \in [n]} d_p(x, x_j)$ .
Then our predictions robustly and correctly predicts  $\max \{N_{i,1}^S,N_{i,2}^S\}$  examples in  $B_{i}$  and hence  $h_{\mathcal{U}}(\mathcal{A};S) = \frac{1}{n}\sum_{i = 1}^{k - 1}\max \{N_{i,1}^S,N_{i,2}^S\}$ . We will show it later that
$$
\mathbb {E} \left[ \max  \{N _ {i, 1} ^ {S}, N _ {i, 2} ^ {S} \} \right] \geq \frac {C (S) | B _ {i} |}{m} + \frac {1}{4} \sqrt {\frac {C (S) | B _ {i} |}{m}}.
$$
By summing over the blocks, we can lower bound the expected total number of robustly correct predictions made by  $\mathcal{A}^{small}$ :
$$
\sum_ {i = 1} ^ {k - 1} \mathbb {E} \left[ \max \{N _ {i, 1} ^ {S}, N _ {i, 2} ^ {S} \} \right] \geq \frac {C (S) \cdot n}{m} + \frac {k - 1}{4} \sqrt {\frac {C (S) \cdot n}{2 (k - 1) m}} \geq \frac {C (S) \cdot n}{m} + \frac {1}{8} \sqrt {\frac {C (S) \cdot n k}{m}}.
$$
To obtain a lower bound of  $C(S)$  that is independent of the choice of  $S$ , for each  $S$  we define  $g_{S}$  satisfying:
$$
g _ {S} (x) = \left\{ \begin{array}{l l} c _ {i} & \forall x \in \mathcal {X} _ {S}, \\ g _ {S} (x _ {j ^ {\prime}}) & \text {w h e r e} j ^ {\prime} = \arg \min  _ {i \in [ n ]} d _ {p} (x, x _ {j}) \end{array} \right. \text {o t h e r w i s e},
$$
and let  $\mathcal{H}_n = \{g_S : S \in (\operatorname{supp}(\mathcal{D}_{\mathcal{X}}) \times \mathcal{Y})^n\}$ . It is easy to see that  $\mathcal{A}^{small}(S) \in \mathcal{H}_n$  for all  $S$ . Hence by the definition of  $\Phi_{\mathcal{D}_{\mathcal{X}}}(\mathfrak{n})$ , we have
$$
\Phi_ {\mathcal {D} _ {\mathcal {X}}} (n) = \min  _ {g _ {S} \in \mathcal {H} _ {n}} \Pr \left\{\kappa_ {\mathcal {U}} \left(g _ {s}\right) (x) \neq \bot \right\} \leq C (S).
$$
$$
\sum_ {i = 1} ^ {k - 1} \mathbb {E} \left[ \max \{N _ {i, 1} ^ {S}, N _ {i, 2} ^ {S} \} \right] \geq \frac {\Phi_ {\mathcal {D} _ {X}} (n) \cdot n}{m} + \frac {1}{8} \sqrt {\frac {\Phi_ {\mathcal {D} _ {X}} (n) \cdot n k}{m}}.
$$
Normalizing by  $n$  proves the desired result.
It remains to lower bound  $\mathbb{E}\left[\max \left\{N_{i,1}^{S}, N_{i,2}^{S}\right\}\right]$ . Since  $N_{i,1}^{S}$  and  $N_{i,2}^{S}$  follow a binomial distribution with parameters  $\left(|B_i|; \frac{C(S)}{m}\right)$ ,  $E[N_{i,1}^{S} + N_{i,2}^{S}] = \frac{2C(S)|B_i|}{m}$  for all  $S$ . Let  $N'$  be an independent copy
of  $N_{i,2}^{S}$ .  $N_{i,1}^{S}$  and  $N^{\prime}$  are negatively correlated, hence
$$
\begin{array}{l} \mathbb {E} \left[ \left| N _ {i, 1} ^ {S} - N _ {i, 2} ^ {S} \right| \right] \geq \mathbb {E} \left[ \left| N _ {i, 1} ^ {S} - N ^ {\prime} \right| \right] \geq \mathbb {E} \left[ N _ {i, 1} ^ {S} - \frac {C (S)}{m} \left| B _ {i} \right| \right] \\ \geq \sqrt {\frac {\mathbb {E} \left[ \left(N _ {i , 1} ^ {S} - \frac {C (S)}{m} | B _ {i} |\right) ^ {2} \right]}{2}} \\ \geq \sqrt {\frac {C (S) \left| B _ {i} \right|}{2 m} \left(1 - \frac {C (S)}{m}\right)} \\ \geq \sqrt {\frac {C (S) | B _ {i} |}{4 m}} \\ \end{array}
$$
For all  $S$
Then we conclude that
$$
\max \{N _ {i, 1} ^ {S}, N _ {i, 2} ^ {S} \} = \frac {N _ {i , 1} ^ {S} + N _ {i , 2} ^ {S}}{2} + \frac {| N _ {i , 1} ^ {S} - N _ {i , 2} ^ {S} |}{2} \geq \frac {C (S) | B _ {i} |}{m} + \frac {1}{4} \sqrt {\frac {C (S) | B _ {i} |}{m}},
$$
We then present  $\mathcal{A}^{big}(C)$  and the theoretical analysis for  $k = \Omega (n / m)$ .
Algorithm 3  $\mathcal{A}^{big}(C)$
1. Let  $t \coloneqq 1 + \frac{Ck}{9\ln m}$
2. Define query  $f_{1}, \ldots, f_{k}$  satisfying:
(a) For  $1 \leq j \leq t, f_1(x_j), \ldots, f_k(x_j)$  are uniformly chosen from all sequences in  $[m]^k$  that have each element in  $[m]$  appearing exactly  $k / m$  times.
(b) For  $j > t, f_{i}(x_{j}) = \bot$  for all  $i = 1,\dots ,k$
(c) For  $x \notin \mathcal{X}_S$ ,  $f_i(x) = f_i(x_{j'})$  for each  $i \in [k]$ , where  $j' = \arg \min_{i \in [n]} d_p(x, x_j)$ .
3. Output  $\hat{f}$  such that:
$$
\hat{f} (x_{j}) = \left\{ \begin{array}{ll}\arg \max_{y\in [m]}\sum_{i:f_{i}(x_{j}) = y}\operatorname {Acc}_{\mathcal{U}}(f_{i};S)^{3}, & \quad 1\leq j\leq t,\\ 1, & \quad j > t. \end{array} \right.
$$
and predicts the rest of  $x$  by  $\hat{f} (x) = \hat{f} (x_{j'})$ , where  $j^{\prime} = \arg \min_{i\in [n]}d_{p}(x,x_{j})$ .
Theorem 4. Let  $k > \frac{18m\log m}{\Phi_{\mathcal{D}\chi}(n)}$  and let  $\mathcal{D}_m^n = \mathcal{D}_{\mathcal{X}}^n\times \mu_m^n$
$$
h _ {\mathcal {U}} (\mathcal {A} ^ {b i g} (\Phi_ {\mathcal {D} _ {\mathcal {X}}} (n)), \mathcal {D} _ {m} ^ {n}) \geq \frac {\Phi_ {\mathcal {D} _ {\mathcal {X}}} (n)}{m} + \frac {\Phi_ {\mathcal {D} _ {\mathcal {X}}} (n) \cdot k}{1 4 4 n \log m}.
$$
Proof. For  $l \in [m]$ , let  $A_{l}$  be the total number of robustly and correctly predicted examples by all the queries that predict the first examples as 'l', i.e.
$$
\begin{array}{l} A _ {l} := n \cdot \sum_ {i: \forall x ^ {\prime} \in \mathcal {U} (x _ {1}), f _ {i} \left(x ^ {\prime}\right) = l} \operatorname {A c c} _ {\mathcal {U}} \left(f _ {i}; S\right) = \sum_ {i: \forall x ^ {\prime} \in \mathcal {U} (x _ {1}), f _ {i} \left(x ^ {\prime}\right) = l} \sum_ {j = 1} ^ {n} \mathbb {1} \left\{f _ {i} \left(x ^ {\prime}\right) = y _ {j}, \forall x ^ {\prime} \in \mathcal {U} \left(x _ {j}\right) \right\} \\ = W _ {0} \cdot \mathbb {1} \{l = y _ {1} \} + \sum_ {i: \forall x ^ {\prime} \in \mathcal {U} \left(x _ {1}\right), f _ {i} \left(x ^ {\prime}\right) = l} \sum_ {j = 2} ^ {t} \mathbb {1} \left\{f _ {i} \left(x ^ {\prime}\right) = y _ {j}, \forall x ^ {\prime} \in \mathcal {U} \left(x _ {j}\right) \right\}, \tag {3} \\ \end{array}
$$
where  $W_0 \leq \frac{k}{m}$  is the number of queries that predict the whole perturbation set  $\mathcal{U}(x_1)$  as  $y_{1}$ .
2Breaking ties randomly.
$$
M _ {l} := \sum_ {i: \forall x ^ {\prime} \in \mathcal {U} (x _ {1}), f _ {i} \left(x ^ {\prime}\right) = l   j = 2} \sum^ {t} \mathbb {1} \left\{f _ {i} \left(x ^ {\prime}\right) = y _ {i}, \forall x ^ {\prime} \in \mathcal {U} \left(x _ {i}\right) \right\},
$$
then for  $l \neq y_1$ ,
$$
A _ {y _ {1}} - A _ {l} = W _ {0} + M _ {y _ {1}} - M _ {l}.
$$
Since  $\mathbb{1}\{f_i(x') = y_j, \forall x' \in \mathcal{U}(x_j)\}$  are independent for  $j \neq j'$ , and are negatively associated across  $i$  for any fixed  $j$ , therefore, by the Chernoff bound, it satisfies that for  $\epsilon < 1$
$$
\mathbf {P r} \left\{| M _ {l} - \mathbb {E} \left[ M _ {l} \right] | > \epsilon \mathbb {E} \left[ M _ {l} \right] \right\} \leq 2 \exp (- \frac {\epsilon^ {2}}{3} \mathbb {E} \left[ M _ {l} \right]).
$$
Suppose  $\epsilon$  satisfies that  $\epsilon \mathbb{E}[M_l] \leq \frac{k}{2m}$  and  $\frac{\epsilon^2}{3} \mathbb{E}[M_l] \geq 3 \log m$ , then by Eq. (3) and the union bound
$$
\mathbf {P r} \left\{\underset {l \in [ m ]} {\arg \max } A _ {l} \neq y _ {1} \right\} <   (m - 1) \cdot \frac {2}{m ^ {3}} \leq \frac {1}{4},
$$
and with probability at least  $3 / 4$ ,  $\hat{f} (x_{1}) = y_{1}$ .
We now derive the bounds on  $\epsilon$ . To this end, we first need to bound  $\mathbb{E}[M_l]$ . Define  $\mathcal{H}_n$  as that in the proof of Theorem 3. Let  $\widetilde{\mathcal{H}}_n$  denotes the corrupted set of  $\mathcal{H}_n$ . For each  $\tilde{g} \in \widetilde{\mathcal{H}}_t$ , define  $\tilde{g}_{\perp} : \mathcal{X} \to \tilde{\mathcal{Y}}$  that takes value  $\perp$  on  $\mathcal{U}(x_j)$  for  $j = t + 1, \ldots, n$  and coincides with  $\tilde{g}$  otherwise, that is,
$$
\tilde {g} _ {\perp} (x) = \left\{ \begin{array}{l l} \perp , & x \in \cup_ {j = t + 1, \ldots , n} \mathcal {U} (x _ {j}), \\ \tilde {g} (x), & \text {o t h e r w i s e}. \end{array} \right.
$$
Let  $\mathcal{G}_n$  be the set which contains all such  $\tilde{g}_{\perp}$ . It is easy to see that  $\kappa_{\mathcal{U}}(f_1), \ldots, \kappa_{\mathcal{U}}(f_k) \in \mathcal{G}_n$ . Note that  $\min_{g \in \mathcal{G}_n} \operatorname*{Pr}\{g(x) \neq \perp\} = \Phi_{\mathcal{D}_X}(n)$ , we have
$$
\frac {\Phi_ {\mathcal {D} _ {x}} (n)}{m} \leq \mathbb {1} \left\{f _ {i} \left(x ^ {\prime}\right) = y _ {j}, \forall x ^ {\prime} \in \mathcal {U} \left(x _ {j}\right) \right\} \leq \frac {1}{m}.
$$
Now for each  $l$  by the linearity of expectations,
$$
(t - 1) \cdot \frac {k}{m} \cdot \frac {\Phi_ {\mathcal {D} _ {\mathcal {X}}} (n)}{m} \leq \mathbb {E} [ M _ {l} ] \leq (t - 1) \cdot \frac {k}{m} \cdot \frac {1}{m}.
$$
Thus  $\epsilon \in \mathbb{E}[M_l]$  holds for
$$
\epsilon \leq \frac {m ^ {2}}{(t - 1) k} \frac {k}{2 m} = \frac {m}{2 (t - 1)},
$$
and  $\frac{\epsilon^2}{3}\mathbb{E}\left[M_l\right]\geq 3\log m$  holds for  $\epsilon \geq \sqrt{9\log m\cdot\frac{m^2}{\Phi_{\mathcal{D}_{\mathcal{X}}}(n)(t - 1)k}}$ . Therefore, we can find a suitable  $\epsilon$  whenever
$$
\sqrt {9 \log m \cdot \frac {m ^ {2}}{\Phi_ {\mathcal {D} _ {\mathcal {X}}} (n) (t - 1) k}} \leq \frac {m}{2 (t - 1)} <   1.
$$
If we choose  $t = 1 + \frac{\Phi_{\mathcal{D}_{\mathcal{X}}}(n)k}{36\log m}$  and  $k > \frac{18}{\Phi_{\mathcal{D}_{\mathcal{X}}}(n)}m\log m$ , the aforementioned conditions hold. Note that  $\operatorname*{Pr}\left\{\kappa_{\mathcal{U}}(\hat{f}) \neq \bot \right\} \geq \Phi_{\mathcal{D}_{\mathcal{X}}}(\bar{n})$ , hence the expected number of robustly and correctly predicted labels is at least
$$
\frac {3}{4} \cdot t + \frac {\Phi_ {\mathcal {D} _ {\mathcal {X}}} (n)}{m} (n - t) \geq \frac {\Phi_ {\mathcal {D} _ {\mathcal {X}}} (n) \cdot n}{m} + t \cdot (\frac {3}{4} - \frac {\Phi_ {\mathcal {D} _ {\mathcal {X}}} (n)}{m}) \geq \frac {\Phi_ {\mathcal {D} _ {\mathcal {X}}} (n) \cdot n}{m} + \frac {\Phi_ {\mathcal {D} _ {\mathcal {X}}} (n) \cdot k}{1 4 4 \log m},
$$
In this work, we study the overfitting bias in the context of robust multiclass learning. We formally define the adaptive algorithms in an adversarial setting and analyze the average case performance that can be achieved by an adaptive algorithm. Upper bounds and lower bounds are both derived, and are matching within logarithmic factors when the number of test samples and distribution of data features are fixed.
[1] Leslie Rice, Eric Wong, and J. Zico Kolter. Overfitting in adversarially robust deep learning. In ICML, volume 119, pages 8093-8104, 2020.
[2] Avrim Blum and Moritz Hardt. The ladder: A reliable leaderboard for machine learning competitions. In ICML, volume 37 of JMLR Workshop and Conference Proceedings, pages 1006-1014, 2015.
[3] Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize toImagenet? In ICML, volume 97 of Proceedings of Machine Learning Research, pages 5389-5400, 2019.
[4] Cynthia Dwork, Vitaly Feldman, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Aaron Leon Roth. Preserving statistical validity in adaptive data analysis. In STOC, pages 117-126, 2015.
[5] Vitaly Feldman, Roy Frostig, and Moritz Hardt. The advantages of multiple classes for reducing overfitting from test set reuse. In ICML, volume 97, pages 1892-1900, 2019.
[6] Jayadev Acharya and Ananda Theertha Suresh. Optimal multiclass overfitting by sequence reconstruction from hamming queries. In ALT, volume 117 of Proceedings of Machine Learning Research, pages 3-21, 2020.
[7] Paul Erdos and Alfréd Rényi. On two problems of information theory. Magyar Tud. Akad. Mat. Kutató Int. Közl, 8:229-243, 1963.
[8] Vasek Chvátal. Mastermind. Combinatorica, 3(3):325-329, 1983.
[9] Nader H. Bshouty. Optimal algorithms for the coin weighing problem with a spring scale. In  $COLT$ , 2009.
[10] Cynthia Dwork, Vitaly Feldman, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Aaron Roth. The reusable holdout: Preserving validity in adaptive data analysis. Science, 349(6248):636-638, 2015.
[11] Cynthia Dwork, Vitaly Feldman, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Aaron Roth. Generalization in adaptive data analysis and holdout reuse. In NeurIPS, pages 2350-2358, 2015.
[12] Raef Bassily, Kobbi Nissim, Adam D. Smith, Thomas Steinke, Uri Stemmer, and Jonathan R. Ullman. Algorithmic stability for adaptive data analysis. In STOC, pages 1046-1059, 2016.
[13] Vitaly Feldman, Roy Frostig, and Moritz Hardt. Open problem: How fast can a multiclass test set be overfit? In  $COLT$ , volume 99 of Proceedings of Machine Learning Research, pages 3185-3189, 2019.
[14] Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian J. Goodfellow, and Rob Fergus. Intriguing properties of neural networks. In *ICLR*, 2014.
[15] Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In ICLR, 2015.
[16] Zekai Wang, Tianyu Pang, Chao Du, Min Lin, Weiwei Liu, and Shuicheng Yan. Better diffusion models further improve adversarial training. In ICML, volume 202 of Proceedings of Machine Learning Research, pages 36246-36263, 2023.
[17] Dong Yin, Kannan Ramchandran, and Peter L. Bartlett. Rademacher complexity for adversarially robust generalization. In ICML, volume 97 of Proceedings of Machine Learning Research, pages 7085-7094, 2019.
[18] Omar Montasser, Steve Hanneke, and Nathan Srebro. VC classes are adversarially robustly learnable, but only improperly. In  $COLT$ , volume 99 of Proceedings of Machine Learning Research, pages 2512-2530, 2019.
[19] Jingyuan Xu and Weiwei Liu. On robust multiclass learnability. In NeurIPS, 2022.
[20] Zhengyu Zhou and Weiwei Liu. Sample complexity for distributionally robust learning under chi-square divergence. Journal of Machine Learning Research, 24(230):1-27, 2023.
[21] Zekai Wang and Weiwei Liu. Robustness verification for contrastive learning. In ICML, volume 162 of Proceedings of Machine Learning Research, pages 22865-22883, 2022.
[22] Xin Zou and Weiwei Liu. Generalization bounds for adversarial contrastive learning. Journal of Machine Learning Research, 24:114:1-114:54, 2023.
[23] Hanshu Yan, Jiawei Du, Vincent Y. F. Tan, and Jiashi Feng. On robustness of neural ordinary differential equations. In ICLR, 2020.
[24] Xiyuan Li, Zou Xin, and Weiwei Liu. Defending against adversarial attacks via neural dynamic system. In NeurIPS, 2022.
[25] Xinsong Ma, Zekai Wang, and Weiwei Liu. On the tradeoff between robustness and fairness. In NeurIPS, 2022.
[26] Boqi Li and Weiwei Liu. WAT: improve the worst-class robustness in adversarial training. In Brian Williams, Yiling Chen, and Jennifer Neville, editors, AAAI, pages 14982-14990, 2023.
[27] Yao-Yuan Yang, Cyrus Rashtchian, Hongyang Zhang, Ruslan Salakhutdinov, and Kamalika Chaudhuri. A closer look at accuracy vs. robustness. In NeurIPS, 2020.
[28] Daniel Cullina, Arjun Nitin Bhagoji, and Prateek Mittal. Pac-learning in the presence of adversaries. In NeurIPS, pages 228-239, 2018.