SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Moreover, the above result can be even strengthened to any approximate empirical risk minimizer, that is a minimizer with an additive  $\Theta(1/m^2)$  error over the empirical risk Shalev-Shwartz et al. [2009], Amir et al. [2021b]. But even further, for the particular structure of  $f$ , which is a regularized linear objective, by [Sridharan et al., 2008, Thm 1] we have:
$$
\mathbb {E} \left[ \Delta_ {D} \left(w _ {S}\right) \right] \leq \tilde {\mathcal {O}} \left(\mathbb {E} \left[ \Delta_ {S} \left(w _ {S}\right) \right] + 1 / m\right).
$$
Together with Theorem 1 we obtain the following algorithmic-independent result:
Theorem 2. There exists a family of distributions  $\mathcal{D}$ , such that for any algorithm  $A$  and  $m > 3$ , there exists a distribution  $D \in \mathcal{D}$  such that if  $\Delta_S(w_S) \leq 1/54$  and  $|S| > m$  then:
$$
I (w _ {S}, S) = \tilde {\Omega} \left(\frac {d}{m ^ {6} \cdot (\mathbb {E} [ \Delta_ {S} (w _ {S}) ] + 1) ^ {5}}\right),
$$
but for any algorithm  $A$  and distribution  $D \in \mathcal{D}$ :
$$
\mathbb {E} \left[ \Delta_ {D} \left(w _ {S}\right) \right] = \tilde {\mathcal {O}} \left(\mathbb {E} \left[ \Delta_ {S} \left(w _ {S}\right) \right] + 1 / m\right).
$$
In other words, even without algorithmic assumptions, we can construct a class of distributions which make the problem easy to learn, but the information bounds are still dimension dependent. In
particular, for any algorithm such that  $\Delta_S(w_S) = O(1 / m)$  we will have that the bound in Eq. (2) is order of:
$$
\mathbb {E} \left[ \Delta_ {D} \left(w _ {S}\right) \right] = \tilde {O} \left(\sqrt {d / m}\right). \tag {5}
$$
Comparison to uniform convergence bounds: Notice that Eq. (5) is the standard generalization bound that can be obtained for any ERM algorithm in the setting of stochastic convex optimization. In detail, through a standard covering number argument (Shalev-Shwartz et al., 2009, thm 5) it is known that, when  $f$  is 1-Lipschitz (not necessarily convex even):
$$
\sup  _ {w \in \mathcal {W}} \underset {S} {\mathbb {E}} \left[ L _ {D} (w) - \frac {1}{m} \sum_ {i = 1} ^ {m} f (w, z _ {i}) \right] \leq \tilde {O} (\sqrt {d / m}).
$$
In other words, the bound we obtain in Eq. (5) can be obtained for any algorithm, with minimal assumptions, irrespective of the mutual information between output and sample. We remark, though, that, through a similar covering argument, one can show that indeed there are algorithms where one can recover the above bound via information-theoretic reasoning.
Proposition 1. Given a 1-Lipschitz function  $f$ , there exists an algorithm  $A$  that given input sample  $S$  outputs a parameters  $w_{S} \in \mathcal{W}$ : such that
$$
\frac {1}{m} \sum_ {i = 1} ^ {m} f \left(w _ {S}, z _ {i}\right) \leq \min  _ {w ^ {\star} \in \mathcal {W}} \frac {1}{m} \sum_ {i = 1} ^ {m} f \left(w ^ {\star}, z _ {i}\right) + \sqrt {\frac {d}{m}}, \tag {6}
$$
$$
I (w _ {S}, S) = \tilde {O} (d \log m).
$$
$$
L _ {D} \left(w _ {S}\right) - L _ {D} \left(w ^ {\star}\right) = \tilde {O} (\sqrt {d / m}).
$$
Sketch. The result follows a simple covering argument. In particular, it is known that there exists a finite subset  $\bar{\mathcal{W}}\subseteq \mathcal{W}$  of size  $|\bar{\mathcal{W}} | = O\left(\sqrt{m}^{d}\right)$  such that for every  $w\in \mathcal{W}$  there is  $\bar{w}\in \bar{\mathcal{W}}$  such that  $\| w - \bar{w}\| \leq \sqrt{d / m}$  (e.g. Wu [2017]). Now, we consider an ERM that is restricted to the set  $\bar{W}$ . By Lipschitness we have that Eq. (6) holds. The information is bounded by the entropy of the algorithm and we have that
$$
I \left(W _ {S}, S\right) \leq H \left(W _ {S}\right) \leq \log | \bar {\mathcal {W}} | = O (d \log m).
$$
The genearlization gap can be bounded via Eq. (2) (or standard union bound).
CMI-bounds Similarly to our setting, also in the setting of PAC learning, Livni and Moran [2020], Bassily et al. [2018] provided limitations to information theoretic generalization bounds. Specifically, they showed that such bounds become vacous in the task of learning thresholds, and the information between output and sample may be unbounded. To resolve this issue, which happens because of the numerical precision required by a thresholds learner, Steinke and Zakynthinou [2020] introduced generalization bounds that depend on conditional mutual information (CMI). They provided CMI bounds that can be derived from VC bounds, compression bounds etc... which in a nutshell means that they are powerful enough to achieve tight learning rates for VC classes such as thresholds. However, the issue in SCO is not comparable. As we show in Proposition 1 already the classical information-theoretic bounds are powerful enough to demonstrate generalization bounds that can be derived via union bound or uniform convergence. In PAC learning, these bounds are also tight, but in SCO such bounds are dimension-dependent. In that sense, there is no analog result to the limitation of learning thresholds. Partly because there is no need for infinite precision in SCO. In SCO, though, we require a separation from uniform convergence bounds, or even more strongly - from dimension dependent bounds. While Haghifam et al. [2020] does demonstrate certain algorithmic-dependent limitations for GD and perturbed GD algorithms, one might hope that, similar to PAC learning, here too CMI-bounds might be able to capture optimal dimension-independent rates for some algorithms.
More formally, given a distribution  $D$  over  $\mathcal{Z}$ , we consider a process where we draw i.i.d two random samples  $Z = S_{1} \times S_{2}$ , where  $S_{1} = \{z_{1}^{0}, \ldots, z_{m}^{0}m\} \sim D^{m}$  and  $S_{2} = \{z_{1}^{1}, \ldots, z_{m}^{1}\} \sim D^{m}$ . Then we define a sample  $S$  by randomly picking  $z_{i} = z_{i}^{0}$  w.p.  $1/2$  and  $z_{i} = z_{i}^{1}$  w.p.  $1/2$  (independently from
$z_{1},\ldots ,z_{i - 1},z_{i + 1},\ldots ,z_{m})$  . Then, Steinke and Zakynthinou [2020] showed that, similarly to Eq. (17), the generalization of an algorithm can be bounded in terms of
$$
\operatorname {g e n e r a l i z a t i o n} (A) = O \left(\sqrt {\operatorname {C M I} _ {m} (A) / m}\right), \quad \operatorname {C M I} _ {m} (A) = I \left(w _ {S}, S \mid Z\right). \tag {7}
$$
Recall that, given a random variable  $Z$ , the conditional mutual information between r.v.  $X$  and r.v.  $Y$  is defined as:
$$
I \left(X; Y \mid Z\right) = \mathbb {E} _ {Z} \left[ \right. \mathbb {E} _ {Y} \left[ \right. D _ {K L} \left( \right.P _ {X \mid Z} \left\| \right. P _ {X \mid Y, Z}\left. \right)\left. \right]\left. \right].
$$
One can show that  $\mathrm{CMI}_m(A) = O(m)$ . The simplest way to achieve a dimension independent generalization bound would be to subsample. If algorithm  $A$  observes a sample  $S$  of size  $m$  and subsamples  $f(m)$  samples, then we trivially obtain the generalization gap:  $O(1 / \sqrt{f(m)})$ . Taking Eq. (7) into consideration by subsampling  $O(\sqrt{m})$  examples, Eq. (7) will lead to a dimension independent generalization bound of  $O(1 / \sqrt[m]{m})$ . So it is possible to obtain dimension independent bounds through the CMI framework, nevertheless it is still not clear if we can beat the above, nonoptimal and naive subsampling bound
Open Problem. Is there an algorithm  $A$  in the SCO setup that can achieve
$$
C M I _ {m} (A) = \tilde {\sigma} (\sqrt {m}),
$$
$$
\mathbb {E} \left[ \Delta_ {D} (A) \right] = O (1 / \sqrt {m}).
$$
4 Technical overview
We next outline the key technical tools and ideas that lead to the proof of Theorem 1. A full proof of Theorem 1 is provided in the supplementary material. As discussed we draw our idea from the privacy literature Steinke and Ullman [2015], Kamath et al. [2019], Bun et al. [2014] and build on fingerprinting Lemmas Boneh and Shaw [1998], Tardos [2008] to construct our "information attacks". We employ a simplified and well-tailored version of the fingerprinting Lemma, due to Kamath et al. [2019], to construct a lower bound on the correlation between the output and the data. From that point our proof differs from standard privacy attacks.
Given the above correlation bound, we now want to lower bound the mutual information between the two correlated random variables (i.e. the output of the algorithm and the empirical mean). Surprisingly, we could not find in the literature an existing lower bound on the mutual information between two correlated random variables. The next Lemma provides such a lower bound and as such may be of independent interest.. We next depict these two technical Lemmas that we need for our proof - the fingerprinting Lemma due to Kamath et al. [2019] and a lower bound on the mutual information between two correlated random variables.
For describing the fingerprinting Lemma, let us denote by  $U[-1/3, 1/3]$  the uniform distribution over the interval  $[-1/3, 1/3]$ , and given  $p \in [-1/3, 1/3]$ , we denote by  $Z_{1:m} \sim U_m(p)$  a random process where we draw  $m$  i.i.d random variables  $Z_1, \ldots, Z_m$  where  $Z \in \{\pm 1\}$  and  $\mathbb{E}[Z] = p$ :
Lemma 1 (Fingerprinting Lemma (Kamath et al. [2019])) For every  $f: \{\pm 1\}^m \to [-\frac{1}{3}, \frac{1}{3}]$ , we have:
$$
\underset {P \sim U [ - 1 / 3, 1 / 3 ]} {\mathbb {E}} \underset {Z _ {1: m} \sim U _ {m} (P)} {\mathbb {E}} \left[ \frac {1 - 9 P ^ {2}}{9 - 9 P ^ {2}} \cdot (\hat {f} (Z _ {1: m}) - P) \cdot \sum_ {i = 1} ^ {m} (Z _ {i} - P) + (\hat {f} (Z _ {1: m}) - P) ^ {2} \right] \geq \frac {1}{2 7}
$$
The above theorem shows that if a random variable  $\hat{f}(Z_{1:m})$  uses the sample to non-trivially estimate the random variable  $P$ , then the output must correlate with the empirical mean. Notice that, in particular, it means that  $\hat{f}(Z_{1:m})$  is not independent of  $Z_{1:m}$  and certain information exists. Our next Lemma quantifies this statement and, as far the author knows, is novel. The proof is provided in the supplementary material
Lemma 2. Let  $X$  and  $Y$  be two random variables such that  $X$  is bounded by 1,  $\mathbb{E}(X) = 0$  and  $\mathbb{E}(Y^2) \leq 1$ . If  $\mathbb{E}[XY] = \beta$  then:
$$
\sqrt {I (X , Y)} \geq \frac {\beta^ {2}}{2 \sqrt {2}}.
$$
The proof of Lemma 2 is provided in the supplementary material
With Lemmas 1 and 2 at hand, the proof idea is quite straightforward. For every  $z \in \{-1/\sqrt{d}, 1/\sqrt{d}\}^d$  we define the following loss function:
$$
f (w, z) = \sum_ {t = 1} ^ {d} (w (t) - z (t)) ^ {2} = \| w - z \| ^ {2}. \tag {8}
$$
For every distribution  $D$ , one can show that the minimizer  $w^{\star} = \arg \min L_{D}(w)$  is provided by  $w = \mathbb{E}[z]$ . By a standard decomposition we can show:
$$
\begin{array}{l} L _ {D} \left(w _ {S}\right) - L _ {D} \left(w ^ {\star}\right) = \mathbb {E} \left[ \left\| w _ {S} + \left(w ^ {\star} - z\right) - w ^ {\star} \right\| ^ {2} - \left\| w ^ {\star} - z \right\| ^ {2} \right] \\ = \mathbb {E} \left[ \left\| w _ {S} - w ^ {\star} \right\| ^ {2} + 2 (w _ {S} - w ^ {\star}) \cdot (w ^ {\star} - z) + \left\| w ^ {\star} - z \right\| ^ {2} - \left\| w ^ {\star} - z \right\| ^ {2} \right] \\ = \mathbb {E} \left[ \left\| w _ {S} - w ^ {\star} \right\| ^ {2} \right] + 2 \mathbb {E} \left[ \left(w _ {S} - w ^ {\star}\right) \cdot (\mathbb {E} [ z ] - z) \right] \\ = \mathbb {E} \left[ \| w _ {S} - w ^ {\star} \| ^ {2} \right] \tag {9} \\ \end{array}
$$
Now, for simplicity of this overview we only consider the case that  $\Delta(w_S) = \Omega(1)$  is some non-trivial constant, for example we may assume that  $\Delta(w_S) < 1/54$  and let us show that for every coordinate,  $t$ , the mutual information between  $w_S(t)$  and  $\sum z_i(t)$  is order of  $\Omega(1/m^2)$ . Then by standard chain rule, and proper derivations the end result can be obtained.
Indeed, one can observe that if  $\Delta(w_S) < 1/54$ , and since  $w^{\star} = \mathbb{E}[z] = P$  we have that  $\mathbb{E}((\sqrt{d} w_S(t) - P(t))^2) \leq 1/54$ . In turn, we can use Lemma 1, and show that in expectation over  $P$  we also have:
$$
\mathbb {E} \left[ \left(\sqrt {d} w _ {S} (t) - P\right) \cdot \left(\sum_ {i = 1} ^ {m} \left(\sqrt {d} z _ {i} (t) - P\right)\right) \right] = \Omega (1).
$$
We now use Lemma 2, and convexity, to lower bound the individual sum of the mutual information,  $\sum_{i=1}^{m} I(w_s(t); z_i(t)|P)$ . By standard technique we conclude that there exists  $P$  for which the mutual information is bounded.
The above outline does not provide a bound that scales with the accuracy of  $\Delta(w_{S})$ , and we need to be more careful in our analysis for the full derivation. The detailed proof is provided in the supplementary material.
Acknowledgments The author would like to thank the anonymous reviewers that suggested to incorporate the lower bound for the individual mutual information term. This turned out to simplify some of the proofs. The author would like to thank Shay Moran, and the research was funded in part by the ERC grant (GENERALIZATION, 10139692), as well as an ISF Grant (2188 \ 20). Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or the European Research Council Executive Agency. Neither the European Union nor the granting authority can be held responsible for them. The author is a recipient of a Google research scholar award and would like to acknowledge his thanks.
D. Aldous. Random walks on finite groups and rapidly mixing markov chains. In Séminaire de Probabilités XVII 1981/82, pages 243-297. Springer, 1983.
N. Alon, S. Ben-David, N. Cesa-Bianchi, and D. Haussler. Scale-sensitive dimensions, uniform convergence, and learnability. Journal of the ACM (JACM), 44(4):615-631, 1997.
G. Aminian, Y. Bu, L. Toni, M. Rodrigues, and G. Wornell. An exact characterization of the generalization error for the gibbs algorithm. Advances in Neural Information Processing Systems, 34:8106-8118, 2021.
C. Zhang, S. Bengio, M. Hardt, B. Recht, and O. Vinyals. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM, 64(3):107-115, 2021.
R. Zhou, C. Tian, and T. Liu. Individually conditional individual mutual information bound on generalization error. IEEE Transactions on Information Theory, 68(5):3304-3316, 2022.