SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Roi Livni
Department Electrical Engineering
rlivni@tauex.tau.ac.il
We examine the relationship between the mutual information between the output model and the empirical sample and the generalization of the algorithm in the context of stochastic convex optimization. Despite increasing interest in information-theoretic generalization bounds, it is uncertain if these bounds can provide insight into the exceptional performance of various learning algorithms. Our study of stochastic convex optimization reveals that, for true risk minimization, dimension-dependent mutual information is necessary. This indicates that existing information-theoretic generalization bounds fall short in capturing the generalization capabilities of algorithms like SGD and regularized ERM, which have dimension-independent sample complexity.
One of the crucial challenges facing contemporary generalization theory is to understand and explain the behavior of overparameterized models. These models have a large number of parameters compared to the available training examples. But nonetheless, they tend to perform well on unseen test data. The significance of this issue has become more pronounced in recent years, as it has become evident that many state-of-the-art learning algorithms are highly overparameterized Neyshabur et al. [2014], Zhang et al. [2021]. The classical generalization bounds, which are well designed to describe the learning behavior of underparameterized models, seem to fail to explain these algorithms.
Understanding the success of overparameterized models seems challenging. Partly, due to the counter-intuitive nature of the process. Common wisdom suggests that, inorder to learn, one has to have certain good bias of the problem at hand, and that in learning we need to restrict ourselves to a class of models that cannot overfit the data. This intuition has been justified by classical learning models such as PAC learningValiant [1984] as well as regression Alon et al. [1997]. In these classical models, it can be even demonstrated Vapnik and Chervonenkis [2015], Blumer et al. [1989] that learning requires more examples than the capacity of the class of model to be learnt, and that avoiding interpolation is necessary for generalization. These results, though, are obtained in distribution-independent setups where one assumes worst-cast distributions over the data.
For this reason, researchers have been searching for new, refined models, as well as improved generalization bounds that incorporate distributional as well as algorithmic assumptions. A promising approach, in this direction, tries to connect the generalization performance to the amount of information the learner holds regarding the data Russo and Zou [2019], Xu and Raginsky [2017], Bassily et al. [2018]. For example, Xu and Raginsky [2017] demonstrated an upper bound on the generalization gap which informally states:
$$
\text {g e n e r a l i z a t i o n} w _ {S}) = O \left(\sqrt {\frac {I (w _ {S} , S)}{| S |}}\right) \tag {1}
$$
Namely, given an empirical sample  $S$ , and an output model  $w_{S}$ , the difference between its empirical error and its true error can be upper bounded by  $I(w_{S}, S)$ , the mutual information between these two random variables. Notice that Eq. (1) does not depend on any trait of the class of feasible models to be considered. In particular, it does not depend, apriori, on number of "free parameters", or a complexity measure such as VC dimension, the dimension of  $w_{S}$ , and not even on some prior distribution. However, it remains a question whether this method and technique can be useful in analysing state-of-the-art learning algorithms. While there has been a lot of work trying to establish the success of learning algorithms in various setups Neu et al. [2021], Xu and Raginsky [2017], Bu et al. [2020], Aminian et al. [2021], Pensia et al. [2018], many of the established bounds are opaque, and often there is no comprehensive end-to-end analysis that effectively illustrates how generalization is to be bounded by Eq. (1) and simultaneously obtain good empirical performance. In fact, there is also evidence Carlini et al. [2021], Feldman [2020] that memorizing data is required, in some regimes, for effective learning. Towards better understanding, we will focus, in this work, on the setup of Stochastic Convex Optimization Shalev-Shwartz et al. [2009] (SCO), and provide accompanying lower bounds to Eq. (1) that will describe how much mutual information is necessary for learning.
SCO as a case study for overparametrization: SCO is a very clean and simple setup where a learner observes noisy instances of (Lipschitz) convex functions, defined in  $\mathbb{R}^d$ , and is required to minimize their expectation. On the one hand, it provides simple, amenable to rigorous analysis, definitions of learnability and learning. On the other hand, this model is the cradle of prototypical algorithms such as Gradient Descent (GD) and Stochastic Gradient Descent (SGD), as well as accelerated methods, which are the workhorse behind state-of-the-art optimization methods.
Moreover, SCO is an ideal model for understanding overparameterization. It is known Feldman [2016] that in this setup,  $\Omega(d)$  examples are needed in order to avoid overfitting. In fact, even concrete algorithms such as GD and regularized-GD may overfit unless they observe dimension-dependent sample size Amir et al. [2021a,b]. In other words, the capacity of the model and its ability to overfit does indeed scale with the dimension. Nevertheless, it is also known that some algorithms do learn with far fewer examples. For example SGDHazan et al. [2016], Regularized-ERMShalev-Shwartz et al. [2009], Bousquet and Elisseeff [2002] and a stable variant of GD Bassily et al. [2020] all learn with  $O(1/\varepsilon^2)$  examples, a dimension independent magnitude. To put it differently, learning in SCO is not just a question of finding the empirical risk minimizer, but also a question of how – what algorithm was used, and learning is not demonstrated by naive uniform convergence bounds that scale with the number of parameters in the model.
Therefore, SCO is a natural candidate to study how information theoretic bounds play a role in learning. We might even hope that these bounds shed light on why some algorithms succeed to learn while others fail. Existing algorithms don't avoid memorizing the data, but it is unclear if holding information on the data is necessary. So we start here with the simplest question:
What is the smallest amount of mutual information required for learning in SCO?
Our main result shows that, in contrast with the dimension-independent learnability results in this setup, the information between the model and the sample has to be dimension-dependent. As such, the complexity of the class appears implicitly in Eq. (1). As a result, carrying  $\Omega(d)$  bits of information over the sample is necessary for learning at optimal rates, and Eq. (1) doesn't yield the optimal generalization performance of algorithms such as Regularized ERM, SGD and stable-GD.
Information-theoretic generalization bounds have a long history of study in ML theory McAllester [1998, 1999], Langford and Shawe-Taylor [2002]. Generalization bounds that directly relate to the information between output and input of the learner initiated in the works of Xu and Raginsky [2017], Bassily et al. [2018], Russo and Zou [2019], Bassily et al. [2018] demonstrated limitations for such generalization bounds, for proper ERM learners, and Livni and Moran [2020] showed that any learner (proper or not), that learns the class of thresholds must leak unbounded amount of information. In
this work we focus on stochastic optimization and on learning Lipschitz functions. In the setup of SCO the above is not true, and one can construct learners that leak  $\tilde{O}(d)$  bits of information (see Proposition 1). But we would like to know whether information-theoretic bounds behave like the uniform convergence bounds (dimension dependent) or capture the minmax learning rates (dimension independent).
Several lines of works applied the analysis of Xu and Raginsky [2017] to provide algorithmic-dependent analysis in the context of stochastic optimization. Pensia et al. [2018] and followup improvements Rodríguez-Gálvez et al. [2021], Negrea et al. [2019], Haghifam et al. [2020] provided information-theoretic generalization bounds for Stochastic Gradient Langevine Dynamics (SGLD) and Neu et al. [2021] extends the idea to analyze vanilla SGD. Aminian et al. [2021] also provides full characterization of the closely related, Gibbs Algorithm and shows that the information can inverseley scale with the sample bounds. The bounds in these works are implicitly dimension independent which may seem contradictory to the result established here. Importantly, the above bounds may depend on hyper-parameters such as noise in SGLD and temperature in Gibbs algorithm, and these hyperparameters may affect the optimization performance of the algorithms. The bounds we obtain are applicable only to algorithms with non-trivial true risk, which depends on hyperparameter choice, and as such there is no real contradiction. Taken together, the noise, for example, in SGLD needs to scale with the dimension in order to obtain non-trivial information-theoretic bounds, but that will lead to a large empirical error. Similarly, if the temperature in the Gibbs algorithm doesn't scale with the dimension, one can potentially achieve small empirical error but at the expense of high information.
Most similar to our work, recently, Haghifam et al. [2022] provided the first set of limitations to information theoretic generalization bounds. They focus on the Gradient Descent method and perturbed variants of it, and provide limitations to both MI bounds as well as conditional mutual information (CMI) bounds Steinke and Ullman [2015] and their individual sample variants Bu et al. [2020], Negrea et al. [2019], Haghifam et al. [2020], Zhou et al. [2022]. In contrast, we focus on the mutual information bound (as well as its individual sample version of Bu et al. [2020]), but we provide bounds that are irrespective of the algorithm.
The key idea behind our lower bound proof builds on privacy attacks developed in the differential privacy literature Bun et al. [2014], Kamath et al. [2019], Steinke and Ullman [2015]. In the context of classification, lower and upper bounds techniques Alon et al. [1997], Bun et al. [2020] were successfully imported to obtain analogous information-theoretic bounds Livni and Moran [2020], Pradeep et al. [2022]. In optimization, though, bounds behave slightly different and therefore, bounds from classification cannot be directly imported to the context of optimization.
2 Setup and Main Results
We begin by describing the classical setup of Stochastic Convex Optimization (SCO), following Shalev-Shwartz et al. [2009]. In this model, we assume a domain  $\mathcal{Z}$ , a parameter space  $\mathcal{W} \subseteq \mathbb{R}^d$  and a function  $f(w,z)$ , termed loss function. The function  $f$  satisfies that for every  $z_0 \in \mathcal{Z}$ , the function  $f(w,z_0)$  as a function over the parameter  $w$  is convex and  $L$ -Lipschitz.
For concreteness, we treat  $L$  as a constant,  $L = O(1)$ , and we concentrate on the case where  $\mathcal{W}$  is the unit ball. Namely:
$$
\mathcal {W} = \{w: \| w \| \leq 1 \}.
$$
As we mostly care about lower bounds, these won't affect the generality of our results. Given a distribution  $D$ , the expected loss of a parameter  $w$  is given by
$$
L_{D}(w) = \underset {z\sim D}{\mathbb{E}}[f(w,z)].
$$
The excess true risk of  $w$ , with respect to distribution  $D$ , is denoted as:
$$
\Delta_ {D} (w) = L _ {D} (w) - L _ {D} (w ^ {\star}), \quad \text {w h e r e}, L _ {D} (w ^ {\star}) := \min  _ {w \in \mathcal {W} _ {z \sim D}} \mathbb {E} [ f (w, z) ].
$$
We also denote the excess empirical risk, given sample  $S = \{z_{1},\dots ,z_{m}\}$ :
$$
\Delta_ {S} (w) = \frac {1}{m} \sum_ {i = 1} ^ {m} f (w, z _ {i}) - \min  _ {w \in \mathcal {W}} \frac {1}{m} \sum_ {i = 1} ^ {m} f (w, z _ {i}).
$$
Leranbility We will focus here on the setup of learning in expectation. In particular, a learning algorithm  $A$  is defined as an algorithm that receives a sample  $S = (z_{1},\dots ,z_{m})$  and outputs a parameter  $w_{S}^{A}$ . We will normally suppress the dependence of the parameter  $w_{S}^{A}$  in  $A$  and simply write  $w_{S}$ . The algorithm  $A$  is said to learn with sample complexity  $m(\varepsilon)$  if it has the following property: For every  $\varepsilon >0$ , if  $S$  is a sample drawn i.i.d from some unknown distribution  $D$ , and  $|S|\geq m(\varepsilon)$  then:
$$
\underset {S \sim D ^ {m}} {\mathbb {E}} \left[ \Delta_ {D} (w _ {S}) \right] \leq \varepsilon .
$$
A closely related setup requires that the learner succeeds with high probability. Standard tools such as Markov's inequality and boosting the confidence Schapire [1990] demonstrate that the two definitions are essentially equivalent in our model.
Information Theory We next overview basic concepts in information theory as well as known generalization bounds that are obtained via such information-theoretic quantities. We will consider here the case of discrete random variables. We elaborate more on this at the end of this section, and how our results extend to algorithms with continuous output. Therefore, throughout, we assume a discrete space of possible outcomes  $\Omega$  as well as a distribution  $\mathbb{P}$  over  $\Omega$ . Recall that a random variable  $X$  that takes values in  $\mathcal{X}$  is said to be distributed according to  $P$  if  $\mathbb{P}(x = X) = P(x)$  for every  $x \in \mathcal{X}$ . Similarly two random variables  $X$  and  $Y$  that take values in  $\mathcal{X}$  and  $\mathcal{Y}$  respectively have joint distribution  $P$  if
$$
\mathbb {P} (x = X, y = Y) = P (x, y).
$$
For a given  $y \in \mathcal{Y}$ , the conditional distribution  $P_{X|y}$  is defined to be  $P(x|y = Y) = \frac{P(x,y)}{\sum_x P(x,y)}$ , and the marginal distribution  $P_X: \mathcal{X} \to [0,1]$  is given by  $P_X(x) = \sum_y P(x,y)$ . If  $P_1$  and  $P_2$  are two distributions defined on a discrete set  $\mathcal{X}$  then the KL divergence is defined to be:
$$
D _ {K L} \left(P _ {1} \| P _ {2}\right) = \sum_ {x \in \mathcal {X}} P _ {1} (x) \log \frac {P _ {1} (x)}{P _ {2} (x)}.
$$
Given a joint distribution  $P$  that takes values in  $\mathcal{X} \times \mathcal{Y}$  the mutual information between random variable  $X$  and  $Y$  is given by
$$
I (X; Y) = \underset {Y} {\mathbb {E}} \left[ D _ {K L} \left(P _ {X | Y} \right\| P _ {X}\right).
$$
We now provide an exact statement of Eq. (1)
Theorem (Xu and Raginsky [2017]). Suppose  $f(w, z)$  is a bounded by 1 loss function. And let  $A$  be an algorithm that given a sample  $S = \{z_1, \dots, z_m\}$  drawn i.i.d from a distribution  $D$  outputs  $w_S$ . Then
$$
\underset {S} {\mathbb {E}} \left[ L _ {D} \left(w _ {S}\right) - \frac {1}{m} \sum_ {i = 1} ^ {m} f \left(w, z _ {i}\right) \right] \leq \sqrt {\frac {2 I \left(w _ {S} , S\right)}{m}}. \tag {2}
$$
Remark on Continuous Algorithms As stated, we focus here on the case of algorithms whose output is discrete, and we also assume that the sample is drawn from a discrete set. Regarding the sample, since we care about lower bounds and our constructions assume a discrete set, there is no loss of generality here. Regarding the algorithm's output, in the setup of SCO there is also no loss of generality in assuming the output is discrete. Indeed, we can show that if there exists a continuous algorithm with sample complexity  $m_0(\varepsilon)$ , and bounded mutual information over the sample, then there exists also a discrete algorithm with sample complexity  $m(\varepsilon) = m_0(O(\varepsilon))$  with even less mutual information.
To see that notice that, since we care about minimizing a Lipschitz loss function, given any accuracy  $\varepsilon$ , we can take any finite  $\varepsilon$ -approximation subset of the unit ball and simply project our output to this set. Namely, given output  $w_{S}$ , where  $S > m(\varepsilon)$ , we output  $\bar{w}_{S}$ , the nearest neighbour in the  $\varepsilon$ -approximation sub set. Because  $L_{D}$  is  $O(1)$  Lipschitz, we have that
$$
\left| L _ {D} \left(\bar {w} _ {S} ^ {A}\right) - L _ {D} \left(w ^ {\star}\right) \right| \leq \left| L _ {D} \left(\bar {w} _ {S} ^ {A}\right) - L _ {D} \left(w _ {S} ^ {A}\right) \right| + \left| L _ {D} \left(w _ {S} ^ {A}\right) - L _ {D} \left(w ^ {\star}\right) \right| \leq O (\varepsilon),
$$
hence up to a constant factor the algorithm has the same learning guarantees. On the other hand, due to data processing inequality:
$$
I (\bar {w} _ {S} ^ {A}, S) \leq I (w _ {S} ^ {A}, S).
$$
2.1 Main Result
Eq. (2) provides a bound over the difference between the expected loss of the output parameter and the empirical loss. Without further constraints, it is not hard to construct an algorithm that carries little information on the sample. But, to obtain a bound over the excess risk of the parameter  $w_{S}$ , one is required not only to obtain a generalization error gap but also to non-trivially bound the empirical risk. The following result shows that requiring both has its limits:
Theorem 1. For every  $0 < \varepsilon < 1/54$  and algorithm  $A$ , with sample complexity  $m(\varepsilon)$ , there exists a distribution  $D$  over a space  $\mathcal{Z}$ , and loss function  $f$ , 1-Lipschitz and convex in  $w$ , such that, if  $|S| \geq m(\varepsilon)$  then:
$$
I \left(w _ {S}; S\right) \geq \sum_ {i = 1} ^ {m} I \left(w _ {S}, z _ {i}\right) = \tilde {\Omega} \left(\frac {d}{\varepsilon^ {5} \cdot m ^ {6} (\varepsilon)}\right). \tag {3}
$$
Theorem 1 accompanies the upper bound provided in Eq. (2) and shows that, while the generalization gap is bounded by the mutual information, the mutual information inversely scales with the optimality of the true risk. Taken together, for any algorithm with non-trivial learning guarantees, there is at least one scenario where it must carry a dimension-dependent amount of information on the sample or require a large sample. Indeed, either  $m(\varepsilon) = \Omega(\sqrt[d]{d / \varepsilon^5})$ , (and then, trivially, the sample complexity of the algorithm scales with the dimension) or, via Eq. (3), we obtain dimension dependent mutual information, and in turn, Eq. (2) is non-vacuous only if the sample is larger than the dimension.
The first inequality is standard and follows from standard chain rule argument (see e.g. [Bu et al., 2020, proposition 2]). The second inequality lower bounds the information with the individual samples. Recently, Bu et al. [2020] obtained a refined bound that improves over Xu and Raginsky [2017] by bounding the generalization with the information between the individual samples and output. Theorem 1, together with subadditivity of the square function, shows that the individual sample bound of Bu et al. [2020] can also become dimension dependent.
3 Discussion
Our main result shows a necessary condition on the mutual information between the model and empirical sample. Theorem 1 shows that for any algorithm with non-trivial learning guarantees, there is at least one scenario where it must carry a dimension-dependent amount of information on the sample or require a large sample. A natural question, then, is whether natural structural assumptions may circumvent the lower bound and allow to still maintain meaningful information theoretic bounds in slightly different setups. To initiate a discussion on this we begin by looking deeper into the concrete construction at hand in Theorem 1. We notice (see Section 4 and the supplementary material) that the construction we provided for Theorem 1 relies on a distribution  $D$  that is always supported on functions of the form:
$$
f (w, z) = \| w - z \| ^ {2} = \| w \| ^ {2} - 2 w \cdot z + 1, \quad z \in \{- 1 / \sqrt {d}, 1 / \sqrt {d} \} ^ {d}. \tag {4}
$$
The constant 1 has no effect over the optimization, nor on the optimal solution, therefore we can treat  $f$  as equivalent to the following function
$$
f \equiv \| w \| ^ {2} - 2 w \cdot z.
$$
The distribution over the element  $z$  is also quite straightforward and involves only bias sampling of the coordinates. The function  $f$ , then, is arguably the simplest non-linear convex function that one can think of and as we further discuss it holds most if not all of the niceties a function can hold that allow fast optimization – it is strongly convex, smooth, and in fact enjoys generalization bounds that can even be derived using standard uniform convergence tools. Indeed for any choice  $w_{S}$
$$
\begin{array}{l} \underset {S \sim D ^ {m}} {\mathbb {E}} L _ {D} (w _ {S}) - \frac {1}{m} \sum_ {i = 1} ^ {m} f (w _ {S}, z _ {i}) = 2 \underset {S \sim D ^ {m}} {\mathbb {E}} \sup  _ {\| w \| \leq 1} [ \frac {1}{m} \sum_ {i = 1} ^ {m} w \cdot z _ {i} - \underset {z \sim D} {\mathbb {E}} [ w \cdot z ] ] \\ \leq O (1 / \sqrt {m}) \\ \end{array}
$$
Where the last inequality follows from a standard Rademacher bound over the complexity of linear classifiers (see Shalev-Shwartz and Ben-David [2014]). In other words, while under further structural
assumptions one might hope to obtain meaningful information theoretic bounds, it should be noted that such structural assumptions must exclude a highly simplistic class of functions that are in fact extremely easy to learn and even enjoy dimension independent uniform convergence bounds.
We now discuss further the niceties of this function class and the implications to refined algorithmic/distributional-dependent generalization bound via the information-theoretic bounds. We begin by analyzing algorithms that achieve the minimax rate.
Algorithmic-dependent generalization bounds As discussed, it is known that SGD Hazan et al. [2016], regularized-ERM Shalev-Shwartz et al. [2009], Bousquet and Elisseeff [2002], as well as stabilized versions of Gradient Descent Bassily et al. [2020] have the following minmax rate for learning  $O(1)$ -Lipschitz convex functions over an  $O(1)$ -bounded domain:
$$
\mathbb {E} \left[ \Delta_ {D} \left(w _ {S}\right) \right] = O \left(1 / \sqrt {m}\right).
$$
Plugging the above in Eq. (3) we obtain that any such algorithm must carry at least  $\Omega(d / m^{5/2})$  bits of information. which entails an information theoretic bound in Eq. (2) of  $O(\sqrt{d} / m^{7/4})$ . This exceeds the true excess risk of such algorithms when  $d \gg m$  and becomes a vacuous bound when we don't assume the sample scales with the dimension (even though the algorithm perfectly learns in this setup). Now, we do not need Theorem 1 to obtain generalization gaps over these concrete algorithms, as a more direct approach would do. But, one might hope that by analyzing noisy versions of these algorithms, or some other forms of information-regularization, we could obtain some insight on the generalization of these algorithms. But, our lower bound applies to any algorithm with meaningful information-theoretic generalization bound. In particular, adding noise, for example, either makes the algorithm diverge or the noise is too small to delete enough information.
Distributional-dependent generalization bounds Next, we would like to discuss the implications to distributional assumptions. The above discussion shows that any trait of an algorithm that makes it optimal is not captured by the amount of information it holds on the data (in the setup of SCO). Distinct from SCO, in practice, many of the underlying problems can be cast into binary classification where it is known Blumer et al. [1989] that without further distributional assumptions learnability cannot be separated from uniform convergence as in SCO.
An interesting question, then, is if information theoretic generalization bounds can be used to obtain distribution-dependent bounds.
The function  $f(w,z)$  in Eq. (4) is known to be strongly-convex for every  $z$ . Recall that a function  $f$  is called 1-strongly convex if  $f - \frac{1}{2}\| w\|^2$  is convex. It is known Shalev-Shwartz et al. [2009] that any ERM over a strongly convex will achieve suboptimality:
$$
\mathbb {E} \left[ \Delta_ {D} \left(w _ {S}\right) \right] = O (1 / m).
$$