Information bottleneck theory of high-dimensional regression: relevancy, efficiency and optimality
Authors: Vudtiwat Ngampruetikorn
David J. Schwab
Venue: NeurIPS 2022 Accept (2022)
Keywords: Information bottleneck
Information theory
Information efficiency
Optimal representation

Abstract: Avoiding overfitting is a central challenge in machine learning, yet many large neural networks readily achieve zero training loss. This puzzling contradiction necessitates new approaches to the study of overfitting. Here we quantify overfitting via residual information, defined as the bits in fitted models that encode noise in training data. Information efficient learning algorithms minimize residual information while maximizing the relevant bits, which are predictive of the unknown generative models. We solve this optimization to obtain the information content of optimal algorithms for a linear regression problem and compare it to that of randomized ridge regression. Our results demonstrate the fundamental trade-off between residual and relevant information and characterize the relative information efficiency of randomized regression with respect to optimal algorithms. Finally, using results from random matrix theory, we reveal the information complexity of learning a linear map in high dimensions and unveil information-theoretic analogs of double and multiple descent phenomena.

Information bottleneck theory of high-dimensionall regression: relevancy, efficiency and optimality
Vudtiwat Ngampruetikorn, David J. Schwab
Initiative for the Theoretical Sciences, The Graduate Center, CUNY
ingampruetikonngg... cundy.ec..cm.eit
Abstract
Avoiding overfitting is a central challenge in machine learning, yet many large neural networks readily achieve zero training loss. This puzzling contradiction necessitates new approaches to the study of overfitting Here we quantify overfitting via residual information, defined as the bits in fitted models that encode noise in training data. Information efficient learning algorithms minimize residual infor- mation while maximizing the relevant bits, which are predictive of the unknown generative models. We solve this optimization to obtain the information content of optimal algorithms for linear regression problem and compare it to that of randomized ridge regression. Our results demonstrate the fundamental trade-off between residual and relevant information and characterize the relative information efficiency of randomized regression with respect to optimal algorithms. Finally, using results from random matrix theory, we reveal the information complexity of learning a linear map in high dimensions and unveil enformation-theoree analogs double and multiple descent phenomena.
Information bottleneck
Conventional wisdom identifies overfitting as being detrimental to generalization performance, yet modern machine learning is dominated by models that perfectly fit training data. Recent attempts to resolve this dilemma have offered much needed insight into the generalization properties of perfectly fitted models [1, 2]. However investigations of overfitting beyond generalization error have received less attention. In this work we present quantitative analysis overfitting based on information theory and, in particular, the information bottleneck (IB) method [3].
The essence of learning the ability to find useful and generalizable representations of training data. An example of such a representation is a fitted model which may capture statistical correlations be- tween two variables (regression and pattern recognition) or the relative likelihood of random variables (density estimation). While what makes a representation useful is problem specific, a good model generalizes well that is, it consistent with test data even though they are not used training.
Achieving good generalization requires information about the unknown data generating process. Maximizing this information is intuitive strategy, yet extracting too many bits from the training data hurts generalization [4, 5J. This fundamental trade-off underpins the IB principle, which formalizes the notion of maximally efficient representation as an optimization problem [3]]
min Qrs I(; W) (y I)I(T; W).
(1)
Here W denotes the data generating process. The conditional distribution @ris denotes a learning algorithm which defines a stochastic mapping from the training data S to the hypothesis or fitted
Note that this minimization is identical to that of the original IB method since /(S:TIW)= under Markov constraint S W.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

model T (Fig la). The relevant information, I(T; W), is the bits in T thht are informative of the generative model W. On the other hand, the residual information, I[; W), is the bits in 7 that are specific to each realization of the training data S and thus are not informative of W. In other words the residual bits measure the degree of overfitting. The parameter y controls the trade-off between these two informations.
The IB method has found success in a diverse array of applications, from neural coding [6. 7], developmental biology [8] and statistical physics [9-11] to clustering [12], deep learning [13-15] and reinforcement learning [16].
Indeed the IB principle has emerged as a potential candidate for a unifying framework for understand- ing learning phenomena [15, 1 17-19] and number of recent works have explored deep connections between information-thherriicc quantities and generalization properties [4, 5, 20-28j. However a direct application of information theory practical learning algorithms is often limited by the difficulty in estimating information, especially in high dimensions. While recent advances in characterizing variational bounds of mutual information have enabled a great deal of scalable, information-thes theory inspired learning methods [13, 29, 30J, these bounds are generally loose and may not reflect the true behaviors of information.
To this end we consider a tractable problem of learning a linear map. We show that the level of overfitting, measured by the encoded residual bits, is nonmonotonic in sample size, exhibiting maximum near the crossover between under- and overparametrized regimes. We also demonstrate that additional maxima can develop under anisotropic covariates. As the residual information bounds analog
the generalization gap [4, 5], its nonmonotonicity can be viewed as an information-thhoreee of (sample- wise) multiple descent- the existence of disjoint regions in which more data hurt general- ization (see, e.g., Refs [31, 32)). Using an IB optimal representation as a baseline, we show that the information efficiency of a randomized least squares regression estimator exhibits sample-wise non- monotonicity with a minimum near the residual information peak. Finally we discuss how redundant coding of relevant information in the data gives rise to the nonmonotonicity of the encoded residual bits and how additional maxima emerge from covariate anisotropy (Sec 4).
1.1 Generative model
Linear map We consider training data of iid samples, S ((x1, (KN.NN)) each of which is a pair of P dimensional input vector E and scalar response ER fl. We assume linear relation between the input and response.
= W x; + and ~ --)(
(2)
where W denotes the unknown linear map and E; a scalar Gaussian noise with mean zero and variance 2 In other words the responses and the inputs are related via Gaussian channel
Y X. W N(X NoxT,ok W,
(3)
where we define Yy TERN and Xx) EWR
Fixed design-We adopt the fixed design setting in which the inputs (design matrix) X are de- terministic and only the responses Y are random variables (see, e.g., Ref [33, Ch 3)). As re- sult, the mutual information between the training data S and any random variable A is given by I(A::)=((A::.))- -AAA)) In the following analyses, we use S and interchangeably.
Random effects In addition we work in the random effects setting (sec, c.g., [34, 35] for recent applications of this setting) in which the true regression parameter W is Gaussian vector,
W ~ N(O, 4 Ir).
(4)
Here we define the covariance such that the signal strength, B = independent of P.
1.2
Information optimal algorithm
The data generating process above results in training data Y and true parameters W that are Gaussian correlated (under the fixed design setting). In this case the IB ciiimiiationmiiimiiii residual in-
formation YY; W) while maximizing relevant information I(T; 11)-admit exact solution [36].

---

characterized by the eigenmodes of the normalized regression matrix,
x Ey = IN + N
with 1== o² N wf
(5)
where 1" denotes the scaled noise-to-signal ratio. The relevant and residual informations of an optimal representation 7 read [36]
I(;:W)= = Σ max(O, In((1 - y()l--))//v))
(6)
I(;:Y Y [W) = =
1 ma In(y(1 v;))),
where v; denote the eigenvalues of Ey and y parametrizes the IB trade-off [see Eq (l)). In our setting convenient recast the summations above integrals (see Appendix A for derivation),
[(F:W)= = P - dF (v) In
d the
We + 1"1
P f dF"(w) In(v/ve) I(T:W), I W) = 2 J02C
where V== X /N and 1e denote the empirical covariance and its cumulative spectral distribution, respectively." In addition we introduce the parameter d. =/'///(> /(y 1) which controls the number and the weights of eigenmodes used in constructing the optimal representation f. In the limit 4. -* the residual information diverges logarithmically (Fig Id) and the relevant information converges to the available relevant information in the data (Fig lc),
I(T: W)
We-
I(Y:)) & Saso dF (0) In 1 + 1/1)).
(10)
Increasing &e from zero decreases both residual and relevant informations, tracing out the optimal frontier until the lower spectral cutoff Ve reaches the upper spectral edge at which both informations vanish (Fig lb) and beyond which no informative solution exists [36-38].
1.3 Information efficiency
The exact characterization of the IB frontier provides a useful benchmark for iiformatioonthhorete analyses of learning algorithms, not least because it allows a precise definition of information efficiency. That is, we can now ask how many more residual bits a given algorithm needs to encode, compared to the IB optimal algorithm, in order to achieve the same level of relevant information. Here we define the information efficiency Uu as the ratio between the residual bits encoded in the outputs of the IB optimal algorithm and the algorithm of interest- and T, respectively- some fixed relevance level μ, i.e.,
I(T; Y W)
I(T:W) I(T; W) م 1(Y:W)
7p
subject
(11)
1(T: Y W)
I(Y: W)
Since the optimal representation minimizes residual bits at fixed μ (Fig la), the information efficiency ranges from zero to one, 0s < In addition we have 0< 1, resulting from the data processing inequality I(T; W) = (YY; W) for the Markov constraint T +- Y W (see, e.g., Ref [39]).
2 Gibbs-posterior least squares regression
We consider one of the best-known learning algorithms: least squares linear regression. Not only this algorithm widely used practice, it has also proved particularly well-suited setting for analyzing learning in the overparametrized regime [40-47]. Indeed it exhibits some of the most intriguing features of overparametrized learning, including benign overfitting and double descent
Thh arguments of the logarithms in Eqs (6-7) are always nonnegative since the data processing inequality means that the IB problem well-defined only for y> 1. and the eigenvalues of normalized regression matrix always range from zero one [36].
Note that the eigenvalues of X and x x are identical except for the number of zero modes.

---

b
l:w IN
Generative models
Training data
1.0
0.3
Priw
0.2
W
Y
ty I.P
@T
0.0
-NIP
Learning algorithm
0.5
frrw/NN
NjP
Fitted models (data representation)
0.01
N N/P
0.01
100
Information optimal algorithm
0.0
f arg min@rir Y W) s.t. I(T:W)
0.5
0.01
0.1
T(YW)
Fr IN
Figure 1: Information optimal algorithm A learning algorithm Qr mapping from training data Y to fitted models T. Information optimal algorithms minimize residual bits I(T:Y Y W are uninformative of the unknown generative model W-an fixed relevance level μt, defined as the ratio between the encoded and available relevant bits, / ; W) and I(Y; W). b-d The information content of optimal algorithms for learning linear map (Sec 1.1) various measurement densities N/P (see color bar). Optimal algorithms cannot increase the relevance level without encoding more residual bits. Increasing N/P reduces the residual bits per sample but only when s P. This results from the change in sample size dependence of relevant bits in the data from linear to logarithmic around N = (inset). That is, available relevant bits in each sample become redundant around N x P and increasingly as N increases further. Learning algorithms use this redundancy to better distinguish signals from noise, thereby requiring fewer residual bits per sample. e-d The IB fronticrs in (b) are parametrized by spectral cutoff dh [see Eqs (8-9)]. Here set w² //² =1 and let P, N oo the same rate such that the ratio N/ /P remains fixed and finite. The empirical spectral distribution For follows the standard Marchenko-Pare Pastur law (see Sec 4).
which describe the surprisingly good generalization performance of overparametrized models and its nonmonotonic dependence on model complexity and sample size [31, 42, 48).
Inferring model from data generally requires assumption the hypothesis space, as well as learning algorithm, which outputs a hypothesis according to some criteria that rank how well each hypothesis explains the data. Linear regression restricts the model class to a linear map, parametrized by T TER", between an input x, and predicted response ft.
class of models, which defines
$1 = T Xj. Xi.
(12)
Minimizing the mean squared error 1 Σ (f: 2 yields a closed form solution for the estimated regressor, -Xx" However, this requires X to be invertible and thus does not work in the overparametrized regime which the sample covariance is not full rank and infinitely many models have vanishing mean squared error.
There are several approaches to break this degeneracy but perhaps the simplest and most studied is the ridge regularization which adds to the mean squared error the preference for model parameters with small 42 norm, resulting the regularized loss function
L(T.X.1) 1 -= x-T11; + +|
(13)
where d > O controls the regularization strength. Minimizing this loss function leads to unique solution = 2NIp) xY even when N P.
Gibbs posterior-Waiteee ridge regression works in the overparametrized regime, it is deterministic algorithm which does not readily lend itself to information-thooon theoretic analyses because the mutual information between two deterministically related continuous random variables diverges. Instead consider the Gibbs posterior (or Gibbs algorithm) which becomes a Gaussian channel when defined with the ridge regularized loss in Eq (13),
QriX.Y e -L(TYX ~ T N प.P XY, 1 28 4+2F
(14)
Here B denotes the inverse temperature. In the zero temperature limit B oo this algorithm returns the usual ridge regression estimate Tí (the mean of the above normal distribution) with probability

---

approaching one. Whilst randomized ridge regression needs not take the form above, Gibbs posteriors are attractive, not least because they naturally emerge, for example, from crommation-regulaiizee risk minimization [5] (see also Ref [27] for recent discussion).
Markov constraint-Thhe generative model Pylw, true parameter distribution Pw and learning algorithm QrIV [Eqs (3-4) & (14)] completely describe the relationship between all random variables of interest through the Markov factorization of their joint distribution (Fig la),
Pr.YWw Pw & Priw @ QTIY.
(15)
Note that Py (x.w and Qry- Qrixx in the fixed design setting (see Sec 1.1).
Information content of Gibbs regression
We now turn to the relevant and residual informations of the models that result from the Gibbs regression algorithm [Eq (14)]. Since all distributions appearing on the rhs of Eq (15) are Gaussian, the relevant and residual informations are given by
I W) = det M, 27/ and I(T:Y W) = In det Ir T
(16)
Here we use the fact that Σr = Σnyr due the Markov constraint [Eqq (15)J. The covariance Σ is defined by the learning algorithm in Eg (14). To obtain I and Σr, we marginalize out Y and W in order from Pr.y.w [Eqs (3-4) & (14-15)] and obtain
TIW W- N Tedlp
4
W,
2B Todrr + N (T+Mlp))
(17)
T N (0,
F+Hl + o N (4+M))) + (W+M))))
(18)
Substituting the covariance matrices above into Eq (16) yields (see Appendix B for derivation)
P f
22
I(T:W) 2 6>0
aF
(v) In
1+
(19)
w +
+ +1) Bo
(v
2Bo² v
I W)
d (v) In
1
(20)
N 1 V+1
6>0
The integration domains are restricted to positive real numbers since the eigenvalues of a covariance matrix are non-negative and the integrands vanish at d/ =0.
In the zero temperature limit 00, the residual information diverges (as expected from a determin- istic algorithm [23, 24]: see also Fig 2c) whereas the relevant information approaches the mutual information between the data Y and the true parameter W,
I(T: W) 8-300
I(::)) = _ Suzo dF (1)) In(1 + +d22)..
(21)
Relevant and residual informations decrease with B until they vanish as B 0+ at which Gibbs posteriors become completely random (Fig 2a-c).
3.1 Zero temperature limit
At first sight appears that our analyses are not applicable in the high-information limit since the residual information diverges for both the optimal algorithm and Gibbs regression (see Figs 1d & 2c). However the rates of divergence differ. Here we use this difference to characterize the efficiency of Gibbs regression in the zero temperature limit co.
We first analyze the limiting behaviors of relevant information. From Eq (10), we see that the relevant information ratio of the IB solution approaches at the =0. Perturbing the in Eq (8) away from zero results in a linear correction to the relevant information,
Ve P
I
I(T: W) I(Y:
+F)
(22)
2' 2 420
5

---

lT:w (lwx
b
fr:w /N
d
7
IrrwiPP /P
0.3
1.0
1.0
0.2
100
0.8
/
0.0
0.9
0.01
0.5
1
IT:YIW/N IN
0.7
0.6
0.5
0.01
0.3
1= 10-
IB frontier
0.4
0.1
10-1
IB optimal
0.0
0
0.5
0.01 100
0.01
100
0.01
100
I w /N
N ///B2
N/P
N/P
Figure 2: Gibbs regression a-c The information content of Gibbs regression [Eq (14)) at N/P=1 and various regularization strengths A (see color bar). The information curves of Gibbs regression are bounded by the IB frontier (dotted curve). b-c The inverse temperature / controls the stochasticity of Gibbs posteriors [Eqs (19-20)]. Both relevant and residual bits decrease with temperature and vanish in the limit B - O where Gibbs posteriors become completely random. d-e Information efficiency [Eq (11)] and residual information of Gibbs regression with 1= 10 vy measurement ratio at various relevance levels μ (see legend). d Gibbs regression approaches optimality in the limits, N P and N & P. and becomes least efficient at N/P=1. Residual bits of Gibbs regression and optimal algorithm (dotted) grow linearly with N when N s P. This growth is similar to that of the available relevant bits (Fig lb inset). But while the available relevant bits always increase with N, the residual bits decrease as N exceeds P. Here we set w² o²//=²= and let P, N - oo at the same rate such that the ratio N/P remains fixed and finite. The eigenvalues of the sample covariance follow the standard Marchenko-Pasturr law (see Sec 4).
Keeping only the leading correction and recalling that μ ==/(f; ([/(())/(( [Eq (11)), we obtain
lim Ve Swzo d (1b In(1 + 111))
(1 p).
(23)
n- l'
SozodF dFF(w))
Similarly expanding the relevant information Gibbs regression [Eq (19)] around / 00 yields f( (F
+ + +0
(T:W) ((Y:W) 2B #>0
(24)
v +
As result, the correspondence between the low low-eemperature and high informationn limits reads N Sozo drx (w) In(1 + +11²")
btd (1 4)-
(25)
2B Sazo dF (1)
V+F
We turn to the residual bits. Expanding Eq (9) around v. =0 and Eq (20) around 8=1= leads to
I(;;Y () = = ( "F"(w)
In Ve P
v vs x + +((c)
dF
In
(26)
#>0
x
2
w=O
1(T;Y W) = = f d (v) In
N
[ dF In
P
d
v v+1 + 1
(27)
2B82
2 420
#>0
From Eqs (23) (25), see that the residual informations above have the same logarithmic sin-
gularity, In(1 - 44), at == 1. Therefore their difference remains finite even as - 1. Combining
Eqs (23) & (25-27) and recalling our definition of information efficiency Π4 = -:::I(TTY W)
[Eq (11) gives
Suzo dFF (v)
Saso dF (v)
d+d B+F
-1
B+T
lim fl = In(1 (4) In Sazo dF (1))
(28)
Sozo dFP P (16)
Note that Jensen's inequality guarantees that the terms in the parentheses sum to a non-negative value.
It is worth pointing out that, at 1= the correction term in Eq (28) vanishes and the efficiency of deterministic Gibbs regression becomes minimally sensitive to algorithmic noise. Incidentally, this value of à also minimizes the L2 prediction error of ridge regression in the asymptotic limit [40].

---

4 High dimensional limit
To place our results in the context of high dimensional learning, we specialize to the thermodynamic limit in which sample size and input dimension tend to infinity at fixed ratio that is, N, P at N/P=nE (0, oo). While it is easy to grow the dimension of the true parameter W (Sec 1.1), we have so far not specified how the design matrix X, and thus the training data Y, should scale in this limit.
To this end, we consider setting in which the design matrix is generated from X= = / where Z E PxN is a matrix with iid entries drawn from a distribution with zero mean and unit variance, and I E Px is covariance matrix." If Σ admits limiting spectral density as P co, then the empirical spectral distribution F becomes deterministic [49, 50].
To aid interpretation of our results, we frame all of the following discussions from the perspective that the input dimension P is held fixed and change in measurement density N=N/P results only from a change in sample size N.
4.1 Isotropic covariates
For E=/p, the empirical spectral distribution converges to to the standard Marchenko- Pasttr law [49]
dF (1) n V(v+ (v 4-) dil for 4/ < E < to.
(29)
2TT
where Vー= (1== and / (0))= 1 n)). We use this spectral distribution in Figs 1-2. Optimal algorithm-In Fig lb, the IB optimal frontiers illustrate the fundamental trade-off; optimal algorithms cannot encode fewer residual bits without becoming less relevant. Figure Ic-d shows that encoded relevant and residual bits go down as thc increases and fewer eigenmodes contribute to the IB optimal representation [Eqs (8-9)]. However relevant and residual informations exhibit different behaviors at high information; as Vi. 0 relevant information plateaus whereas residual information diverges logarithmically (see also Eq (26)).
Gibbs regression-Fils Figure 2a depicts the information content of Gibbs regression at different regular- ization strengths [Eqs (19-20)] and illustrates the fundamental trade-off, similarly to the IB frontier (dotted) but at a lower relevance level. Here the information curves are parametrized by the inverse temperature B which controls the algorithmic stochasticity; Gibbs posteriors become deterministic as 00 and completely random at f=0 [Eq (14)). In Fig 2b-c, we see that Gibbs regression encodes fewer relevant and residual bits as temperature goes up. Decreasing Gibbs temperature results in an increase in encoded information. In the serperruprr limit, the relevant bits saturate while the residual bits grow logarithmically (cf. Fig lc-d; see Sec 3.1 for detailed analysis of this limit). The amount of encoded information depends also on the regularization strength A. Figure 2b-c shows that, at fixed temperature, increase in a leads to less information extracted. However this does not necessarily mean that a larger A hurts information efficiency. Indeed a lower temperature can compensate for the decrease in information. In Fig 2a, see that the information curves be closer to the optimal frontier as A increases. In general the maximum efficiency occurs at an intermediate regularization strength that depends on data structure and measurement density (see also Sec 4.2).
Elficiency-Figure 2d displays the information efficiency of Gibbs regression at different relevant information levels (see See 1.3). We see that the efficiency approaches optimality (11 1) in the limits л a-0 0 and n oo, Away from these limits, Gibbs regression requires more residual bits than the optimal algorithm to achieve the same level of relevance with efficiency minimum around n= 1. We also see that the efficiency of Gibbs regression decreases with relevance level (see also Supplementary Figure in Appendix D).
Extensivity-Learnin is qualitatively different in the over- and underparametrized regimes. In Fig 2e we see that both optimal algorithms and Gibbs regression exhibit nonmonotonic dependence on sample size. In the overparametrized regime n < 1, the residual information is extensive in sample size, i.e it grows linearly with N. This scaling behavior mirrors that of the relevant bits in the data (Fig 1b inset). But unlike the available relevant bits which continue to grow in the data-abundant regime, albeit sublinearly- the encoded residual bits decrease with sample size in this limit (see also
"This prescription includes the case where input vectors are drawn iid from x; ~N Σ) for (I.....N).
7

---

a
ly IP
= 0.001
r=0.01
r=0.1
r=1
100
Ve
0.1
0.01
0.01
0.01
0.001
10-4
0.01
100 10*
IB optimal
N
0.6
w 1yy
1.0
0.4
0001
0.5
0.2
10
1
0.0
0.0
0
0.5
100
10*
100 N/P
10*
1 100 10
100
10
!r:rw//
N/P
N/P
N N/P
Figure 3: Multiple descent under anisotropic covariates The relevant bits in the data decreases slightly as the anisotropy ratio departs from one (see legend). When s the available relevant information grow linearly with N. Strong anisotropy sees this growth start becoming sublinear at smaller N/P. The frontiers at P We see that while less relevant information is available in the anisotropic case, it takes fewer residual bits to achieve the same relevance level as the isotropic case (see legend in a). c The empirical spectral density the sample covariance at different anisotropy ratios (see labels). Each vertical line is normalized by its maximum. We see that anisotropy splits the spectral continuum into two bands which merge into one as N/P decreases. The solid line depicts the IB cutoff &e [Eqs (8-9)] for the relevance level = The residual information of optimal algorithms (dotted) and Gibbs regression at various regularization strengths (see color bar) for ==0.8 8 and different anisotropy ratios (same labels as in c). Here we set w² / and let P, N - co at the same rate such that the ratio N/P remains fixed and finite. The eigenvalues of the sample covariance follows the general Marchenko- coachenko-Pastur theorem (see Sec 4.2).
Supplementary Figure in Appendix D). The resulting maximum is an information-theorttic analog of double descent- the decrease in overfitting level (test error) the number of parameters exceeds sample size (decreasing n) [3].
Redundancy Indeed Indeed we could have anticipated the extensive behavior of the residual bits in the overparametrized regime (Fig 2e). this limit, the extensivity available relevant bits implies that the data encode relevant information with no redundancy. In other words, the relevant bits in one observation do not overlap with that another. As result, the dominant learning strategy to treat each sample separately and extract the same amount of information from each of them, thus resulting in extensive residual information. relevant bits in the data becomes increasingly redundant (Fig lb inset). Learning algorithms exploit this redundancy to better distinguish signals from noise, thereby encoding fewer residual bits.
the data-abundant regime, on the other hand, the coding of
4.2 Anisotropic covariates
To explore the effects of anisotropy, we consider a two-scale model in which the population spectral distribution F an equal mixture two point masses at s, and s.. We normalize the trace of ratio, does not depend on f²_ we set E/P=(s, + +=)/2=1 =I such that B[[ xx) = =B = As a result the anisotropy in our two-scale model is parametrized completely by the eigenvalue ratio r=s_/s+.
the population covariance such that the signal variance, and thus the signal-to-
Unlike the isotropic case, the limiting empirical spectral distribution does not admit closed form expression. We obtain F by solving the Silverstein equation and inverting the resulting Stieltjes transform [51] (see Appendix C). Figure 3c depicts spectral density various anisotropy ratios and measurement densities. At high measurement densities 2 1, anisotropy splits the continuum part of the spectrum into two bands, corresponding to the two modes of the population covariance. These bands broaden decreases and eventually merge into one in the overparametrized limit.

---

Available information In Fig 3a, we see that anisotropy decreases the relevant information in the data, but does not affect its qualitative behaviors: the available relevant bits are extensive in the overparametrized regime and subextensive in the data-abundant regime. Although fewer relevant bits are available, learning needs not be less information efficient. Indeed the IB frontiers in Fig 3b illustrate that it takes fewer residual bits in the anisotropic case to reach the same level of relevance as in the isotropic case. This behavior is also apparent in Fig 3d (dotted) as we increase anisotropy levels (from right to left panels).
Anisotropy effects-Anisonee Anisotropy affects optimal algorithms via the different scales in the population covariance. The signals along high-varianee (easy) directions are stronger and, as a result, the coding of relevant bits in these directions becomes subextensive and redundant around == 1/2 (the proportion of easy directions) instead of at one (Fig 3a). This earlier onset of redundancy allows for more effective signal-noise discrimination in the anisotropic case (Fig 3b d). In the data-abundant limit, however, low-variance (hard) directions become important as learning algorithms already encode most of the relevant bits along the easy directions. Indeed the hard directions are harder for more anisotropic inputs and thus the required residual bits increase with anisotropy in the limit n oo (Fig 3d).
Triple descent-Perhaps the most striking effect of anisotropy is the emergence of an information- theoretic analog of (sample-wise) multiple descent, which describes disjoint regions where more data makes overfitting worse (larger test error) [32, 52]. In Fig 3d, we see that an additional residual information maximum emerges at large n. This behavior is a consequence of the separation of scales. The first maximum at n~ 1 originates from easy directions and the other maximum at higher from hard directions. In fact the IB cutoff v/. in Fig 3c demonstrates that the residual information maxima roughly coincide with the inclusion of all high-variance modes around n~ 1 and low-variance modes at higher n. In addition we note that for optimal algorithms the first maximum shifts to a lower n as the anisotropy level increases. This observation is consistent with the fact that the onset of redundancy of relevant bits in the data occurs at smaller π in the anisotropic case (Fig 3a).
Gibbs PrrresioonAnisotren
makes Gibbs regression depend more strongly on regularization strengths. see Fig 3. In particular the information efficiency decreases with A near the first residual information minimum around n~ but this dependence reverses near the second maximum and at larger n. This behavior is expected. Inductive bias from strong regularization helps prevent noise from poisoning the models at small n. But when the data become abundant, regularization is unnecessary.
Conclusion & Outlook
We use the information bottleneck theory to analyze linear regression problems and illustrate the fundamental trade-off between relevant bits, which are informative of the unknown generative processes, and residual bits, which measure overfitting. We derive the information content of optimal algorithms and Gibbs posterior regression, thus enabling a quantitative investigation of information efficiency. In addition our analytical results on the zero temperature limit of the Gibbs posterior offer glimpse of a connection between information efficiency and optimally tuned ridge regression. Finally, using results from random matrix theory, we reveal the information complexity of learning linear map in high dimensions and unveil an information-theoret analog of multiple descent phenomena. Since residual information is upper bound on the generalization gap [4, 5], we believe that this information nonmonotonicity could be connected to the original double descent phenomena. But remains to be seen how deep this connection is.
Our work paves the way for a number of different avenues for future research. While we only focus on isotropic regularization here, it would be interesting to understand how structured regularization informatiion-fffcienn
affects information extraction. Bayesian regression, and other classes of learning problems, e.g., classification and density estimation, are also in order. An investigation of information efficiency based on other f-divergence could lead to new insights into generalization. In particular an exact relationship exists between residual Jeffreys information and generalization error of Gibbs posteriors [27]. Finally exploring how coding redundancy in training data quantitatively affects learning phenomena in general would make for an exciting research direction (see, c.g., Refs [17, 19]).
analyses of different algorithms, such as
'Note that Fig 3c does not show the zero modes which are present at < 1. The fact that the spectral continuum appears to be above the IB cutoff small does not mean all eigenmodes used in the IB solution.

---

Acknowledgments and Disclosure of Funding
This work was supported in part by the National Institutes of Health BRAIN initiative (R01EB026943), the National Science Foundation, through the Center for the Physics of Biological Function (PHY- 1734030), the Simons Foundation and the Sloan Foundation.
References
[1] M. Belkin, D. J. Hsu, and P. Mitra, Overlitting perfect fitting? Risk bounds for classification and regression rules that interpolate, in Advances in Neural Information Processing Systems, Vol. 31, edited by S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa- Bianchi, and R. Garnett (Curran Associates, Inc., 2018).
[2] M. Belkin, A. Rakhlin, and A. B. Tsybakov, Does data interpolation contradict statistical optimality?? in Proceedings of the Twenty-Second International Conference on Artificial Intelligence and Statistics, Proceedings of Machine Learning Research, Vol. 89, edited by K. Chaudhuri and M. Sugiyama (PMLR, 2019) pp. 1611-1619.
[3] N. Tishby, F. C. Pereira, and Bialek, The information bottleneck method, in 37th Allerton Conference on Communication, Control and Computing, edited by B. Hajek and R. S. Sreenivas (University of Illinois, 1999) pp. 368-377.
[4] D. Russo and J. Zou, Controlling Bias in Adaptive Data Analysis Using Information Theory, in Proceedings of the 19th International Conference on Artificial Intelligence and Statistics, Vol. 51, edited by A. Gretton and C. C. Robert (PMLR, 2016) pp. 1232- 1240.
[5] A. Xu and M. Raginsky, Information-theoreti analysis of generalization capability of learning algorithms, in Advances in Neural Information Processing Systems, Vol. 30, edited by I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Gamett (Curran Associates, Inc., 2017).
[6] S. Palmer, O. Marre, M. J. Berry, and Bialek, Predictive information in sensory population. Proceedings of the National Academy of Sciences 112, 6908 (2015).
[7] S. Wang, I. Segev, A. Borst, and S. Palmer, Maximally efficient prediction in the early fly visual system may support evasive flight maneuvers, PLOS Computational Biology 17, c1008965 (2021).
[8] M. Bauer, M. D. Petkova, T. Gregor, E. F. Wieschaus, and W. Bialek, Trading bits in the readout from a genetic network, Proceedings of the National Academy of Sciences 118, e2 11001111100 1118 (2021).
[9] S. Still, D. A. Sivak, A. J. Bell, and G. E. Crooks, Thermodynamics of Prediction, Physical Review Letters 109, 120604 (2012).
[10] A. Gordon, A. Banerjee, M. Koch-Janusz, and Z. Ringel, Relevance in the Renormalization Group and in Information Theory, Physical Review Letters 126, 240601 (2021).
[11] A. G. Kline and S. F. Palmer, Gaussian information bottleneck and the non-perturbative renormalization group, New Journal of Physics 24, 033007 (2022).
[12] D. Strouse and D. J. Schwah, The Information Bottleneck and Geometric Clustering, Neural Computation 31, 596 (2019).
[13] A. A. Alemi, I. Fischer, J. V. Dillon, and K. Murphy, Deep Variational Information Bottleneck, in Interna- tional Conference on Learning Representations (2017).
[14] A. Achille and S. Soatto, Information Dropout:
Learning Optimal Representations Through Noisy Compu- tation, IEEE Transactions on Pattern Analysis and Machine Intelligence 40, 2897 (2018).
[15] A. Achille and S. Soatto, Emergence of Invariance and Disentanglement in Deep Representations, Journal of Machine Learning Research 19, (2018).
[16] A. Goyal, R. Islam, D. Strouse, Z. Ahmed, H. Larochelle, M. Botvinick, S. Levine, and Y. Bengio, Transfer and Exploration via the Information Bottleneck, in International Conference on Learning Representations (2019).
[17] Bialek, I. Nemenman, and N. Tishby, Predictability, Complexity, and Learning, Neural Computation 13, 2409 (2001).
[18] O. Shamir, S. Sabato, and N. Tishby, Learning and generalization with the information bottleneck, Theoret- ical Computer Science 411, 2696 (2010), Algorithmic Learning Theory (ALT 2008).
[19] W. Bialek, S. E. Palmer, and D. J. Schwab, What makes it possible to learn probability distributions in the natural world? (2020).
10

---

[20] I. Alabdulmohsin, An Thoorration-Theoretiee Route from Generalization in Expectation to Generalization in Probability, in Proceedings of the 20th International Conference Artificial Intelligence and Statistics. Proceedings Machine Learning Research, Vol. 54, edited by A. Singh and Zhu (PMLR, 2017) pp- 92-100.
[21] I. Nachum and A. Yehudayoff, Average-Case Information Complexity of Learning, in Proceedings of the 30th International Conference on Algoritmmic Learning Theory, Proceedings of Machine Learning Research, Vol. 98, edited by A. Garivier and S. Kale (PMLR, 2019) pp. 633-646.
[22] J. Negrea, M. Haghifam, G. K. Dziugaite, A. Khisti, and D. M. Roy, Theoreiiion-hheeretiic Generalization Bounds for SGLD via Data- aata-Dependent Estimates, in Advances in Neural Information Processing Systems, Vol. 32, edited by H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc E. Fox, and R. Garnett (Curran Associates, Inc., 2019).
[23] Y. Bu, S. Zou, and V. V. Veeravalli, Tightening Mutual Information- Based Bounds on Generalization Error, IEEE Journal on Selected Areas in Information Theory 1, 121 (2020).
[24] T. Steinke and I.. 7akynthinou, Reasoning About Generalization via Conditional Mutual Information, in Proceedings of Thirty Third Conference on Learning Theory, Proceedings of Machine Learning Research, Vol. 125, edited by J. Abernethy and S. Agarwal (PMLR, 2020) pp. 3437-3452.
[25] M. Haghifam, J. Negrea, A. Khisti, M. Roy, and G. K. Dziugaite, Sharpened Gencralization Bounds based on Conditional Mutual Information and an Application to Noisy, Iterative Algorithmss Advances in Neural Information Processing Systems, Vol. 33, edited by H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin (Curran Associates, Inc., 2020) pp. 9925-9935.
[26] G. Neu, G. K. Dziugaite, M. Haghifam, and D. M. Roy, Information- Theoretic Generalization Bounds for Stochastic Gradient Descent, in Proceedings of Thirty Fourth Conference on Learning Theory, Proceedings of Machine Learning Research, Vol. 134, edited by M. Belkin and S. Kpotufe (PMLR, 2021) pp. 3526- 3545.
Thecorrti
[27] G. Aminian, Y. Bu, L. Toni, M. Rodrigues, and G. Wornell, An Exact Characterization of the Generalization Error for the Gibbs Algorithm, in Advances in Neural Information Processing Systems, Vol. 34, edited by M. Ranzato, A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan (Curran Associates, Inc., 2021) pp. 8106-810 8118.
[28] Q Chen, C. Shui, and M. Marchand, Generalization Bounds For Meta-L An Analysis, in Advances in Neural Information Processing Systems, Vol. 34, edited by M. Ranzato, A. Beygelz- imer. Y. Dauphin, P. Liang. and J. W. Vaughan (Curran Associates, Inc., 2021) pp. 35878-25890.
[29] M. Chalk, O. Marre, and G. Tkacik, Relevant sparse codes with variational information bottleneck, in Advances in Neural Information Processing Systems, Vol. 29, edited by D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett (Curran Associates, Inc., 2016) pp. 1957-1965.
[30] B. Poole, S. Ozair, A. Van Den Oord, A. Alemi, and G. Tucker, On Variational Bounds of Mutual Information, in Proceedings of the 36th International Conference Machine Learning, Proceedings of Machine Learning Research, Vol. 97, edited by K. Chaudhuri and R. Salakhutdinov (PMLR, 2019) pp.
5171-5180.
[31] P. Nakkiran, G. Kaplun, Y. Bansal, T. Yang, B. Barak, and I. Sutskever, Deep Double Descent: Where Bigger Models and More Data Hurt, in International Conference on Learning Representations (2020).
[32] S. d'Ascoli, L. Sagun, and G. Biroli, Triple descent and the two kinds of overfitting: where & why do they appear?, in Advances in Neural Information Processing Systems, Vol. 33, edited by H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin (Curran Associates, Inc., 2020) pp. 3058-3069.
[33] T. Hastie, R. Tibshirani, and J. Friedman, The Elements of Statistical Learning, 2nd ed. (Springer, 2009).
[34] Y. Sheng and E. Dobriban, One-shot Distributed Ridge Regression in High Dimensions, in Proceedings of the 37th International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 119, edited by H. D. III and A. Singh (PMLR, 2020) pp. 8763-8772.
[35] S. Liu and E. Dobriban, Ridge Regression: Structure, Volsd-aali,,,on, and Sketching, in International Conference on Learning Representations (2020).
[36] G. Chechik, A. Globerson, N. Tishby, and Weiss, Information bottleneck for Gaussian variahles, Joumal of Machine Learning Research 6, 165 (2005).
[37] T. Wu and I. Fischer, Phase Transitions for the Information Bottleneck in Representation Learning, in International Conference on Learning Representations (2020).
[38] V. Ngampructikom and D. J. Schwab, Perturbation Theory for the Information Bottlencek, in Advances in Neural Information Processing Systems, Vol. 34, edited by M. Ranzato, A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan (Curran Associates, Inc., 2021) pp. 21008-21018.
[39] T. M. Cover and J. A. Thomas, Elements of Information Theory, 2nd ed. WWiley-Interscicen 2006).
11

---

[40] E. Dobriban and S. Wager. High-dinensional asymptotics of prediction: Ridge regression and classification,
The Annals of Statistics 46, 247 (2018).
[41] T. Hastie, A. Montanari, S. Rosset, and R. J. Tibshirani, Surprises in high-dimensional ridgeless least squares interpolation. The Annals of Statistics 50, 949 (2022).
[42] L. Bartlett, M. Long, G. Lugosi, and Tsigler. Benign overfitting linear regression, Proceedings of the National Academy of Sciences 117, 30063 (2020).
[43] M. Emami, M. Sahrace-Ardakan P. Pandit, S. Rangan, and A. Fletcher, Generalization Error of Generalized Linear Models High Dimensions, in Proceedings of the 37th International Conference on Machine Learning, Procoedings of Machine Learning Research, Vol. 119, edited by H. D. III and A. Singh (PMLR, 2020) pp. 2892-2901.
[44] 7. Li, C. Xie, and Q. Wang, Provahle More Data Hurt in High Dimensional Least Squares Estimator (2020).
[45] D. Wu and J. Xu, On the Optimal Weighted 62 Regularization in Overparameterized Lincar Regression, in Advances in Neural Information Processing Systems, Vol. 33, edited by H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin (Curran Associates, Inc., 2020) pp. 101 10112-10123.
[46] G. Mel and S. Ganguli, A theory of high dimensional regression with arbitrary correlations between input features and target functions: sample complexity, multiple descent curves and hierarchy of phase transitions, in Proceedings of the 38th International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 139, edited by M. Meila and T. Zhang (PMLR, 2021) pp. 7578-7587.
[47] D. Richards, J. Mourtada, and L. Rosasco, Asymptotics of Ridge(less) Regression under General Source Condition, in Proceedings of The 24th International Conference on Artificial Intelligence and Statistics, Proceedings of Machine Learning Research, Vol. 130, edited by A. Banerjee and K. Fukumizu (PMLR, 2021) pp. 3889-3897.
[48] M. Belkin, D. Hsu, Ma, and Mandal, Reconciling modern machine-learning practice and the classical bias- varianae trade-off, Proceedings the National Academy Sciences 116, 15849 (2019).
[49] V. A. Marčenko and L. A. Pastur, Distribution of eigenvalues for some sets of random matrices, Mathematics of the USSR-Sborik 457 (1967).
[50] J. Silverstein, Strong Convergence of the Empirical Distribution of Eigenvalues of Large Dimensional Random Matrices, Journal of Multivariate Analysis 55, 331 (1995).
[51] Silverstein and S. Choi, Analysis of the Limiting Spectral Distribution Large Dimensional Random Matrices, Journal of Multivariate Analysis 54, 295 (1995).
[52] L. Chen, Y. Min, M. Belkin, and A. Karbasi, Multiple Descent: Design Your Own Generalization Curve, in Advances in Neural Information Processing Systems, Vol. 34, edited by M. Ranzato, A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan (Curran Associates, Inc., 2021) pp. 8898-8912.
12

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? Yes] Our numerical results are based on specific models, which describe in full Sees 1.1, 2 & 4.
(c) Did you discuss any potential negative societal impacts of your work? [N/A] This is theoretical work about the fundamentals of learning and we do not anticipate any potential negative societal impacts.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] See Sec 1.1.
(b) Did you include complete proofs of all theoretical results? [Yes] We include all deriva- tions some which appear in the Appendix.
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or as URL)? [N/A]
(b) Did you specify all the training details (c.g., data splits, hyperparameters, how they were chosen)? [N/A]
(c) Did you report error bars (e.g.. with respect the random seed after running experi- ments multiple times)? [N/A]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [N/A]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you eite the creators? [N/A]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you't 're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
13


Reviews:
[{"review_id": "fWvOqFx7MUN", "reviewer": "Reviewer_Lvpt", "summary": "The paper studies overfitting in linear regression using information bottleneck and Gibbs-posterior least squares regression. The authors define information efficiency in equation (11), which is a ratio of conditional MIs between the optimal solution of Gaussian variables given in reference [36] and an algorithm of interest. Since linear regression objective requires $XX^T$ to be invertible, and ridge regularization yields deterministic solution, the authors propose to use Gibbs posterior (Eq (14)). The information efficiency is derived in Eq (28). They further specialize it to the thermodynamic limit.\n\n** After reading the author rebuttal, where additional information on the assumptions and conceptual framework are provided. I agree that this could be a positive first step towards understanding overfitting using IB. My main concern is about the information efficiency defined in Eq (11). As the authors have provided more detail on the motivation of this definition, I am willing to raise my score.**", "questions": "The assumptions require the design matrix to be fixed and that the error follows Gaussian distribution. Would this affect the overfitting?\n\nIt is hard to obtain information theoretic results for the linear regression and ridge regularization in Eq (12) and Eq (13). Therefore, the authors apply Gibbs posterior. How would the results deviate from the original linear regression model?", "limitations": "Not applicable", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\nIB has received a lot of attentions in the study of neural networks. Linear regression using Gaussian channels has also been extensively studied. This paper provides an idea of using Gibbs posterior to obtain the information theoretic analysis of overfitting in linear regression.\n\nWeakness:\nIt is not clear to me why the information efficiency defined in Eq (11) is useful for the study of overfitting. In NNs, the IB objective is derived from a rate-distortion problem. Here, if the goal is to achieve good generalization, the objective should be derived from optimizing generalization errors. Moreover, some assumptions are restrictive and not well justified. See questions below for other comments.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "cmvyK-U4_V", "reviewer": "Reviewer_L3Rw", "summary": "This paper studied the over-fitting problem in linear models from an information-theoretic perspective. More specifically, the author defined the concept of information efficiency, which is the ratio between the mutual information values of the optimal representation and the learned representation to the scalar response. To further analyse the behavior of the information efficiency, the author considered the ridge regression with the Gibbs posterior, and derived the limit of the information efficiency with the relevance level going to one and temperature going to zero. Besides, the author checked the high dimensional limits of information under isotropic and anisotropic covariates setups.", "questions": "1. Is there any constraint on the optimal representation in equation (6) and (7)?\n\n2. I am confused about the author's definition of information bottleneck with linear map. In my understanding, the information bottleneck method would minimize the mutual information between input X and representation T as I(T; X) , where the author minimized the mutual information between mapping W and the representation T as I(W;T). Could the author provide more explanation about this?", "limitations": "The derivation of the proposed theorem is under the simplest linear model, with some assumptions, which limits the generalization ability of the method for analyzing more complicated scenarios.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n1. Different from the common usage of information bottleneck, which always served as a regularizer in learning processes, the author quantitatively analyzed the information efficiency of bottleneck representation with asymptopic limits.\n\n2. The author observed the behaviors of mutual information from multiple perspectives such as efficiency, extensivity and redundancy.\n\n3. The author smartly used the Gibbs posterior with zero-temperature limit to avoid the deterministic ridge regression.\n\nWeakness:\n1. The author's analysis is based on the simple linear model problem, which is too simple to apply the information bottleneck theory(since one can directly obtain the closed-form solution). Besides, I am wondering how the derived theorem can be further used into more complex practical scenarios, e.g. with nonlinearly neural networks. \n\n2. The analysis of high dimensional limit with anisotropic covariates only consider the spectral distribution as a mixture of two-point masses. What if one considers more general scenarios?\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "UjQxz2xVahK", "reviewer": "Reviewer_jTSy", "summary": "This paper relies on information theory to investigate linear regressions, analysing linear regressors through an information bottleneck lens.\nBasically, if we have a fixed input matrix X and we imagine a process where we:\n1. Sample a generative linear parameter vector W;\n2. Sample a set of targets Y according to W*X + GaussianNoise;\n3. Learn a vector T, which tries to reconstruct W from both X and Y.\n\nWe can decompose the amount of information T encodes about the training data (X; Y) as:\nI(T; Y) = I(T; Y | W) + I(T; W)\n\nNote that X does not appear in these equations, as it is assumed to be fixed.\n\nThe paper then gives interpretations to each of these mutual information terms and analyses them: I(T; W) is the relevant learned bits, and I(T; Y | W) is the residual bits.\n\nUsing these terms the paper first analyses the behavior of an information theoretically optimal algorithm, i.e. one which minimises the term I(T; Y | W) subject to I(T; W)/I(Y; W)=\\mu for several different values of \\mu.\nThey then present the trade-off of how much relevant vs residual information a model can extract from Y when given a dataset and parameters of size N/P.\n\nThe paper then analyses the information content of a learning algorithm which fits a linear regression using a Gibbs posterior.\nThey show Gibbs regression is optimal when N << P or N >> P, where optimality is measured in terms of I(T_{optimal}; Y | W) / I(T_{Gibbs}; Y | W)---i.e. how much more residual information the Gibbs procedure extracts than an optimal method---subject to it extracting the same information about the parameters I(T_{optimal}; W) = I(T_{Gibbs}; W).\nThe paper than shows that the amount of residual information per parameter a Gibbs regression extracts I(T; Y | W) / P first grows with N/P when N<P and then decreases for N>P. It then relates these results to the double descent phenomenon.\n\n\n\n========= Update post author--reviewer discussion ============\n\nAs noted in my review and responses. The main issue with the paper (in my reading) is that presenting results which control for $\\mu=\\frac{I(T; W)}{I(Y; W)}$ being constant---as opposed to $I(T; W)$---could lead to statistical artifacts which superficially seem interesting, but which do not reflect deeper insights into how these models work. Further, the authors arguments in favour of continuing to control only for $\\mu=\\frac{I(T; W)}{I(Y; W)}$ did not convince me.\n\nIn the authors' last response, however, they pointed me to the results which I requested (i.e., the ones controlling for $I(T; W)$ being constant) and the results seem to hold there. I thus remove my objection against this paper being accepted and raise my score to a 6.\n\nIn case this paper gets accepted, I would suggest the authors create a plot showing the model's performance (e.g., expected mean squared error) as a function of both $I(T; W)$ and $I(T; Y | W)$. This could make clearer the connection between model performance and both residual and relevant information.", "questions": "* What does the arrow notation in equation 23/25/28 represent?\n\n* Do you think the results presented in Figures 2d and 2e could be a byproduct of plotting those curves for fixed \\mu values? Why should we interpret a simultaneous reduction in I(T; W) and  I(T; Y | W) for N<<P as being related to the double descent phenomenon?\n\n* If I understood correctly, in all plots where the x-axis is N/P only the N changes, right? Why display it as N/P instead of N then?\n\n* Figure captions say \"Here we set [...] and let 𝑃, 𝑁 →∞.\" What does it mean to let P, N -> \\infty, while only N/P changes in the x-axis?\n\n* Nitpick: Equation 1 discusses random variables S, T, and W . Why use S instead of Y?", "limitations": "N/A.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This paper's strengths are:\n* This paper is very interesting and relevant in its chosen topic of study;\n* Framing the information bottleneck in terms of the amount of information the learned parameters T hold about the generative parameters W (instead of the typical way of discussing how a set of representations Z hold information about the target Y vs the input X) seems new to me, although it was not presented as such.\n* Relating the information bottleneck theory to double descent could be very impactful and an interesting way to interpret that phenomenon.\n\nThe paper's weaknesses are:\n* Some of the results in the paper---Figures 2d, and 2e, i.e. the ones related to the double descent phenomenon---are (perhaps) unintuitive. I tried to analyse them carefully, but I am still not convinced about them. Specifically, while the authors interpret these plots as providing an information-theoretic explanation for double descent, I think the shape of these curves might simply be a byproduct of the experiment design used to plot them. Specifically, each of the colored curves are conditioned such that  \\mu=I(T; W)/I(Y; W) is constant. The curves then show that, as models get to the regime of N<P, there is a reduction on the residual information I(T; Y | W). The paper then relates this to double descent. However, couldn't this be just a byproduct of a reduction in I(Y; W)? In other words, to keep \\mu constant the curves are displaying as a single thing the results for models with very different \"performances\"/relevant information I(T; W)---might this be what is driving the curves to look like this?\n* In my opinion, the paper's presentation in Section 3.1 is lacking. While throughout most of the paper the authors try to motivate and explain the equations they present, Section 3.1 is harder to follow and the equations there are less well described. Further, I did not fully understand what the arrow notation in equation 23/25/28 represent. Are those limits?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept