Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper studied the over-fitting problem in linear models from an information-theoretic perspective. More specifically, the author defined the concept of information efficiency, which is the ratio between the mutual information values of the optimal representation and the learned representation to the scalar response. To further analyse the behavior of the information efficiency, the author considered the ridge regression with the Gibbs posterior, and derived the limit of the information efficiency with the relevance level going to one and temperature going to zero. Besides, the author checked the high dimensional limits of information under isotropic and anisotropic covariates setups.
strengths_and_weaknesses: Strength:
1. Different from the common usage of information bottleneck, which always served as a regularizer in learning processes, the author quantitatively analyzed the information efficiency of bottleneck representation with asymptopic limits.

2. The author observed the behaviors of mutual information from multiple perspectives such as efficiency, extensivity and redundancy.

3. The author smartly used the Gibbs posterior with zero-temperature limit to avoid the deterministic ridge regression.

Weakness:
1. The author's analysis is based on the simple linear model problem, which is too simple to apply the information bottleneck theory(since one can directly obtain the closed-form solution). Besides, I am wondering how the derived theorem can be further used into more complex practical scenarios, e.g. with nonlinearly neural networks. 

2. The analysis of high dimensional limit with anisotropic covariates only consider the spectral distribution as a mixture of two-point masses. What if one considers more general scenarios?

questions: 1. Is there any constraint on the optimal representation in equation (6) and (7)?

2. I am confused about the author's definition of information bottleneck with linear map. In my understanding, the information bottleneck method would minimize the mutual information between input X and representation T as I(T; X) , where the author minimized the mutual information between mapping W and the representation T as I(W;T). Could the author provide more explanation about this?
limitations: The derivation of the proposed theorem is under the simplest linear model, with some assumptions, which limits the generalization ability of the method for analyzing more complicated scenarios.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper relies on information theory to investigate linear regressions, analysing linear regressors through an information bottleneck lens.
Basically, if we have a fixed input matrix X and we imagine a process where we:
1. Sample a generative linear parameter vector W;
2. Sample a set of targets Y according to W*X + GaussianNoise;
3. Learn a vector T, which tries to reconstruct W from both X and Y.

We can decompose the amount of information T encodes about the training data (X; Y) as:
I(T; Y) = I(T; Y | W) + I(T; W)

Note that X does not appear in these equations, as it is assumed to be fixed.

The paper then gives interpretations to each of these mutual information terms and analyses them: I(T; W) is the relevant learned bits, and I(T; Y | W) is the residual bits.

Using these terms the paper first analyses the behavior of an information theoretically optimal algorithm, i.e. one which minimises the term I(T; Y | W) subject to I(T; W)/I(Y; W)=\mu for several different values of \mu.
They then present the trade-off of how much relevant vs residual information a model can extract from Y when given a dataset and parameters of size N/P.

The paper then analyses the information content of a learning algorithm which fits a linear regression using a Gibbs posterior.
They show Gibbs regression is optimal when N << P or N >> P, where optimality is measured in terms of I(T_{optimal}; Y | W) / I(T_{Gibbs}; Y | W)---i.e. how much more residual information the Gibbs procedure extracts than an optimal method---subject to it extracting the same information about the parameters I(T_{optimal}; W) = I(T_{Gibbs}; W).
The paper than shows that the amount of residual information per parameter a Gibbs regression extracts I(T; Y | W) / P first grows with N/P when N<P and then decreases for N>P. It then relates these results to the double descent phenomenon.



========= Update post author--reviewer discussion ============

As noted in my review and responses. The main issue with the paper (in my reading) is that presenting results which control for $\mu=\frac{I(T; W)}{I(Y; W)}$ being constant---as opposed to $I(T; W)$---could lead to statistical artifacts which superficially seem interesting, but which do not reflect deeper insights into how these models work. Further, the authors arguments in favour of continuing to control only for $\mu=\frac{I(T; W)}{I(Y; W)}$ did not convince me.

In the authors' last response, however, they pointed me to the results which I requested (i.e., the ones controlling for $I(T; W)$ being constant) and the results seem to hold there. I thus remove my objection against this paper being accepted and raise my score to a 6.

In case this paper gets accepted, I would suggest the authors create a plot showing the model's performance (e.g., expected mean squared error) as a function of both $I(T; W)$ and $I(T; Y | W)$. This could make clearer the connection between model performance and both residual and relevant information.
strengths_and_weaknesses: This paper's strengths are:
* This paper is very interesting and relevant in its chosen topic of study;
* Framing the information bottleneck in terms of the amount of information the learned parameters T hold about the generative parameters W (instead of the typical way of discussing how a set of representations Z hold information about the target Y vs the input X) seems new to me, although it was not presented as such.
* Relating the information bottleneck theory to double descent could be very impactful and an interesting way to interpret that phenomenon.

The paper's weaknesses are:
* Some of the results in the paper---Figures 2d, and 2e, i.e. the ones related to the double descent phenomenon---are (perhaps) unintuitive. I tried to analyse them carefully, but I am still not convinced about them. Specifically, while the authors interpret these plots as providing an information-theoretic explanation for double descent, I think the shape of these curves might simply be a byproduct of the experiment design used to plot them. Specifically, each of the colored curves are conditioned such that  \mu=I(T; W)/I(Y; W) is constant. The curves then show that, as models get to the regime of N<P, there is a reduction on the residual information I(T; Y | W). The paper then relates this to double descent. However, couldn't this be just a byproduct of a reduction in I(Y; W)? In other words, to keep \mu constant the curves are displaying as a single thing the results for models with very different "performances"/relevant information I(T; W)---might this be what is driving the curves to look like this?
* In my opinion, the paper's presentation in Section 3.1 is lacking. While throughout most of the paper the authors try to motivate and explain the equations they present, Section 3.1 is harder to follow and the equations there are less well described. Further, I did not fully understand what the arrow notation in equation 23/25/28 represent. Are those limits?
questions: * What does the arrow notation in equation 23/25/28 represent?

* Do you think the results presented in Figures 2d and 2e could be a byproduct of plotting those curves for fixed \mu values? Why should we interpret a simultaneous reduction in I(T; W) and  I(T; Y | W) for N<<P as being related to the double descent phenomenon?

* If I understood correctly, in all plots where the x-axis is N/P only the N changes, right? Why display it as N/P instead of N then?

* Figure captions say "Here we set [...] and let 𝑃, 𝑁 →∞." What does it mean to let P, N -> \infty, while only N/P changes in the x-axis?

* Nitpick: Equation 1 discusses random variables S, T, and W . Why use S instead of Y?
limitations: N/A.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper studies overfitting in linear regression using information bottleneck and Gibbs-posterior least squares regression. The authors define information efficiency in equation (11), which is a ratio of conditional MIs between the optimal solution of Gaussian variables given in reference [36] and an algorithm of interest. Since linear regression objective requires $XX^T$ to be invertible, and ridge regularization yields deterministic solution, the authors propose to use Gibbs posterior (Eq (14)). The information efficiency is derived in Eq (28). They further specialize it to the thermodynamic limit.

** After reading the author rebuttal, where additional information on the assumptions and conceptual framework are provided. I agree that this could be a positive first step towards understanding overfitting using IB. My main concern is about the information efficiency defined in Eq (11). As the authors have provided more detail on the motivation of this definition, I am willing to raise my score.**
strengths_and_weaknesses: Strengths:
IB has received a lot of attentions in the study of neural networks. Linear regression using Gaussian channels has also been extensively studied. This paper provides an idea of using Gibbs posterior to obtain the information theoretic analysis of overfitting in linear regression.

Weakness:
It is not clear to me why the information efficiency defined in Eq (11) is useful for the study of overfitting. In NNs, the IB objective is derived from a rate-distortion problem. Here, if the goal is to achieve good generalization, the objective should be derived from optimizing generalization errors. Moreover, some assumptions are restrictive and not well justified. See questions below for other comments.
questions: The assumptions require the design matrix to be fixed and that the error follows Gaussian distribution. Would this affect the overfitting?

It is hard to obtain information theoretic results for the linear regression and ridge regularization in Eq (12) and Eq (13). Therefore, the authors apply Gibbs posterior. How would the results deviate from the original linear regression model?
limitations: Not applicable
review_title: null

Meta-review: metareview: The technical concerns of the reviewers were cleared in the discussion and based on the final reviews and my own reading the paper seems technically sound. While the considered model is very simple and hence relevance to practice is hard to foresee, the contribution of analyzing the IB method in a simple setting is considered valuable and of interest to the community. We hence recommend acceptance of the paper. The reviews and the subsequent discussion provide many suggestions that should help the authors to improve the presentation of their results. In particular, adding the motivation for the information efficiency and plots suggested by reviewer jTSy seem of interest. 
confidence: Less certain
Decision: Accept