Title: PAC-Bayes Bounds for Multivariate Linear Regression and Linear Autoencoders
Venue: NeurIPS 2025
Accepted: yes
Keywords: PAC-Bayes bound

multivariate linear regression

linear autoencoder

recommender system
Abstract: Linear Autoencoders (LAEs) have shown strong performance in state-of-the-art recommender systems. However, this success remains largely empirical, with limited theoretical understanding.  In this paper, we investigate the generalizability -- a theoretical measure of model performance in statistical learning -- of multivariate linear regression and LAEs. We first propose a PAC-Bayes bound for multivariate linear regression, extending the earlier bound for single-output linear regression by Shalaeva et al., and establish sufficient conditions for its convergence. We then show that LAEs, when evaluated under a relaxed mean squared error, can be interpreted as constrained multivariate linear regression models on bounded data, to which our bound adapts. Furthermore, we develop theoretical methods to improve the computational efficiency of optimizing the LAE bound, enabling its practical evaluation on large models and real-world datasets. Experimental results demonstrate that our bound is tight and correlates well with practical ranking metrics such as Recall@K and NDCG@K.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper provides a novel theoretical understanding of Linear AutoEncoders (LAEs) through PAC-Bayes theory. First, the authors extend an existing result of Shalaeva et al. to the case of multivariate linear regression and explicit sufficient conditions to ensure convergence of such bounds. Then, they adapt their results to LAEs by focusing on the particular case of multivariate linear regression on bounded data. From such theoretical results, they derive a novel learning algorithm for LAEs and conclude their work with some experiments.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: \n- The question of PAC-Bayes for LAEs is, to my knowledge, new.\n- Authors avoid classical assumptions in PAC-Bayes such as subgaussianity for their specific setting\n\nWeaknesses\n- The paper is hard to read and there is little discussion about the results.\n- Lit review is incomplete in the main text?\n- The motivation behind the various lemma and theorem is not always clear (e.g Lemmas 4.2,4.3 Proposition 5.3)\n\nQUESTIONS:\n- L. 208 what is $X_{*,j}$ in practice? The star suggest this is a theoretical quantity in some sense, is this true? \n- Lemma 4.2 why highlighting the specific form of $R_{true}$ given this quantity cannot be computable in practice?\n- Theorem 3.2: can you compare your result PAC-Bayes bounds existing for linear regression here (instead of the appendix)? For instance, with either [1,2] or Shalaeva's work to understand precisely what is original in your bound?\n- Theorem 3.2 is obtained from the general Alquier's bound, which is often achieved by subgaussian or subexponential assumptions, which is a choice you do not make here. In recent works PAC-Bayes bounds for heavy-tailed losses (i.e. admitting finite variances) have been obtained via supermartingales argument see [3,4]). Couldn't you reach tighter results using these bounds? This would avoid the exponential moment which may ba hard to control and (potentially) computationally instable\n- l. 87 the assumption on $\\mathcal{D}$ is not valid throughout the whole paper, you should detail it\n- l.196, isn't it the exact same statement than Theorem 3.2? What is new here?\n- l. 214, 'And the PAC-Bayes bound for LAEs is formed by applying the above two constraints to (8)' what does this mean?  \n- l. 252 You affirm that 'neural network models typically do not admit closed-form solutions for the optimal ρ for (11).' This is actually not true if you choose to assimilate a neural net to its vectors of weights. (11) is minimized by Gibbs posteriors (see e.g. [5, Section 5.1])\n- l. 266-267 Are such derivation useful in the main text?\n- It seems that \n- Proposition 5.3 Again, why highlighting the specific form of the exponential moment (a function of $R_{true}$) given this quantity cannot be computable in practice?\n- l. 291 you say this exponential moment is much easier to compute. How can you make this bound tractable as it is supposed to be unknown on real-life experiments?\n\n\nIn conclusion, while some theoretical novelties seems to appear in this work (PAC-Bayes bounds for the specific case of multilinear regression without light-tailed assumptions), additional discussions and comparisons with modern methods in PAC-Bayes is severely lacking and make the paper hard to parse.  As I am not an expert in LAE, I may miss the particular relevance of the proposed results but, in my humble opinion, the paper has not been truly pedagogical about it. I believe then it needs to be enriched and rewritten before acceptance. \n\n\nReferences\n[1] Germain et al., PAC-Bayesian Theory meets Bayesian inference, 2016\n\n[2] Haddouche et al. PAC-Bayes Unleashed: Generalisation Bounds with Unbounded Losses, 2021\n\n[3] Haddouche et al. PAC-Bayes Generalisation Bounds for Heavy-Tailed Losses through Supermartingales; 2023\n\n[4] Chugg et al. A unified recipe for deriving (time-uniform) PAC-Bayes bounds, 2024\n\n[5] Catoni, A PAC-Bayesian approach to adaptive classification, 2003\n\nLIMITATIONS:\nLiterature review is missing to understand the precise novelty of such results. Also, a strength of Theorem 3.2, which is to avoid assumptions such as subgaussianity, is not highlighted.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 1, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper extends PAC-Bayes theory to multivariate linear regression, deriving a novel generalization bound that remains tight for full-rank models under unbounded loss. It adapts this bound to Linear Autoencoders by framing them as constrained multivariate regression on bounded data, addressing challenges like zero-diagonal weight constraints and input-target dependencies. Experiments demonstrate tightness and correlation with recommendation metrics like Recall@K and NDCG@K.\n\nStrengths:\n\nFirst PAC-Bayes bound for multivariate regression and LAEs, filling a theoretical gap.\nRigorous convergence analysis with sufficient conditions.\nComputationally efficient methods for bound optimization.\nStrong empirical validation on real-world datasets.\n\nLimitations:\nThe author has elaborately described his framework and has carried out numerous theoretical studies and experiments to verify its effectiveness.\nDue to the large number of formulas and symbols, some parts may appear rather messy (for example, the lower half of the second page). It is hoped that the author can adjust the format and structure of the paper to make it more aesthetically pleasing and easy to read.\n\nSTRENGTHS AND WEAKNESSES:\nThis is the first PAC-Bayes bound for multivariate regression and LAEs, and fill a theoretical gap.\nThe rigorous convergence analysis with sufficient conditions shows the effectiveness of the framework.\nThe author has proved it is a computationally efficient methods for bound optimization.\nThe framework has a strong empirical validation on real-world datasets.\n\nQUESTIONS:\nThe author has elaborately described his framework and has carried out numerous theoretical studies and experiments to verify its effectiveness.\nDue to the large number of formulas and symbols, some parts may appear rather messy (for example, the lower half of the second page). It is hoped that the author can adjust the format and structure of the paper to make it more aesthetically pleasing and easy to read.\n\nLIMITATIONS:\nThe author has elaborately described his framework and has carried out numerous theoretical studies and experiments to verify its effectiveness.\nDue to the large number of formulas and symbols, some parts may appear rather messy (for example, the lower half of the second page). It is hoped that the author can adjust the format and structure of the paper to make it more aesthetically pleasing and easy to read.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-26"}

comments: SUMMARY:
This paper develops a PAC-Bayes generalization bound tailored for multivariate linear regression. The authors then derive a sufficient conditions to guarantee its theoretical convergence under certain assumption (stated in Thm 3.3), e.g. Gaussian data assumption. They then adapt this framework to handle bounded data distributions and then specifically apply it to analyze Linear Autoencoders (LAEs). After that, under Gaussian data assumption, the authors presents a practical computational method for this LAE-specific bound. At the end, experimental results on validating the tightness of the proved bounds are provided.

STRENGTHS AND WEAKNESSES:
Strengths:
(1) The paper provides the first PAC-Bayes bound for LAEs.
(2) The paper derives a sufficient condition for convergence in the case of Gaussian data.
(3) The paper also fixes the issues in Shalaeva's bound and proves the claimed bound in Shalaeva's work.

Weaknesses:
(1) The calculation of PAC-Bayes bound requires the data distribution to be known.
(2) A couple of the results requires the Gaussian data assumption.

QUESTIONS:
My question will be mainly about the assumptions on input data. I think it will help greatly if the paper can provide more detailed justifications on those assumptions, e.g., maybe they are not common, but they are important and indeed appearing in XX by evidence/literature XX.

My next question might be helpful (hopefully) for enriching the paper or future direction, if possible. Can the value of the PAC-Bayes bound after a bunch of update steps during the training process help adaptively improve the training process? Is it possible?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 3
DATE: 2025-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work develops PAC-Bayes generalization bound for multivariate linear regression and extends that bound to the Linear AutoEncoders (LAEs) as a constrained form of multivariate linear regression under the bounded data assumption. To make the theory practical, an algorithm for computing the bound is developed. The bound is further applied to Linear Autoencoder (LAE) recommender systems using the EASE model for empirical validation. Experiments conducted on standard recommendation datasets confirm the bound's effectiveness in a practical setting.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n- Bounds for a more general case: The paper provides a nice theoretical extension of PAC-Bayes bounds to multivariate linear regression case, while also formalizing and clarifying the the established convergence proof conditions from Shalaeva et al.\n- Extension to Linear AutoEncoder: The paper does a good job of linking its theory to Linear Autoencoders (LAEs) in recommender systems. The derived multivariate bound is adjusted to account for the specific case of LAEs, like the zero-diagonal constraint on the weight matrix and how the input/target data is structured in implicit feedback scenarios (Section 4.2).\n- The paper also provides an algorithm to compute the bound in practice. A neat closed-form solution for the optimal posterior is also provided under the Gaussian data assumption (Theorem 5.2). The theoretical results are backed with real-world tests in Table 1, which highlight the tightness of the bound.\n\n**Weaknesses**\n\n- Limited applicability to low-rank scenario: The practical computation methods, particularly the closed-form solutions in Section 5, are derived under the assumption that the weight matrix W follows a multivariate Gaussian distribution, which implies it is full-rank. This framework does not directly apply to low-rank structure: W = UV^T. Such low-rank models are common in recommendation systems, e.g. in collaborative filtering and matrix completion. Adapting the theory to impose distributions on the factors U and V would break the Gaussian assumption for W, rendering the current computational methods inapplicable. This limits the scope of the paper's impact.\n- Mismatch Between Theoretical Loss and Recall/NDCG metrics: The commonly used metrics in recommendation systems (Recall/NDCG) are set-based ranking metrics (e.g. computed as Recall@K and NDCG@k for a set of k predictions). It is not clear to me why the chosen loss ($(||Y - WX||_F^2)$) in the analysis is correlated to these metrics. A model that is optimal under squared error is not guaranteed to be optimal for ranking.\n\nQUESTIONS:\nRelated to weakness 2: Why do we see correlation in the used loss  and the ranking metrics, given that optimal loss may not be the best outcome for ranking?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}

comments: SUMMARY:
The authors develop PAC-Bayes bounds for
1) multivariate linear regression problems (that is, when the output variable Y is a vector),
2) linear autoencoders (LAE).

STRENGTHS AND WEAKNESSES:
General comments:
- the paper seems technically correct, and is overall well written,
- but while authors know the basic PAC-Bayes techniques, they ignore the existing literature on PAC-Bayes bounds for multivariate regression and for VAEs.

1) Multivariate regression:
Covered as a special case of https://www.mdpi.com/1099-4300/25/2/333 (take Z= Identity),
Also included in https://arxiv.org/abs/2206.08619
One observation: the rates in the above references are in 1/n (n = sample size), while all rates derived in this submission cannot be better than 1/sqrt(n).

2) LAEs:
They should compare with existing bound on VAEs:
https://proceedings.mlr.press/v151/cherief-abdellatif22a.html
https://proceedings.neurips.cc/paper_files/paper/2023/hash/b29500824d22ee9bbd25e4cd97c49b55-Abstract-Conference.html
It's not OK not to cite these papers, and the results stated here cannot be assessed if there is no rigorous comparison with these existing works.

Minor comments:
- the claim that Alquier et al.'s 2016 bound cannot be used in practice because Psi depends on the unknown data-generating process is a little weird, as the paper provides many examples where we can upper bound Psi by distribution independent quantities.

QUESTIONS:
1) Please compare the results to the existing works on multivariate regression and LAE/VAE's.
2) Explain what are the improvements over these results.

LIMITATIONS:
Yes.
TITLE: Official Review
CLARITY: 2
IMPACT: 1
ORIGINALITY: 1
DATE: 2025-07-08