Title: Sharp Matrix Empirical Bernstein Inequalities
Venue: NeurIPS 2025
Accepted: yes
Keywords: random matrix

empirical bernstein inequality

concentration inequality
Abstract: We present two sharp, closed-form empirical Bernstein inequalities for symmetric random matrices with bounded eigenvalues. By sharp, we mean that both inequalities adapt to the unknown variance in a tight manner: the deviation captured by the first-order $1/\sqrt{n}$ term asymptotically matches the matrix Bernstein inequality exactly, including constants, the latter requiring knowledge of the variance. 
Our first inequality holds for the sample mean of independent matrices, and our second inequality holds for a mean estimator under martingale dependence at stopping times.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proves two empirical Bernstein inequalities for sums of symmetric random matrices, extending the empirical Bernstein bounds from the scalar to the matrix setting. Suppose $T_\\alpha$ satisfies $\\Pr(\\lambda_{\\max}(\\sum_i X_i) \\geq T_\\alpha) \\leq \\alpha$, where $X_i$'s are mean-zero symmetric matrices. The standard (non-empirical) matrix Bernstein inequality gives\n$$\nT_\\alpha = \\sqrt{\\frac{2\\log(d/\\alpha) \\|V\\|}{n}} + O\\left(\\frac{\\log(1/\\alpha)}{n}\\right),\n$$\nwhere $V = \\frac{1}{n} \\sum_i \\mathbb{E} X_i^2$, and each $-I \\preceq X_i \\preceq I$. In the empirical versions, the unknown variance $V$ is replaced with a data-dependent estimator.\n\nSpecifically, the paper shows show that\n$$\nT_\\alpha = \\sqrt{\\frac{2 \\log(nd/((n-1)\\alpha)) \\Vert V_n^\\ast \\Vert }{n}} + O\\left( \\frac{\\log(nd/\\alpha)}{n} \\right),\n$$\nwhere $V_n^* = \\frac{1}{n} \\sum_{i\\ \\text{odd}} (X_i - X_{i+1})^2$ is the paired sample variance. This result is shown to be sharp asymptotically. The sharpness is obtained through an unbalanced union bound, improving on the scalar case of Maurer and Pontil. \n\nThe paper also proves a second empirical Bernstein inequality under martingale dependence, generalizing the classical matrix Freedman inequality. The technique is based upon the self-normalized supermartingales due to Waudby-Smith and Ramdas (2023) but generalized to the matrix case.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: The paper generalizes the classical matrix Bernstein/Freedman inequalities to the empirical case and obtains sharp bounds. \n\nWeaknesses: Previous work of Kroshnin and Suvorikova seems to have a good bound with much simpler proofs. I do not find this paper's improvement very important in applications (unless one cares a lot about the constants).\n\nQUESTIONS:\nCan you comment more on Kroshnin and Suvorikova’s result? Their result contains an \\hat{z} term but that is just 1 in many cases. If you only want a bound that is tight in order (without the \\hat{z} term in Kroshnin and Suvorikova’s result) instead of having sharp constants, is there a simpler proof?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-18"}

comments: SUMMARY:
This paper proposes two sharp, closed-form empirical Bernstein inequalities for symmetric random matrices with bounded eigenvalues. The first inequality (Theorem 3.1) applies to the sample mean of independent matrices and uses a paired sample variance estimator. The second (Theorem 4.2 and Corollary 4.3) extends to martingale-dependent sequences.

STRENGTHS AND WEAKNESSES:
The paper is technically sound, and proves novel matrix Empirical Bernstein inequalities that are asymptotically tight. However, the authors only suggest that the matrix EB inequalities can be used to obtain tighter confidence interval, and do not relate it to machine learning problems, so I believe that it is out of the scope of NeuIPS2025 and should be submitted to statistics journals or conferences instead.

QUESTIONS:
1. Could you suggest any potential applications of the matrix EB inequalities? e.g. for bandit problems or PAC-Bayesian bounds. Including such applications might make this paper more suitable for machine learning conferences.

2. Can you prove that the second EB inequality is sharper than the first one for iid matrices, or only empirically so?

3. Is there a techinical difficulty in extending the first EB inequality to odd numbers? What is the difference between paired sample variance and sample variance in order to give a bound?

4. On the techinical side, what is the main difficulty in extending EB inequality to matrices?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-23

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper generalizes scalar empirical Bernstein inequalities to the matrix case. Known scalar empirical Bernstein inequalities give a bound on the deviation of the sample mean from the true mean in terms of the empirical variance for i.i.d. samples, or a bound on the deviation of a particular weighted average from the conditional mean in terms of some empirical quantity in the general case. For matrices, a Bernstein type inequality bounds the largest eigenvalue of the centralized average of n i.i.d symmetric matrices in terms of some matrix V that upper bounds the matrices variance w.r.t. to the Loewner partial order. Empirical variants are notoriously harder to attain. The authors formulate a couple of sharp matrix Bernstein empirical inequalities, where sharp means that its deviation terms approache the non-empirical Bernstein inequality deviation term a.s. including constants.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: the paper is clear and concise, the discussion on the problem, known results and contributions of the paper are thorough.\nWeaknesses: I am not convinced that the results are significant enough, and find the originality of the paper to be lacking. For example, I would like to see more convincing arguments why empirical Bernstein inequalities for matrices are important. In the scalar problem, the empirical inequality allows one to obtain an empirical confidence interval for the true mean given the sample average, when the true variance is unknown. Can something similar be said about the matrix case? As far as originality, it seems to me that the proofs are derivative of previous works, e.g., Theorem 3.1. is obtained by applying the matrix Bennett-Bernstein inequality twice and using the union bound in a non-trivial way.\n\nQUESTIONS:\nIn eq. (58), where does the factor 2 in the parentheses originate from? This is an average of dxd matrices, so shouldn't be d/alpha instead of 2d/alpha?\n\nLIMITATIONS:\nThe authors adequately addressed the limitations and potential negative societal impact of their work.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper considers the question of proving matrix empirical Bernstein-type concentration inequalities. In its simplest form, the Bernstein inequality is a well-known bound that is used to reason about the empirical mean of independent random variables with common mean and bounded support that satisfy a second moment condition. The second moment condition involves a variance bound $\\sigma^2$ which is usually not known in practical settings.\nThere are a number of generalizations and stronger versions of this result that one can consider. First, it is important to consider ``empirical\" versions of this inequality, where the unknown bound $\\sigma^2$ is replaced by an estimate for the variance. This has the advantage that the deviation of the empirical mean from the true mean can then be quantified precisely, instead of being expressed as a function of the unknown variance bound. Another generalization involves going beyond independent random variables, and considering various kinds of dependencies. Finally, it is worth considering generalizations from scalars to vectors and matrices.\nThe present work gives empirical Bernstein-type concentration inequalities for matrices, both for independent random matrices and for random matrices that exhibit a martingale dependence. Both inequalities exhibit sharp constants.\n\nSTRENGTHS AND WEAKNESSES:\nI think this is an interesting work, but I have some concerns regarding its overall merit. For that reason, I will try to discuss both positive and negative aspects of the work below.\n\nStrengths:\n1) The main results are interesting in their own right. Indeed, having empirical versions of standard concentration inequalities generalized to the matrix setting and with sharp constants constitutes an interesting contribution to the literature.\n2) The writing is generally accessible.\n\nWeaknesses:\n1) It is mentioned that it is generally harder to establish concentration results for random matrices, compared to how hard it is to achieve this for random variables or vectors. However, I felt there was not an adequate discussion of the technical challenges that the authors encountered when attempting to establish the results in this work. Instead, it felt like the results mostly follow some approaches which have appeared in prior work (e.g., Audibert et al. [2009], ], Maurer and Pontil [2009], Tropp [2012], etc).\n2) No application of the results is given. The paper makes the case about the importance of the result based on the fact that empirical versions of concentration inequalities are generally useful in practice, and that constants are sharp. However, I feel the authors could have made a stronger case about the importance of the results, by a giving a concrete application.\nOverall, I do not consider this to be a bad paper by any means, but I think it might be a bit marginal. I am voting borderline accept for the time being.\n\nJ.-Y. Audibert, R. Munos, and C. Szepesvari. Exploration–exploitation tradeoff using variance estimates in multi-armed bandits. Theoretical Computer Science, 410(19):1876–1902, 2009.\n\nA. Maurer and M. Pontil. Empirical Bernstein bounds and sample variance penalization. In 22nd Annual Conference on Learning Theory, 2009.\n\nJ. A. Tropp. User-friendly tail bounds for sums of random matrices. Foundations of Computational Mathematics, 12:389–434, 2012.\n\nQUESTIONS:\nCan the authors provide a more detailed discussion of the challenges involved in proving the results? Also, can they give more concrete examples of problems where using these concentration bounds would yield significant benefits?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-23"}