Title: Perturbation Bounds for Low-Rank Inverse Approximations under Noise
Venue: NeurIPS 2025
Accepted: yes
Keywords: Low-rank approximation

Pseudoinverse

Robustness  Numerical linear algebra

Noisy matrices
Abstract: Low-rank pseudoinverses are widely used to approximate matrix inverses in scalable machine learning, optimization, and scientific computing. However, real-world matrices are often observed with noise, arising from sampling, sketching, and quantization. The spectral-norm robustness of low-rank inverse approximations remains poorly understood.  We systematically study the spectral-norm error $\| \tilde{A}_p^{-1} - A_p^{-1} \|$ for an $n\times n$ symmetric matrix $A$, where $A_p^{-1}$ denotes the best rank-\(p\) approximation of $A^{-1}$, and $\tilde{A} = A + E$ is a noisy observation. Under mild assumptions on the noise, we derive sharp non-asymptotic perturbation bounds that reveal how the error scales with the eigengap, spectral decay, and noise alignment with low-curvature directions of $A$. Our analysis introduces a novel application of contour integral techniques to the \emph{non-entire} function $f(z) = 1/z$, yielding bounds that improve over naive adaptations of classical full-inverse bounds by up to a factor of $\sqrt{n}$. Empirically, our bounds closely track the true perturbation error across a variety of real-world and synthetic matrices, while estimates based on classical results tend to significantly overpredict. These findings offer practical, spectrum-aware guarantees for low-rank inverse approximations in noisy computational environments.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work explores the problem of bounding the error for low rank approximation of inverse of a noisy matrix. Specifically, for PSD matrix $A$ and observed noisy matrix $\\tilde{A} = A + E$, with noise matrix $E$, let $\\tilde{A_p}^{-1}$ and $A_p^{-1}$  be the best rank-$p$ approximation to $\\tilde{A}$ and $A$ respectively; the work provides an upper bound on $||\\tilde{A_p}^{-1} - A_p^{-1}||$ in terms of $||E||$, smallest eigenvalue $\\lambda_n$ and the gap in eigenvalues $\\delta_p = \\lambda_{n-p} - \\lambda_{n-p+1}$ of matrix $A$. The bounds obtained in this work are data dependent, and for certain regimes provides tighter results than previously known, under the assumption that the norm of noise matrix $||E|| \\leq \\min (\\lambda_n, \\delta_{n-p} )$. \nTo prove the bound, following up on the recent line of work, they apply contour bootstrapping to the function $f(x) = 1/x$. This involves bounding the error term by an integral over the contour that contains $p$ smallest eigenvalues of matrix $A$. The authors then bound this integral.\nThey also discuss extensions to symmetric matrices (can have negative eigenvalues) and also provide bounds for this case.\nEmpirical evidence on synthetic and real-world datasets is provided by comparing their bound and previously known bound, showing that their bounds provide better characterization of actual error.\n\nSTRENGTHS AND WEAKNESSES:\nThe bound obtained in this work for $||\\tilde{A_p}^{-1} - A_p^{-1}||$ is $5 \\Big( \\frac{||E||}{\\lambda_n^2} + \\frac{||E||}{\\lambda_{n-p} \\delta_{n-p}} \\Big)$ whereas the previous known bound is $ \\Big( \\frac{8||E||}{3\\lambda_n^2} + \\frac{2}{\\lambda_{n-p}} \\Big)$. As a result, their bound is better whenever $||E|| << \\delta_{n-p}$. Essentially, for small noise and large eigenvalue separation, the bound obtained in this work is better.\n\n\nThough, for instances where gap can be of the order $O(\\log{n})$, the bound obtained seems worse compared to the EY-N bound.\nFor justifying the assumption of $||E|| < \\delta_{n-p}$, the authors provide discussion assuming sub-Gaussian noise, on the variance of noise that ensures this condition w.h.p which is fine as long as $m >> n$, otherwise the permissible noise scale can be much smaller - $\\tilde{O}(1/\\sqrt{n})$. This also potentially limits the applicability of the bound for high-dimensional data, which is typically the case for modern machine learning applications.\n\nQUESTIONS:\nPlease refer the previous section for discussion. No specific questions for the authors.\n\nLIMITATIONS:\nThe scope of work is limited to analyzing the function $f(x) = 1/x$ and the bounds offer improved characterizations of error for certain parameter regime.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the spectral‐norm error between the best rank-$p$ approximations of the inverse of a symmetric matrix $A$ and its noisy observation $\\widetilde A = A + E$. The authors introduce a novel “contour bootstrapping” technique applying contour‐integral methods to the non-entire function $f(z)=1/z$ to localize resolvent expansions around the smallest eigenvalues. Under the written condition, they prove the bound which improves classical full-inverse estimates by up to a $\\sqrt n$ factor in regimes of fast spectral decay.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n- The adaptation of contour‐integral (Riesz projector) methods to $f(z)=1/z$ under noise is non-trivial and addresses a gap in perturbation theory for low-rank pseudoinverses.\n- The paper connects the theory to applications (e.g., differential privacy, structural engineering) by analyzing real-world matrices and demonstrating that typical noise levels satisfy the gap condition for safe usage.\n\n**Weaknesses:**\n - The requirement $4\\|E\\| < \\delta_{n-p}$ may be hard to check in black-box or data-streaming settings, as estimating tail eigengaps can be as expensive as the inverse itself .\n- The analysis assumes a fixed matrix $A$; extensions to time-varying or adaptive contexts (e.g., iterative solvers) are not addressed.\n- While theoretically elegant, the custom rectangular contour $\\Gamma$ (Section 3) depends on spectral extremal values ($\\lambda_1,\\lambda_n,\\delta_{n-p}$); practical implementation may require careful numerical tuning and stability analysis.\n\nQUESTIONS:\n- Can you propose a low-cost procedure (e.g., randomized sketch) to approximate $\\delta_{n-p}$ and $\\lambda_n$ sufficiently well to certify the gap condition without full eigendecomposition?\n- How sensitive is the bound to slight misestimation of $\\lambda_1$ or $\\delta_{n-p}$ when constructing $\\Gamma$? Have you observed any numerical instabilities?\n-  Is there a practical way to measure or bound $\\|E\\|$ within the low-curvature eigenspace to tighten the bound further in specific applications (e.g., quantization noise)?\n- Section A mentions general symmetric $A$; can your approach handle non-square or non-Hermitian matrices (e.g., via pseudospectrum)?\n\nLIMITATIONS:\nSee Weaknesses", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 1, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work provides perturbation bounds for the low-rank inverse approximation of noisy matrices, which is non-asymptotic and spectrum-adaptive. It is demonstrated that the two assumptions required for the conclusion, that the noise needs to be bounded by the smallest eigenvalue and the $(n-p)$th eigengaps, should be applicable for many real-world scenarios. The proof is accomplished by first bounding the perturbation with a contour integration, and then applying a developed contour bootstrapping and designing a bespoke counter. Empirical study shows the results are validated by some real-world cases, and are tighter than the EY–N bound.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- This paper studies a foundamental problem with potentially wide downstream applications, including optimization, scientific computing, signal processing, etc.\n- The theoretical results are claimed to be the first non-asymptotic spectral-norm perturbation bounds for low-rank approximations of noizy matrix inversion, which provide a $sqrt(n)$-level gain compared to the previous EY-N bound.\n- Empirical results validate that the proposed bound is tighter than the previous EY-N bound.\n\nWeaknesses:\n- In the main theoretical results, besides that the noise matrix bounded by the spectrum, the matrix $A$ is required to be real symmetric and positive semi-definite, and the noise matrix $E$ is required to be symmetric. These requirements will also limit the application of the results.\n- Only several specific matrices are checked in the empirical study, including the 69-by-69 1990 US Census covariance matrix, the 1083-by-1083 BCSSTK09 stiffness matrix, and synthetic matrices from discretised Hamiltonians. Some examples with real-world applications will better demonstrate the value of this work.\n\n==========update after rebuttal============\n\nThe theoretical results are extended to general symmetric matrices in the Appendices.\n\nQUESTIONS:\nNone.\n\nLIMITATIONS:\nThe authors mentioned several limitations in the manuscript, including the spectral quality dependencies and limitation on static matrices.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper prove new perturbation bounds for $\\left\\| \\tilde{A}_p^{-1} - A_p^{-1} \\right\\|$, where $A_p$ is the best rank $p$ approximation of $A$, and $\\tilde{A} = A + E$ is a noisy version of $A$. Based on techniques involving contour integrals, the authors prove a new bound that is spectral-gap dependent and sharper than previous versions. Previously, using basic classical techniques such as the Neumann expansion and the Eckart–Young–Mirsky theorem, one can prove that \n$$\\left\\|\\left(\\tilde{A}^{-1}\\right)_p-A_p^{-1}\\right\\| \\leq 8\\|E\\|/3\\lambda\\_n^2 + 2/\\lambda\\_{n-p}$$\nUnder the additional assumption that $4\\|E\\|\\leq \\delta\\_{n-p} = \\lambda\\_{n-p} - \\lambda\\_{n-p+1}$,  the authors improve the bound to \n$$\\left\\|\\left(\\tilde{A}^{-1}\\right)_p-A_p^{-1}\\right\\| \\leq \\frac{4\\|E\\|}{\\lambda\\_n^2}+\\frac{5\\|E\\|}{\\lambda\\_{n-p} \\delta\\_{n-p}} $$\nThe key improvement is that as $\\|E\\|\\to 0$, the upper bound also goes to zero.\n\nSTRENGTHS AND WEAKNESSES:\nThis paper is a pretty easy read: both the results and the proof are clearly presented. The authors basically prove a sharper perturbation bound using techniques in complex analysis. These techniques are fairly common in the numerical linear algebra literature.  However, I do think the main results presented in this work are some what expected: the new bound on the error takes into account the spectral gap, spectral decay, so it is naturally \"sharper\", because it is more specific. This is also my view of the experimental section, which shows that the author's bound is tighter that classical ones. \n\nOne other concern I have is that this is one of the only papers I've reviewed at this conference that derives a purely theoretical result for numerical linear algebra.  If judged as a applied math paper, I think the theoretical results are a bit incremental, since the baseline result here can be derived in just 3 or 4 lines, based on very classic ideas. In fact, there is not really a baseline paper to compare to, other than the bound that one can derive by using standard techniques. But on the other hand, it is also hard to view this as a machine learning paper, because the lack of direct connections to applications. Therefore, I would not recommend accepting this paper.\n\nQUESTIONS:\nn/a\n\nLIMITATIONS:\nn/a", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies perturbation bounds for low-rank matrix inverse approximation under noise. Specifically, given a symmetric matrix $A$, and noise $E$, this work studies the spectral-norm of the matrix $\\tilde{A} - A$, where $\\tilde{A} = A+E$. The goal here is to derive sharp asymptotic error bounds that scales appropriately with the eigengap, spectral decay, and noise alignment for even the smallest non-zero eigenvalues of A. The main paper studies this problem when $A$ is positive-definite, and in the appendix the authors study the general symmetric case.  The core of the analysis is the novel use of contour integral representations of matrix functions and how these are used to provide tighter error bounds. The theoretical results are complemented with empirical analysis of 2 real and one synthetic matrix, which demonstrates how well the error bounds here mimic the true error outperforming Eckart-et al-Neumann bound.\n\nSTRENGTHS AND WEAKNESSES:\nI should preface by saying that I am not an expert in the field of random matrix theory. \n### Strengths\n- I believe the application of contour integrals for finding the smallest eigenvalues is novel, and opens the door for using this technique across other applications including but not limited to approximation of the Schatten-norms, spectral density estimation, among others.\n\n### Weaknesses\n- Equation (2) is derived in Section F. upon checking I noticed that the authors are using Weyl's inequality to bound the eigengap between $||\\tilde{A}^{-1} - \\tilde{A}||$ as $$||\\tilde{A}^{-1} - \\tilde{A}|| \\geq \\tilde{\\lambda}^{-1}_{n-p} - {\\lambda}\\_{n-p}^{-1}.$$ \n\nHowever, as far as I know Weyl's is $\\max_{i \\in [n]} |\\lambda_i^A - {\\lambda}_i^B| \\leq ||A-B||$. Are the authors using some specific Weyl's inequality? If yes, please cite it accordingly, or if there is any assumption to this statement, it should be stated in the theorem statement and not assumed in the proofs.\n\n- While the authors state the bounds are for PSDs in the main paper, the first line in Setup (line 95) assumes for $A \\in \\mathbb{R}^{n\\times n}$ that $\\lambda_1 \\geq \\ldots \\geq \\lambda_n > 0$ which is PD matrix. While $E$ is assumed to be symmetric, in Step 1 within Proof overview, eigenvalues of $\\tilde{A}$ are also assumed to be positive, which confused a bit. What is the point when $E$ cannot perturb the eigenvalues of $A$ to result in indefinite $\\tilde{A}$?\n\n- In the same line in Step 1 of Proof overview, the authors use this nice trick to swap the norm and the integrals. However $dz$ should be $|dz|$, which complicates the whole proof (I am unsure if the proof breaks though). The same issue exists in appendix $A$. Is $dz$ and $|dz|$ interchangable in this context? Surely not, right, as one has to hand two cases $\\geq 0$ and $< 0$ separately?\n\n- In line 803, the authors state that there exists some permutation of $[n]$ such that $\\sigma_{n-p} = \\lambda_{\\pi(p)}$. The $\\lambda$ here should be an absolute value, as otherwise, yes there is some perturbation when this is true, but such perturbation might not lead to consecutive values be guaranteed to be $> 2||E||$, which is a required assumption in Theorem A.1\n\nQUESTIONS:\nPlease refer to the Weakness section above. Given those are resolved, I would like to understand what the authors think would this result mean for problems like the Schatten-1 norm or the spectral density estimation under noisy environments.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-18"}