Title: The Price of Sparsity: Sufficient Conditions for Sparse Recovery using Sparse and Sparsified Measurements
Venue: NeurIPS 2025
Accepted: yes
Keywords: Sparse linear regression

Data sparsification

Sparse support recovery

Sparse data

Compressed sensing

High-dimensional statistics

Information-theoretic limits
Abstract: We consider the problem of recovering the support of a sparse signal using noisy projections. While extensive work has been done on the dense measurement matrix setting, the sparse setting remains less explored. In this work, we establish sufficient conditions on the sample size for successful sparse recovery using sparse measurement matrices. Bringing together our result with previously known necessary conditions, we discover that, in the regime where $ds/p \rightarrow +\infty$, sparse recovery using a sparse design exhibits a phase transition at an information-theoretic threshold of $n_{\text{INF}}^{\text{SP}} = \Theta\left(s\log\left(p/s\right)/\log\left(ds/p\right)\right)$ for the number of measurements, where \(p\) denotes the signal dimension, $s$ the number of non-zero components of the signal, and $d$ the expected number of non-zero components per row of measurement. This expression makes the price of sparsity explicit: restricting each measurement to $d$ non‑zeros inflates the required sample size by a factor of $\log{s}/\log\left(ds/p\right)$, revealing a precise trade‑off between sampling complexity and measurement sparsity. Additionally, we examine the effect of sparsifying an originally dense measurement matrix on sparse signal recovery. We prove in the regime of $s = \alpha p$ and $d = \psi p$ with $\alpha, \psi \in \left(0,1\right)$ and $\psi$ small that a sample of size $n^{\text{Sp-ified}}_{\text{INF}} = \Theta\left(p / \psi^2\right)$ is sufficient for recovery, subject to a certain uniform integrability conjecture, the proof of which is work in progress.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies sparse recovery with the added structure that the design matrix itself may be sparse (i.e. sparse measurements). The authors derive the information-theoretic threshold which delineates the sample complexity for when asymptotic support recovery of the signal is possible/impossible. Under a certain conjecture, the authors also describe how one might sparsify a dense measurement matrix and still obtain signal recovery; this can be useful from the standpoint of storage and computation resources.\n\nSTRENGTHS AND WEAKNESSES:\n__Strengths__\n\nThe paper studies a canonical setting and the problem of studying sparse measurement matrices is very interesting. The problem is clearly of interest to the NeurIPS community, and the paper provides interesting practical implications as well. The paper's main result, a sharp characterization of the price of sparsity, is quite fundamental and nicely explains to what degree information in the measurement matrix is important. The writing is very clear, and the arguments are transparent. I especially appreciated the proof sketch of Theorem 1. The paper is also very honest about its limitations. I enjoyed reading this paper.\n\n__Weaknesses__\n\nIn my view, the only weaknesses of the paper are the ones the authors themselves identify. Strictly speaking, the upper and lower bounds are not directly comparable since they apply to different settings. Yet I, like the authors, don't believe the gap is fundamental and could be closed. Of course, it does weaken the ultimate conclusions of the paper. Specifically, the conclusion of Theorem 1 + Corollary 2 cannot be interpreted as a strict characterization of the information-theoretic threshold. As the authors note in Remark 2.2, the parameter spaces between upper and lower bounds are different, and the notion of recovery is also different. Likewise, the results regarding sparsification are conditional on a conjecture. It would be really great to have all this cleaned up (especially concerning Theorem 1 + Corollary 2), especially for a top venue like NeurIPS. However, I am sympathetic to the authors and also am willing to speculate the gap is not fundamental.\n\nQUESTIONS:\n- The authors are transparent about the gaps. Though it may not be possible to quickly resolve them, I think it would substantially improve the paper if the authors could describe, even at a high-level, what the difficulties are in closing the gaps. Such a description would be helpful in convincing even the skeptical reader that the issue is not so serious (as the authors believe).\n\n- The authors need to acknowledge and cite the missing data literature, as this is very closely related to the sparsification considered in the latter part of the paper. For example, the authors need to discuss Loh & Wainwright, \"High-dimensional regression with noisy and missing data: Provable guarantees with nonconvexity\" as well as discuss related references.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-06-18"}

comments: SUMMARY:
The paper studied the problem of support recovery of a sparse vector (signal) of dimension p from n linear projections, under the less common but interesting assumption that each projection vector is sparse (with d non-zero elements on average). In other words, support recovery of a sparse vector using a sparse sensing matrix. 
The main results are:
 (1) A sufficient condition on the number of measurements guaranteeing recovery using the MLE in terms of p, d, and s (the number of non-zero entries in the signal) when ds/p-> infinity. Combined with previous results (Wang, Wainwright, and Ramchandran, 2016), this new condition yields an information-theoretic phase transition. 
 (2) Sufficient conditions on the number of samples n when a dense sensing matrix undergoes a sparsification process.

STRENGTHS AND WEAKNESSES:
Strengths:
- The topic is within the scope of NeurIPS. 
- The motivation for considering a sparse sensing matrix to save computation is reasonable. 
- The contributions are significant.

Weaknesses:
- As the authors mentioned, the analysis in the paper leaves several challenges (mostly discussed in Remark 2.2). These challenges could probably be resolved with relatively little work, so it appears that a more complete work is within reach.
- I suggest explaining in the introduction the connection between support recovery and MSE or other natural error criteria.  
- There are a few undefined terms: H(\alpha), ||s||_0, in the abstract, it is unclear that n_{inf}^{sp} refers to the number of measurements,

QUESTIONS:
Referring to an extended setting outlined in Lines 67-68, it is unclear what sparsity means.

LIMITATIONS:
Yes. Especially in Remark 2.2.
TITLE: Official Review
CLARITY: 4
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-01

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors study how taking a sparse measurement matrix in sparse regression (or compressed sensing) problems affects the sample complexity of these problems. Previous work showed that, at a given level of sparsity, recovery of the hidden signal is impossible with too few statements. This work shows a matching positive information-theoretic result, that above this number of samples the (computationally intractable) maximum likelihood estimator does asymptotically recover the (support of the) signal. Thus this result settles the information-theoretic threshold of sparse regression with a sparse measurement matrix. To account for the possibility that the measurement matrix is prescribed rather than designed by the user, the authors also prove a similar result (conditional on a technical conjecture) for a random sparsification of a given dense measurement matrix. That is, a measurement is *generated* with a dense measurement matrix, but *recovered* by a sparsification of that matrix.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: The main result is well-motivated and a valuable addition to the literature on this subject that closes a clear open problem of pinning down the information-theoretic threshold for this problem. The authors clearly describe the state of affairs and their contribution, and the background information on the problem is presented very nicely and leaves (to me) no lingering concerns. The idea of sparsification is also a nice one and gives a valuable complement to the first main result about sparse measurement matrices. The authors are also very clear and explicit about even minor limitations and restrictions of their result, which I appreciated.\n\nWeaknesses: The proof techniques for analyzing the MLE are fairly standard, and it seems (though I have not gone through them in detail) that while sparsity changes things it does not make these calculations require fundamentally different tools. Also, a minor point: the introduction of sparsification is a little bit unclear, and I would present this problem in the way that I describe it in the summary, emphasizing that the point is to consider using a different matrix for measuring versus recovering.\n\nQUESTIONS:\n- Since the idea of sparsification is an algorithmic one and the precise means of sparsification is presumably flexible, could other ways of sparsifying a matrix rather than completely at random that have stronger guarantees on spectral properties they preserve be useful? This area has been widely studied with the advent of general-purpose matrix concentration inequalities, and for example many examples are discussed in Tropp's monograph \"An introduction to matrix concentration inequalities\".\n\n- What does \"slow decay of sparsity\" mean in regard to the computational threshold (p2), and could the authors be slightly more specific about the status of the algorithmic threshold? Is the algorithmic threshold for a sparse measurement matrix actually known, and known to be comparable to the performance of LASSO? I assume not quite, since (to my knowledge) the question of the overlap gap property under sparsity of the measurement matrix has not been addressed before?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies sparse signal recovery under the sparse measurement matrix assumption. The paper shows an information-theoretic sample complexity threshold for successful sparse recovery, i.e., the phase transition happens at $n^{SP}_{INF}= \\Theta(s\\log{(p/s)}/\\log{(ds/p)})$ where d is the non-zero entries of the measurement matrix, s is the non-zero entries of the signal and p is the dimension of the signal. Furthermore, the paper shows a simplified sample complexity for a specific regime where $s$ and $d$ is asymptotically proportional to $p$.\n\nSTRENGTHS AND WEAKNESSES:\nStrength: The problem set up (i.e., the sparse measurement assumption) looks interesting. The result in Theorem 1 looks solid and can be related to previous work.\nWeakness: The major concern I have is around the assumption of $\\sigma^2=\\Theta (1)$ and therefore are dropped in later results (e.g. Corollary 2 and Theorem 3). The noise level $\\sigma^2$ plays a very important role in SNR and SNR is the major key component (if not the most important)  that determines how difficult the recovery problem is and thus how much sample would be roughly needed.\n If we assume $\\sigma^2=\\Theta (1)$, then $SNR \\rightarrow \\infty$ when $ds/p \\rightarrow \\infty$ which is an easy task. Previous works that study the linear problem under asymptotic regime mostly need to control $SNR=\\Theta(1)$ or at least the results depend on $||\\beta^*||$ or $\\sigma^2$ so that we can derive the corresponding sample complexity when $SNR=\\Theta(1)$. For example, in [13] (referenced in this paper), the abstract points out that \"high signal to noise ratio $k/\\sigma^2 = \\Omega (1)$ regime\". In the light of this, I would recommend relaxing the assumption of $\\sigma^2=\\Theta(1)$ and revising the introduction and the results other than Theorem 1 accordingly.\n\nQUESTIONS:\nBesides the issue I mentioned in above, I would like a summary table that compares the sample complexity for different cases among the previous work and this paper.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}