Sparse Bayesian Generative Modeling for Compressive Sensing

Authors: benedikt  böck
sadaf  syed
wolfgang  utschick
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 18:01:03

Abstract: This work addresses the fundamental linear inverse problem in compressive sensing (CS) by introducing a new type of regularizing generative prior. Our proposed method utilizes ideas from classical dictionary-based CS and, in particular, sparse Bayesian learning (SBL), to integrate a strong regularization towards sparse solutions. At the same time, by leveraging the notion of conditional Gaussianity, it also incorporates the adaptability from generative models to training data. However, unlike most state-of-the-art generative models, it is able to learn from a few compressed and noisy data samples and requires no optimization algorithm for solving the inverse problem. Additionally, similar to Dirichlet prior networks, our model parameterizes a conjugate prior enabling its application for uncertainty quantification. We support our approach theoretically through the concept of variational inference and validate it empirically using different types of compressible signals.

Main paper: 
 introduction: 
 Research in CS has shown that it is possible to reduce the number of measurements far below the one determined by the Nyquist sampling theorem while still being able to extract the information-carrying signal from the acquired observations. The fundamental problem in CS is an ill-posed linear inverse problem, i.e., the goal is to recover the signal x * ∈ R N of interest from an under-determined set of measurements y ∈ R M with M ≪ N , related byEQUATIONwhere x * is compressed by the measurement matrix A and potentially corrupted by additive noise n ∼ N (0, σ 2 n I). Since the under-determined observation y does not carry enough information alone to faithfully reconstruct x * , additional prior (or model) knowledge about x * is required to make the inverse problem "well-posed". Classical CS algorithms such as Lasso regression or orthogonal matching pursuit (OMP) address this problem by incorporating the model knowledge that x * is sparse or compressible with respect to some dictionary [1] . Nowadays, modern deep learning (DL)based approaches such as unfolding algorithms, generative model-based CS and un-trained neural networks (NNs) expand the possibilities by learning prior knowledge from a training set or designing the network architecture to be biased towards a certain class of signals [2] [3] [4] [5] [6] . These approaches typically require a (potentially large) training set of ground-truth data samples, or their architecture is specifically biased towards natural images. However, in many applications, ground-truth training data might not be easily accessible. In, e.g., electron microscopy, the amount of electron dose has to be restricted to not induce damage on the probe resulting in low-contrast noisy data samples [7] . The sensors in wearable electrocardiography (ECG) monitoring devices generally provide noisy signals with artifacts [8] , which limits the ability to learn from patient-specific data in real-world settings. Another example is the wireless 5G communication standard, where mobile users receive compressed and noisy so-called channel observations on a frequent basis (cf. [9] ) while acquiring ground-truth channel information requires costly measurement campaigns. Thus, in many applications, it is either impossible or prohibitively expensive to collect lots of ground-truth training data while corrupted data is readily available. This highlights the necessity for methods that can learn from corrupted data samples.In this paper, we propose a new learnable prior for solving the inverse problem (1) , which can learn from only a few compressed and noisy data samples and, thus, requires no ground-truth information in its training phase. Additionally, it applies to any type of signal, which is compressible with respect to some dictionary. Our approach shares similarities with generative model-based CS [4, 10] . There, a generative model is first trained to capture the exact underlying prior distribution p(x) of the signal x * of interest. It then serves as probabilistic prior to regularize the inverse problem (1) . Similarly, classical CS algorithms like Lasso regression also impose a probabilistic prior on the sparse representation of x * with respect to some dictionary [11] . In contrast to modern generative models, however, these priors do not have any generation capabilities but solely bias the inverse problem (1) towards sparse solutions. This observation indicates that a probabilistic prior does not necessarily need to capture the exact prior distribution p(x) to effectively regularize (1) . The proposed model in this work builds upon this insight and forms a trainable but simultaneously sparsity-inducing prior.For that, we aim to combine the adaptability of generative models to training data with the property of many types of signals x * to be compressible with respect to some dictionary. As a result, our model learns statistical information in the signal's sparse/compressible domain. Examples of signals with a specific statistical structure in their compressible representation include piecewise smooth functions and natural images, whose wavelet coefficients approximately build a connected sub-tree or wireless channels, which are burst-sparse in their angular domain [12, 13] .Related Work. Early work on CS, which considers statistical structure in the wavelet domain of images, is given in [14, 15] . Based on the theoretical foundation in [16] and the concept of SBL [17, 18] , these papers introduce a hierarchical Bayesian model, which is used to apply variational inference and Markov chain Monte Carlo (MCMC) posterior sampling to solve (1) . Training Gaussian mixture models (GMMs), i.e., classical generative models, from compressed image patches of one or a few images has been analyzed in [19] [20] [21] [22] . In this line of research, however, the GMM is fit directly in the pixel domain. Compressive dictionary learning represents a different line of research aiming to learn the dictionary from solely compressed data [23] [24] [25] [26] , and strongly bases on the dictionary learning method K-SVD [27] . More recently, variational autoencoders (VAEs) and generative adversarial networks (GANs) have been studied in the context of CS [4, 10] . There, the VAE/GAN is used to solve the inverse problem (1) by constraining the signal x * of interest to lie in the range of the generative model instead of being sparse with respect to some dictionary. In this context, AmbientGAN as well as CSGAN are extensions that loosen up the training set requirements and can learn from corrupted data [28, 29] . Another related topic is the ability of some generative models, i.e., diffusion models, VAEs and GMMs, to provide an approximation of the conditional mean estimator (CME) [30] [31] [32] . In the case of the latter two, the CME is represented as a tractable convex combination of linear minimum mean squared error (MSE) estimators by exploiting their conditional Gaussianity on a latent space that determines these estimators' means and covariances.Our main contributions are as follows:• We introduce a new type of sparsity-inducing generative prior for the inverse problem (1) , which differs from classical CS algorithms due to its ability to learn from data. On the other hand, it also differs from other modern NN-based approaches due to its ability to promote sparsity in the signal's compressible domain. Moreover, it can learn from a few corrupted data samples and, thus, requires no ground-truth information in its training phase.• We theoretically underpin our approach by proving that its training maximizes a variational lower bound of a sparsity inducing log-evidence.• Building on the notion of conditional Gaussianity, we introduce two specific implementations of the proposed type of prior based on VAEs and GMMs, which do not require an optimization algorithm in their inference phase and come with computational benefits.• By exploiting the shared property with Dirichlet prior networks to parameterize a conjugate prior, we demonstrate how our approach can be applied for uncertainty quantification.• We validate the performance on datasets containing different types of compressible signals. 12 Background and MethodNotation. The operations b -1 , √ b and |b| 2 denote the respective element-wise operation for the vector b. Moreover, diag(B) represents the vectorized diagonal of the matrix B, while diag(b) denotes the diagonal matrix with the vector b on its diagonal. 
problem formulation: 
 We consider the typical CS setup, in which we measure N -dimensional ground-truth samples x i by a known measurement matrix A ∈ R M ×N (or A i ∈ R M ×N ) with M ≪ N , where each observation is potentially corrupted by noise n i drawn from additive white Gaussian noise (AWGN) n ∼ N (0, σ 2 n I). We assume all x i to be compressible with respect to a known dictionary matrix D ∈ R N ×S , i.e., y i = ADs i + n i with x i = Ds i and s i ∈ R S being approximately sparse. All s i are assumed to be independent and identically distributed (i.i.d.), i.e., s i ∼ p(s), where p(s) is unknown with s exhibiting non-trivially dependent entries. Typical signals which fit into this category are, e.g., natural images, piecewise smooth functions, and wireless channels (cf. Section 1). Our approach in this work allows to either solely have access to corrupted training observations Y = {y i } Nt i=1 or training tuples with ground-truth informationG = {(s i , y i )} Nt i=1 or W = {(x i , y i )} Nt i=1 .We first train the proposed model using Y (or, alternatively, G or W) to serve as an effective prior for (1). Our goal is then to estimate a ground-truth signal x * of a newly observed y. We also define X = {x i } Nt i=1 and S = {s i } Nt i=1 .In SBL for CS, the idea for solving the inverse problem ( 1) is to assign a parameterized prior to s * , i.e., the compressible representation of the signal x * of interest with x * = Ds * [18] . It assumesy|s * ∼ p(y|s * ) = N (y; ADs * , σ 2 I), s * ∼ p γ (s) = N (s; 0, diag(γ)).(2) Given a single observation y, the parameters γ (and sometimes σ 2 ) are estimated by an expectationmaximization (EM) algorithm maximizing the corresponding log-evidence log p γ (y) implicitly defined by (2) . After that, it is utilized that p γ (s) forms a conjugate prior of p(y|s * ) resulting in a closed-form posterior p γ (x|y) providing all necessary information to estimate x * . In [18] , a variational interpretation of SBL is given, addressing why this approach yields sparse results, even though p γ (s) does not inherently promote sparsity. It is shown that there exists a C > 0 such that for all γ > 0 and sEQUATIONThe function t(s), however, is a well-known improper but sparsity-inducing prior, used as a noninformative prior for scale parameters [33] . Let another statistical model be given by p(y|s * ) (cf. ( 2)) and prior t(s) instead of p γ (s) with implicitly defined log-evidence log π (s) (y). Due to t(s) being improper, the log-evidence log π (s) (y) forms a not normalized but valid log-likelihood. Moreover, this model is sparsity-inducing due to the sparsity promoting characteristics of t(s). Based on (3), it holds that log π (s) (y) ≥ log p γ (y) and, thus, log p γ (y) can be embedded in variational inference, forming a tractable variational lower bound of an intractable log-evidence log π (s) (y) of actual interest. Moreover, applying the EM algorithm to (2) is "evidence maximization over the space of variational approximations to a model (i.e., p(y|s * ) and t(s)), with a sparse, regularizing prior" [18] .GMMs and VAEs are generative models aiming to learn an unknown distribution p(x) from a training set X . Both models represent p(x) as the marginalization with conditionally Gaussian p(x|•) and an additional latent variable k (or z) [34] [35] [36] , i.e.,EQUATIONEQUATIONThe tunable parameters ({ρ k , µ k , C k } K k=1 for GMMs and θ for VAEs) are learned by optimizing the model's log-evidence i log p (f ) (x i ) (f ∈ {GMM, VAE}) over X . In the case of GMMs, this is typically done by an EM algorithm alternating between the so-called e-and m-step. The e-step determines the closed-form posteriors p t (k|x i ) (for all i) via the Bayes rule using the model's parameters in the tth iteration. The m-step then updates the model's parameters by maximizing i E pt(k|xi) [log p(x i , k)]. In case of VAEs, however, the posterior p θ (z|x) (and log p (VAE) (x)) are intractable and the EM algorithm cannot be applied. Therefore, a tractable distribution q ϕ (z|x) = N (z; µ ϕ (x), diag(σ2 ϕ (x)) with variational parameters ϕ is introduced, which approximates p θ (z|x), and µ ϕ (x) and σ 2 ϕ (x) are generated by a NN encoder. Equivalently, µ θ (z) and C θ (z) are generally realized by a NN decoder. The objective to be maximized is the evidence lower bound (ELBO) L(θ, ϕ) serving as a tractable lower bound for the intractable log-evidence i log p θ (x i ), i.e.,EQUATIONwith D KL (•) being the Kullback-Leibler (KL) divergence. Generally, the GMM's and VAE's training critically depends on iteratively characterizing and updating their posteriors p t (k|x i ) and q ϕ (z|x i ).The goal of this section is to derive a class of generative models, for which we can guarantee that, on the one hand, it is sparsity inducing, but on the other hand, it is trainable and can learn from solely compressed and noisy observations Y resulting in an effective probabilistic prior for (1).To incorporate the bias towards sparsity, we start with SBL discussed in Section 2.2 and combine it with the VAE's and GMM's main principle for their adaptability to complicated distributions, i.e., introducing a latent variable z (or k) on which we condition with a parameterized Gaussian (cf. ( 4) and ( 5)). 2 More specifically, we exploit the Gaussianity of p γ (s) in (3) and modify it to a parameterized Gaussian conditioned on some latent variable z with arbitrarily parameterized p δ (z) while explicitly keeping its mean zero and its covariance matrix diagonal, i.e., p θ (s|z) = N (s; 0, diag(γ θ (z))). The resulting set of statistical models referred to as sparse Bayesian generative models, is given by y|s ∼ p(y|s) = N (y; ADs, σ 2 I), s|z ∼ p θ (s|z) = N (s; 0, diag(γ θ (z))) , z ∼ p δ (z). (7) Training principle. Our proposed training scheme is independent of specific realizations for γ θ (z) and p δ (z), which is why they are kept general in this section. Specific parameterizations are discussed in Section 3.2 and 3. (z) = 1 σ 2 D T A T AD + diag(γ -1 θ (z)) -1 , µ s|y,z θ (z) = 1 σ 2 C s|y,z θ (z)D T A T y (8)equal those in SBL [18] . The equivalent formulas for σ 2 = 0 are given in Appendix C. By considering the decomposition p θ,δ (s, z|y) = p θ (s|z, y)p θ,δ (z|y), we conclude that our proposed set of statistical models in (7) exhibits posteriors, which are partially tractable in closed form independent of any specifics of γ θ (z) and p δ (z). The remaining p θ,δ (z|y) resembles the standard GMM's and VAE's posterior, and we can either apply the EM algorithm or variational inference to maximize the corresponding log-evidence. Both will be discussed in Section 3.2 and 3.3, respectively.Estimation scheme. The goal after the training is to use the learned statistical model in (7) for regularizing the inverse problem (1) and estimating x * from a newly observed y. Generally, the CME E[x|y] is a desirable estimator due to its property of minimizing the MSE. We utilize insights from [32, 22] , which derive approximate closed forms of the CME using VAEs and GMMs. More precisely, by using the law of total expectation, we approximate the CME byEQUATION)The inner expectation is given by the learned µ s|y,z θ (z) in ( 8), while, depending on its specific characteristics, p θ,δ (z|y) also exhibits a closed form or a variational approximation has been learned during training (cf. Section 3.2), with which we approximate the outer expectation by a Monte-Carlo estimation. Based on the perception-distortion trade-off, the CME is not optimal from a perceptual point [37] , and deviating estimators from the CME approximation might be beneficial. More detailed descriptions of the CME approximation as well as further estimators based on the specific implementations in Section 3.2 and 3.3 are given in Appendix G.3 Theoretical Analysis and Specific Parameterizations 
sparsity guarantees: 
 In Section 2.4, we motivate constraining p θ (s|z) to be a zero mean conditional Gaussian with diagonal covariance matrix by SBL and its sparsity inducing property. In the following, we rigorously show that constraining p θ (s|z) in this manner is sufficient to maintain the sparsity inducing property for any statistical model following (7) despite the additionally included latent variable z. More precisely, we show that any statistical model in (7) with some specified parameterized p δ (z) and γ θ (z) exhibits a log-evidence that is interpretable as a variational lower bound of a sparsity inducing log-evidence. For that, we establish the following theorem. Theorem 3.1. Let p θ (s|z) = N (s; 0, diag(γ θ (z)) (i.e., it is defined according to (7)), let z be either continuous or discrete and finite, and let γ θ (z) > 0. Then, there exists a constant C > 0 such that for all s, θ, δ with any arbitrary distribution p δ (z)EQUATION)The integral corresponds to a summation for discrete and finite p δ (z).The proof is given in Appendix A. Based on Theorem 3.1, it holds thatEQUATIONwith log π (s) (y) being the log-evidence of y|s ∼ p(y|s) and the improper but sparsity promoting prior t(s), whereas log p θ,δ (y) is implicitly defined by (7) . Consequently, we can interpret the maximization of log p θ,δ (y) over (θ, δ) as the evidence maximization over the space of variational approximations to a model with a sparsity inducing and regularizing prior. However, contrary to classical SBL, we have the additional degree of freedom to choose the specifics of γ θ (z) and p δ (z) without any constraint while maintaining the connection to sparsity.The integral version of p θ,δ (s) in (10) strongly resembles the VAE's decomposition of p(x) in ( 5).In fact, by constraining p δ (z) = p(z) = N (z; 0, I), the distribution p θ,δ (s) = p θ (s) corresponds to a subset of all possible distributions covered by the VAE decomposition in (5) . This motivates to choose p δ (z) in this exact manner and realize γ θ (z) as the output of a NN decoder. Generally, we aim to optimize the log-evidence i log p θ (y i ) in (11) with respect to θ solely given compressed and noisy observations Y. However, equivalent to VAEs, the log-evidence i log p θ (y i ) is intractable for p(z) = N (z; 0, I), which is why a tractable lower bound has to be derived. A tractable lower bound on the log-evidence of a statistical model with latent variables is obtained by subtracting the KL divergence between the conditional distribution of the latent variables given the observations y Encoder q ϕ (z|y) p θ (s| z) and a tractable variational distribution from the log-evidence (cf. ( 6)). For the model in (7) and i.i.d. training data, this results inDecoder AD + ⊙ ⊙ + y N (0, I) N (0, I) N (0, σ 2 I) µ ϕ σ ϕ z √ γ θ sEQUATIONEQUATIONwhere the derivation of ( 13) from ( 12) is given in Appendix B. Based on insights from the training principle explained in Section 2.4, the variational posterior q ϕ (z, s|y i ) in ( 13) can be simplified to q ϕ (z, s|y i ) = p θ (s|z, y i )q ϕ (z|y i ) with closed-form p θ (s|z, y i ) and only requires variational parameters ϕ for q ϕ (z|y i ) to be tractable. As a result, the adapted ELBO LEQUATIONTo ensure the training of standard VAEs to be tractable, E q ϕ (z|yi) [•] is generally approximated by a single-sample Monte-Carlo estimation [36] . We build on the same strategy and approximate ( 14) byEQUATION) with zi ∼ q ϕ (z|y i ) and E p θ (s| zi,yi) [log p(y i |s)] exhibiting a closed-form solution detailed in Appendix D. In addition, D KL (p θ (s| zi , y i )||p θ (s| zi )) is the tractable KL divergence of two multivariate Gaussians, and D KL (q ϕ (z|y i )||p(z)) matches the KL divergence from standard VAEs (cf. [36] ). Both are specified in Appendix E. The adapted ELBO in (15) resembles the classical ELBO of ordinary VAEs with a modified reconstruction loss and an additional KL divergence. Equivalent to ordinary VAEs, we model q ϕ (z|yi ) = N (z; µ ϕ (y i ), diag(σ 2 ϕ (y i ))) and realize (µ ϕ (•), σ 2 ϕ (•)) by a NN encoder, and γ θ (•) by a NN decoder. The adapted ELBO in (15) forms a differentiable and variational approximation of the model's log-evidence i log p θ (y i ). Based on Theorem 3.1 and (11), it holds thatEQUATIONThus, L(θ,ϕ) serves as a variational lower bound of a sparsity-inducing log-evidence. We denote the resulting VAE as Compressive Sensing VAE (CSVAE), and its schematic is presented in Fig. 1 . It resembles the vanilla VAE with the main difference that the encoder takes a compressed observation y as input and the decoder solely outputs conditional variances γ θ ( z) of s|z. In case of varying measurement matrices A i for each training, validation and test sample, we use the least-squares estimate x(LS) i = A T i (A i A T i ) -1 y i insteadBy considering discrete and finite p δ (z) in (7) , i.e., p δ (z = k) = p δ (k) = ρ k , we observe that the resulting p θ,δ (s) strongly resembles the GMM's decomposition of p(x) in (4). This motivates to choose p δ (z) in exactly this manner and p θ (s|z) = p(s|k) = N (s; 0, diag(γ k )), i.e., s is modelled as GMM with zero means and diagonal covariance matrices. Equivalent to CSVAEs in Section 3.2, the distribution p(s|k, y) contained in the model's posterior p(k, s|y) = p(s|k, y)p(k|y) exhibits a closed-form solution (cf. ( 8)). Moreover, due to the linear relation between y and s in (7) and the assumption that s|k is Gaussian, y|k is also Gaussian with zero mean and covariance matrixC y|k k = ADdiag(γ k )D T A T + σ 2 I .(17) and p(k|y) is computable in closed form by Bayes, i.e., p(k|y) = (p(y|k)p(k))/( k p(y|k)p(k)). Thus, p(k, s|y) exhibits a closed form, and an EM algorithm can be applied, whose e-step coincides with computing p(k, s|y). Based on the extended EM algorithm in [22] , where the authors fit a vanilla GMM (4) in the pixel domain using compressed image patches, we formulate an m-step tailored to our model (7) . We denote the resulting model as Compressive Sensing GMM (CSGMM). The m-step aims to update the CSGMM's parameters {ρ k , γ k } by optimizing i E pt(k,s|xi) [log p(x i , s, k)], whose closed-form solution is given in the following lemma. Lemma 3.2. Let the statistical model in (7) be given with discrete p δ (z = k) = p(k) = ρ k and γ θ (z) = γ k , and let Y contain i.i.d. noisy and compressed observations y i . Given p t (k|y i ) and p t (s|k,y i ) = N (s; µ s|yi,k k,t , C s|yi,k k,t) in the tth iteration of an EM algorithm, the updates of{ρ k , γ k } γ k,(t+1) = yi∈Y p t (k|y i )(|µ s|yi,k k,t | 2 + diag(C s|yi,k k,t))EQUATION)form a valid m-step and, thus, guarantee to improve the log-evidence in every iteration.The proof of Lemma 3.2, the CSGMM's training with ground-truth data S and pseudo-code are outlined in Appendix F, H and N (cf. algorithm 4), respectively.From a broader perspective, our proposed algorithm represents two consecutive stages:1) Choose a dictionary D, with respect to which a general signal set C of interest is compressible (e.g. a wavelet basis for the set of natural images). 2) Learn the statistical characteristics of the specific signal subset S ⊂ C of interest in its compressible domain (e.g. the subset of handwritten digits).Due to the strong regularization of stage 1) constraining the search space in stage 2), our model can learn from a few corrupted samples Y. To ensure the search space to be within the set of compressible signals with respect to D, we enforce s|z in (7) and, thus, s to have zero mean (cf. Theorem 3.1).Limitation. On the one hand, this restriction regularizes the problem at hand. On the other hand, it generally introduces a bias, which potentially prevents perfectly learning the unknown distribution of s as it is not always possible to decompose a distribution in this way. As a result, the proposed model is biased towards capturing the sparsity-specific features from Y and being an effective prior for (1) rather than learning a comprehensive representation of the true p(s).G θ : z → x = G θ (z) (e.g., a GAN or VAE) with z ∈ R F , x ∈ R N and F ≪ N [4].Arguably, the notion of compressibility is still included in this setup, since it assumes that every x can be perfectly encoded by only a few F values. Moreover, these models typically assign a simplistic fixed distribution, e.g. p(z) = N (z; 0, I), to their compressible domain. Consequently, while generative model-based CS replaces the dictionary with a learnable mapping G θ and constrains the compressible domain by forcing it to be representable by, e.g., N (0, I), our approach does the exact opposite. We keep the mapping from the compressible into the original domain fixed by a dictionary D and learn a non-trivial statistical model p θ,δ (s) in the compressible domain.Connection to uncertainty quantification. While coming from different motivations, our approach shares similarities with methods from uncertainty quantification, namely so-called prior networks [38] . There, the general idea is to let the NN output the parameters of a conditioned conjugate prior (conditioned on the NN input) instead of directly the quantities of interest (cf. [39, 40] ). As a result, one can use entropy and mutual information measures to quantify different types of uncertainty. Equivalently, our proposed models output the parameters γ θ (z) of a conditioned conjugated prior p θ (s|z), which is why they enable to quantify their uncertainty by the same measures.Computational complexity. It is noteworthy that our algorithm's inference after training requires no optimization algorithm but only consists of, e.g., a feed-forward operation through a comparably small VAE. Thus, our proposed method's signal reconstruction comes with computational benefits and differs in this context from other approaches. On the downside, naively implementing the training requires computing and storing the posterior covariance matrices in (8) , which can lead to a non-negligible computational and memory-related overhead for very high dimensions. To overcome this issue, we derive equivalent reformulations of the CSVAE's and CSGMM's update steps for training that circumvent the explicit computation of these matrices. More precisely, the reformulated update steps solely require the explicit storing and inversion of the observations' covariance matrices (cf. ( 17)), which are typically much lower dimensional. The reformulations are given in Appendix I. (∥ xi -x i ∥ 22 /N ) with xi being the estimation of x i and the SSIM [45] with N test being set to 5000 in any simulation.Baselines & hyperparameters. As non-learnable baselines, we use Lasso [11] and SBL [18] , where we either adjust Lasso's shrinkage parameter on a ground-truth dataset or use the configurations from [4] . As baselines, which can learn from compressed data, we use CSGAN [29] and CKSVD [23] with OMP, sparsity level 4 and 288 learnable dictionary atoms. We solely evaluate CKSVD on piecewise smooth functions due to its requirement of varying measurement matrices for observing the training samples. For CSGAN on MNIST, we use the configuration specified in [29] . 4 In any simulation, we use K = 32 components for the proposed CSGMM and the proposed CSVAE's en-and decoders contain two fully-connected layers with ReLU activation and one following linear layer, respectively. We utilize Adam for optimization [46] , and only consider the CME approximation of the estimators (cf. Section 2.4) since we did not observe notable differences to the alternative estimators in Appendix G. We utilize an overcomplete Daubechies db4 dictionary [47] in all estimators with fixed dictionaries. For all estimators except CSGAN, all images and estimates are normalized and clipped between 0 and 1, respectively. For CSGAN, images are normalized between -1 and 1 following [4, 29] , and for evaluation, test images and their estimations are then re-normalized between 0 and 1. For a more detailed overview of hyperparameter configurations, see Appendix K. 
results: 
 Reconstruction. In Fig. 2 The estimation of CSGANs is restricted to lie on the learned manifold, which has been adjusted based on the training dataset. In consequence, when the test sample x * contains details that cannot be found in the training dataset, CSGANs cannot reconstruct these but rather output a similar but realistic representative from the manifold. On the contrary, our proposed estimation scheme in Section 2.4 is not restricted to a manifold but rather infers conditional distributions (cf. ( 8)) over the whole linear space. Moreover, Fig. 2 c ) and d) demonstrate that the proposed model can effectively learn from only a few hundred compressed and noisy samples. In Fig. 2 i ), CSVAE and CSGMM are compared using different dictionaries types. The overall nMSE performance remains good for different dictionaries except for applying the models directly in the pixel domain.In Fig. 2 f), g) and h), the nMSE over varying M (see f)), varying N t (see g)) and exemplary reconstructed samples (see h)) are shown for the set of piecewise smooth functions. Here, we only evaluated the proposed CSVAE. 5 Compared to the baselines Lasso, SBL, and CKSVD, the CSVAE performs the best in distortion (see f) and g)) as well as perception (see h)). In Fig. 3 a )-e), the same plots are shown for the celebA dataset. Despite the significantly larger dimension, the results are consistent with those for MNIST and the set of piecewise smooth functions. Uncertainty quantification. One possibility to determine the model's uncertainty is to determine the differential entropy h(z|y) of the CSVAE's encoder distribution q ϕ (z|y). Since q ϕ (z|y) is a Gaussian with diagonal covariance matrix (cf. Section 3.2), h(z|y) can be calculated efficiently in closed form. In Fig. 3 Additional results. The results in Fig. 2 and 3 display the performance for CSGAN and the proposed CSVAE and CSGMM trained on solely compressed data. However, all three models can also learn from ground-truth data (cf. Appendix H). In Appendix L, we include this comparison for MNIST, provide results on FashionMNIST, analyze the estimators' robustness, and plot further reconstructions for all datasets. Moreover, in Appendix M, we analyze the runtime for training and reconstruction in more detail and give an overview of the used compute resources. 
conclusion: 
 In this work, we introduced a new type of learnable prior for regularizing ill-conditioned inverse problems denoted by sparse Bayesian generative modeling. Our approach shares the property of classical CS methods of utilizing compressibility, but at the same time, it incorporates the adaptability to training data. Due to its strong regularization towards sparsity, it can learn from a few corrupted data samples. It applies to any type of compressible signal and can be used for uncertainty quantification. While this work focused on setting up the sparse Bayesian generative modeling framework, extensions to learnable dictionaries, circumventing the inversion of the observations' covariance matrices during training, and perception-emphasizing estimators are part of future work. 
 
 Appendix: To prove Theorem 3.1, we restate (3), i.e., there exists a C > 0 such that for all γ > 0 and sEQUATION)Based on (19) , there exists a C > 0 such that for all θ, s, z with γ θ (z) > 0EQUATION)By multiplying both sides in (20) with p δ (z) and integrating over z, we conclude that there exists a C > 0 such that for all θ, s, δ with γ θ (z) > 0EQUATIONindependent of the particular choice of p δ (z). In case of discrete and finite p δ (z) the integral corresponds to a summation.We write out the KL divergence and utilize that p θ (y) is independent of the expectation, i.e.,EQUATIONEQUATIONIn the noise-free case, the closed-form covariance matrix and mean of p θ (s|z, y) are given byEQUATIONµ s|y,z θEQUATIONwith (•) † being the Moore-Penrose inverse [18] .E p θ (s| zi,yi) [log p(y i |s)] = E p θ (s| zi,yi) - 1 2 M log(2πσ 2 ) + 1 σ 2 ∥y i -ADs∥ 2( zi ) and C s|yi, zi θ ( zi ) are the mean and covariance matrix of p θ (s| zi , y i ). Moreover [46] ,EQUATIONwith q ϕ (z|y i ) = N (z; µ ϕ (y i ), diag(σ 2 ϕ (y i ))) and µ j,ϕ (y i ) and σ 2 j,ϕ (y i )) being the jth entry of µ ϕ (y i ) and σ 2 ϕ (y i ), respectively. Additionally, N L denotes the CSVAE's latent dimension.The optimization problem we aim to solve is given byEQUATIONFirst, we reformulate the objective asEQUATIONMoreover, we observe that log p(y i |s) does not depend on {ρ k , γ k } and we can leave it out from the optimization problem. Additionally,EQUATION)where we inserted p(s|k) = N (s; 0, diag(γ k )) (cf. ( 7)) and γ k,j denotes the jth entry of γ k . In the next step, letE pt(s|yi,k) |s j | 2 = |µ s|yi,k k,j | 2 + C s|yi,k k,j,j , where µ s|yi,k k,j and C s|yi,k k,j,j denote the jth entry of µ s|yi,k k,t and the diagonal of C s|yi,k k,tin the tth iteration, respectively. Thus, our optimization problem of interest can be restated asEQUATIONand Lagrangian multiplier ν. Taking the derivative of L with respect to γ m,q and ρ m and setting it to zero leads toEQUATIONand, thus,EQUATIONEQUATIONwhere ( 41) comes from the normalization of k ρ k = 1.CSVAE. The CME approximation in ( 9) with the proposed CSVAE in Section 3.2 is given byEQUATIONwith Z containing samples zi ∼ q ϕ (z|y). Alternatively, one can estimate x * based on a newly observed y in the following way. We use the mean µ ϕ (y) as maximum a posteriori (MAP) estimate based on q ϕ (z|y) and, then, estimate x * byx * CSVAE,MAP = D E p θ (s|µ ϕ (y),y) s|y, µ ϕ (y) = Dµ s|y,µ ϕ (y) θEQUATION)This method is applied in [32] to reduce computational complexity, but also deviates from the CME approximation and potentially improves the perceptual quality of the reconstruction.CSGMM. In case of the proposed CSGMM in Section 3. An alternative estimator of x * is given byEQUATIONwith kMAP = argmax p(k|y) [22] . A pseudo-code for all estimators is provided in Appendix N (cf. algorithm 2-6).The CSGMM as well as the CSVAE can both be trained given ground-truth datasets. For training the CSGMM, this dataset either contains the compressible ground-truth signals s i , i.e., S, or the signals x i themself, i.e., X . On the other hand, for training the CSVAE, both datasets must also contain their corresponding observations y i . Thus, the training dataset must be G or W (cf. Section 2.1).To train the CSGMM, the training goal is to maximize the log-evidence of the training dataset. In case of having access to S, the vanilla EM algorithm is employed with the modification of enforcing the GMM's means to be zero and covariance matrices to be diagonal in every update step. More precisely, after computing the posteriors p t (k|s i ) by the Bayes rule in the tth iteration, the m-step is given byEQUATIONIn case of having access to X , the same modified EM algorithm from Section 3.3 is employed by exchanging AD with D and setting the noise variance σ 2 to some small value.To train the CSVAE using the dataset G, we also aim optimize the log-evidence of the training datasetEQUATIONBy considering that log p θ (y i |s i ) does not depend on the CSVAE's parameters (see (7) ), introducing additional variational parameters ϕ and approximating p θ (z|y, s) with q ϕ (z|y), we apply the standard reformulations of VAEs, which ends up in an ELBO resembling the standard ELBO in ( 6), given byEQUATIONwith zi ∼ q ϕ (z|y i ) and p θ (s i ) being defined in (10) with p(z) = N (z; 0, I). In the case of training the CSVAE with W, the same training procedure from Section 3.2 is applied, resulting in the modified ELBO L(θ,ϕ) approximated byEQUATIONwhere we set the variational distribution q ϕ (s, z|x i , y i ) of the latent variables given the observations to q ϕ (z|y i )p θ (s|z, x i ). Consequently, instead of computing the posterior p θ (s|z, y i ), we use p θ (s|z, x i ), which is given by (8) with D instead of AD and setting the noise variance σ 2 to some small value. Additionally, to compute E p θ (s| zi,xi) [log p(x i |s)] we replace y i , M and AD in (29) with x i , N and D, respectively.In a first step towards a computationally more efficient implementation for training the CSVAE and CSGMM, we reformulate the expressions of the conditional mean µ s|y,z θ (z) and conditional covariance matrix C s|y,z θ (z) from [17, 18] in (8) . To do so, we observe that conditioned on z, s and y in (7) are jointly Gaussian. In consequence, we can alternatively apply the standard formulas for computing the moments of a conditional distribution for a jointly Gaussian setup [49] , i.e., µ s|y,z θEQUATIONEQUATIONwithEQUATIONs,y|z θ (z) = diag(γ θ (z))D T A T . Importantly, we do not need to explicitly compute C s|y,z θ (z) according to (51). The subsequent reformulations differ between CSGMM and CSVAE.CSGMM. For the CSGMM, only the diagonal entries of C s|y,z θ (z) must be explicitly computed, i.e., diag C s|y,z θEQUATIONThe matrix C for which we can directly use (53). In this way, we circumvent explicitly computing the full posterior covariance matrices in (8) , rendering training the CSGMM more efficient.The objective for training the CSVAE is given in (15) , which consists of the modified reconstruction loss in (29) as well as the KL divergences in (30) and (31) . While µ s|y,z θ (z) in ( 29) can be directly computed using (50) , we reformulate the trace-term in (29) to circumvent the explicit computation of C s|y,z θ (z). To do so, we apply the following steps EQUATIONThe artificial 1D dataset of piecewise smooth functions is generated in the following way:EQUATIONwhere h (j) m,i ∼ Bernoulli(0.5) and either takes -0.4 or 0.4, a (m) i ∼ N (0, 0.1 2 ), η (m) i ∼ U(0, 2π), g 1 ∼ U(0, 2) and g 2 ∼ U(2, 4). After generation, each x i (t) is sampled equidistantly N = 256 times, and its samples are stored in vectors x i , which then constitute the ground-truth dataset X . Exemplary signals and their wavelet decomposition are given in Fig. 4 and 5, respectively.MNIST & FashionMNIST. The non-learnable baselines for the simulations on MNIST are Lasso as well SBL. We apply Lasso directly in the pixel domain with its shrinkage parameter λ set to 0.1 in line with [4] . For SBL, we use an overcomplete db4 dictionary with symmetric extension [47] . Moreover, although we do not include noise in the simulations for MNIST, we set σ 2 in (2) to an increment corresponding to 40dB SNR to be able to apply the computationally efficient reformulations from Appendix I. For CSGAN, we use the exact hyperparameter configurations specified in [29] . For CSGMM, we set the number K of components to 32 and iterate until the increments of the training dataset's log-evidence reach the tolerance parameter of 10 -3 , a standard stopping criterion for GMMs [50] . The CSVAE encoders and decoders contain two fully connected layers with ReLU activation and one following linear layer, respectively. The widths of the layers are set in a way such that for the first two layers, the width increases linearly from the input dimension to 256, while the final linear layer maps from 256 to the desired dimension (i.e., either S for the decoder or twice the latent dimension for the encoder). The latent dimension is set to 16, the learning rate is set to 2 • 10 -5 , and the batch size is set to 64. We use the Adam optimizer for optimization [46] . We once reduce the learning rate by a factor of 2 during training and stop the training, when the modified ELBO in (15) for a validation set of 5000 samples does not increase. For CSGMM as well as CSVAE we set σ 2 in ( 7) to an increment corresponding to 40dB SNR to be able to apply the training reformulation from Appendix I. We also use the same overcomplete db4 dictionary as for SBL. Moreover, we use N s = 64 samples to approximate the outer expectation in (9) .Piecewise smooth function. For the simulations on the set of piecewise smooth functions, we adjust the shrinkage parameter of Lasso based on a ground-truth validation dataset of 5000 samples once for every M . Moreover, we also use the overcomplete db4 dictionary for Lasso as for all other dictionary-based estimators. Instead of choosing 256, we choose 128 as the maximum width of the en-and decoder layers. Otherwise, the hyperparameters remain the same as for the simulations on MNIST.CelebA. For the simulations on celebA, we set the shrinkage parameter λ of Lasso to 0.00001 (cf. [4] ). As celebA contains colored images, we choose a block-diagonal dictionary with the overcomplete db4 dictionary three times along the diagonal and zero matrices in all off-diagonals. SBL [18] CSGAN [29] (trained on ground truth)(trained on ground truth, ours) CSGMM (trained on ground truth, ours) Each block corresponds to one color channel. We use this dictionary for all estimators on celebA. The batch size to set to 32. Otherwise, all hyperparameters are chosen in the same way as for MNIST.Generally, we did no detailed network architecture search for the proposed CSVAE since we observed no considerable change in performance by testing out different architectures.Robustness comparison. The proposed CSVAE relies on approximating the outer expectation in (9) by Monte-Carlo sampling. Similar to this approximation, the baseline CSGAN also applies several random restarts, i.e., it estimates the ground-truth sample several times for a single observation and chooses the best-performing estimation by comparing their tractable measurement errors [29, 4] . In this way, both methods can be compared in terms of the number N s of repeated estimations they perform for a single observation. In Fig. 6 a ) and b), we compare the nMSE and the SSIM for both with respect to N s . We evaluate their performance on the MNIST dataset with M = 200 and N t = 20000. It can be seen that the proposed CSVAE achieves already good performance for a single Monte-Carlo sample, i.e., N s = 1, while CSGAN is significantly worse when only using one restart (i.e., N s = 1) compared to having many restarts.MNIST with training on ground-truth data. In Fig. 6 c )-f), the results from Fig. 2 a)-d ) are extended with the performance of the corresponding models trained on ground-truth data, which are represented by the dashed curves. It should be noted that ground-truth information here refers to the MNIST images x i themself. In consequence, CSGAN as well as the proposed CSGMM are trained on X , while CSVAE is trained on W (cf. Section 2.1 and Appendix H). For the sake of readability, we leave out the error bars in these plots. Generally, the CSGAN, as well as the proposed CSVAE and CSGMM, benefit from the additional information during training in terms of the distortion metrics nMSE and SSIM. For the nMSE, the overall performance comparison remains the same, while for SSIM CSGAN trained on X outperforms CSVAE and CSGMM trained on W and X , respectively. In Fig. 7 , additional exemplary reconstructed MNIST images are shown for all models trained on compressed as well as ground-truth data X in case of the CSGAN and CSGMM and W in case of the CSVAE. It can be seen that perceptually, the CSGAN significantly benefits from the ground-truth FashionMNIST. In Fig. 6 g) and h), the performance of SBL, CSGAN, and the proposed CSVAE and CSGMM for varying observation dimensions M and fixed number N t = 20000 of training samples is shown. While for small M , CSGAN outperforms the proposed CSVAE and CSGMM, its performance saturates for increasing M , and it performs worse than CSVAE and CSGMM. In this case, CSGAN's regularization to enforce the reconstruction to be in the generator's domain is beneficial for strongly compressed observations (i.e., for small M ). In Fig. 8 , exemplary reconstructed FashionMNIST images are shown.Additional exemplary reconstructions. In Fig. 9 , 10 and 11, additional exemplary reconstructions for the piecewise smooth functions, MNIST as well as celebA are shown.All models have been simulated on an NVIDIA A40 GPU except for the proposed CSGMM, whose experiments have been conducted on an Intel(R) Xeon(R) Gold 6134 CPU @ 3.20GHz. We report the number of learnable parameters, the time used for training as well as the average reconstruction time after training for simulations with piecewise smooth functions, MNIST and celebA in Table 1 , 2 and 3, respectively. The average reconstruction time of estimating x * from a newly observed y has been measured for the MAP-based estimators in Appendix G. While the reported numbers give an overview of the comparison between the different tested models, it is important to note that we did not aim to fully optimize our simulations for computational efficiency.Algorithm 1 summarizes one iteration of the training procedure for the CSVAE. Algorithm 4 does the same for the CSGMM. In algorithm 2 and 3, the pseudo-code of the CME approximation and the MAP-based estimator using the CSVAE is presented, respectively (cf. ( 42) and ( 43)). In algorithm 5 and 6, the same is given for the CSGMM (cf. ( 44) and ( 45)) ← --γ • The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Justification: There are no potential societal impacts of our work, which we think must be specifically highlighted.EQUATION• Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.Source code is available at https://github.com/beneboeck/sparse-bayesian-gen-mod.For the sake of readability, we exclusively use z throughout the remainder of this section to denote the continuous as well as the discrete and finite random variable, on which we condition.The celebA dataset is released under a custom license for non-commercial research use.The original work of CSGAN[29] does not provide results for varying measurement matrices as well as for celebA trained on solely compressed data. We also did not find a working hyperparameter configuration, so we leave out CSGAN for the piecewise smooth functions and the celebA dataset.We leave out the CSGMM due to the need to compute separate posterior covariance matrices for all training samples by using varying measurement matrices (cf. (8)), which leads to a computational overhead.For the CSGMM we use the notation from Section 3.3 


Decision: This paper proposed one compressed sensing method with sparsity inducing conditionally Gaussian generative models. All the reviewers agree that this idea is novel and interesting. After my own reading, I also agree with that and thus I recommend acceptance. Nevertheless, as pointed out by most reviewers, especially Reviewer 34q1 and WoRD,  the authors should carefully revise the manuscript as required/suggested in the rebuttal and discussions to improve the quality, which is very important.

Reviews: review: paper_summary: paper_summary: The paper proposes a set of methods to learn a generative model over the sparse representations of  a signal. The dictionary basis is fixed and given, and the signal of interest is assumed to be sparse in this basis. Therefore, the generative model learns to provide such sparse representations. The method learns the generative prior from the noisy and compressed observations. The authors leverage the notion of conditional Gaussianity and propose VAE and GMM based models for learning such priors. The choice enables the inference with only forward operation.
strength_weakness: strength_weakness: Overall, the paper combines some of the existing ideas from prior works (for example posterior updates from SBL, VAE ELBO derivations), and therefore, the novelty seems marginal. To be more concrete, it *seems* straightforward that if one wants to do variational Bayes, then the VAE style decomposition is the standard approach. If there are any additional challenges, the authors should clarify.

The idea of forward pass only inference has been tried before (see for example the paper “Solving Linear Inverse Problems Provably via Posterior Sampling with Latent Diffusion Models” NeurIPS 2023).  

Learning from the compressed samples can be compelling although not new as mentioned by the authors themselves (for example CSGAN). The proposed method requires further elaboration on the loss with respect to learning from uncompressed measurements substantiated with more experiments.

The experiment section is a bit thin given the limited technical novelty, particularly with respect to the baselines. Given a lot of prior works in the literature on learning based solvers for inverse problems (for example, all variations of generative priors based on flows and diffusions, unfolded algorithms like lista and its variations), the authors have a more difficult task of placing their contribution within the prior work. The paper is not very convincing in showing what is the unique direction in which the field is moved forward.
questions: questions: Some of my questions about the novelty was implicit in my comments above.
-	Can authors provide some comment on the performance loss with respect to uncompressed generative prior?
-	How many MC samples are used at inference for (9)?
-	Generative priors generally suffer from convergence issues and require multiple restarts. Could the authors comment if their forward-only operation has a similar issue? 
-	It would be good to incorporate error bars in the plots and numerical results. Given the MC step, I expect a higher variance for the proposed approach.
limitations: limitations: See comments above.
paper_summary: paper_summary: The paper introduces a novel training algorithm for generative models used as priors in linear inverse problems, with a specific focus on compressed sensing. The authors propose a training principle that regularizes the prior to learn a sparse representation of the signal of interest, implemented in Variational Autoencoders (VAEs) and Gaussian Mixture Models (GMMs). This is achieved by learning two posterior distributions p(y∣s) and p(s∣z) over a set of parameters through optimization of the Evidence Lower Bound (ELBO) derived in Equation 15.

During the inversion phase, the true signal is estimated by sampling from the posterior distribution E[x∣y] as given in Equation 9. Unlike many contemporary generative priors, this method does not require solving an optimization algorithm to achieve the estimate. This approach simplifies the inversion process, making it more computationally efficient.

The performance of these implementations is validated on datasets containing different types of compressible signals. The results demonstrate the effectiveness of the proposed method in accurately reconstructing signals. Additionally, the paper provides theoretical support on the tightness of the lower bound of the estimate, further validating the robustness and reliability of the proposed approach.
strength_weakness: strength_weakness: The authors provide theoretical motivation; however, they do not address how the potential estimation of the measurement y impacts the estimation of the true signal x. Additionally, building on this point, the CSVAE may experience representation error, as noted in [5], which is not mentioned as a limitation. While the proposed method does improve upon [5], this remains a source of error.


Section 4 is limited in the following ways:

* The only metric reported for natural images is the normalized mean square error, while other commonly used metrics, such as PSNR and SSIM, are not included.

* Given the computational focus of this paper, it is concerning that all experiments were conducted on low-resolution grayscale images, especially considering the A40 GPU budget.

* The baselines appear to be less contemporary compared to current literature on compressed sensing inversion algorithms. [Normalizing Flow](https://proceedings.mlr.press/v119/asim20a/asim20a.pdf) and [Diffusion Models](https://openreview.net/pdf/4f6f0e2347a3d6f9a88b39e445f77c1e7503064e.pdf) have largely replaced GANs due to issues with representation error. Similar to the proposed method, Diffusion Generative Priors do not require gradient-based approaches to estimate the posterior. There are hyperlinks to the work in the responses. 

* Considering this paper is training generative before be used downstream, there seems to be no reported metrics about training other than computational resources.
questions: questions: * Could the authors please provide perceptual metrics (i.e. SSIM, LPIPS) for the reconstruction comparison experiments in Figures 2 and 3? 
* Could the authors elaborate on the computational overhead provided in footnote 4 and explain the reason for utilizing only grayscale images when reporting having a computational budget of NVIDIA A40 GPU for CSVAE? 
* Could the authors please explain the advantages the proposed method has over contemporary generative prior such as normalizing flow and diffusion-based priors? 
* One concept I am unclear about is how do you check if the proposed method is overly biased towards the measurements y? This issue might arise because the conditional distribution P(y|s)  has equally likely probabilities when Y=AS, but does not guarantee S will be consistent with training samples depending on how sparse s is in practice.
limitations: limitations: Please refer to the weakness portion of the review.
paper_summary: paper_summary: The authors present an elegant new approach for dictionary based compressive sensing wherein the sparsity inducing prior is tuned from data by maximizing a lower bound on the evidence. This is a new paradigm for compressive sensing which appears to improve on reconstruction error over standard approaches and does not require solving an optimization problem at inference time.
strength_weakness: strength_weakness: - Since their approach offers a new perspective on sparse Bayesian learning, it would have been helpful to compare the sparsity of predictions in addition to the nMSE.
- I understand that space is limited, but I found the description of the experiments to be a bit terse. 
- It wasn't clear to me when a practitioner should choose CSVAE vs. CSGMM
questions: questions: - How does your approach compare to methods for sparse Bayesian learning with proper priors; for example Louizos et. al. "Bayesian Compression for Deep Learning." NeurIPS 2017?
- Have you tried evaluating the quality of error bars from the predictive posterior so that your approach might be used in the context of UQ?
- Can you explain the reason for the numerical instabilities in the noise free simulations for some of the image datasets?
- Have you analyzed the histogram of the posterior over $s$? Can you suggest how the practitioner who is interested in a sparse $s$ might $0$ entries of the vector?
limitations: limitations: I think the authors have done a good job addressing the potential limitations of their approach in section 3.4. It would be interesting to analyze how well generative priors trained on a particular dataset transfer to similar datasets.
paper_summary: paper_summary: This paper introduces a new type of sparsity inducing generative prior for the inverse problem. The authors theoretically underpin our approach by proving that its training maximizes a variational lower bound of a sparsity inducing log-evidence.
strength_weakness: strength_weakness: 1. The comparison methods include Lasso, CKSVD, and CSGAN, all of which are presented before 2018. The comparison methods are too old.
2. I think the author should clarify the specific application scenarios for this research direction.
3. In Figure 7, compared with CSGAN, the results of CSVAE and CSGMM are not very clear.
questions: questions: 1. In Table 2: Resources for simulations on MNIST (M = 200, Nt = 20000, Fig. 2 a)), CSGAN has much less parameters than the proposed CSVAE , why does CSGAN take much more training time?
2. Could the reconstruction results be influenced by the amount of noise?
limitations: limitations: I think the application scenes of this work is not very clear. The datasets used for comparison are very naive, colorful images with more textures should be used. And the authors should include more recent comparison methods, existing comparison methods are all presented before 2018.

Comments: title: Thanks for the response
Answer to Reviewever K63t
Answer to Reviewer y1C8
Further Questions
Follow-up on the answers
Please review the rebuttal and respond
comment: I would like to thank the authors for the answer. I increased my score to weak accept.

I feel the above discussion can be helpful to include in the revised version of the paper, particularly to make the story more coherent. They might even think about reflecting it in the title and abstract if permitted. I still feel that the story of the paper is about learning from compressed measurements, as it does not have a detailed case study on uncompressed measurements. Their arguments that SBL is particularly useful because of incorporating uncertainties and because of estimating the ground truth samples is compelling to me. As a stretch goal, the authors could think of ablation experiments to verify this hypothesis (for example by freezing the variance params in SBL  - not sure if it makes sense though).
We thank the reviewer for the reviewer's comments on our rebuttal and for the willingness to increase the score if all of the reviewer's concerns are addressed.

Moreover, we thank the reviewer for pointing out that our contribution might be misunderstood as we focus more on the learning aspect in our abstract and contributions instead of the "learning from compressed measurements"-aspect.
It seems to us that the key aspects of our contributions depend on the individual background of the reader.
Coming from a "SBL for compressive sensing" perspective, emphasizing the ability to learn from data in an initial training phase might be more natural. However, we agree that for the "generative models for compressive sensing" community, the additional aspect of being able to learn from compressed measurements is crucial to accurately putting our work's contribution into concurrent literature. 

In Fig. 6 and 7, we compare CSGAN, CSVAE, and CSGMM trained on uncompressed (ground-truth) data (dashed curves) as well as compressed data (solid curves). Perceptually, CSGAN benefits more from uncompressed (ground-truth) information during training than our proposed CSVAE and CSGMM. In the case of the distortion metric nMSE, all approaches perform approximately equally better when being trained on uncompressed data. In total, the performance gains to the CSGAN baseline seem to be more prominent when it comes to being trained on compressed training samples.

From our perspective, this indicates that our proposed approach is particularly relevant in those cases where only compressed training samples are accessible and/or the online (inference) latency plays a crucial role. The intuition behind why the approach is capable of successfully learning from compressed data is the following:

After each parameter update step during training, the proposed CSGMM and CSVAE do the following:

- They estimate the corresponding ground-truth samples in the training data (Eq. (8) right side)
- Additionally and arguably even more importantly, they quantify the uncertainty of this estimation in terms of the estimations' covariance matrix (Eq. (8) left side).
- After doing so, both proposed models incorporate both (the estimation as well as the error quantification) in the subsequent update step. In the case of CSGMM, this is done by its corresponding M-step (Lemma 3.2). In the case of CSVAEs, this is done in its modified reconstruction loss in Appendix D, as well as the additional KL divergence in Eq. (15), which does not appear in standard VAEs. 

This mechanism (especially the aspect of quantifying the estimation's error and incorporating it in the next update step) does not appear in CSGAN's methodology of learning from compressed data. This is because CSGAN estimates the ground-truth samples after each update step but only provides these estimations to its discriminator without considering any uncertainty. From an intuitive point of view, this is the reason why our proposed approach seems to be superior when it comes to compressed training samples. 

We would like to thank the reviewer again, and based on the reviewer's comments, we will revise our work to better phrase our work's contribution regarding learning from compressed measurements. Moreover, we will consider extending our discussion section to explicitly discuss the intuitive learning aspects of our proposed method described above.
We would like to thank the reviewer for commenting on our rebuttal. We agree with the reviewer that the MNIST dataset generally provides enough ground-truth training samples for training state-of-the-art machine learning-based compressive sensing techniques. However, we indeed modified the dataset for training our proposed method. In all our simulations in the main paper (Fig. 2 and 3) and most results in the Appendix (all except the dashed results in Fig. 6), the parameter $M$ does not only refer to the dimension of the compressed observation, which is to be reconstructed but also to all training samples (Section 4.1 - Measurement matrix and evaluation metric). For example, all training samples in Fig 2 a) have been compressed by a Gaussian measurement matrix to have dimension $M$ before being used for training, which is why none of the state-of-the-art methods trained on ground truth data could have been used in all these settings. We will revise our work to make this more prominent in our final version. 

We agree that Deep Image Prior and Deep Decoder could have been considered as further baselines. Both are cited in our introduction. However, we decided against both as these methods only apply to natural images and cannot be trained [1], whereas our setting is about reconstructing any compressible type of signals based on learning from corrupted training samples. Therefore, CSGAN and CKSVD seem to us to be the closest baselines from the "generative model"- and "dictionary learning"-community. We hope the reviewer agrees with us that both untrained neural-network baselines are, thus, not required for our work.

[1]  D. Ulyanov, A. Vedaldi, and V. Lempitsky, “Deep image prior,” International Journal of Computer Vision, vol. 128, no. 7, p. 1867–1888, Mar. 2020.
Yes, I agree with both points "to our best knowledge, contemporary generative priors for compressive sensing generally rely on the assumption of being trained on many ground-truth training samples" and "the ability to learn from strongly compressed signals and exhibit signals/images in this dimensional regime." However, one of the datasets used is MNIST, which has sufficient data for diffusion-based prior or normalizing flows and these priors are not limited to images they can perform inference on 1-D signal. Please correct me if something was done differently to the training set that could not be adopted for this framework.  Furthermore, even in the case where there is limited data, there are untrained priors such as [Deep Decorder](https://arxiv.org/pdf/1810.03982) or [Deep Image Prior](https://arxiv.org/pdf/1711.10925) that can be used as comparisons. 


Based on the response above you have implicity answered my question about the equally likely probabilities along the line of Y=AS.
I would like to thank the reviewers for their detailed response, I will update my score accordingly based on the rebuttal.
I would like to thank the authors for taking time to answer my questions and comments. I will discuss some of the responses a bit further, and that, with the intention of building more confidence and increasing my score, so hopefully the authors can bear with the process, and their time is well spent at the end. 

* To begin, I agree with the authors that “the capability to learn from compressed data” is not minor, as I mentioned in my original review.
* I am also satisfied with the answers to my questions regarding MC samples, error bars, and convergence issues. 
* **Regarding deriving ELBO from SBL:** As far as I can see, the inequality in (7) is the standard step in ELBO bounds. The decomposition in (13) follows from the basic probability decomposition for $\mathbf{z}, \mathbf{s}$  and $\mathbf{y}$ (given in Appendix B). Then, the computation of the bound follows from MC sampling like classical VAE (also referenced by the authors themselves, namely ref. 36), and the tractable computation of some terms in Appendix D. The derivations of Appendix D (eq. 26-29) are not exactly the same as VAE paper but builds very similarly on Gaussian likelihood assumption and the linearity of expectation.  So, unfortunately, I still feel that these derivations are based on standard linear algebra and probability techniques, very similar to the standard VAE. We can therefore focus on other comments regarding the novelty.
* **Regarding “1. One technical contribution from our side is to extend the statistical model…”:** 
I agree with the reviewer that the proposed method enables “the learning of a nontrivial sparsity inducing prior with training data”, however the actual contribution is “the learning of a nontrivial sparsity inducing prior” from **compressed measurements**, as the authors emphasize (although it is not so clear from the title and not so pronounced in the abstract).  

So, the question is: does the method perform equally better from uncompressed data? If not, what makes the method better in case of learning from compressed data? I feel combining two contributions (namely learning a nontrivial sparsity prior AND learning from compressed data) and selecting the baselines accordingly makes the message convoluted: is the paper about a better generative prior or about learning from compressed data? Answering such questions is important for the paper, which is proposing a general method for solving a particular problem.
Thank you for the careful rebuttal. I thins the authors have addressed my concerns. I hope you can revise the final paper to point out exemplary applications more specifically. I have improved my rating.
Dear Reviewers,

Thank you once again for your time and effort in reviewing for NeurIPS 2024.

The authors have already submitted their rebuttal to your comments, and the discussion phase has now begun. For those who have not yet responded, please review the authors' responses carefully at your earliest convenience. Your timely feedback on whether your previous concerns have been adequately addressed is crucial to the reviewing process.

Thank you very much!

Best regards,

Area Chair
Thank you for your thoughtful response. I think extending the Appendix to include 
a more concrete discussion of CGSMM versus CSVAE would be helpful.

While not a requirement, I hope you do 
end up including some analysis of the posterior and sparsity since this will 
make your work relevant to a broader community.