Sparse Bayesian Generative Modeling for Compressive Sensing
Authors: Benedikt Böck
Sadaf Syed
Wolfgang Utschick
Venue: NeurIPS 2024 poster (2024)
Keywords: Compressive sensing
variational inference
sparse bayesian learning
variational autoencoder
Gaussian mixture model
generative model
Primary area: probabilistic_methods
TL;DR: In this work, a new type of sparsity inducing generative prior for compressive sensing is introduced.

Abstract: This work addresses the fundamental linear inverse problem in compressive sensing (CS) by introducing a new type of regularizing generative prior. Our proposed method utilizes ideas from classical dictionary-based CS and, in particular, sparse Bayesian learning (SBL), to integrate a strong regularization towards sparse solutions. At the same time, by leveraging the notion of conditional Gaussianity, it also incorporates the adaptability from generative models to training data. However, unlike most state-of-the-art generative models, it is able to learn from a few compressed and noisy data samples and requires no optimization algorithm for solving the inverse problem. Additionally, similar to Dirichlet prior networks, our model parameterizes a conjugate prior enabling its application for uncertainty quantification. We support our approach theoretically through the concept of variational inference and validate it empirically using different types of compressible signals.

Sparse Bayesian Generative Modeling for Compressive Sensing
Benedikt Böck, Sadaf Syed, Wolfgang Utschick
TUM School of Computation, Information and Technology
Technical University of Munich
{benedikt. boeck, sadaf. wtw.aa.sseddduddeeeeaoo utschickk)@tmmm de
boock,
Abstract
This work addresses the fundamental linear inverse problem in compressive sensing (CS) by introducing method utilizes ideas from classical dictionary-based CS and, in particular, sparse Bayesian learning (SBL). to integrate a strong regularization towards sparse solu- tions. At the same time, by leveraging the notion of conditional Gaussianity, it also incorporates the adaptability from generative models to training data. However, unlike most state-of state-of-the-ar fi--tha-rt art generative models, it is able to learn from few com- pressed and noisy data samples and requires no optimization algorithm for solving the inverse problem. Additionally, similar to Dirichlet prior networks, our model parameterizes a conjugate prior enabling its application for uncertainty quantifi- cation. We support our approach theoretically through the concept of variational inference and validate empirically using different types of compressible signals.
type of regularizing generative prior. Our proposed
1 Introduction
Research in CS has shown that it possible to reduce the number of measurements far below the one determined by the Nyquist sampling theorem while still being able to extract the information-carryinn signal from the acquired observations. The fundamental problem in CS an ill-posed linear inverse problem, i.e., the goal is to recover the signal x E R of interest from an under-determined set of measurements y E R.M with M < N, related by
Ax" + n.
(1)
where x= is compressed by the measurement matrix A and potentially corrupted by additive noise ~ [NO,, o, I). Since the under-determined observation does not carry enough information alone to faithfully reconstruct x², additional prior (or model) knowledge about x is required to make the inverse problem "well-posed". Classical CS algorithms such as Lasso regression or orthogonal matching pursuit (OMP) address this problem by incorporating the model knowledge that x= is sparse or compressible with respect to some dictionary [I. Nowadays, modern deep learning (DL)- based approaches such as unfolding algorithms, generative model-based CS and un-trained neural networks (NNs) expand the possibilities by learning prior knowledge from a training set or designing network architecture to be biased towards certain class of signals 2 These approaches typically require a (potentially large) training set of ground-truth data samples, or their architecture is specifically biased towards natural images. However, in many applications, ground-truth training data might not be easily accessible. In, e.g.. electron microscopy, the amount of electron dose has to be restricted to not induce damage on the probe resulting in low-contrast noisy data samples [8). The sensors in wearable electrocardiography (ECG) monitoring devices generally provide noisy signals with artifacts 19]. which limits the ability to learn from patient-specine data real-world settings. Another example is the wireless 5G communication standard, where mobile users receive compressed and noisy so-called channel observations on frequent basis (cf. [0) while acquiring ground-truth
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

channel information requires costly measurement campaigns. Thus, in many applications, it is either impossible or prohibitively expensive to collect lots of ground truth training data while corrupted data is readily available. This highlights the necessity for methods that can learn from corrupted data samples.
In this paper, we propose a new learnable prior for solving the inverse problem (.). which can learn from only a few compressed and noisy data samples and, thus, requires no ground-truth information in its training phase. Additionally, it applies to any type of signal, which is compressible with respect to some dictionary. Our approach shares similarities with generative model- CS 4 There, a generative model is first trained to capture the exact underlying prior distribution p(z) of the signal x" of interest. It then serves as probabilistic prior to regularize the inverse problem (I) Similarly, classical CS algorithms like Lasso regression also impose probabilistic prior on the sparse representation of x" with respect to some dictionary 12] In contrast to modern generative models, however, these priors do not have any generation capabilities but solely bias the inverse problem (I) towards sparse solutions. This observation indicates that a probabílistic prior does not necessarily need to capture the exact prior distribution p(x) to effectively regularize (I. The proposed model in prior. For that, we aim to combine the adaptability of generative models to training data with the property of many types of signals x² to be compressible with respect to some dictionary. As a result, our model learns statistical information in the signal's 's sparse/compressible domain. Examples of signals with a specific statistical structure in their compressible representation include piecewise smooth functions and natural images, whose wavelet coefficients approximately build a connected sub-tree or wireless channels, which are burst-sparse in their angular domain 13 14].
this work builds upon this insight and forms a trainable but simultaneously sparsity-inducinn
Related Work. Early work on CS, which considers statistical structure in the wavelet domain of images, is given in (55] [6]. Based on the theoretical foundation in [17] and the concept of SBL 18 19, these papers introduce hierarchical Bayesian model, which is used to apply variational inference and Markov chain Monte Carlo (MCMC) posterior sampling to solve (T]. Training Gaussian
mixture models (GMMs), i.e., classical generative models, from compressed image patches of one or a few images has been analyzed in 23). In this line of research, however, the GMM is fit directly in the pixel domain. Compressive dictionary learning represents different line of research aiming to learn the dictionary from solely compressed data (4427). and strongly bases on the dictionary learning method K-SVD (2) More recently, variational autoencoders (VAEs) and generative adversarial networks (GANs) have been studied in the context of CS [ 11. There, the VAE/GAN is used to solve the inverse problem (1 by constraining the signal x of interest to lie in the range of the generative model instead of being sparse with respect to some dictionary. In this context, AmbientGAN well as CSGAN are extensions that loosen up the training set requirements and can learn from corrupted data 29 30 Another related topic is the ability of some generative models, i.e., diffusion models, VAEs and GMMs, to provide an approximation of the conditional mean estimator (CME) (31 In the case of the latter two, the CME is represented as tractable convex combination of linear minimum mean squared error (MSE) estimators by exploiting their conditional Gaussianity on latent space that determines these estimators' means and covariances.
Our main contributions are follows:
We introduce a new type of sparsity-inducing generative prior for the inverse problem I.. which differs from classical CS algorithms due its ability learn from data. On the other hand, it also differs from other modern NN-based approaches due to its ability to promote sparsity in the signal's compressible domain. Moreover, can learn from a few corrupted data samples and, thus, requires no ground-trum truth information in its training phase.
We theoretically underpin our approach by proving that its training maximizes a variational lower bound of a sparsity log-evidence.
Building on the notion of conditional Gaussianity, we introduce two specific implementa- tions of the proposed type of prior based on VAES and GMMs, which do not require an optimization algorithm in their inference phase and come with computational benefits.
By exploiting the shared property with Dirichlet prior networks to parameterize conjugate prior, we demonstrate how our approach can be applicd for uncertainty quantification.
We validate the performance on datasets containing different types of compressible signals
Soorree code is available htttpp///ggihuuu..cc////bbeee///
saayyeessiinggn-mmoooooo mod

---

2 Background and Method
Notation.
The operations b b-1, b and 16|2 denote the respective element-wise operation for the vector b. Moreover, diag(BB B) represents the vectorized diagonal of the matrix B, while diag(b) denotes the diagonal matrix with the vector b on its diagonal.
2.1 Problem Formulation
We consider the typical CS setup, in which we measure N-diiensional ground-truth samples æ; by known measurement matrix A RRMx (or A, R with M < N, where each observation is potentially corrupted by noise n, drawn from additive white Gaussian noise (AWGN) (O0, of I). We assume all æ; to be compressible with respect known dictionary matrix D E RNS' i.e., y: ADs; + n. with 2; Ds; and s; E R being approximately sparse. All s; are assumed to be independent and identically distributed (i.i.d), i.e. 8; ~ p(s), where p(s) is unknown with exhibiting non-trivially dependent entries. Typical signals which fit into this category are, e.g., natural images, piecewise smooth functions, and wireless channels (cf. Section D. Our approach in this work allows to either solely have access to corrupted training observations y = } or training tuples with ground-truth information G = {(s,, b or W = {(x,, }:)} 1=1- We first train the proposed model using 3/ (or, alternatively, or W) to serve as an effective prior for (T). Our goal is then to estimate a ground-truth signal x² of a newly observed y.
We also define X' = { xx [
N. and S = {s:
2.2 Sparse Bayesian Learning for Compressive Sensing
In SBL for CS, the idea for solving the inverse problem (T) is to assign a parameterized prior to s*, i.e, the compressible representation of the signal x² of interest with x² Dss [99]. It assumes y|s* ~ p(y)s") NNy ADs", o² I), s" ~ pp)(( = (s; 0, diag(r)).
(2)
Given a single observation y. the parameters y (and sometimes o²) are estimated by an expectation- maximization (EM) algorithm maximizing the corresponding log-evidence log Py(y) implicitly defined by ( After that, it is utilized that (s( forms conjugate prior of p(y)s*) resulting in closed-form posterior (x ly) providing all necessary information to estimate x². In [I9], variational interpretation of SBL is given, addressing why this approach yields sparse results, even though py(ss) (s) does not inherently promote sparsity. It is shown that there exists a C. > 0 such that for
all y > 0 and s
(a) 2 C
The function ts)), however, is a well-known improper but sparsity- inducing prior, used as a non- informative prior for scale parameters (34). Let another statistical model be given by p(yls*) (cf. (2) and prior (()) instead of (8) with implicitly defined log-evidence ( Due to ((s) being improper, the log-evidence log n (>>) forms not normalized but valid ligeliiel Moreover, this model is sparsity-inde -induiy- due to the pparsity-promotinn characteristics of t(s). Based on (3)
(y) and, thus, log (B) can be embedded in variational inference, forming a tractable variational lower bound of an intractable log-evidence log ( (z)) of actual interest. Moreover, applying the EM algorithm to 2 is "evidence maximization over the space of variational approximations to a model (i.e., p(y) s*) and t())), with sparse, regularizing prior" [19
holds that log π (*) (y) 2 loog (g))
2.3 Gaussian Mixture Models and Variational Autoencoders
GMMs and VAEs are generative models aiming to learn an unknown distribution p(æ) from a training set X. Both models represent p(x) as the marginalization with conditionally Gaussian p(æl-) and an additional latent variable l (or z) 37], i.c.,
K Σ p(k)p(zlk)) Σ
p GMMM (x)
Pa Px:N(x: pe: Ct),
(4)
D ( p(=)po(<lz))d= / - (LE (o())) (5)
3

---

The tunable parameters (fpk: Me,, Cx] = for GMMs and @ for VAEs) are learned by optimizing the model's log-evidence Σ; (epp(((r)) ( E (GMM, VAE)) over X. In the case of GMMs, this is typically done by an EM algorithm alternating between the so-called e- and m-step. The e-step determines the closed-form posteriors pr (r(k)ar) (e;) (for all i) via the Bayes rule using the model's parameters in the čth iteration. The m-step then updates the model's parameters by maximizing E; Pp. (klbe) [log p( (r,, }-)] In case of VAEs, however, the posterior po(=|æ) (and log ( (xr) are intractable and the EM algorithm cannot be applied. Therefore, a tractable distribution (0(2)x) N((: pa (x), (o (z)) with variational parameters Φ is introduced, which approximates po 21) and (ar) and Φ (a:) are generated by NN encoder. Equivalently, (=)(2) and Co(z) are generally realized by a NN decoder. The objectíve to be maximized the evidence lower bound (ELBO)
L(O, Φ) serving as a tractable lower bound for the intractable log-evidence Σ, log po (x;)),
(x;) 2 L(0,0)
Σ (log)o((;) (1o a., [poo (.))) O.ER
(6)
DEE
with (+) being the Kullback-Leibler (KL) divergence. Generally, the GMM's and VAE's training critically depends on iteratively characterizing and updating their posteriors pr (k|[),) and qs (z(
2.4 Proposed Method
The goal of this section is to derive class of generative models, for which we can guarantee that, on the one hand, it is sparsity-inducing, but on the other hand, it is trainable and can learn from solely compressed and noisy observations y resulting in an effective probabilistic prior for
To incorporate the bias towards sparsity, we start with SBL discussed in Section 2.2 and combine it with the VAE's and GMM's main principle for their adaptability to complicated distributions, i.e., introducing latent variable z (or k) on which we condition with a parameterized Gaussian (cf. 4 and
5 More specifically, we exploit the Gaussianity of (s()) in (3) and modify it to a parameterized Gaussian conditioned on some latent variable x with arbitrarily parameterized ps ps(2) while explicitly
keeping its mean zero and its covariance matrix diagonal, i.e.. pe s x) = N(s O, diaa((7((2)) (z))). The resulting set of statistical models referred to as sparse Bayesian generative models, is given by
y s ~ p(yls) (y; ADs, I). s|2 pe(s = N (s; O, diag(ro(z))). ~~ pz(z). (7)
Training principle. Our proposed training scheme is independent of specific realizations for Yo (z)
and p=(x), which is why they are kept general in this section. Specific parameterizations are discussed in Section 3.2 and 3.3] The training goal is to maximize the model's 's log-evidence Σ; log ps.q (y:) implicitly defined by (7) over training set of compressed and potentially noisy observations V. For that, we rely on the main training principles for statistical models including latent variables, i.e., the EM algorithm and variational inference (cf. Section 2.3) A key requirement of these principles is the ability to track and update the model's posterior (i.e., Pe a z y for 7 or approximations of it over the training iterations (cf. Section 2.3). In classical SBL, the prior distribution (y(s) forms conjugate prior of p(y]s) (cf. (2)) and, thus, the posterior Pyyssy yy is tractable in closed form. We utilize this property by observing that conditioned on some z, (7) coincides with the classical SBL model. Consequently, s/2 is conditioned conjugate prior of in (7) with tractable Gaussian posterior pe (s)2..22 (z.2) y). whose closed-form -orm covariance matrix and mean
Co Celu.((2) = ( DEDD"AAAA + diag(To (x))))
elu. (z)
1 Col. (2)DT 0²
Hg
(8)
equal those in SBL 19]. The equivalent formulas for 0² are given in Appendix C By
considering the decomposition pa, 70,5(8,220 xly) pa [s|2, y)po. (2(y), we conclude that our proposed set of statistical models in 7 exhibits posteriors. which are partially tractable in closed form independent of any specifics of (0(2) and (z). ) The remaining Po.s (z yy) resembles the standard
GMM's and VAE's posterior, and we can either apply the EM algorithm or variational inference to
maximize the corresponding log-evidence. Both will be discussed in Section 3.2 3.3 respectively.
FFor the sake of readability. we exclusively use throughout the remainder of this section to denote the continuous as well as the discrete and finite random variable, on which we condition.

---

Estimation scheme. The goal after the training is to use the learned statistical model in 7 for regularizing the inverse problem ( and estimating x" from a newly observed y. Generally, the CME E a y is desirable estimator due to its property of minimizing the MSE. We utilize insights from 3 23 which derive approximate closed forms of the CME using VAEs and GMMs. More precisely, by using the law of total expectation, we approximate the CME by
E E[ = E [ [alyy =] [u] D E E sly. 2]1y] R3 D Bpo ((ll) (ep(()))))) [sly, =]
(9)
The inner expectation is given by the learned 14 (z) in while, depending on its specific characteristics, po,8 (zly) also exhibits a closed form variational approximation has been learned 3.2). with which we approximate the outer expectation by Monte- Carlo estimation. Based on the perception perception-distorion trade-off, the CME is not optimal from a perceptual point [8], and deviating estimators from the CME approximation might be beneficial. More detailed descriptions of the CME approximation as well as further estimators based on the specific implementations Section 3.2 and 3.3 are given in Appendix G
during training (cf. Section
3 Theoretical Analysis and Specific Parameterizations
3.1 Sparsity Guarantees
motivate constraining (s(2)) to be zero mean conditional Gaussian with diagonal covariance matrix by SBL and its sparsity-inducing property. In the following, we rigorously show that constraining po x) in this manner is sufficient maintain the sparsity-inducing property for any statistical model following 7 despite the additionally included latent variable z. More precisely, we show that any statistical model in 7 with some specified parameterized ps(z) and (z)))) exhibits a log-evidence that is interpretable as a variational lower bound of a sparsity- inducing log-evidence. For that, establish the following theorem.
In Section 2.4
Theorem 3.1. Let (=(s))) (ss O, diag(rr (z)) (i.e., it is defined according ), let be either continuous or discrete and finite, and let ( >> 0. Then, there exists a constant C > O such that
for all 0, 8 with any arbitrary distribution (z)
po. = -f (iig))(()) S t(s) = C.
N
(10)
|s:
The integral corresponds to summation for discrete and finite p=(2).
The proof is given in Appendix A. Based on Theorem 3.1 it holds that
2 Peg (8)) for all 0,
(1I)
with log n (y) being the log-evidence of yls ~ p(y]s) and the improper but sparsity-promoting prior t(s), whereas log p (t) is implicitly defined by 7 Consequently, can interpret the maximization of log Pe.6 (3) over (0, 8) as the evidence maximization over the space variational approximations to a model with sparsity-induce classical SBL, we have the additional degree of freedom to choose the specifics of (z(2) and pa (z) without any constraint while maintaining the connection to sparsity.
and regularizing prior. However, contrary to
3.2 Compressive Sensing VAE
The integral version of pe,s in 10) strongly resembles the VAE's decomposition of p(z) in (5).
In fact, by constraining Pa (z(z) p(z) = (z; 0, I), the distribution Pe. (s) Po(s) corresponds
to subset of all possible distributions covered by the VAE decomposition in (5. This motivates to choose p8(2 pz(z) ) in this exact manner and realize Yo (z) as the output of NN decoder. Generally, we aim to optimize the log-evidence Σ, log (3)) in [1] with respect to O solely given compressed and noisy observations y. However, equivalent to VAEs, the log-evidence Σ. log is intractable for p(z)= N2 I) which is why tractable lower bound has to be derived. A tractable lower bound on the log-evidence of a statistical model with latent variables obtained by subtracting the KL divergence between the conditional distribution of the latent variables given the observations
5

---

2
Encoder 2s (=(2))) y
Decoder
v
V
BAD
po (
(O
(O0.1)
Figure 1: schematic of the sparsity-inducing CSVAF.
and tractable variational distribution from the log-evidence (cf. (6)). For the model in 7 and i.dd. training data, this results in
Σ n:yY
log po 2 L
(0,
(CSVAE)
Σ
log (3()
DKL s| lpp (zy))
(12)
V.EY
log p(v p(yi (s) (#)2)
Σ wey
(13)
q (2, s sly:)
where the derivation of 13 from 12 is given in Appendix B] Based on insights from the training
principle explained in Section 2.4. the variational posterior (x s s|y:) in (13) can be simplified
to 4s(2, s|y.) p yo (o) 34) with closed-form pe(s)z; z, y/) and only requires variational
parameters & for (z(z))) to be tractable. As result, the adapted ELBO L (9)0))
(CSWAB) equals
log plue ())0(())))(2))
Σ (x.ell)))) B.EV
(Roo()lu))) [pocelz...) [o(s(s() N.EV
Σ
po (8 2, 2.
(14)
DKL (Po 9:) pp)
DRL ( Dxu(os(llu))ll<()))
To ensure the training of standard VATs to be tractable, (go(slya) is generally approximated by single-sample Monte-Carlo estimation 37
We build on the same strategy and approximate (14) by
(0.0)
Σ B.EY
(E [s]] (xL(g<<
DxL (pe (po(s)ã, y1
ppo (s (()
(15)
with 2: ~ (2(2)))) and (a w. [log exhibiting closed-fom solution detailed in Ap-
pendix D In addition, DKL (pel 2: up )) is the tractable KL divergence of two multivariate
Gaussians, and (KL y. [l(()) matches the KL divergence from standard VAEs (cf. (37).
Both are specified in Appendix E The adapted ELBO in (15) resembles the classical ELBO of ordinary VAEs with a modified reconstruction loss and an additional KI. divergence. Equivalent to
ordinary VAES, we model go (2)))) = N2; (z (Br)). diag( a (1+))) and realize ( (-)) by
NN encoder, and (-(-) )) by NN decoder. The adapted ELBO in (15) forms differentiable and variational approximation of the model's log-evidence E, log po( Based on Theorem 3.1 and
it holds that
Σ N.VV
log ( (z)) N
Σ Log(o(y:) N.EV
(00)] for all 0,
(16)
(CSVAE) Thus, - ((0.)) serves as a variational lower bound of a sparsity-inducing log-evidence. We denote the resulting VAE as Compressive Sensing VAE (CSVAE), and its schematic is presented in Fig. [ It resembles the vanilla VAE with main difference that the encoder takes a compressed observation as input and the decoder solely outputs conditional variances (Y(()) of s|z. In case of varying measurement matrices A, for each training, validation and test sample, we use the least-squares estimate (LS)) AT (A:AT))- "y instead of y, as encoder input. The CSVAE's training with ground-truth data and pscudo-code for training outlined in Appendix Had N (cf. algorithm
3.3 Compressive Sensing GMM
By considering discrete and finite (2) in 7 i.e., Ps(2 = k) = ps = Ph. we observe that the
resulting (s. strongly resembles the GMM's decomposition of p(æ) in 4 This motivates to
choose ps (=) in exactly this manner and po (s|2) p(s(k) Nss;; : O diag (Tkg()))) 1). i.e., 8 modelled
GMM with zero means and diagonal covariance matrices. Equivalent to CS in Section 3.2

---

the distribution p(s/k, y) contained in the model's posterior p(k, sly) p(s]k, y)p(k y) exhibits closed-form solution (cf. (8). Moreover, due to the linear relation between y and in (7) and the assumption that l is Gaussian, ykk is also Gaussian with zero mean and covariance matrix
(17) and pkkyy) is computable in closed form by Bayes, i.e., p(klp) (p(g))))(k))/C p(u (p()))
c A D AT o
Thus, p(k: aly) exhibits a closed form, and an EM algorithm can be applied, whose e-step coincides with computing p(k: s y)) Based on the extended EM algorithm in [23], where the authors fit vanilla GMM (4) in the pixel domain using compressed image patches, we formulate an m-step tailored to model 77). We denote the resulting model as Compressive Sensing GMM (CSGMM), The m-step aims to update the CSGMM's parameters {pk: } by optimizing E, E (p.((,,)))))))) [log p(æ,, 8, k)]. whose closed-fomn solution is given in the following lemma.
Lemma 3.2. Let the statistical model in 7 be given with discrete ps(z k)= p(k)= = Pt and (0(2)= = Th. and let y contain i..dd noisy and compressed observations y:- Given Pe (kelys) and pe(s|k: y.) = (s; PE,t
C...t & Calur. in the tth iteration of an EM algorithm, the updates of {pk: 7ss
(k,(++1) Emey Pc (ell)) (o
Kluet Klu diag(CC (C vi.k))
-RR(t++1) Eney (klpp))
(18)
Ey.eyPi(ke (kly.)
12/
form a valid m-step and, thus, guarantee to improve the log-evidence in every iteration.
The proof of Lemma
3.2 the CSGMM's training with ground-truth data S and pseudo-code are outlined in Appendix and ( algorithm 4 respectively.
3.4 Discussion
From a broader perspective, our proposed algorithm represents two consecutive stages:
1) Choose dictionary D, with respect to which a general signal set C of interest is compress- ible (e.g. a wavelet basis for the set of natural images).
2) Learn the statistical characteristics of the specific signal subset S C C of interest in its compressible domain (e.g. the subset of handwritten digits).
Due to the strong regularization of stage I) constraining the search space in stage 2), our model can learn from a few corrupted samples y. To ensure the search space to be within the set of compressible signals with respect to D, we enforce s| in (7) and, thus, s to have zero mean (cf. Theorem 3.1)
Linitation. On the one hand, this restriction regularizes the problem at hand. On the other hand, it generally introduces a bias, which potentially prevents perfectly learning the unknown distribution of it is not always possible decompose distribution in this way. As result, the proposed model is biased towards capturing the crusity--peccfiu features from y and being an effective prior for (T) rather than learning comprehensive representation of the true p(s).
Distinction from generative model-based CS. Generative model-based CS builds on a generator Go :z + x Go(z) (e.g., GAN or VAE) with z E R.P, E R. and F & N 4. Arguably, the notion of compressibility is still included in this setup, since assumes that every æ can be perfectly encoded by only a few F values. Moreover, these models typically assign simplistic fixed distribution, e.g. p(=) = (2; 0, I), to their compressible domain. Consequently, while generative model-based CS replaces the dictionary with learnable mapping Ge and constrains the compressible domain by forcing it to be representable by, e.g., (O, I). our approach does the exact opposite. We keep the mapping from the compressible into the original domain fixed by dictionary D and learn non-trivial statistical model Po.& (s) in the compressible domain.
Connection to uncertainty quantification. While coming from different motivations, our approach shares similarities with methods from uncertainty quantification, namely so-called prior networks 33.. There, the general idea is to let the NN output the parameters of a conditioned conjugate prior (conditioned on the NN input) instead of directly the quantities of interest (cf. 40 one can use entropy and mutual information measures to quantify different types of uncertainty. Equivalently, our proposed models output the parameters 70(2) of a conditioned conjugated prior pe(s) (e(s)z), which is why they enable to quantify their uncertainty by the same measures.
As a result,
7

---

Computational complexity. It is noteworthy that our algorithm's 's inference after training requires no optimization algorithm but only consists of, e.g., frewwrd operation through a comparably small VAR. Thus, our proposed method's signal reconstruction comes with computational benefits and differs in this context from other approaches. On the downside, naively implementing the training requires computing and storing the posterior covariance matrices in (8) which can lead to a non-neglige negliiiii computational and memory-related overhead for very high dimensions. To overcome this issue, we derive equivalent reformulations of the CSVAE's and CSGMM's update steps for training that circumvent the explicit computation of these matrices, More precisely, the reformulated update steps solely require the explicit storing and inversion of the observations' covariance matrices (cf. ) which are typically much lower dimensional. The reformulations are given in Appendix
Experiments
4.1 Experimental Setup
Datasets. We evaluate the performance on four datasets. We use the MNIST dataset (N 784) for evaluation 42 (CC-BY-SA 3.0 license)]. Moreover, we use an artificial ID dataset of piecewise smooth functions of dimension N = 256. For that, we combine HeaviSine functions with polynomials of quadratic degree and randomly placed discontinuities. which are frequently used for CS and compressible in the wavelet domain [7]. The generation and examples are outlined in Appendix
I We also use dataset of 64 x 64 cropped celebA images (N 3 64 12288) 44 and evaluate on the MNsSi dataset (N = 784) in Appendix
45 (MIT license)].
Measurement matrix & evaluation metric. For the simulations piecewise smooth functions, we use a separate measurement matrix A, for each training, validation and test sample, where all A; contain i..d. Gaussian entries A, (0, M For all remaining simulations, measurement matrix each with equally distributed i.i.d. Gaussian entries for the whole dataset. We evaluate the distortion performance by a normalized MSE nMSE = 1/Niest Σ ( æ; 221l//N) with 2; being the estimation of x; and the SSIM 46 with Neest being set to 5000 any simulation.
one fixed
Baselines & hyperparameters. As non-learnable baselines, we use Lasso and SBL 19 where we either adjust Lasso's 's shrinkage parameter on ground trrth dataset or use the configurations from []. As baselines, which can learn from compressed data, we use CSGAN (30) and CKSVD (24 with OMP, sparsity level and 288 learnable dictionary atoms. We solely evaluate CKSVD on piecewise smooth functions due to its requirement of varying measurement matrices for observing the training samples. For CSGAN on MNIST, we use the configuration specified in 30 In any simulation, we use K = 32 components for the proposed CSGMM and the proposed CSVAE's en- and decoders contain two fully-connected layers with ReLU activation and one following linear layer, respectively. We utilize Adam for optimization 47 and only consider the CME approximation of the estimators (cf. Section2.4] since we did not observe notable differences to the alternative estimators in Appendix G utilize an overcomplete Daubechies db4 dictionary [48] in all estimators with fixed dictionaries. For all estimators except CSGAN, all images and estimates are normalized and clipped between 0 and 1, respectively. For CSGAN, images are normalized between -1 and 1 following [4] 30), and for evaluation, test images and their estimations are then re-normalized between 0 and 1. For a more detailed overview of hyperparameter configurations, see Appendix K
4.2 Results
Reconstruction. In Fig. 2 and 3] the reconstruction performance of all estimators for MNIST, the piecewise smooth functions, and celehA is shown. All trainable models are trained on compressed training samples y' of dimension M without any ground-truth information during training (cf. Section 2.1. The models are then used to estimate test signals æ; from observations of the same dimension M (cf. (T) for evaluation. In the case of the piecewise smooth functions, we also add noise of a 10dB signal-to-noise ratio (SNR) for training and testing, i.e., of = [||| Ax l 10) in (Τ.
'The celebA dataset is released under custom license for nom-commencial research use.
"The original work of CSGAN (30) does not provide results for varying measurement matrices as well as for celebA trained on solely compressed data. We also did not find working hyperparameter configuration, so we leave out CSGAN for the piecewise smooth functions and the celebA dataset.
8

---

Lasso 12
SBL
15
CKSVD 24
CSGAN
30
CSVAE (ORES)
CSGMM (ours)
a) MNIST
<) MNIST
f) piecewise smooth
g piccewise smooth
0.1
0.09
0.11
0.11
nMSE
0.07
0.07
0.08
0.08
0.05
0.04
0.03
0.05
0.05
0.01
00.01
0.02
0.02
200
2 10
2 10
2 10a
10a
10a
10
M
Ne
N₂
b) MNIST
d) MNIST
CSGAN SBL. Original
4
0.8
9
0.8
SSIN
0.6
0.6
4
0
0.4
0.4
0.2
Q
9
0
80
120
160
200
2 10
2
10
2
10
M
Ne
CSGMM CSVAF
(ours) (ours)
4
9
0
7
Original
Lasso
4
7
0
7
SBL
CKSVD
type
DCT
Haar
db3
None
CSVAF
CSVAE
0,0183
0.0170
0.0088
0.0064
0.05
(ouPS)
CSGMM
0.0199
0.0196
0.0100
0.0085
0.0624
Figure 2: a) and b) nMSE and SSIM over M (N₂ = 20000, MNIST), c) and d) nMSE and SSIM over N, (M = 160, MNIST), e) exemplary reconstructed MNIST images (M = 200, N, = 20000), 10dB, N, 10000, piece-wise smooth fct.), g) nMSE over N₂ (SNRaB = 10dB, MM 100, piece-wise smooth fct.), h) exemplary reconstructed piece-wise wise smooth fet. (M 100, N₁ 1000), i) nMSE comparison of dictionaries (MNIST, M 160, N, 20000).
aMSE over M (SNRas
In Fig. 2 a)-d), the 1MSE and SSIM on MNIST is shown for a) and b) varying observation dimensions M and the fixed number N₂ 20000 of training samples, and c) and d) vice versa with fixed M = 160. The error bars represent standard deviations. In terms of both distortion metrics nMSE and SSIM. CSVAE and CSGMM perform overall the best. In Fig. (c), exemplary reconstructed MNIST images for M = 200 and N, = 20000 are shown. Perceptually, CSGAN emphasizes different reconstruction aspects than the proposed CSVAE and CSGMM. While CSVAE and CSGMM successfully recover details of the MNIST images, CSGAN prioritizes the similarity in contrast between bright and dark areas. This can be explained by their different estimation strategies. The estimation of CSGANs is restricted to lie on the learned manifold, which has been adjusted based on the training dataset. In consequence, when the test sample x contains details that cannot be found in the training dataset, CSGANs cannot reconstruct these but rather output a similar but realistic representative from the manifold. On the contrary, our proposed estimation scheme in Section 2.4 is not restricted to manifold but rather infers conditional distributions (cf. (8) over the whole linear space. Moreover, Fig. c)) and d) demonstrate that the proposed model can effectively learn from only a few hundred compressed and noisy samples. In Fig. ii.. CSVAE and CSGMM are compared using different dictionaries types. The overall nMSE performance remains good for different dictionaries except for applying the models directly in the pixel domain.
In Fig. 2 D. g) and h), the nMSE over varying M (see f)), varying N, (see g)) and exemplary reconstructed samples (see h)) are shown for the set of piecewise smooth functions. Here, we only evaluated the proposed CSVAE$J performs the best in distortion (see f) and g)) as well as perception (see h)). In Fig. a))-), the same plots are shown for the celebA dataset. Despite the significantly larger dimension, the results are consistent with those for MNIST and the set of piecewise smooth functions.
the baselines Lasso, SBL, and CKSVD, the CSVAE
*e leave out the CSGMM due to the need to compute separate posterior covariance matrices for all training samples by using varying measurement matrices (cf. ( which leads computational overhead.

---

Lasso
P
SBL 19
CSVAE (OurS)
CSGMM (ours)
labels of the lest samples y
a) celebA
b) celebA
f) trained compressed zeros
0.008
uMSE
0.03
Counts
80 15 8
0.006
0.02
0.001
0.01
0.002
900
1800
2700
500
h(*ly)
M
N₂
Original
c) cclebA
d) celebA
0.8
SSIM
0.6
0.8
0.7
Lasso
0.2
0.6
900
1800
2700
500
1581
5000
M
N₂
SBL
g) MNIST (M- 200,
20000)
Training time (minutes)
Avg reconstrue- tion time (ms)
Model
CSVAE
Lasso SBL
73.85
294. 17
CSGAN
1481
480.81
CSOMM
CSGMM (ours)
45
2.64
(ours)
CSVAE (ours)
88
2.02
Figure a) and b) nMSE and SSIM over M (N; 5000), c) and d) nMSE and SSIM over Ne (M 1800), e) exemplary reconstructed celebA images (M 2700, N₂ 5000). f) histogram of hx for compressed test MNIST images of digits 0.1 and 7, where the CSVAE is trained on compressed zeros, g) training and reconstruction time for MNIST (M 200, N. = 20000).
Uncertainty quantification. One possibility determine the model's uncertainty is determine the differential entropy h=2ly) of the CSVAE's encoder distribution qo (=lyy) Since qa (=ly) is Gaussian with diagonal covariance matrix (ef. Section 3.2). h(z yy) can be calculated efficiently in closed form. In Fig. 3 f). histogram is shown with values of h(zly). The CSVAE is trained on solely compressed MNIST zeros. We then forward compressed zeros, ones and sevens and evaluate h( 2 (ly. It can be seen that the CSVAE's encoder identifies the observations that are distinct to the training dataset by providing larger entropy values on average for the compressed ones and sevens.
Runtime. Fig. g shows our measured training time as well as average reconstruction time of the MAP-based estimators (cf. Section G for MNIST, M 200, N; 20000. CSGMM and CSVAE are considerably faster than the baselines validating the corresponding discussion in Section 3.4
Additional results. The results in Fig. and 3 display the performance for CSGAN and the proposed CSVAE and CSGMM trained on solely compressed data. However, all three models can also learn from ground-truth data (cf. Appendix [). In Appendix D we include this comparison for MNIST, provide results on FashionMNIST, analyze the estimators' robustness, and plot further reconstructions for all datasets. Moreover, in Appendix M we analyze the runtime for training and reconstruction in more detail and give an overview of the used compute resources.
5 Conclusion
In this work, we introduced new type of learnable prior for regularizing ill-conditioned inverse problems denoted by sparse Bayesian generative modeling. Our approach shares the property of classical CS methods of utilizing compressibility, but the same time, it incorporates the adaptability to training data. Due to its strong regularization towards sparsity, it can learn from a few corrupted data samples. applies to any type compressible signal and can be used for uncertainty quantification. While this work focused on setting up the sparse Bayesian generative modeling framework, extensions learnable dictionaries, circumventing the inversion of the observations' covariance matrices during training. and perceptioon-emhhasziigg estimators part of future work.
10

---

Acknowledgments and Disclosure of Funding
This research was supported by Rohde & Schwarz GmbH & Co. KG. The authors gratefully acknowledge the financial support and resources provided by the company. The authors also sincerely appreciate the valuable discussions with Dominik Semmler and Benedikt Fesl.
References
[1] Y. Eldar and G. Kutyniok, Compressed Sensing: Theory and Applications. University Press, 11 2012.
Cambridge
[2] K. Gregor and Y. LeCun, "Learning fast approximations of sparse coding," in Proceedings of the 27th International Conference on International Conference on Machine Learning, ser. ICML' 10. Madison, WI, USA: Omnipress, 2010, p. 399-406.
[3] Liu, X. Chen, Z. Wang, and W. Yin, "ALISTA: Analytic weights are as good as learned weights in LISTA," in International Conference on Learning Representations, 2019.
[4] A. Bora, A. Jalal, E. Price, and A. G. Dimakis, "Compressed sensing using generative models," in Proceedings of the 34th International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, vol. 70. PMLR, Aug 2017, pp. 537-546.
[5] D. Ulyanov, A. Vedaldi, and V. Lempitsky, "Deep image prior," International Journal of Computer Vision, vol. 128, no. 7, p. 1867-1888, Mar. 2020.
[6] R. Heckel and Hand, "Deep decoder: Concise image representations from untrained non- convolutional networks," in 7th International Conference on Learning Representations, ICLR 2019, New Orleans. LA, USA, May 6-9, 2019. OpenReview.net, 2019.
[7] Z. Kadkhodaie and Simoncelli, "Stochastic solutions for linear inverse problems using the prior implicit in denoiser," in Advances in Neural Information Processing Systems, M. Ranzato, A. Beygelzimer, Y. Dauphin, P. Liang, and W. Vaughan, Eds., vol. 34. Curran Associates, Inc., 2021, pp. 13 242-13 254.
[8] T.-O. Buchholz, M. Jordan, G. Pigino, and F. Jug, "Cryo-care: Content-aware image restoration for cryo-transmission electron microscopy data," in 2019 IEEE 16th International Symposium on Biomedical Imaging (ISBI 2019), 2019, pp. 502-506.
[9] G. Revach, T. Locher, N. Shlezinger, R. J. G. van Sloun, and R. Vullings, "Hkf: Hicrarchical Kalman filtering with online learned evolution priors for adaptive ECG denoising," IEEE Transactions on Signal Processing, vol. 72, pp. 3990-4006, 2024.
[10] W. Kim, Y. Ahn, J. Kim, and B. Shim, "Towards deep learning-aided wireless channel estimation and channel state information feedback for 6G," Journal Communications and Networks, vol. 25, no. 1, pp. 61-75, 2023.
[11] A. Jalal, L. Liu, A. G. Dimakis, and C. Caramanis, "Robust compressed sensing using generative models," " in Neural Information Processing Systems, 2020.
[12] R. Tibshirani, "Regression shrinkage and selection via the Lasso," Journal of the Royal Statisti- cal Society (Series B), vol. 58, pp. 267-288, 1996.
[13] R. Baraniuk, "Optimal tree approximation with wavelets," Proceedings of SPIE The Interna- tional Society for Optical Engineering, vol. 3813, pp. 196-207, Dec. 1999.
[14] A. Liu, V. K. N. Lau, and W. Dai, "Exploiting burst-sparsity in massive MIMO with partial channel support information," IEEE Transactions on Wireless Communications, vol. 15, no. 11, pp. 7820-7830, 2016.
[15] L. He, H. Chen, and L. Carin, "Tree-structured compressive sensing with variational Bayesian analysis," IEEE Signal Processing Letters, vol. 17, no. 3. pp. 233-236, 2010.
[16] I.. Нe and I.. Carin, "Exploiting structure in wavelet-based Bayesian compressive sensing." IEEE Transactions on Signal Processing, vol. 57, no. 9, pp. 3488-3497, 2009,

---

[17] R. G. -Braaniuk, V. Cevher, M. F. Duarte, and C. Hegde, "Model- based compressive sensing," IEEE Transactions on Information Theory, vol. 56, no, 4, pp. 1982-2001, 2010.
[18] M. E. Tipping, "Sparse Bayesian learning and the relevance vector machine," J. Mach. Learn. Res., vol. 1, p. 211-244, sep 2001.
[19] D. Wipf and B. Rao, "Sparse Bayesian learning for basis selection," " IEEE Trunsactions on Signal Processing, vol. 52, no. 8, pp. 2153-2164, 2004.
[20] G. Yu and G. Sapiro, "Statistical compressed sensing of Gaussian mixture models," IEEE Transactions on Signal Processing, vol. 59, no. 12, pp. 5842 2442-5858, 2011.
[21] Renna, R. Calderbank, L. Carin, and M. R. D. Rodrigues, "Reconstruction of signals drawn from a Gaussian mixture via noisy compressive measurements," IEEE Transactions on Signal Processing, vol. 62, no. pp. 2265-2277, 2014.
[22] G. Yu, G. Sapiro, and S. Mallat, "Solving inverse problems with piecewise linear estimators: From Gaussian mixture models to structured sparsity," IEEE Transactions on Image Processing, vol. 21, no. 5, pp. 2481-2499, 2012.
[23] J. Yang, X. Liao, X. Yuan, P. Llull, D. J. Brady, G. Sapiro, and L. Carin, "Compressive sensing by learning a Gaussian mixture model from measurements," IEEE Transactions on Image Processing, vol. 24, no. 1, pp. 106-119, 2015.
[24] Pourkamali Anaraki and S. M. Hughes, "Compressive K-SVD," 2013 IEEE International Conference on Acoustics, Speech and Signal Processing. 2013, pp. 5469 1473..
[25] F. P. Anaraki and S. Hughes, "Memory and computation efficient PCA via very sparse random projections," in Proceedings of the 31st International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, vol. 32, no. 2. Bejing, China: PMLR, 22-24 Jun 2014, pp. 1341-1349.
[26] F. Pourkamali-Anaran S. Becker, and S. M. Hughes, "Efficient dictionary learning via very sparse random projections," in 2015 International Conference on Sampling Theory and Applica- tions (SampTA), 2015, pp. 478 482.
[27] Chang, B. Tolooshams, and D. Ba, "Randnet: Deep learning with compressed measurements of images," in 2019 IEEE 29th International Workshop on Machine Learning for Signal Processing (MLSP), 2019, pp. 1-6.
[28] M. Aharon, M. Elad, and A. Bruckstein, "K-SVD: An algorithm for designing overcomplete dictionaries for sparse representation." " IEEE Transactions on Signal Processing, vol. 54, no. 11, pp. 4311-4322, 2006.
[29] A. Bora, E. Price, and A. G. Dimakis, "AmbientGAN: Generative models from lossy measure- ments," in 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 2018, Conference Track Proceedings. OpenReview.nee, 2018.
[30] M. Kabkab, P. Samangouci, and R. Chellappa, "Task-aware compressed sensing with generative adversarial networks," in AAAI Conference on Artificial Intelligence, 2018.
[31] B. Fesl, Böck, Strasser, M. Baur, M. Joham, and Utschick, "On the asymptotic mean square error optimality of diffusion models," 2024, arXiv:2403.02957.
[32] M. Koller, Fesl, N. Turan, and W. Utschick, "An asymptotically MSE-optimal estimator based on Gaussian mixture models," IEEE Transactions on Signal Processing, vol. 70, pp. 4109-4123, 2022.
[33] M. Baur, B. Fesl, and W. Utschick, "Leveraging variational autoencoders for parameterized MMSE estimation," IEEE Transactions Signal Processing, vol. 72, pp. 3731-3744, 2024.
[34] O. Berger, Statistical decision theory and Bayesian analysis. 1985.
New York: Springer- Apringer-Verllg,
12

---

[35] G. J. McLachlan and D. Peel, Finite mixture models. 2000.
Wiley Series in Probability and Statistics,
[36] C. M. Bishop, Pattern Recognition and Machine Learning (Information Science and Statistics), 1tt ed. Springer, 2007.
[37] D. P. Kingma and M. Welling, "An introduction to variational autoencoders," Foundations and Trends@ in Machine Learning, vol. 12, no. 4, 307-392, 2019.
[38] Y. Blau and T. Michaeli, "The perception-distorie tradeoff," in 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. IEEE, Jun. 2018
[39] D. T. Ulmer, C. Hardmeier, and J. Frellsen, "Prior and posterior networks: survey on evidential deep learning methods for uncertainty estimation." Transactions on Machine Learning Research, 2023.
[40] A. Malinin and M. Gales, "Predictive uncertainty estimation via prior networks," in Advances in Neural Information Processing Systems, S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa- Bianchi, and R. Garnett, Eds., vol. 31. Curran Associates, Inc., 2018.
[41] A. Amini, W. Schwarting, A. Soleimany, and D. Rus, "Deep evidential regression," Advances in Neural Information Processing Systems, vol. 33, 2020.
[42] Y. Lecun, L. Bottou, Y. Bengio, and Haffner, "Gradient-based learning applied document recognition," Proceedings of the IEEE, vol. 86, no. 11, pp. 2278-2324, 1998.
[43] D. L.. Donoho and I. M. Johnstone, "Adapting to unknown smoothness via wavelet shrinkage," Journal of the American Statistical Association, vol. 90, no. 432, pp. 1200-1224, 1995.
[44] Z. Liu, P. Luo, X. Wang, and X. Tang, "Deep learning face attributes in the wild," in Proceedings of International Conference Computer Vision (ICCV), December 2015.
[45] H. Xiao, K. Rasul, and R. Vollgraf, "Fashion-MNIST" a novel image dataset for benchmarking machine learning algorithms," 2017, arXiv:1708.0777
[46] Z. Wang, A. Bovik, H. Sheikh, and E. Simoncelli, "Image quality assessment: from error visibility to structural similarity," IEEE Transactions on Image Processing, vol. 13, no. pp. 600-612, 2004.
[47] D. Kingma and J. Ba, "Adam: A method for stochastic optimization," in International Confer- ence on Learning Representations (ICLR), San Diega, CA, USA, 2015.
[48] G. R. Lee, R. Gommers, F. Waselewski. K. Wohlfahrt, and A. O'Leary, "Pywavelets: A python package for wavelet analysis," Journal of Open Source Software, vol. 4, no. 36, p. 1237, 2019.
[49] R. Blahut, Principles and practice of information theory. Publishing Co., Inc., 1987.
USA: Addison- Wesley Longman
[50] P. Bickel and K. Doksum, Mathematical Statistics: Basic Ideas and Selected Topics. Chapman and Hall/CRC, 2015.
[51] Pedregosa, G. Varoquaux, A. Gramfort, Michel, B. Thirion, Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay, "Scikit-learn: Machine learning in Python," Journal of Machine Learning Research, vol. 12, pp. 2825-2830, 2011.
13

---

A Proof of Theorem 3.1
To prove Theorem 3.1 restate (3), i.e., there exists C >
such that for all y > 0 and
(s; diag(n)) S ((8) C
(19)
88
Based 19 there exists C such that for all 0, 8, z with Ya(2)
NV
1
Po s 2) (s; 0. diag(rro (z))) - t(s) C
(20)
By multiplying both sides in (0 with (z(z) and integrating over z, we conclude that there exists
C' > 0 such that for all 0, s, ô with 70 z)) >
po.s (s) p6 (s|2) 1 pp(())((sddz = t(s)
(21)
independent of the particular choice of ps(z). In case of discrete and finite ps(z) the integral corresponds to a summation.
B Derivation of the adapted ELBO for CSVAES
We write out the KL divergence and utilize that pe (zy is independent of the expectation, i.e.
L ()
(CSVAE)
Σ
log po()))
(xo slx))lle sly:))
(22)
BEEY
log po po(si)po(z,slg
E (2o(2.0)))) log p(vils po ( (00())2)00) ((2))((2)
= Σ
(0.2.
(z, 8/y:)
WEY
WEY
(23)
C Conditional Posterior in the Noise-Free Case
In the noise-free case, the closed-form covariance matrix and mean of pe (s|2, y) are given by
Calux (2) = (I diag( (77())) (ADdiag(v<r(())
vr((z)) AD (aag()a(2))
(24)
elv.= (z) = diag( (roag(/7((z))
(ADdiag (ADdiag(Vno(3))) y.
(25)
with (-)1 being the Moore-Penrose inverse 19
D Closed-Form Solution of the CSVAE Reconstruction Loss
(ogtess... = Ep 2..))
-
1
M log(2too?)
Φ yy ADx113) (26)
= M log (
+ 0²
1
E Epo(s s. |1 y, ADs (27)
T TD' A yi + tr traDsss T ''DT ATAA)))
= -
(M M loggg222ee (2TT²)
+ -²
11ael12 シे AD
p4
* v
(ž,)
14
a lB (z,) T (28)
+u((AD (Cllu:
14
| 2
(2
1 (z: TT) DT DTAT))
=-
- M (log(2mo²) - [1y/ ADI 9.
( Ir(ADD |M
(Z) A
(29)
where pig
2ly.. (2.) and Ce v.., ( are the mean and covariance matrix of pe 2 y.).
14

---

E Closed-Form Solution of the CSVAE KL Divergences
By inserting the closed form of the KL. divergence between two Gaussians (cf. 49 we get
log det (diag(rro (z))) log det (cg) (Crlu. (&:)
(po
(30)
+tr diag ((7a(2)) ( C M. (Z:) + 14 elv.. (2,) diag
vi.ki (2:)
where Me
elur. (2,) and Co M. (2,) are the mean and covariance matrix of po y:). Moreover
471.
DKL Dku(oo
Σ (1 log " 13. of (y:))
(31)
with (zly)) NNx uo (3i) ( and (v)) and 03.4 (3/:) being the jth entry of (v:) and o (y:), respectively. Additionally, N₂ denotes the CSVAE's latent dimension.
F Proof of Lemma 3.2
The optimization problem aim to solve is given by
(P) (c.(e)))) argmax (pe.m) D:EY
Σ
(p.(k))))))) [logp(y.. s.t. Σ Ph: =1
(32)
First, we reformulate the objective as
(k. v.) [logp(y;, k:] eEY
Σ (k.cla))) (logp(y:(s) + logp(s(k)) log)(k)), (33)
Moreover, we observe that log p(yi(s)) does not depend on {kk: and we can leave it out from the optimization problem. Additionally,
Σ (klvr))
[pa(ele..)) ((s/(s)))) + log p(k)]]
BIEY
(34)
5
Ep.(sly).)) [|sg|
Σ Σ WE 1=1
Pe(klys)
S log 2nt +
Σ
log Vhes +
+ log Pk
where we inserted p(s(k) (ss 0, (aagg()))))))))))))))))))) (cf. and 7,. denotes the jth entry of Yk- In the next step, let (p)(elu..k) [0, ME.
kly, + C l where Ma. alyn.k and C.
RJJ denote the jth
entry of kk, "'v..k and the diagonal of Ck. problem of interest can be restated as
d in the tth iteration, respectively. Thus, optimization
(klla.) (+1 Slog 2T + Σ
aluu. Pkk
+ Cluls
)
argmax ΣΣ
log Thit Ykg
log pk
(mx-TT)) wEyk=1
s.t. Σ 1
(35)
with Lagrangian
L= Σ po(k))))
K (-1
S log 2n +
Σ
5
log yk.j +
1
1l Cellick
))
Yru
B.EY 1m
1=1
(36)
log + v(1
Σ Pk)
15

---

and Lagrangian multiplier v. Taking the derivative of L with respect to Ym.g and pm and setting it to zero leads
c =
1
Σ
pe(mly))
1
1mm.0
+ C br
0
(37)
d'mma
2
Ymm.
Vin.g
D.EY
(m
C=
1
(38)
3pm
Pm
D:Ey
and, thus,
Eme Pe(mlyr)
(t.1)
Ev.Ey Pe(ml)gi)
(39)
Pm,(t+1) Eyecy Pc(ml
(40)
w= |1
(41)
where 41 comes from the normalization of Σn Pk =
G Estimators Based on the CSVAE and CSGMM
CSVAE. The CME approximation in 9 with the proposed CSVAE in Section 3.2 is given by
D
i CSVAE.CME
E [p((s(E,.)
[sly. 2;
(42)
Z
2:E2
with containing samples 2 ~ (0(2 8). Alternatively, one can estimate x" based on a newly
observed y in the following way. We the mean (3 as maximum posteriori (MAP) estimate based on g 90 (zly) and, then, estimate x by
AASAAEAPP = D (po(ellug())..) [sly, (1)) Dug alv. (1)(
(43)
This method is applied in 33 to reduce computational complexity, but also deviates from the CME
approximation and potentially improves the perceptual quality of the reconstruction.
CSGMM. In case of the proposed CSGMM in Section 3.3 the CME approximation in exhibits a closed form, i.e.,
CSGMM.CME
D
D p(kly) [s]kk v).
(44)
An alternative estimator of xt is given by
CSGMMMAP = DE
p(a (ouerr [KMAP: y)
(45)
with KMAP arggax p(ky) 23 A pseudo-code for all estimators is provided in Appendix N (cf.
algorithm 6
H CSVAE and CSGMM Training on Ground-Truth Data
The CSGMM as well as the CSVAE can both be trained given ground-truth datasets. For training the CSGMM, this dataset either contains the compressible ground-truth signals 8,, i.e., S, the signals x; themself, i.e., X. On the other hand, for training the CSVAE, both datasets must also contain their corresponding observations y:. Thus, the training dataset must be or W (cf. Section 2.1
CSGMM. To train the CSGMM, the training goal is to maximize the log-evidence of the training dataset. case having access to S, the vanilla EM algorithm is employed with the modification of enforcing the GMM's means to be zero and covariance matrices to be diagonal in every update
16

---

step. More precisely, after computing the posteriors (r(k)s;) by the Bayes rule in the tth iteration, the m-step is given by
Pr Ph.(++1) Pi
(k,++1)
(46)
[8]
In case of having access to X, the same modified EM algorithm from Section 3.3 is employed by exchanging AD with D and setting the noise variance o² some small value.
CSVAE. To train the CSVAE using the dataset G, we also aim optimize the log-evidence of the training dataset
Σ (#..M))GG
Po (8t, y:)
Σ
log po (3r 8:) + log po;;)
(47)
(#:
By considering that log pe (3:))ss:) does depend on the CSVAE's parameters (see 77). introducing additional variational parameters & and approximating pe(2|y, 8) with (0(=)y), we apply the standard reformulations of VABs, which ends up in ELBO resembling the standard ELBO in (6). given by
L(0,0) 6)
Σ (a.N))Eg
log (esposs
(xo
(48)
with 2; ~ (=(y))) and po (8())) being defined in (T0) with p(z) 0,; In the case of training the CSVAE with W, the same training procedure from Section 3.2 is applied, resulting in the modified ELBO L(o approximated by
Σ DEW
(2o (=lv)) (=())))((())) - DRI. 2,, az, (s (=)))))
(49)
where we set the variational distribution ( z 2; Mi) of the latent variables given the observations to q y, (s)2,, Consequently, instead of computing the posterior po(s|z, y:). we use
po (o(s] 2;), which is given by (8) with D instead of AD and setting the noise variance o² some small value. Additionally, to compute (epo(e(.....) [log pl (m [s]] we replace y:, M and AD in (29 with æ,, N and D. respectively.
I Implementation Aspects for Reducing the Computational Overhead
In a first step towards computationally more efficient implementation for training the CSVAE and CSGMM, we reformulate the expressions of the conditional mean μ40 </y.= (z) and conditional
do so, we observe that conditioned on z, s and in (7) are jointly Gaussian. In consequence, we can alternatively apply the standard formulas for computing the moments of conditional distribution for jointly Gaussian setup [50] i.c.
covariance matrix Cg C=lv.= from 18
ely. <l)=*(2) C (z)
(C (z) y
Hg
(50)
Gelv. (z) = (iagg(o(=)) (z) (cxl
(2)
(51)
with
Cæll= (z) (odiag(rr(z)))) AT 1r++d I²I
(52)
and C d AT Importantly, do not need to explicitly compute C. ( according to (11) The subsequent reformulations differ between CSGMM and CSVAE.
CSGMM. For the CSGMM, only the diagonal entries of Caly. (2) must be explicitly computed, i.e.,
diag (Colu: = (=) - diag C. (x)) O Fz))
-1
C Cg. (x))²
(53)
17

---

The matrix Cg (z) (z) (i.e., Oh (cglh) has been precomputed for the con- ditional mean in 50 and subsequently determining diag Ch (CC't)""(C""" (c) )) only
requires OSSM) operations. Moreover, (Ckll) has already been determined for evaluating the posterior distributions p(k(y)) in the preceding e-step (cf. Section 3.3 The closed-form m-step in (8) only takes diag (Cll. for which we can directly use (53 In this way, we circumvent explicitly computing the full posterior covariance matrices in rendering training the CSGMM more efficient.
CSVAE.
The objective for training the CSVAE is given in 15]. which consists of the modified
reconstruction loss in 229 as well as the KL divergences in 330 and 31
While 14 <iv. (z) in (29)
be directly computed using (50), we reformulate the trace-term in 29 to circumvent the explicit
computation of Co C (z). To do so, we apply the following steps
tr( tr(ADo (Z:)D"AA) = ir (C (Z A AD)
(54)
8 El
(Zi)
o² tr (o) (())))))))))
diag(no' (21)) = = S
1=l y0,r
(55)
with Caluu.E (ž.) being the jth diagonal entry of C C (2,) and y0, being the jth entry
of 7o (2). (z,)..
For the derivation, we mainly apply the formula of C &elvi. (2:) in (8. By observing
that the 2² in (55) cancels out with the 1/0² in (29), the trace-term (29 cancels out with -
diag
C (E:) in (30 We also reformulate log det (C (Ex) in (30).
rendering the explicit computation of & obsolete, i.e.,
(Co) (C
- log det ( DT ATAT diag(Tø ((:)) = (56)
log det
((22TT
1 DT AT ATAT ATAAAAD AD + diag(rø (εt))
llog det
(iag()o(2;)) diag(( ((x)) (57)
((
1 DJ A AD
- log det
+ diag(1) 7o ((.)) diag(rve(:)))
+ log det (ε:))) = (58)
02
- log det
DJ ATATADDIE ADdiaa(rro
+ log dogddidiia((ne (ž+))) = (59)
σ²
log det
ADdiag( (70 (2:)) AT
+ logdet(diag(no) (ã:)) = (60)
1 I) log det C/ (2,) + logdet 0²
log let
(61)
M logoo 0² - log det Cyi ž; (ž.) + log det(diag( (ž,))) (62)
where the formula of & in K). &
(2,) A Ddiag (oo (2,
DT AT
o² as
well as Sylvester's determinant theorem for the reformulation from 59 to 60
In general, equivalent reformulations can also be applied when the CSVAE and CSGMM are trained on ground-truie -tttt data (cf. Appendix H by replacing C &: (2,) with C m. (2,).
'For the CSGMM we use the notation from Section 3.3
18

---

0.6
0.5
0
-0.5
100
200
100
200
100
200
Figure 4: Exemplary signals within the 1D dataset of piecewise smooth functions.
1.5
-155
100
200
0
100
200
0
100
200
Figure 5: Wavelet Transforms of the exemplary signals in Fig.
J
Implementation of the ID Dataset of Piecewise Smooth Functions
The artificial ID dataset of piecewise smooth functions is generated in the following way:
(6 t a sin(4 77 () t [0. g1)
(1)
(1) =
G 2 + sin + 71 (2)). t [91, 92)
(63)
== 1 + sin(4nt + 11 (8)). t E [g2, 4)
(2)
where 1 (2 Bernouli(0.5)) and either takes 0.4 or 0.4, d;
((0,0.1²) 1/ (m) U(0, 2m)
91 (0, 2) and 92 (22 4). After generation, each (t) is sampled equidistantly N 256 times, and its samples are stored in vectors 2,, which then constitute the ground-truth dataset X'
Exemplary signals and their wavelet decomposition are given in Fig. 4 and [ respectively.
K Detailed Overview of Hyperparameter Configurations
MNIST & FashionMNIST. The non-learnable baselines for the simulations on MNIST are Lasso as well SBL. We apply Lasso directly the pixel domain with its shrinkage parameter À set 0.1 in line with [4]. For SBL, we use an overcomplete db4 dictionary with symmetric extension [4]] Moreover, although we do not include noise in the simulations for MNIST, we set o² in (2) to an increment corresponding to 40dB SNR to be able to apply the computationally efficient reformul aations from Appendix[] [ For CSGAN, we use the exact hyperparameter configurations specified in 30 For CSGMM, we set the number K of components to 32 and iterate until the increments of the training dataset's log-evidence reach the tolerance parameter of 10 -0-3 a standard stopping criterion for GMMs [51] The CSVAE encoders and decoders contain two fully connected layers with ReLU activation and one following linear layer, respectively. The widths the layers are set a way such that for the first two layers, the width increases linearly from the input dimension to 256, while the final linear layer maps from 256 to the desired dimension (i.e., either S for the decoder or twice the latent dimension for the encoder). The latent dimension is set to 16, the learning rate is set to 2 10 10-5, and the batch size is set to 64. We use the Adam optimizer for optimization [47). We once reduce the learning rate by factor of 2 during training and stop the training, when the modified ELBO in (15) for a validation set of 5000 samples does not increase. For CSGMM as well as CSVAR we set o² in 7 to an increment corresponding to 40dB SNR to be able to apply the training reformulation from Appendix [ We also use the same overcomplete db4 dictionary as for SBL. Moreover, we use N, 64 samples to approximate the outer expectation in 9
Piecewise smooth function. For the simulations on the set of piecewise smooth functions, we adjust the shrinkage parameter of Lasso based on ground-truth validation dataset of 5000 samples once for every M. Moreover, we also use the overcomplete db4 dictionary for Lasso as for all other dictionary-basse estimators. Instead of choosing 256, we choose 128 as the maximum width of the en- and decoder layers. Otherwise, the hyperparameters remain the same as for the simulations on MNIST.
CelebA. For the simulations on celebA, we set the shrinkage parameter À of Lasso to 0.00001 (cf. celebA contains colored images, we choose a block-diagonal dictionary with the overcomplete db4 dictionary three times along the diagonal and zero matrices in all off-diagonals.
19

---

CSGAN 30
CSVAE (trained J/
CSGMM (trained V/
SBL
10
(trained }
CSGAN
CSVAE
CSGMM
(friiiee on greand truth)
(trained on ground truth, oun)
(trained ground fruth, ours)
a) MNIST
MNIST
MNIST
FashionMNIST
0.04
0.13
0.05
nMSE
0.03
0.09
0.02
0.03
0.03
0.05
0.01
0.01
0.01
0.01
16
64
120
160
200
2 10"
2 2-10"
2. 10"
80
120
160
200
N.
M
N₂
M
b) MNIST
d) MNIST
D MNIST
h) FashionMNIST
0.8
0.8
0.8
SSIM
0.6
0.7
0.6
0.6
0.4
0.6
0.4
0.2
0.5
64
120
160
200
2 10
2. 10
2 10
80
120
160
200
N.
M
N₂
M
Figure 6: a) and b) uMSE and SSIM comparison over number N. of estimations per observation 20000), c)-f) uMSE and SSIM performance of models trained on well models trained ground-truth data (dashed curves)
(MNIST, M = 200.N,
compressed data (solid curves) W and over c) and d) M (N₂ = 20000, MNIST), and e) and f) N; (M = 160, MNIST), g) and h) performance comparison on FashionMNIST.
Each block corresponds to one color channel. We use this dictionary for all estimators on celebA. The batch size to set to 32. Otherwise, all hyperparameters are chosen in the same way as for MNIST.
Generally, we did no detailed network architecture search for the proposed CSVAE since observed no considerable change in performance by testing out different architectures.
L Additional Results
Robustness comparison. The proposed CSVAE relies on approximating the outer expectation in (9) by Monte-Carde sampling. Similar to this approximation, the baseline CSGAN also applies several random restarts, i.e., it estimates the ground-truth sample several times for a single observation and chooses the best-perfin rest-performin estimation by comparing their tractable measurement errors 500 4 In this way, both methods can be compared terms of the number N, repeated estimations they perform for single observation. In Fig. a) and b), we compare the nMSE and the SSIM for both with respect to N,. We evaluate their performance on the MNIST dataset with M = 200 and N 20000. It can be seen that the proposed CSVAE achieves already good performance for single Monte-Carlo sample, i.c., N, = 1, while CSGAN is significantly worse when only using one restart (i.c., N₂ 1) compared to having many restarts.
MNIST with training on ground-truth data. In Fig. ())-f), the results from Fig. a)-d) extended with the performance of the corresponding models trained on ground-truth data, which are represented by the dashed curves. should be noted that ground-truth information here refers the MNIST images e; themself. In consequence, CSGAN as well as the proposed CSGMM are trained on X, while CSVAE is trained we leave out the error bars in these plots. Generally, the CSGAN, as well as the proposed CSVAE and CSGMM, benefit from the additional information during training in terms of the distortion metrics nMSE and SSIM. For the nMSE, the overall performance comparison remains the same, while for SSIM CSGAN trained on X' outperforms CSVAE and CSGMM trained on W and X, respectively. In Fig- 7 additional exemplary reconstructed MNIST images are shown for all models trained on compressed as well as ground-truth data X' case of the CSGAN and CSGMM and W in case of the CSVAE It can be seen that perceptually, the CSGAN significantly benefits from the ground-truth
W (cf. Section 2.1 and Appendix HD.. For the sake of readability,
20

---

SBL Originn
2
7
9
5
6
5
4
1
3 3
4
2
7
2
a) reconstructions from models solely trained on compressed data y without any ground-truth information
CSOMN CSVAE CSGAN (OEN)) (OUDS)
2
4
5
6
4
1
3
2
2
7
2
2
4
9
5
6
5
4
1
3
4
2
2
2
4
9
6
5
4
3
4
2
7
2
b) recomstructions from models trained on ground-truth data X' W
CSGAN
2
Y
9
5
6
5
O
7
4
1
3
2
7
2
CSGMA CSVAL
(OUNS) (Oun)
2
4
9
5
6
5
4
1
3
4
2
7
2
2
4
5
6
5
4
1
3
4
2
7
2
Figure 7: Exemplary reconstructed MNIST images for M 200, N, 20000 from a) models,
which are solely trained on compressed data (with observations of dimension M), and b) models, which are trained on ground truth data.
information during training, while the proposed CSVAE and CSGMM perform similarly in perception for training set of compressed or ground-truth samples. This highlights their effective regularization effect explained in Section 3,4
FaShionMNIST. In Fig. g and h). the performance of SBL, CSGAN, and the proposed CSVAE and CSGMM for varying observation dimensions M and fixed number N₂ = 20000 of training samples is shown. While for small M, CSGAN outperforms the proposed CSVAE and CSGMM. its performance saturates for increasing M, and it performs worse than CSVAE and CSGMM. In this case, CSGAN's regularization to enforce the reconstruction to be in the generator's 's domain beneficial for strongly compressed observations (i.e., for small M). In Fig. g exemplary reconstructed MNSS images are shown.
Additional exemplary reconstructions. In Fig. and 11 additional exemplary reconstructions for the piecewise smooth functions, MNIST as well as celebA are shown.
M Overview of Compute Resources
All models have been simulated on an NVIDIA A40 GPU except for the proposed CSGMM, whose experiments have been conducted on an Intel(R) Xeon(R) Gold 6134 CPU @ 3.20GHz. We report the number of learnable parameters, the time used for training well the average reconstruction time after training for simulations with piecewise smooth functions, MNIST and celebA in Table [ 3 respectively. The average reconstruction time of estimating x" from newly observed y has been measured for the MAP-based estimators in Appendix G While the reported numbers give an overview of the comparison between the different tested models, it is important note that did not aim to fully optimize our simulations for computational efficiency.
N
Pseudo-Code for the Training and Inference of the CSVAE and CSGMM
Algorithm| summarizes one iteration of the training procedure for the CSVAE Algorithm does the same for the CSGMM. In algorithm 2 and 3. the pseudo-code of the CME approximation and the MAP-based estimator using the CSVAE E presented, respectively (cf. and (43 ). algorithm
and 6 the same is given for the CSGMM (cf. (44) and (45)
21

---

CSGMM CSYAR CSDAN SBL Original
(Oun)) (cuns)
Figure 8: Exemplary reconstructed FashionMNIST images (M 200, N₂ 20000, Fig.
g). h))
Original
mm
Liso
SBL.
hithy
OSVAE CKSVD (oun))
Figure 9: Exemplary reconstructed piece-win wiseewwieeae smooth functions (M 140, N, 10000, Fig.
SBL Orginol
7
4
2
3
5
\
2
4
6
3
5
5
0
4
1
7
4
3
5
CSGMA CSVAR USGAN
1
7
4
/
3
5
1
3
4
3
G
O
O
4
(NWrS) (001S)
7
4
3
5
\
2
4
6
3
5
5
0
7
4
2
3
5
4
6
3
5
0
Figure 10: Exemplary reconstructed MNIST images (M
160, N.
20000, Fig.
a))
Original
Lasso
SBL
CSVAE
(OUPS)
CSGMM (ours)
Figure 11: Exemplary reconstructed celebA images (M 2700, N = 5000, Fig. a))
22

---

Table 1: Resources for simulations on piecewise smooth fet. (M = 120, N, = 20000, Fig. ())
Model
# Parameters
Training time (hours)
Avg reconstruction time in ms
SBI.
137.23
CKSVD
0.43
2.38
CSVAE (ours)
109376
0.24
1.47
Table 2: Resources for simulations on MNIST (M = 200, N, = 20000, Fig. a)).
Model
# Parameters
Training time (hours)
Avg reconstruction time in ms
SBL
294.17
CSGAN
208817
24.68
480.81
CSGMM (ours)
46208
0.75
2.64
CSVAE (ours)
566467
1.48
2.02
Table 3: Resources for simulations on celebA (M = 1 1800, N; = 5000, Fig. 3 a)).
Model
# Parameters
Training time (hours)
Avg reconstruction time in
SBL
6067.70
CSGMM (ours)
555104
5.33
329.34
CSVAE (ours)
5063138
17.88
62.59
Algorithm Update Step the Training Phase of the CSVAE
Input: parameters
iteration g(t) (i.e., decoder) and (t (i.e., the encoder), batch Vistch meas.
matrix (or corresponding batch meas. matrices {A:)), diet. D, noise o², optimizer Adam, in the fth iteration, learning rate X
Output: parameters in the (t + 1)th iteration g(++1) ((++1)
for = to [Dbssch do
1) Mages (BI), dates (yI) kinodder
y/
2) draw 2, ~ 2a(e)) (z(31) = ((2; Moto (vi); (g1)) (via reparameterization trick 37
3) Vatcs (E:)
Decoder
4) (CNl: (Z:), 1e
plu.. (z:)
(Tote) (Z:); A, A.D. y3:) cla. (2,), A, D. 0²
5) By Pacoo (HlE, log p(yn [s)
Appendin
(nglon
6) DKL (g (zl))))l((z))
Appendix:
(Mate) (yr), σ x (gr:)
7) DKL (Pomt)
(s/²;)
Appendix E
Culle 0is)
(Zi); gie)
v. (2:),
21)) Jaces (E:)
end for (CSWA) ((1))
(5). 6). Piaaectl
gi++1) ((()))
Adam, (L (CSVAE)
(01)) (01)) X, ((t) (
Algorithm 2 CME Approximation with the CSVAE the Inference Phase (cf. (42)
Input: observation y, encoder (Ho()), ar))) (-),, decoder (Y))), meas. matrix A, dict. D. noise o², cardinality IZ| 42
Output: CME approximation diMe
1) p (y), (g)
Encuder
for = to 2 do
2) draw E, (zlv) = ((z)
3) (2,)
Dosandr
4) H4
a1 2.2. (2.)
(7a(2)) A, o2
end for
5) MêME
D/1 11 (E.))
23

---

Algorithm 3 MAP-based Estimator with the CSVAE in the Inference Phase (cf.
Input: observation y, encoder (ua(-), ++)),, decoder (e((), meas, matrix A, dict. noise o²
Output: Encoder
MAP-based estimator FMAP
1) (D(
3) "o (z)())
Decoder
Ma (3)
4) 1e
17
(7 A, A,
5) ØMAP
Dμ4
(H)(W)
Algorithm 4 One EM Step in the Training Phase of the CSGMM with one fixed A
Input: parameters in the (th iteration {7k Pkk 1k=1 training set V, meas, matrix A, dict. D,
(8) K
noise o²
Output: parameters in the (t + 1)th iteration (Tk .kk 1==1 for =1 to K do
((+1) ((+1) K
1) Carlk
D.A.o²
7
2) diag C k
, A.D,A
for = 1 to | do
3) Pt (kelys) (Bayes)
(Culk (8) ket -Pk
4) Mk,t
(Oy,k, D.A, 0², 02,01 y1.
end for
5) (7k
Pk .Pk (1+1) (1+2)
Lemana
3R (fm(tluu))
Aly [2] M4. i==
diag (Crluut))
end for
Algorithm 5 CME Approximation with the CSGMM in the Inference Phase (cf. (44)
Input: observation y. GMM (Pk, 7ts 1==1 meas, matrix A, diet. D, noise o²
Output: CME approximation CCME for l: to K do 1) Cylk
(7k: o.
2) p(ky))
(Bayes)
(CE PP
3) M elu.k
CHll D. A.A.A
end for 4) @CMF
Ek=1 p(k (1 blu.
Algorithm MAP-hased Estimator with the CSGMM the Inference Phase (cf. (45))
Input: observation y. GMM (Pk, 7h k= meas. matrix A, dict. D, noise o²
Output: MAP-based estimation #MAP for 1) 1)Ckle
to do Carlk
(7h,, D,A,
2) p(kly) (Bayes) (CHik Pk:
end for
3) RMAP argmax pkkly) y
Calhan Keas D. A, 8,
4)
5) BAMP
DH
Du
#l KMAP
koop
24

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: All claims made in the abstract and introduction are discussed in Section 2.4 3.1
3.2
3.3 and
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
Limitations are discussed in Section 3.4
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
25

---

Justification:
Assumptions of Theorem 3.1 and Lemma 3.2 are stated in the theorem and
lemma themself. while rigorous proofs of Theorem and Lemma 3.2 are provided in Appendix A and respectively. Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proof's provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: All necessary information reproduce the results is stated Section 4.1 and Appendix K Appendix IN provides additional pseudo-code to facilitate the reproduction. Moreover, a link a GitHub repository with our source code is provided in Section
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
26

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Three of the four used datasets (MNIST, FashionMNIST, celebA) are freely accessible, while detailed description of generating the fourth dataset (piecewise smooth functions) is given in Appendix []] Moreover, link to GitHub repository containing the source code is provided in Section
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (httpss:/miis.cc))
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ //nips httpppsccppbblcgguuuuuuee /obblii/gguddss//ddeeubiiii (ooodeSummsssionnll ssioiii for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e-g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All necessary information is stated in Section 4.1 Moreover, refer the link to a GitHub repository with source code Section !
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We used reasonable high number of samples, see Section 4.1 ally, we provide error bars representing the results' standard deviations.
Addition-
Guidelines:
The answer NA means that the paper does not include experiments.
27

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The compute resources are provided in Appendix M
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics hcttta///nearrap...cpppp./////ee
Answer: [Yes]
Justification: We have carefully read the NeurlPS Code of Ethics and confirm that our research conforms with it in every respect.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there special consid- eration due to laws or regulations in their jurisdiciio))
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There are no potential societal impacts of our work, which we think must be specifically highlighted.
28

---

Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We do not see any potential risk for misuse of our proposed model.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer:
[Yes]
Justification: We cite and explicitly state the licenses and terms of use of all used datasets in Section 4.1
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
29

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
If assets are released, the license, copyright information, and terms use in the datasets.paperssiii paperswi thcode. www.com//aaaa datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
All details are outlined in Section and
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Our work does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our work does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
30

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
31


Reviews:
[{"review_id": "3Kluiq7joh", "reviewer": "Reviewer_K63t", "summary": "The paper proposes a set of methods to learn a generative model over the sparse representations of  a signal. The dictionary basis is fixed and given, and the signal of interest is assumed to be sparse in this basis. Therefore, the generative model learns to provide such sparse representations. The method learns the generative prior from the noisy and compressed observations. The authors leverage the notion of conditional Gaussianity and propose VAE and GMM based models for learning such priors. The choice enables the inference with only forward operation.", "questions": "Some of my questions about the novelty was implicit in my comments above.\n-\tCan authors provide some comment on the performance loss with respect to uncompressed generative prior?\n-\tHow many MC samples are used at inference for (9)?\n-\tGenerative priors generally suffer from convergence issues and require multiple restarts. Could the authors comment if their forward-only operation has a similar issue? \n-\tIt would be good to incorporate error bars in the plots and numerical results. Given the MC step, I expect a higher variance for the proposed approach.", "limitations": "See comments above.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The idea of using purely forward operations for inference is quite nice, since most of the solutions based on generative priors require gradient descent on the latent space and suffer from latency among other problems. Furthermore, learning the prior from compressed observations makes the approach more practical as it does not need ground truth signal.", "weaknesses": "Overall, the paper combines some of the existing ideas from prior works (for example posterior updates from SBL, VAE ELBO derivations), and therefore, the novelty seems marginal. To be more concrete, it *seems* straightforward that if one wants to do variational Bayes, then the VAE style decomposition is the standard approach. If there are any additional challenges, the authors should clarify.\n\nThe idea of forward pass only inference has been tried before (see for example the paper “Solving Linear Inverse Problems Provably via Posterior Sampling with Latent Diffusion Models” NeurIPS 2023).  \n\nLearning from the compressed samples can be compelling although not new as mentioned by the authors themselves (for example CSGAN). The proposed method requires further elaboration on the loss with respect to learning from uncompressed measurements substantiated with more experiments.\n\nThe experiment section is a bit thin given the limited technical novelty, particularly with respect to the baselines. Given a lot of prior works in the literature on learning based solvers for inverse problems (for example, all variations of generative priors based on flows and diffusions, unfolded algorithms like lista and its variations), the authors have a more difficult task of placing their contribution within the prior work. The paper is not very convincing in showing what is the unique direction in which the field is moved forward.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer K63t for the thorough review and the appreciation of the proposed method’s ability to reconstruct via a single forward operation. In the following, we address the raised weaknesses and questions.\n\n**To Weaknesses (first paragraph):** \nWe agree with the reviewer that the idea of realizing variational Bayes via a VAE-style decomposition is straightforward. However, this is not our only technical contribution and there are additional challenges and contributions from our side.\n\n1) One technical contribution from our side is to extend the statistical model of SBL for compressive sensing in Eq. (2) by an additional (conditional) latent variable $\\mathbf{z}$ with arbitrarily parameterized distribution (Eq. (7)). While this extension might look minor, this idea enables the learning of a nontrivial sparsity inducing prior with training data. In contrast, the original SBL for compressive sensing in Section 2.2 cannot learn from data but rather applies the EM algorithm to adjust a naive prior during inference. To our best knowledge, we propose the first model, which incorporates SBL for compressive sensing while simultaneously being able to learn from training data in an initial offline phase.\n\n2) We also rigorously prove that our introduced statistical model in Eq. (7) is sparsity inducing (Theorem 3.1). We consider this as\na further technical contribution.\n\n3) The idea of using a VAE-style decomposition might be straightforward. However, we think that it is not trivial to show that there exists a tractable ELBO for the statistical model in Eq. (7) (which is different from the standard VAE ELBO). We derive this ELBO from (12) to (15) with additional closed-form formulas in Appendix D. Moreover, for the CSGMM, we derive the closed form of its m-step in Lemma 3.2, which solves the non-trivial optimization problem in Eq. (30).\n\nAltogether, we would like to kindly disagree that our work has ”limited technical novelty” and hope the reviewer agrees on this point.\n\n**To Weaknesses (fourth paragraph):**\nWe understand the reviewer’s perspective that our baselines might seem to ignore some of the state-of-the-art compressive sensing methods. We are aware of the rapidly improving diffusion-based and unfolding-based compressive sensing techniques recently published. We also partially cite the corresponding pioneering work (such as the ALISTA paper) in our introduction. However, these state-of-the-art models have to learn from lots of ground-truth data. We want to emphasize that we neither aim nor claim to outperform these techniques as we think that it is illusory to expect a model, which is trained on a few and heavily compressed data samples (as our model), might outperform state-of-the-art pre-trained techniques trained on many ground-truth samples. From a (natural) image processing perspective, the capability to learn from compressed data might seem minor as many ground-truth natural images are often accessible. However, we think there are several applications that strongly rely on this ability and being able to outperform what seems to us as the state-of-the-art model in this regime (CSGAN) is certainly a contribution from our perspective. We additionally would like to refer to our answer to 34q1 (Reviewer 4) - Weaknesses 2) for exemplary applications where the currently available state-of-the-art models are not applicable, as this reviewer asked explicitly for applications. Having said that, we will revise our work in the final version to better place our work in the current research area. \n\n**To Question 1 and Weaknesses (third paragraph):**\nIn Appendix I, we included the comparison between CSGAN and the proposed CSGMM and CSVAE trained on ground-truth and compressed data. Moreover, in the original work of CSGAN, it was shown that CSGAN can outperform the standard GAN when being trained on ground-truth data for compressive sensing. Since our main focus lies on the capability to train from compressed data, the informational gain from further baselines trained on ground-truth data seems to us rather limited (see also our previous answer). We hope the reviewer agrees with us on this.\n\n**To Question 2 and Question 4:**\nThe number of Monte Carlo samples is 64 (Section 4.1 - Hyperparameters). While we understand the reviewer’s intuition of the CSVAE estimator to exhibit high variance, the opposite is the case. In fact, the proposed approach is highly robust w.r.t. the Monte Carlo approximation and already one/a few sample(s) are sufficient for approximation. In fact, CSGAN also relies on (Monte-Carlo-like) estimating $\\mathbf{x}^*$ several times, which is why it is possible to explicitly compare both methods in this regard. In Fig. 1 (left) of the attached pdf, the nMSE over MC samples for CSVAE and CSGAN is shown for MNIST reconstructions with observations of dimension 200. It can be seen that taking more than one sample is almost negligible for CSVAE, while only one trial for CSGAN performs significantly worse than computing many trials. We additionally agree with the reviewer that error bars help the reader evaluate the results, which is why we included error bars representing the standard deviations of the estimations. In Fig. 1 (right) and Fig. 2 of the attached PDF, we included the same error bars in all the plots of our main paper.\n\n**To Question 3:**\nAs shown in Fig. 1 (left) in the attached PDF, the inference of our proposed method is robust and requires no multiple starts. This property is based on the differences between the estimation methodology of GANs and our proposed methods. While GANs enforce their estimation to lie on a learned manifold, our proposed method regularizes the estimation by means of a probabilistic prior and aims to approximate the conditional mean estimator.\n\nWe thank the reviewer again. Based on the feedback, we will revise our paper to place our work better in the research area as well as include error bars and the simulations regarding the MC samples.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4vxA7n2gTJ", "reviewer": "Reviewer_y1C8", "summary": "The paper introduces a novel training algorithm for generative models used as priors in linear inverse problems, with a specific focus on compressed sensing. The authors propose a training principle that regularizes the prior to learn a sparse representation of the signal of interest, implemented in Variational Autoencoders (VAEs) and Gaussian Mixture Models (GMMs). This is achieved by learning two posterior distributions p(y∣s) and p(s∣z) over a set of parameters through optimization of the Evidence Lower Bound (ELBO) derived in Equation 15.\n\nDuring the inversion phase, the true signal is estimated by sampling from the posterior distribution E[x∣y] as given in Equation 9. Unlike many contemporary generative priors, this method does not require solving an optimization algorithm to achieve the estimate. This approach simplifies the inversion process, making it more computationally efficient.\n\nThe performance of these implementations is validated on datasets containing different types of compressible signals. The results demonstrate the effectiveness of the proposed method in accurately reconstructing signals. Additionally, the paper provides theoretical support on the tightness of the lower bound of the estimate, further validating the robustness and reliability of the proposed approach.", "questions": "* Could the authors please provide perceptual metrics (i.e. SSIM, LPIPS) for the reconstruction comparison experiments in Figures 2 and 3? \n* Could the authors elaborate on the computational overhead provided in footnote 4 and explain the reason for utilizing only grayscale images when reporting having a computational budget of NVIDIA A40 GPU for CSVAE? \n* Could the authors please explain the advantages the proposed method has over contemporary generative prior such as normalizing flow and diffusion-based priors? \n* One concept I am unclear about is how do you check if the proposed method is overly biased towards the measurements y? This issue might arise because the conditional distribution P(y|s)  has equally likely probabilities when Y=AS, but does not guarantee S will be consistent with training samples depending on how sparse s is in practice.", "limitations": "Please refer to the weakness portion of the review.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "The proposed method appears to be novel and aligns with an intriguing line of work that focuses on training generative priors with the intention of using them for inverse problems.\n\nSection 2 of the paper is well-crafted, effectively elucidating the connection between the proposed method and existing literature.\n\nFor the given baselines, the proposed method yields better reconstructions of the true signal, as verified by commonly used metrics and visual inspection.", "weaknesses": "The authors provide theoretical motivation; however, they do not address how the potential estimation of the measurement y impacts the estimation of the true signal x. Additionally, building on this point, the CSVAE may experience representation error, as noted in [5], which is not mentioned as a limitation. While the proposed method does improve upon [5], this remains a source of error.\n\n\nSection 4 is limited in the following ways:\n\n* The only metric reported for natural images is the normalized mean square error, while other commonly used metrics, such as PSNR and SSIM, are not included.\n\n* Given the computational focus of this paper, it is concerning that all experiments were conducted on low-resolution grayscale images, especially considering the A40 GPU budget.\n\n* The baselines appear to be less contemporary compared to current literature on compressed sensing inversion algorithms. [Normalizing Flow](https://proceedings.mlr.press/v119/asim20a/asim20a.pdf) and [Diffusion Models](https://openreview.net/pdf/4f6f0e2347a3d6f9a88b39e445f77c1e7503064e.pdf) have largely replaced GANs due to issues with representation error. Similar to the proposed method, Diffusion Generative Priors do not require gradient-based approaches to estimate the posterior. There are hyperlinks to the work in the responses. \n\n* Considering this paper is training generative before be used downstream, there seems to be no reported metrics about training other than computational resources.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer y1C8 for the detailed review. In the following, we address the reviewer’s raised weaknesses and questions.\n\n**To Summary:** \nWe would like to point out that we do not learn $p(\\mathbf{y}|\\mathbf{s})$. In fact, keeping $p(\\mathbf{y}|\\mathbf{s})$ fixed by a pre-known dictionary is a key property of our proposed method. Instead, we learn the parameters of a statistical model for $p_{\\mathbf{\\theta},\\mathbf{\\delta}}(\\mathbf{s})$. Moreover, we do not reconstruct $\\mathbf{x}^*$ by posterior sampling, but rather approximate the point conditional mean estimator. This estimation methodology is different from the diffusion-based posterior sampling methods or the MAP-like GAN-based reconstruction which the reviewer might be familiar with.\n\n**To Weaknesses (before bullet points):** The representation error of GAN-based compressive sensing originates from the discrepancy between the learned GAN’s range (i.e., its learned manifold) and the ground-truth signal, which is supposed to be reconstructed. Our proposed method learns no manifold but rather a parameterized statistical model, which is why our proposed method does not experience this source of error.\n\n**To Weaknesses (bullet point 1 and Question 1):** The PSNR is a function of the nMSE, which is why the PSNR encodes the same information as the nMSE, and including both to us seems to provide no additional information. However, we agree with the reviewer that the SSIM provides additional information, which is why we plotted the SSIM for Fig. 2 and 3 in the attached pdf (Fig. 3). Our proposed methods either perform comparably or outperform CSGAN in all settings and outperform all other baselines.\n\n**To Weaknesses (bullet point 2) and Question 2:** The choice of low-resolution grayscale images and 1D signals in our work is motivated by the exemplary applications, we have in mind for our proposed method. We think that there are several applications that crucially depend on the ability to learn from strongly compressed signals and exhibit signals/images in this dimensional regime. For example, in wireless communication (either 1D or 2D low-dimensional) as well as in ECG denoising (1D signals) in wearable technology, data is typically low-dimensional, noisy, and compressed, and ground-truth training data is difficult to obtain. Moreover, in those applications, latency plays a crucial role, which is why we think that our proposed method matches those applications. However, we decided on images and generic 1D signals in our work, as these are the benchmark datasets used for our baselines, and we wanted to introduce our method independent from specific applications. We refer to our answer to reviewer 34q1 (Reviewer 4) - Weaknesses 2) for more details about exemplary applications. \nHaving said that, there is indeed a memory-related limitation for our proposed method related to the computation and processing of the calculated covariance matrices. Sparse Bayesian learning-based techniques generally share this limitation. While it is possible to circumvent the explicit computation of the posterior covariance matrices in Eq. (8) for training the CSVAE and CSGMM, the covariance matrices in Eq. (17) have to be computed and inverted. This is also, what we refer to in footnote 4. In the final version of the paper, we will revise our limitations and explicitly discuss this point in more detail as this property might limit the application of our model in extremely high-dimensional image settings. To overcome this issue and combine it with efficient Sparse Bayesian learning methods is considered future work.\n\n**To Weaknesses (bullet point 3) and Question 3:** To our best knowledge, contemporary generative priors for compressive sensing generally rely on the assumption of being trained on many ground-truth training samples. However, as pointed out in our previous paragraph, there are several applications in which this assumption is difficult or even not possible to realize. In contrast, our proposed method does not require ground-truth training samples in the training phase and, thus, can be applied for those applications. CSGAN\nseems to us to be the current state-of-the-art generative prior, which also relaxes the requirements on the training data by incorporating the knowledge that the prior is to be used for compressive sensing. This is why we compare our method with CSGAN.\n\n**To Weaknesses (bullet point 4):** In the attached PDF (Fig. 4 c) and d)), we included the tracking of the objective functions for CSVAE and CSGMM as an additional metric for successful training.\n\n**To Question 4:** Unfortunately, we cannot fully comprehend the reviewer’s question. Can the reviewer specify what is meant by ”equally likely probabilities when $\\mathbf{y} = \\mathbf{A}\\mathbf{s}$”? The reviewer might refer to a distributional shift between the training and test samples. While we think that it is an important question whether the proposed method is robust against distributional shifts in the test set, we consider this question to be out of the scope of our work as this paper is supposed to introduce the general concept and demonstrate good performance on benchmark datasets.\n\nWe would like to thank the reviewer again. We will revise our final paper and plan to explicitly address the potential limitations with very high-dimensional data, which Sparse Bayesian learning generally suffers from. Moreover, we plan to also include the results on the SSIM distortion metric.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7jbkfjLB7m", "reviewer": "Reviewer_WoRD", "summary": "The authors present an elegant new approach for dictionary based compressive sensing wherein the sparsity inducing prior is tuned from data by maximizing a lower bound on the evidence. This is a new paradigm for compressive sensing which appears to improve on reconstruction error over standard approaches and does not require solving an optimization problem at inference time.", "questions": "- How does your approach compare to methods for sparse Bayesian learning with proper priors; for example Louizos et. al. \"Bayesian Compression for Deep Learning.\" NeurIPS 2017?\n- Have you tried evaluating the quality of error bars from the predictive posterior so that your approach might be used in the context of UQ?\n- Can you explain the reason for the numerical instabilities in the noise free simulations for some of the image datasets?\n- Have you analyzed the histogram of the posterior over $s$? Can you suggest how the practitioner who is interested in a sparse $s$ might $0$ entries of the vector?", "limitations": "I think the authors have done a good job addressing the potential limitations of their approach in section 3.4. It would be interesting to analyze how well generative priors trained on a particular dataset transfer to similar datasets.", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- Approaching dictionary based compressive sensing in this way is novel as far as I'm aware. Not having to solve an optimization problem at inference time has the potential to be impactful.\n- I found the paper to be very well-written. The authors did a good job of motivating their approach by relating it back to relevant background literature and discussing necessary theory.\n- Their results appear to be promising and should hopefully motivate future work in this paradigm.", "weaknesses": "- Since their approach offers a new perspective on sparse Bayesian learning, it would have been helpful to compare the sparsity of predictions in addition to the nMSE.\n- I understand that space is limited, but I found the description of the experiments to be a bit terse. \n- It wasn't clear to me when a practitioner should choose CSVAE vs. CSGMM", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank reviewer WoRD for the positive review and the appreciation of our work. In the following, we address the reviewer’s raised weaknesses and questions.\n\n**To Weaknesses (bullet point 1):** We thank the reviewer for this suggestion and give it serious consideration for the final manuscript.\n\n**To Weaknesses (bullet point 2):** As the final manuscript is allowed to have one additional page, we plan to reorganize the experimental section and describe the simulations in more detail. \n\n**To Weaknesses (bullet point 3):** There are a few aspects to consider when choosing between CSGMM and CSVAE comprising the following:\n\n1) After training CSGMM with training samples observed with one fixed measurement matrix, it is possible to apply the model to reconstruct signals from different measurement matrices even with different dimensions. This is not possible with CSVAEs as the encoder is trained measurement matrix specific. \n\n2) When the training set has been observed using varying measurement matrices, the covariance matrices in Eq. (17) become sample-dependent. Moreover, as the EM algorithm requires all training samples for one update step, storing all these sample-specific covariance matrices might lead to memory-related issues for the CSGMM. This issue does not arise with CSVAEs since there, the update step is done via small training batches.\n\n3) For the online reconstruction, CSGMM has to evaluate $p(k|\\mathbf{y})$ for all components $k$. For very high dimensions, this can get computationally more expensive than the simple forward operation through the equivalent CSVAE's NN encoder.\n\n**To Question 1:** In the mentioned paper, the priors are assigned directly to the weights of the neural network. However, in our neural network-based parameterization of CSVAE, we keep the parameters (i.e., $\\mathbf{\\theta}$) deterministic but rather combine the sparse Bayesian learning framework with the output $\\mathbf{\\gamma}_{\\mathbf{\\theta}}(\\mathbf{z})$ of the CSVAE by interpreting it as the diagonal of a zero-mean Gaussian with diagonal covariance matrix. From our perspective, both settings are rather different and are difficult to compare to each other. We hope the reviewer agrees on this point with us. \n\n**To Question 2:** While we think, that the idea to explicitly evaluate the error bars of the posteriors is interesting, we did not try this out. We appreciate the idea and will give it serious consideration for the final manuscript. Additionally, in the attached PDF, we now included error bars in the results for the nMSE and SSIM.\n\n**To Question 3:** We implemented CSVAE in pytorch. The numerical instabilities concern the backpropagation of Eq (24) in our work, which includes the pseudoinverse. Although (24) is theoretically differentiable w.r.t. $\\mathbf{\\theta}$, it turned out that utilizing the equivalent (8) instead of (24) with some small artificial noise variance (equivalent to 40dB SNR) led to stable training, while directly using (24) sometimes resulted in numerical issues.\n\n**To Question 4:** From our perspective, this question seems to relate to Question 2. We did not analyze the posterior's histogram. We think, there are several possibilities to approximate the compressible estimate of $\\mathbf{s}$ by a sparse one. One way could be to decide on some decision threshold. We refer to [1] for more details.\n\nWe would like to thank the reviewer again and give the reviewer's suggestions regarding the sparsity and posterior analysis a serious consideration for the final manuscript. Additionally, we will extend the Appendix by a discussion about when to choose CSGMM over CSVAE and vice versa.\n\n[1] G. Dziwoki, M. Kucharczyk, \"On a Sparse Approximation of Compressible Signals,\" 2020, Circuits Syst. Signal Process. 39(4): 2232-2243", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yspJ3lah9g", "reviewer": "Reviewer_34q1", "summary": "This paper introduces a new type of sparsity inducing generative prior for the inverse problem. The authors theoretically underpin our approach by proving that its training maximizes a variational lower bound of a sparsity inducing log-evidence.", "questions": "1. In Table 2: Resources for simulations on MNIST (M = 200, Nt = 20000, Fig. 2 a)), CSGAN has much less parameters than the proposed CSVAE , why does CSGAN take much more training time?\n2. Could the reconstruction results be influenced by the amount of noise?", "limitations": "I think the application scenes of this work is not very clear. The datasets used for comparison are very naive, colorful images with more textures should be used. And the authors should include more recent comparison methods, existing comparison methods are all presented before 2018.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "This work can learn from a few corrupted data samples and, thus, requires no ground-truth information in its training phase.", "weaknesses": "1. The comparison methods include Lasso, CKSVD, and CSGAN, all of which are presented before 2018. The comparison methods are too old.\n2. I think the author should clarify the specific application scenarios for this research direction.\n3. In Figure 7, compared with CSGAN, the results of CSVAE and CSGMM are not very clear.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer 34q1 for the comprehensive review. In the following, we address the reviewer’s raised weaknesses and questions.\n\n**To Weaknesses 1:** We would very much appreciate it if the reviewer could specify which comparison methods are missing in our work. While it is difficult to track all the latest literature, we are aware of the rapidly improving diffusion-based and unfolding-based compressive sensing techniques recently published. However, to the best of our knowledge, these techniques generally rely on the assumption of being trained on many ground-truth training samples. CSGAN seems to us to be the current state-of-the-art generative prior, which also relaxes the requirements on the training data by incorporating the knowledge that the prior is to be used for compressive sensing, which is why we compare our method to CSGAN.\n\n**To Weaknesses 2:** While we do not think that we can comprehend all possible applications, we give an overview of three possible applications we can think of, in which the ability to learn from compressed data is crucial and where many state-of-the-art machine learning-based compressive sensing techniques might not be applicable.\n\n1) ECG denoising: The sensors in wearable electrocardiography (ECG) monitoring devices generally provide noisier signals, with more artifacts compared to those typically used in the hospital [1]. However, if one wants to do patient-specific training, the ability to directly train from the data provided by these sensors comes with severe benefits as it is rather unrealistic to first capture clean ground-truth training samples in hospitals for each patient individually.\n\n2) Electron microscopy: In electron microscopy, one generally wants to restrict the total amount of electron dose used for measuring to not interact with (change) or even destroy the measured sample of interest [2]. However, this typically leads to noisy acquisitions with low contrast and resolution. Thus, being able to learn from these corrupted acquisitions also comes with significant benefits.\n\n3) Wireless communication: In, e.g., the current 5G wireless communication standard, mobile users during communication receive compressed observations on a frequent basis (see Eq. (1) in [3]). While current machine learning techniques in wireless communication rely on simulated data or require expensive measurement campaigns for the training data, our proposed method can directly learn from the compressed data that mobile users receive during communication. \n\nAll these applications have in common that they are rather low-dimensional. Moreover, latency plays an important role in ECG denoising as well as wireless communication rendering our proposed method to be specifically interesting for those applications.\n\n**To Weaknesses 3:** These observations come from the different reconstruction aspects of CSGAN and CSVAE/CSGMM. Our estimators approximate the point conditional mean estimators, which minimizes the distortion measure MSE. Therefore, CSVAE- and CSGMM-based reconstruction emphasizes the reconstruction of details, while CSGAN emphasizes the visual contrast. This can be explained by the perception-distortion tradeoff [4]. By comparing the ground truth (first row) with the reconstructed images more closely, one can see that the CSGAN misses out on details, which are reconstructed by CSVAE and CSGMM.\n\n**To Question 1:** The different training times result from the distinct training algorithms between CSGAN and CSVAE/CSGMM. More specifically, the training of CSGAN builds on a min-max optimization problem, where additionally after each update step, a Lasso-like reconstruction algorithm has to be applied. On the other hand, the training of CSVAE optimizes the ELBO derived in Eq. (15).\n\n**To Question 2:** The noise level plays indeed an important role. In fact, since our proposed model can incorporate the noise level in the training process (see Eq. (2)), our proposed method is quite robust against additional noise. To evaluate this, we simulated different noise levels on MNIST and plotted the nMSE and SSIM results in the attached PDF (Fig. 4 a) and b)). One can see that our proposed method exhibits significantly less performance decrease for higher noise levels (i.e., smaller SNR) compared to all other baselines.\n\n**To Limitations:** The choice of low-resolution grayscale images and 1D signals in our work is mainly motivated by the exemplary applications stated above, which are all low-dimensional and can be interpreted as either 1D or grayscaled 2D signals/images. Additionally, the used datasets in our work have been considered as the benchmark datasets for our baselines. \n\nWe would like to thank the reviewer again for the valuable feedback. We will revise our final paper to point out exemplary applications more specifically. Moreover, we plan to also include the simulations on varying noise levels.\n\n[1] G. Revach, T. Locher, N. Shlezinger, R. J. G. v. Sloun, R. Vullings, \"HKF: Hierarchical Kalman Filtering with Online Learned Evolution Priors for Adaptive ECG Denoising,\" 2023, arXiv:2210.12807.\n\n[2] T.-O. Buchholz, M. Jordan, G. Pigino, and F. Jug, “Cryo-care: Content-aware image restoration for cryo-transmission electron microscopy data,” in 2019 IEEE 16th International Symposium on Biomedical Imaging (ISBI 2019), 2019, pp. 502–506.\n\n[3] W. Kim, Y. Ahn, J. Kim, and B. Shim, “Towards deep learning-aided wireless channel estimation and channel state information feedback for 6G,” Journal of Communications and Networks, vol. 25, no. 1, pp. 61–75, 2023.\n\n[4] Y. Blau and T. Michaeli, “The perception-distortion tradeoff,” in 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. IEEE, Jun. 2018.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear Program Chair, Senior Area Chair, Area Chair and Reviewers,

We would like to thank you for taking the time to review our paper and for the valuable feedback. For our response, please refer to our point-by-point responses to each reviewer below. 

You will also find attached a PDF with additional simulation results, in which the plots for the corresponding reviewers are marked with blue boxes and their names.

Sincerely, 

The authors.

Meta-review: This paper proposed one compressed sensing method with sparsity inducing conditionally Gaussian generative models. All the reviewers agree that this idea is novel and interesting. After my own reading, I also agree with that and thus I recommend acceptance. Nevertheless, as pointed out by most reviewers, especially Reviewer 34q1 and WoRD,  the authors should carefully revise the manuscript as required/suggested in the rebuttal and discussions to improve the quality, which is very important.
Decision: Accept (poster)