Title: Sparse Bayesian Generative Modeling for Compressive Sensing
Venue: NeurIPS 2024
Accepted: yes
Keywords: Compressive sensing

variational inference

sparse bayesian learning

variational autoencoder

Gaussian mixture model

generative model
Abstract: This work addresses the fundamental linear inverse problem in compressive sensing (CS) by introducing a new type of regularizing generative prior. Our proposed method utilizes ideas from classical dictionary-based CS and, in particular, sparse Bayesian learning (SBL), to integrate a strong regularization towards sparse solutions. At the same time, by leveraging the notion of conditional Gaussianity, it also incorporates the adaptability from generative models to training data. However, unlike most state-of-the-art generative models, it is able to learn from a few compressed and noisy data samples and requires no optimization algorithm for solving the inverse problem. Additionally, similar to Dirichlet prior networks, our model parameterizes a conjugate prior enabling its application for uncertainty quantification. We support our approach theoretically through the concept of variational inference and validate it empirically using different types of compressible signals.

Sparse Bayesian Generative Modeling for Compressive Sensing

Abstract

This work addresses the fundamental linear inverse problem in compressive sensing (CS) by introducing a new type of regularizing generative prior. Our proposed method utilizes ideas from classical dictionary-based CS and, in particular, sparse Bayesian learning (SBL), to integrate a strong regularization towards sparse solutions. At the same time, by leveraging the notion of conditional Gaussianity, it also incorporates the adaptability from generative models to training data. However, unlike most state-of-the-art generative models, it is able to learn from a few compressed and noisy data samples and requires no optimization algorithm for solving the inverse problem. Additionally, similar to Dirichlet prior networks, our model parameterizes a conjugate prior enabling its application for uncertainty quantification. We support our approach theoretically through the concept of variational inference and validate it empirically using different types of compressible signals.

1 Introduction

Research in CS has shown that it is possible to reduce the number of measurements far below the one determined by the Nyquist sampling theorem while still being able to extract the information-carrying signal from the acquired observations. The fundamental problem in CS is an ill-posed linear inverse problem, i.e., the goal is to recover the signal x ∗ ∈ R N of interest from an under-determined set of measurements y ∈ R M with M ≪ N , related by

where x ∗ is compressed by the measurement matrix A and potentially corrupted by additive noise n ∼ N ( 0 , σ 2 n I ) . Since the under-determined observation y does not carry enough information alone to faithfully reconstruct x ∗ , additional prior (or model) knowledge about x ∗ is required to make the inverse problem 'well-posed'. Classical CS algorithms such as Lasso regression or orthogonal matching pursuit (OMP) address this problem by incorporating the model knowledge that x ∗ is sparse or compressible with respect to some dictionary [1]. Nowadays, modern deep learning (DL)based approaches such as unfolding algorithms, generative model-based CS and un-trained neural networks (NNs) expand the possibilities by learning prior knowledge from a training set or designing the network architecture to be biased towards a certain class of signals [2-7]. These approaches typically require a (potentially large) training set of ground-truth data samples, or their architecture is specifically biased towards natural images. However, in many applications, ground-truth training data might not be easily accessible. In, e.g., electron microscopy, the amount of electron dose has to be restricted to not induce damage on the probe resulting in low-contrast noisy data samples [8]. The sensors in wearable electrocardiography (ECG) monitoring devices generally provide noisy signals with artifacts [9], which limits the ability to learn from patient-specific data in real-world settings. Another example is the wireless 5G communication standard, where mobile users receive compressed and noisy so-called channel observations on a frequent basis (cf. [10]) while acquiring ground-truth

channel information requires costly measurement campaigns. Thus, in many applications, it is either impossible or prohibitively expensive to collect lots of ground-truth training data while corrupted data is readily available. This highlights the necessity for methods that can learn from corrupted data samples.

In this paper, we propose a new learnable prior for solving the inverse problem (1), which can learn from only a few compressed and noisy data samples and, thus, requires no ground-truth information in its training phase. Additionally, it applies to any type of signal, which is compressible with respect to some dictionary. Our approach shares similarities with generative model-based CS [4, 11]. There, a generative model is first trained to capture the exact underlying prior distribution p ( x ) of the signal x ∗ of interest. It then serves as probabilistic prior to regularize the inverse problem (1). Similarly, classical CS algorithms like Lasso regression also impose a probabilistic prior on the sparse representation of x ∗ with respect to some dictionary [12]. In contrast to modern generative models, however, these priors do not have any generation capabilities but solely bias the inverse problem (1) towards sparse solutions. This observation indicates that a probabilistic prior does not necessarily need to capture the exact prior distribution p ( x ) to effectively regularize (1). The proposed model in this work builds upon this insight and forms a trainable but simultaneously sparsity-inducing prior. For that, we aim to combine the adaptability of generative models to training data with the property of many types of signals x ∗ to be compressible with respect to some dictionary. As a result, our model learns statistical information in the signal's sparse/compressible domain. Examples of signals with a specific statistical structure in their compressible representation include piecewise smooth functions and natural images, whose wavelet coefficients approximately build a connected sub-tree or wireless channels, which are burst-sparse in their angular domain [13, 14].

Related Work. Early work on CS, which considers statistical structure in the wavelet domain of images, is given in [15, 16]. Based on the theoretical foundation in [17] and the concept of SBL [18, 19], these papers introduce a hierarchical Bayesian model, which is used to apply variational inference and Markov chain Monte Carlo (MCMC) posterior sampling to solve (1). Training Gaussian mixture models (GMMs), i.e., classical generative models, from compressed image patches of one or a few images has been analyzed in [20-23]. In this line of research, however, the GMM is fit directly in the pixel domain. Compressive dictionary learning represents a different line of research aiming to learn the dictionary from solely compressed data [24-27], and strongly bases on the dictionary learning method K-SVD [28]. More recently, variational autoencoders (VAEs) and generative adversarial networks (GANs) have been studied in the context of CS [4, 11]. There, the VAE/GAN is used to solve the inverse problem (1) by constraining the signal x ∗ of interest to lie in the range of the generative model instead of being sparse with respect to some dictionary. In this context, AmbientGAN as well as CSGAN are extensions that loosen up the training set requirements and can learn from corrupted data [29, 30]. Another related topic is the ability of some generative models, i.e., diffusion models, V AEs and GMMs, to provide an approximation of the conditional mean estimator (CME) [31-33]. In the case of the latter two, the CME is represented as a tractable convex combination of linear minimum mean squared error (MSE) estimators by exploiting their conditional Gaussianity on a latent space that determines these estimators' means and covariances.

Our main contributions are as follows:

We introduce a new type of sparsity-inducing generative prior for the inverse problem (1), which differs from classical CS algorithms due to its ability to learn from data. On the other hand, it also differs from other modern NN-based approaches due to its ability to promote sparsity in the signal's compressible domain. Moreover, it can learn from a few corrupted data samples and, thus, requires no ground-truth information in its training phase.

We theoretically underpin our approach by proving that its training maximizes a variational lower bound of a sparsity-inducing log-evidence.

Building on the notion of conditional Gaussianity, we introduce two specific implementations of the proposed type of prior based on VAEs and GMMs, which do not require an optimization algorithm in their inference phase and come with computational benefits.

By exploiting the shared property with Dirichlet prior networks to parameterize a conjugate prior, we demonstrate how our approach can be applied for uncertainty quantification.

We validate the performance on datasets containing different types of compressible signals. 1

2 Background and Method

√

Notation. The operations b -1 , b and | b | 2 denote the respective element-wise operation for the vector b . Moreover, diag ( B ) represents the vectorized diagonal of the matrix B , while diag ( b ) denotes the diagonal matrix with the vector b on its diagonal.

2.1 Problem Formulation

We consider the typical CS setup, in which we measure N -dimensional ground-truth samples x i by a known measurement matrix A ∈ R M × N (or A i ∈ R M × N ) with M ≪ N , where each observation is potentially corrupted by noise n i drawn from additive white Gaussian noise (AWGN) n ∼ N ( 0 , σ 2 n I ) . We assume all x i to be compressible with respect to a known dictionary matrix D ∈ R N × S , i.e., y i = ADs i + n i with x i = Ds i and s i ∈ R S being approximately sparse. All s i are assumed to be independent and identically distributed (i.i.d.), i.e., s i ∼ p ( s ) , where p ( s ) is unknown with s exhibiting non-trivially dependent entries. Typical signals which fit into this category are, e.g., natural images, piecewise smooth functions, and wireless channels (cf. Section 1). Our approach in this work allows to either solely have access to corrupted training observations Y = { y i } N t i =1 or training tuples with ground-truth information G = { ( s i , y i ) } N t i =1 or W = { ( x i , y i ) } N t i =1 . We first train the proposed model using Y (or, alternatively, G or W ) to serve as an effective prior for (1). Our goal is then to estimate a ground-truth signal x ∗ of a newly observed y . We also define X = { x i } N t i =1 and S = { s i } N t i =1 .

2.2 Sparse Bayesian Learning for Compressive Sensing

In SBL for CS, the idea for solving the inverse problem (1) is to assign a parameterized prior to s ∗ , i.e., the compressible representation of the signal x ∗ of interest with x ∗ = Ds ∗ [19]. It assumes

Given a single observation y , the parameters γ (and sometimes σ 2 ) are estimated by an expectationmaximization (EM) algorithm maximizing the corresponding log-evidence log p γ ( y ) implicitly defined by (2). After that, it is utilized that p γ ( s ) forms a conjugate prior of p ( y | s ∗ ) resulting in a closed-form posterior p γ ( x | y ) providing all necessary information to estimate x ∗ . In [19], a variational interpretation of SBL is given, addressing why this approach yields sparse results, even though p γ ( s ) does not inherently promote sparsity. It is shown that there exists a C > 0 such that for all γ > 0 and s

The function t ( s ) , however, is a well-known improper but sparsity-inducing prior, used as a noninformative prior for scale parameters [34]. Let another statistical model be given by p ( y | s ∗ ) (cf. (2)) and prior t ( s ) instead of p γ ( s ) with implicitly defined log-evidence log π ( s ) ( y ) . Due to t ( s ) being improper, the log-evidence log π ( s ) ( y ) forms a not normalized but valid log-likelihood. Moreover, this model is sparsity-inducing due to the sparsity-promoting characteristics of t ( s ) . Based on (3), it holds that log π ( s ) ( y ) ≥ log p γ ( y ) and, thus, log p γ ( y ) can be embedded in variational inference, forming a tractable variational lower bound of an intractable log-evidence log π ( s ) ( y ) of actual interest. Moreover, applying the EM algorithm to (2) is 'evidence maximization over the space of variational approximations to a model (i.e., p ( y | s ∗ ) and t ( s ) ), with a sparse, regularizing prior' [19].

2.3 Gaussian Mixture Models and Variational Autoencoders

GMMsand VAEs are generative models aiming to learn an unknown distribution p ( x ) from a training set X . Both models represent p ( x ) as the marginalization with conditionally Gaussian p ( x |· ) and an additional latent variable k (or z ) [35-37], i.e.,

The tunable parameters ( { ρ k , µ k , C k } K k =1 for GMMs and θ for VAEs) are learned by optimizing the model's log-evidence ∑ i log p ( f ) ( x i ) ( f ∈ { GMM , VAE } ) over X . In the case of GMMs, this is typically done by an EM algorithm alternating between the so-called e- and m-step. The e-step determines the closed-form posteriors p t ( k | x i ) (for all i ) via the Bayes rule using the model's parameters in the t th iteration. The m-step then updates the model's parameters by maximizing ∑ i E p t ( k | x i ) [log p ( x i , k )] . In case of V AEs, however, the posterior p θ ( z | x ) (and log p ( VAE ) ( x ) ) are intractable and the EM algorithm cannot be applied. Therefore, a tractable distribution q ϕ ( z | x ) = N ( z ; µ ϕ ( x ) , diag ( σ 2 ϕ ( x )) with variational parameters ϕ is introduced, which approximates p θ ( z | x ) , and µ ϕ ( x ) and σ 2 ϕ ( x ) are generated by a NN encoder. Equivalently, µ θ ( z ) and C θ ( z ) are generally realized by a NN decoder. The objective to be maximized is the evidence lower bound (ELBO) L ( θ , ϕ ) serving as a tractable lower bound for the intractable log-evidence ∑ i log p θ ( x i ) , i.e.,

with D KL ( · ) being the Kullback-Leibler (KL) divergence. Generally, the GMM's and VAE's training critically depends on iteratively characterizing and updating their posteriors p t ( k | x i ) and q ϕ ( z | x i ) .

2.4 Proposed Method

The goal of this section is to derive a class of generative models, for which we can guarantee that, on the one hand, it is sparsity-inducing, but on the other hand, it is trainable and can learn from solely compressed and noisy observations Y resulting in an effective probabilistic prior for (1).

To incorporate the bias towards sparsity, we start with SBL discussed in Section 2.2 and combine it with the VAE's and GMM's main principle for their adaptability to complicated distributions, i.e., introducing a latent variable z (or k ) on which we condition with a parameterized Gaussian (cf. (4) and (5)). 2 More specifically, we exploit the Gaussianity of p γ ( s ) in (3) and modify it to a parameterized Gaussian conditioned on some latent variable z with arbitrarily parameterized p δ ( z ) while explicitly keeping its mean zero and its covariance matrix diagonal, i.e., p θ ( s | z ) = N ( s ; 0 , diag ( γ θ ( z ))) . The resulting set of statistical models referred to as sparse Bayesian generative models, is given by

Training principle. Our proposed training scheme is independent of specific realizations for γ θ ( z ) and p δ ( z ) , which is why they are kept general in this section. Specific parameterizations are discussed in Section 3.2 and 3.3. The training goal is to maximize the model's log-evidence ∑ i log p δ , θ ( y i ) implicitly defined by (7) over a training set of compressed and potentially noisy observations Y . For that, we rely on the main training principles for statistical models including latent variables, i.e., the EM algorithm and variational inference (cf. Section 2.3). A key requirement of these principles is the ability to track and update the model's posterior (i.e., p θ , δ ( s , z | y ) for (7)) or approximations of it over the training iterations (cf. Section 2.3). In classical SBL, the prior distribution p γ ( s ) forms a conjugate prior of p ( y | s ) (cf. (2)) and, thus, the posterior p γ ( s | y ) is tractable in closed form. We utilize this property by observing that conditioned on some z , (7) coincides with the classical SBL model. Consequently, s | z is a conditioned conjugate prior of y | s in (7) with tractable Gaussian posterior p θ ( s | z , y ) , whose closed-form covariance matrix and mean

equal those in SBL [19]. The equivalent formulas for σ 2 = 0 are given in Appendix C. By considering the decomposition p θ , δ ( s , z | y ) = p θ ( s | z , y ) p θ , δ ( z | y ) , we conclude that our proposed set of statistical models in (7) exhibits posteriors, which are partially tractable in closed form independent of any specifics of γ θ ( z ) and p δ ( z ) . The remaining p θ , δ ( z | y ) resembles the standard GMM's and VAE's posterior, and we can either apply the EM algorithm or variational inference to maximize the corresponding log-evidence. Both will be discussed in Section 3.2 and 3.3, respectively.

Estimation scheme. The goal after the training is to use the learned statistical model in (7) for regularizing the inverse problem (1) and estimating x ∗ from a newly observed y . Generally, the CME E [ x | y ] is a desirable estimator due to its property of minimizing the MSE. We utilize insights from [33, 23], which derive approximate closed forms of the CME using VAEs and GMMs. More precisely, by using the law of total expectation, we approximate the CME by

The inner expectation is given by the learned µ s | y , z θ ( z ) in (8), while, depending on its specific characteristics, p θ , δ ( z | y ) also exhibits a closed form or a variational approximation has been learned during training (cf. Section 3.2), with which we approximate the outer expectation by a MonteCarlo estimation. Based on the perception-distortion trade-off, the CME is not optimal from a perceptual point [38], and deviating estimators from the CME approximation might be beneficial. More detailed descriptions of the CME approximation as well as further estimators based on the specific implementations in Section 3.2 and 3.3 are given in Appendix G.

3 Theoretical Analysis and Specific Parameterizations

3.1 Sparsity Guarantees

In Section 2.4, we motivate constraining p θ ( s | z ) to be a zero mean conditional Gaussian with diagonal covariance matrix by SBL and its sparsity-inducing property. In the following, we rigorously show that constraining p θ ( s | z ) in this manner is sufficient to maintain the sparsity-inducing property for any statistical model following (7) despite the additionally included latent variable z . More precisely, we show that any statistical model in (7) with some specified parameterized p δ ( z ) and γ θ ( z ) exhibits a log-evidence that is interpretable as a variational lower bound of a sparsity-inducing log-evidence. For that, we establish the following theorem.

Theorem 3.1. Let p θ ( s | z ) = N ( s ; 0 , diag ( γ θ ( z )) (i.e., it is defined according to (7) ), let z be either continuous or discrete and finite, and let γ θ ( z ) > 0 . Then, there exists a constant C > 0 such that for all s , θ , δ with any arbitrary distribution p δ ( z )

The integral corresponds to a summation for discrete and finite p δ ( z ) .

The proof is given in Appendix A. Based on Theorem 3.1, it holds that

with log π ( s ) ( y ) being the log-evidence of y | s ∼ p ( y | s ) and the improper but sparsity-promoting prior t ( s ) , whereas log p θ , δ ( y ) is implicitly defined by (7). Consequently, we can interpret the maximization of log p θ , δ ( y ) over ( θ , δ ) as the evidence maximization over the space of variational approximations to a model with a sparsity-inducing and regularizing prior. However, contrary to classical SBL, we have the additional degree of freedom to choose the specifics of γ θ ( z ) and p δ ( z ) without any constraint while maintaining the connection to sparsity.

3.2 Compressive Sensing VAE

The integral version of p θ , δ ( s ) in (10) strongly resembles the V AE's decomposition of p ( x ) in (5). In fact, by constraining p δ ( z ) = p ( z ) = N ( z ; 0 , I ) , the distribution p θ , δ ( s ) = p θ ( s ) corresponds to a subset of all possible distributions covered by the V AE decomposition in (5). This motivates to choose p δ ( z ) in this exact manner and realize γ θ ( z ) as the output of a NN decoder. Generally, we aim to optimize the log-evidence ∑ i log p θ ( y i ) in (11) with respect to θ solely given compressed and noisy observations Y . However, equivalent to VAEs, the log-evidence ∑ i log p θ ( y i ) is intractable for p ( z ) = N ( z ; 0 , I ) , which is why a tractable lower bound has to be derived. A tractable lower bound on the log-evidence of a statistical model with latent variables is obtained by subtracting the KL divergence between the conditional distribution of the latent variables given the observations

y

Encoder

q

ϕ

(

z

|

y

)

p

θ

(

s

|

˜

z

)

Decoder

AD

+

⊙

⊙

+

y

N

(

0

,

I

)

N

(

0

,

I

)

N

(

0

, σ

2

I

)

µ

ϕ

σ

ϕ

˜

z

√

γ

θ

s

and a tractable variational distribution from the log-evidence (cf. (6)). For the model in (7) and i.i.d. training data, this results in

where the derivation of (13) from (12) is given in Appendix B. Based on insights from the training principle explained in Section 2.4, the variational posterior q ϕ ( z , s | y i ) in (13) can be simplified to q ϕ ( z , s | y i ) = p θ ( s | z , y i ) q ϕ ( z | y i ) with closed-form p θ ( s | z , y i ) and only requires variational parameters ϕ for q ϕ ( z | y i ) to be tractable. As a result, the adapted ELBO L ( CSVAE ) ( θ , ϕ ) equals

To ensure the training of standard V AEs to be tractable, E q ϕ ( z | y i ) [ · ] is generally approximated by a single-sample Monte-Carlo estimation [37]. We build on the same strategy and approximate (14) by

with ˜ z i ∼ q ϕ ( z | y i ) and E p θ ( s | ˜ z i , y i ) [log p ( y i | s )] exhibiting a closed-form solution detailed in Appendix D. In addition, D KL ( p θ ( s | ˜ z i , y i ) || p θ ( s | ˜ z i )) is the tractable KL divergence of two multivariate Gaussians, and D KL ( q ϕ ( z | y i ) || p ( z )) matches the KL divergence from standard VAEs (cf. [37]). Both are specified in Appendix E. The adapted ELBO in (15) resembles the classical ELBO of ordinary VAEs with a modified reconstruction loss and an additional KL divergence. Equivalent to ordinary VAEs, we model q ϕ ( z | y i ) = N ( z ; µ ϕ ( y i ) , diag ( σ 2 ϕ ( y i ))) and realize ( µ ϕ ( · ) , σ 2 ϕ ( · )) by a NN encoder, and γ θ ( · ) by a NN decoder. The adapted ELBO in (15) forms a differentiable and variational approximation of the model's log-evidence ∑ i log p θ ( y i ) . Based on Theorem 3.1 and (11), it holds that

Thus, L ( CSVAE ) ( θ , ϕ ) serves as a variational lower bound of a sparsity-inducing log-evidence. We denote the resulting V AE as Compressive Sensing VAE (CSVAE), and its schematic is presented in Fig. 1. It resembles the vanilla V AE with the main difference that the encoder takes a compressed observation y as input and the decoder solely outputs conditional variances γ θ (˜ z ) of s | z . In case of varying measurement matrices A i for each training, validation and test sample, we use the least-squares estimate ˆ x (LS) i = A T i ( A i A T i ) -1 y i instead of y i as encoder input. The CSVAE's training with ground-truth data and pseudo-code for training are outlined in Appendix H and N (cf. algorithm 1).

3.3 Compressive Sensing GMM

By considering discrete and finite p δ ( z ) in (7), i.e., p δ ( z = k ) = p δ ( k ) = ρ k , we observe that the resulting p θ , δ ( s ) strongly resembles the GMM's decomposition of p ( x ) in (4). This motivates to choose p δ ( z ) in exactly this manner and p θ ( s | z ) = p ( s | k ) = N ( s ; 0 , diag ( γ k )) , i.e., s is modelled as GMM with zero means and diagonal covariance matrices. Equivalent to CSVAEs in Section 3.2,

the distribution p ( s | k, y ) contained in the model's posterior p ( k, s | y ) = p ( s | k, y ) p ( k | y ) exhibits a closed-form solution (cf. (8)). Moreover, due to the linear relation between y and s in (7) and the assumption that s | k is Gaussian, y | k is also Gaussian with zero mean and covariance matrix

and p ( k | y ) is computable in closed form by Bayes, i.e., p ( k | y ) = ( p ( y | k ) p ( k )) / ( ∑ k p ( y | k ) p ( k )) . Thus, p ( k, s | y ) exhibits a closed form, and an EM algorithm can be applied, whose e-step coincides with computing p ( k, s | y ) . Based on the extended EM algorithm in [23], where the authors fit a vanilla GMM(4) in the pixel domain using compressed image patches, we formulate an m-step tailored to our model (7). We denote the resulting model as Compressive Sensing GMM (CSGMM). The m-step aims to update the CSGMM's parameters { ρ k , γ k } by optimizing ∑ i E p t ( k, s | x i ) [log p ( x i , s , k )] , whose closed-form solution is given in the following lemma.

Lemma 3.2. Let the statistical model in (7) be given with discrete p δ ( z = k ) = p ( k ) = ρ k and γ θ ( z ) = γ k , and let Y contain i.i.d. noisy and compressed observations y i . Given p t ( k | y i ) and p t ( s | k, y i ) = N ( s ; µ s | y i ,k k,t , C s | y i ,k k,t ) in the t th iteration of an EM algorithm, the updates of { ρ k , γ k }

form a valid m-step and, thus, guarantee to improve the log-evidence in every iteration.

The proof of Lemma 3.2, the CSGMM's training with ground-truth data S and pseudo-code are outlined in Appendix F, H and N (cf. algorithm 4), respectively.

3.4 Discussion

From a broader perspective, our proposed algorithm represents two consecutive stages:

Choose a dictionary D , with respect to which a general signal set C of interest is compressible (e.g. a wavelet basis for the set of natural images).

Learn the statistical characteristics of the specific signal subset S ⊂ C of interest in its compressible domain (e.g. the subset of handwritten digits).

Due to the strong regularization of stage 1) constraining the search space in stage 2), our model can learn from a few corrupted samples Y . To ensure the search space to be within the set of compressible signals with respect to D , we enforce s | z in (7) and, thus, s to have zero mean (cf. Theorem 3.1).

Limitation. On the one hand, this restriction regularizes the problem at hand. On the other hand, it generally introduces a bias, which potentially prevents perfectly learning the unknown distribution of s as it is not always possible to decompose a distribution in this way. As a result, the proposed model is biased towards capturing the sparsity-specific features from Y and being an effective prior for (1) rather than learning a comprehensive representation of the true p ( s ) .

Distinction from generative model-based CS. Generative model-based CS builds on a generator G θ : z ↦→ x = G θ ( z ) (e.g., a GAN or VAE) with z ∈ R F , x ∈ R N and F ≪ N [4]. Arguably, the notion of compressibility is still included in this setup, since it assumes that every x can be perfectly encoded by only a few F values. Moreover, these models typically assign a simplistic fixed distribution, e.g. p ( z ) = N ( z ; 0 , I ) , to their compressible domain. Consequently, while generative model-based CS replaces the dictionary with a learnable mapping G θ and constrains the compressible domain by forcing it to be representable by, e.g., N ( 0 , I ) , our approach does the exact opposite. We keep the mapping from the compressible into the original domain fixed by a dictionary D and learn a non-trivial statistical model p θ , δ ( s ) in the compressible domain.

Connection to uncertainty quantification. While coming from different motivations, our approach shares similarities with methods from uncertainty quantification, namely so-called prior networks [39]. There, the general idea is to let the NN output the parameters of a conditioned conjugate prior (conditioned on the NN input) instead of directly the quantities of interest (cf. [40, 41]). As a result, one can use entropy and mutual information measures to quantify different types of uncertainty. Equivalently, our proposed models output the parameters γ θ ( z ) of a conditioned conjugated prior p θ ( s | z ) , which is why they enable to quantify their uncertainty by the same measures.

Computational complexity. It is noteworthy that our algorithm's inference after training requires no optimization algorithm but only consists of, e.g., a feed-forward operation through a comparably small VAE. Thus, our proposed method's signal reconstruction comes with computational benefits and differs in this context from other approaches. On the downside, naively implementing the training requires computing and storing the posterior covariance matrices in (8), which can lead to a non-negligible computational and memory-related overhead for very high dimensions. To overcome this issue, we derive equivalent reformulations of the CSVAE's and CSGMM's update steps for training that circumvent the explicit computation of these matrices. More precisely, the reformulated update steps solely require the explicit storing and inversion of the observations' covariance matrices (cf. (17)), which are typically much lower dimensional. The reformulations are given in Appendix I.

4 Experiments

4.1 Experimental Setup

Datasets. We evaluate the performance on four datasets. We use the MNIST dataset ( N = 784 ) for evaluation [42, (CC-BY-SA 3.0 license)]. Moreover, we use an artificial 1D dataset of piecewise smooth functions of dimension N = 256 . For that, we combine HeaviSine functions with polynomials of quadratic degree and randomly placed discontinuities, which are frequently used for CS and compressible in the wavelet domain [43, 17]. The generation and examples are outlined in Appendix J. We also use a dataset of 64 × 64 cropped celebA images ( N = 3 · 64 2 = 12288 ) [44] 3 and evaluate on the FashionMNIST dataset ( N = 784 ) in Appendix L [45, (MIT license)].

Measurement matrix & evaluation metric. For the simulations on piecewise smooth functions, we use a separate measurement matrix A i for each training, validation and test sample, where all A i contain i.i.d. Gaussian entries A i,kl ∼ N (0 , 1 M ) . For all remaining simulations, we use one fixed measurement matrix A each with equally distributed i.i.d. Gaussian entries for the whole dataset. We evaluate the distortion performance by a normalized MSE nMSE = 1 /N test ∑ N test i =1 ( ∥ ˆ x i -x i ∥ 2 2 /N ) with ˆ x i being the estimation of x i and the SSIM [46] with N test being set to 5000 in any simulation.

Baselines & hyperparameters. As non-learnable baselines, we use Lasso [12] and SBL [19], where we either adjust Lasso's shrinkage parameter on a ground-truth dataset or use the configurations from [4]. As baselines, which can learn from compressed data, we use CSGAN [30] and CKSVD [24] with OMP, sparsity level 4 and 288 learnable dictionary atoms. We solely evaluate CKSVD on piecewise smooth functions due to its requirement of varying measurement matrices for observing the training samples. For CSGAN on MNIST, we use the configuration specified in [30]. 4 In any simulation, we use K = 32 components for the proposed CSGMM and the proposed CSVAE's en- and decoders contain two fully-connected layers with ReLU activation and one following linear layer, respectively. We utilize Adam for optimization [47], and only consider the CME approximation of the estimators (cf. Section 2.4) since we did not observe notable differences to the alternative estimators in Appendix G. We utilize an overcomplete Daubechies db4 dictionary [48] in all estimators with fixed dictionaries. For all estimators except CSGAN, all images and estimates are normalized and clipped between 0 and 1 , respectively. For CSGAN, images are normalized between -1 and 1 following [4, 30], and for evaluation, test images and their estimations are then re-normalized between 0 and 1 . For a more detailed overview of hyperparameter configurations, see Appendix K.

4.2 Results

Reconstruction. In Fig. 2 and 3, the reconstruction performance of all estimators for MNIST, the piecewise smooth functions, and celebA is shown. All trainable models are trained on compressed training samples Y of dimension M without any ground-truth information during training (cf. Section 2.1). The models are then used to estimate test signals x i from observations of the same dimension M (cf. (1)) for evaluation. In the case of the piecewise smooth functions, we also add noise of a 10dB signal-to-noise ratio (SNR) for training and testing, i.e., σ 2 n = E [ ∥ Ax ∥ 2 2 ] / ( M · 10) in (1).

80

120

160

200

0

.

01

0

.

04

0

.

07

0

.

1

M

nMSE

2

·

10

2

2

·

10

3

2

·

10

4

0

.

01

0

.

03

0

.

05

0

.

07

0

.

09

N

t

40

80

120

0

.

02

0

.

05

0

.

08

0

.

11

M

10

2

10

3

10

4

0

.

02

0

.

05

0

.

08

0

.

11

N

t

80

120

160

200

0

.

2

0

.

4

0

.

6

0

.

8

M

SSIM

2

·

10

2

2

·

10

3

2

·

10

4

0

.

2

0

.

4

0

.

6

0

.

8

N

t

SBL [19]

CKSVD [24]

CSGAN [30]

CSVAE (ours)

CSGMM (ours)

a) MNIST

c) MNIST

f) piecewise smooth

g) piecewise smooth

b) MNIST

d) MNIST

e)

h)

i)

Original

SBL

CSGAN

CSVAE

(ours)

CSGMM

(ours)

Original

SBL

CKSVD

CSVAE

(ours)

D

type

DCT

Haar

db3

db5

None

CSVAE

0.0183

0.0170

0.0088

0.0064

0.0513

CSGMM 0.0199

0.0196

0.0100

0.0085

0.0624

In Fig. 2 a)-d), the nMSE and SSIM on MNIST is shown for a) and b) varying observation dimensions M and the fixed number N t = 20000 of training samples, and c) and d) vice versa with fixed M = 160 . The error bars represent standard deviations. In terms of both distortion metrics nMSE and SSIM , CSVAE and CSGMM perform overall the best. In Fig. 2 c), exemplary reconstructed MNIST images for M = 200 and N t = 20000 are shown. Perceptually, CSGAN emphasizes different reconstruction aspects than the proposed CSV AE and CSGMM. While CSVAE and CSGMM successfully recover details of the MNIST images, CSGAN prioritizes the similarity in contrast between bright and dark areas. This can be explained by their different estimation strategies. The estimation of CSGANs is restricted to lie on the learned manifold, which has been adjusted based on the training dataset. In consequence, when the test sample x ∗ contains details that cannot be found in the training dataset, CSGANs cannot reconstruct these but rather output a similar but realistic representative from the manifold. On the contrary, our proposed estimation scheme in Section 2.4 is not restricted to a manifold but rather infers conditional distributions (cf. (8)) over the whole linear space. Moreover, Fig. 2 c) and d) demonstrate that the proposed model can effectively learn from only a few hundred compressed and noisy samples. In Fig. 2 i), CSVAE and CSGMM are compared using different dictionaries types. The overall nMSE performance remains good for different dictionaries except for applying the models directly in the pixel domain.

In Fig. 2 f), g) and h), the nMSE over varying M (see f)), varying N t (see g)) and exemplary reconstructed samples (see h)) are shown for the set of piecewise smooth functions. Here, we only evaluated the proposed CSVAE. 5 Compared to the baselines Lasso, SBL, and CKSVD, the CSVAE performs the best in distortion (see f) and g)) as well as perception (see h)). In Fig. 3 a)-e), the same plots are shown for the celebA dataset. Despite the significantly larger dimension, the results are consistent with those for MNIST and the set of piecewise smooth functions.

900

1800

2700

0

.

01

0

.

02

0

.

03

M

nMSE

500

1581

5000

0

.

002

0

.

004

0

.

006

0

.

008

N

t

15

17

19

21

0

20

40

60

h

(

z

|

y

)

Counts

900

1800

2700

0

.

2

0

.

4

0

.

6

0

.

8

M

SSIM

500

1581

5000

0

.

6

0

.

7

0

.

8

N

t

a) celebA

b) celebA

c) celebA

d) celebA

e)

SBL [19]

CSVAE (ours)

CSGMM (ours)

Original

SBL

CSVAE

(ours)

CSGMM

(ours)

f) trained on compressed zeros

g) MNIST (

M

= 200

,

N

t

= 20000

)

labels of the test samples

y

are:

0

,

1

,

7

Model

Training time

(minutes)

Avg reconstruc-

tion time (ms)

-

73.85

SBL

-

294.17

CSGAN

1481

480.81

CSGMM (ours)

45

2.64

CSVAE (ours)

88

2.02

Uncertainty quantification. One possibility to determine the model's uncertainty is to determine the differential entropy h ( z | y ) of the CSVAE's encoder distribution q ϕ ( z | y ) . Since q ϕ ( z | y ) is a Gaussian with diagonal covariance matrix (cf. Section 3.2), h ( z | y ) can be calculated efficiently in closed form. In Fig. 3 f), a histogram is shown with values of h ( z | y ) . The CSVAE is trained on solely compressed MNIST zeros. We then forward compressed zeros, ones and sevens and evaluate h ( z | y ) . It can be seen that the CSVAE's encoder identifies the observations that are distinct to the training dataset by providing larger entropy values on average for the compressed ones and sevens.

Runtime. Fig. 3 g) shows our measured training time as well as average reconstruction time of the MAP-based estimators (cf. Section G) for MNIST, M = 200 , N t = 20000 . CSGMM and CSVAE are considerably faster than the baselines validating the corresponding discussion in Section 3.4.

Additional results. The results in Fig. 2 and 3 display the performance for CSGAN and the proposed CSVAE and CSGMM trained on solely compressed data. However, all three models can also learn from ground-truth data (cf. Appendix H). In Appendix L, we include this comparison for MNIST, provide results on FashionMNIST, analyze the estimators' robustness, and plot further reconstructions for all datasets. Moreover, in Appendix M, we analyze the runtime for training and reconstruction in more detail and give an overview of the used compute resources.

5 Conclusion

In this work, we introduced a new type of learnable prior for regularizing ill-conditioned inverse problems denoted by sparse Bayesian generative modeling. Our approach shares the property of classical CS methods of utilizing compressibility, but at the same time, it incorporates the adaptability to training data. Due to its strong regularization towards sparsity, it can learn from a few corrupted data samples. It applies to any type of compressible signal and can be used for uncertainty quantification. While this work focused on setting up the sparse Bayesian generative modeling framework, extensions to learnable dictionaries, circumventing the inversion of the observations' covariance matrices during training, and perception-emphasizing estimators are part of future work.

Acknowledgments and Disclosure of Funding

This research was supported by Rohde & Schwarz GmbH & Co. KG. The authors gratefully acknowledge the financial support and resources provided by the company. The authors also sincerely appreciate the valuable discussions with Dominik Semmler and Benedikt Fesl.

References

Y. Eldar and G. Kutyniok, Compressed Sensing: Theory and Applications . Cambridge University Press, 11 2012.

K. Gregor and Y. LeCun, 'Learning fast approximations of sparse coding,' in Proceedings of the 27th International Conference on International Conference on Machine Learning , ser. ICML'10. Madison, WI, USA: Omnipress, 2010, p. 399-406.

J. Liu, X. Chen, Z. Wang, and W. Yin, 'ALISTA: Analytic weights are as good as learned weights in LISTA,' in International Conference on Learning Representations , 2019.

A. Bora, A. Jalal, E. Price, and A. G. Dimakis, 'Compressed sensing using generative models,' in Proceedings of the 34th International Conference on Machine Learning , ser. Proceedings of Machine Learning Research, vol. 70. PMLR, Aug 2017, pp. 537-546.

D. Ulyanov, A. Vedaldi, and V. Lempitsky, 'Deep image prior,' International Journal of Computer Vision , vol. 128, no. 7, p. 1867-1888, Mar. 2020.

R. Heckel and P. Hand, 'Deep decoder: Concise image representations from untrained nonconvolutional networks,' in 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019 . OpenReview.net, 2019.

Z. Kadkhodaie and E. Simoncelli, 'Stochastic solutions for linear inverse problems using the prior implicit in a denoiser,' in Advances in Neural Information Processing Systems , M. Ranzato, A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, Eds., vol. 34. Curran Associates, Inc., 2021, pp. 13 242-13 254.

T.-O. Buchholz, M. Jordan, G. Pigino, and F. Jug, 'Cryo-care: Content-aware image restoration for cryo-transmission electron microscopy data,' in 2019 IEEE 16th International Symposium on Biomedical Imaging (ISBI 2019) , 2019, pp. 502-506.

G. Revach, T. Locher, N. Shlezinger, R. J. G. van Sloun, and R. Vullings, 'Hkf: Hierarchical Kalman filtering with online learned evolution priors for adaptive ECG denoising,' IEEE Transactions on Signal Processing , vol. 72, pp. 3990-4006, 2024.

W. Kim, Y. Ahn, J. Kim, and B. Shim, 'Towards deep learning-aided wireless channel estimation and channel state information feedback for 6G,' Journal of Communications and Networks , vol. 25, no. 1, pp. 61-75, 2023.

A. Jalal, L. Liu, A. G. Dimakis, and C. Caramanis, 'Robust compressed sensing using generative models,' in Neural Information Processing Systems , 2020.

R. Tibshirani, 'Regression shrinkage and selection via the Lasso,' Journal of the Royal Statistical Society (Series B) , vol. 58, pp. 267-288, 1996.

R. Baraniuk, 'Optimal tree approximation with wavelets,' Proceedings of SPIE - The International Society for Optical Engineering , vol. 3813, pp. 196-207, Dec. 1999.

A. Liu, V. K. N. Lau, and W. Dai, 'Exploiting burst-sparsity in massive MIMO with partial channel support information,' IEEE Transactions on Wireless Communications , vol. 15, no. 11, pp. 7820-7830, 2016.

L. He, H. Chen, and L. Carin, 'Tree-structured compressive sensing with variational Bayesian analysis,' IEEE Signal Processing Letters , vol. 17, no. 3, pp. 233-236, 2010.

L. He and L. Carin, 'Exploiting structure in wavelet-based Bayesian compressive sensing,' IEEE Transactions on Signal Processing , vol. 57, no. 9, pp. 3488-3497, 2009.

R. G. Baraniuk, V. Cevher, M. F. Duarte, and C. Hegde, 'Model-based compressive sensing,' IEEE Transactions on Information Theory , vol. 56, no. 4, pp. 1982-2001, 2010.

M. E. Tipping, 'Sparse Bayesian learning and the relevance vector machine,' J. Mach. Learn. Res. , vol. 1, p. 211-244, sep 2001.

D. Wipf and B. Rao, 'Sparse Bayesian learning for basis selection,' IEEE Transactions on Signal Processing , vol. 52, no. 8, pp. 2153-2164, 2004.

G. Yu and G. Sapiro, 'Statistical compressed sensing of Gaussian mixture models,' IEEE Transactions on Signal Processing , vol. 59, no. 12, pp. 5842-5858, 2011.

F. Renna, R. Calderbank, L. Carin, and M. R. D. Rodrigues, 'Reconstruction of signals drawn from a Gaussian mixture via noisy compressive measurements,' IEEE Transactions on Signal Processing , vol. 62, no. 9, pp. 2265-2277, 2014.

G. Yu, G. Sapiro, and S. Mallat, 'Solving inverse problems with piecewise linear estimators: From Gaussian mixture models to structured sparsity,' IEEE Transactions on Image Processing , vol. 21, no. 5, pp. 2481-2499, 2012.

J. Yang, X. Liao, X. Yuan, P. Llull, D. J. Brady, G. Sapiro, and L. Carin, 'Compressive sensing by learning a Gaussian mixture model from measurements,' IEEE Transactions on Image Processing , vol. 24, no. 1, pp. 106-119, 2015.

F. Pourkamali Anaraki and S. M. Hughes, 'Compressive K-SVD,' in 2013 IEEE International Conference on Acoustics, Speech and Signal Processing , 2013, pp. 5469-5473.

F. P. Anaraki and S. Hughes, 'Memory and computation efficient PCA via very sparse random projections,' in Proceedings of the 31st International Conference on Machine Learning , ser. Proceedings of Machine Learning Research, vol. 32, no. 2. Bejing, China: PMLR, 22-24 Jun 2014, pp. 1341-1349.

F. Pourkamali-Anaraki, S. Becker, and S. M. Hughes, 'Efficient dictionary learning via very sparse random projections,' in 2015 International Conference on Sampling Theory and Applications (SampTA) , 2015, pp. 478-482.

T. Chang, B. Tolooshams, and D. Ba, 'Randnet: Deep learning with compressed measurements of images,' in 2019 IEEE 29th International Workshop on Machine Learning for Signal Processing (MLSP) , 2019, pp. 1-6.

M. Aharon, M. Elad, and A. Bruckstein, 'K-SVD: An algorithm for designing overcomplete dictionaries for sparse representation,' IEEE Transactions on Signal Processing , vol. 54, no. 11, pp. 4311-4322, 2006.

A. Bora, E. Price, and A. G. Dimakis, 'AmbientGAN: Generative models from lossy measurements,' in 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings . OpenReview.net, 2018.

M. Kabkab, P. Samangouei, and R. Chellappa, 'Task-aware compressed sensing with generative adversarial networks,' in AAAI Conference on Artificial Intelligence , 2018.

B. Fesl, B. Böck, F. Strasser, M. Baur, M. Joham, and W. Utschick, 'On the asymptotic mean square error optimality of diffusion models,' 2024, arXiv:2403.02957.

M. Koller, B. Fesl, N. Turan, and W. Utschick, 'An asymptotically MSE-optimal estimator based on Gaussian mixture models,' IEEE Transactions on Signal Processing , vol. 70, pp. 4109-4123, 2022.

M. Baur, B. Fesl, and W. Utschick, 'Leveraging variational autoencoders for parameterized MMSE estimation,' IEEE Transactions on Signal Processing , vol. 72, pp. 3731-3744, 2024.

J. O. Berger, Statistical decision theory and Bayesian analysis . New York: Springer-Verlag, 1985.

G. J. McLachlan and D. Peel, Finite mixture models . Wiley Series in Probability and Statistics, 2000.

C. M. Bishop, Pattern Recognition and Machine Learning (Information Science and Statistics) , 1st ed. Springer, 2007.

D. P. Kingma and M. Welling, 'An introduction to variational autoencoders,' Foundations and Trends® in Machine Learning , vol. 12, no. 4, p. 307-392, 2019.

Y. Blau and T. Michaeli, 'The perception-distortion tradeoff,' in 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition . IEEE, Jun. 2018.

D. T. Ulmer, C. Hardmeier, and J. Frellsen, 'Prior and posterior networks: A survey on evidential deep learning methods for uncertainty estimation,' Transactions on Machine Learning Research , 2023.

A. Malinin and M. Gales, 'Predictive uncertainty estimation via prior networks,' in Advances in Neural Information Processing Systems , S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, Eds., vol. 31. Curran Associates, Inc., 2018.

A. Amini, W. Schwarting, A. Soleimany, and D. Rus, 'Deep evidential regression,' Advances in Neural Information Processing Systems , vol. 33, 2020.

Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner, 'Gradient-based learning applied to document recognition,' Proceedings of the IEEE , vol. 86, no. 11, pp. 2278-2324, 1998.

D. L. Donoho and I. M. Johnstone, 'Adapting to unknown smoothness via wavelet shrinkage,' Journal of the American Statistical Association , vol. 90, no. 432, pp. 1200-1224, 1995.

Z. Liu, P. Luo, X. Wang, and X. Tang, 'Deep learning face attributes in the wild,' in Proceedings of International Conference on Computer Vision (ICCV) , December 2015.

H. Xiao, K. Rasul, and R. Vollgraf, 'Fashion-MNIST: a novel image dataset for benchmarking machine learning algorithms,' 2017, arXiv:1708.07747.

Z. Wang, A. Bovik, H. Sheikh, and E. Simoncelli, 'Image quality assessment: from error visibility to structural similarity,' IEEE Transactions on Image Processing , vol. 13, no. 4, pp. 600-612, 2004.

D. Kingma and J. Ba, 'Adam: A method for stochastic optimization,' in International Conference on Learning Representations (ICLR) , San Diega, CA, USA, 2015.

G. R. Lee, R. Gommers, F. Waselewski, K. Wohlfahrt, and A. O'Leary, 'Pywavelets: A python package for wavelet analysis,' Journal of Open Source Software , vol. 4, no. 36, p. 1237, 2019.

R. E. Blahut, Principles and practice of information theory . USA: Addison-Wesley Longman Publishing Co., Inc., 1987.

P. Bickel and K. Doksum, Mathematical Statistics: Basic Ideas and Selected Topics . Chapman and Hall/CRC, 2015.

F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay, 'Scikit-learn: Machine learning in Python,' Journal of Machine Learning Research , vol. 12, pp. 2825-2830, 2011.

A Proof of Theorem 3.1

To prove Theorem 3.1, we restate (3), i.e., there exists a C > 0 such that for all γ > 0 and s

Based on (19), there exists a C > 0 such that for all θ , s , z with γ θ ( z ) > 0

By multiplying both sides in (20) with p δ ( z ) and integrating over z , we conclude that there exists a C > 0 such that for all θ , s , δ with γ θ ( z ) > 0

independent of the particular choice of p δ ( z ) . In case of discrete and finite p δ ( z ) the integral corresponds to a summation.

B Derivation of the adapted ELBO for CSVAEs

We write out the KL divergence and utilize that p θ ( y ) is independent of the expectation, i.e.,

C Conditional Posterior in the Noise-Free Case

In the noise-free case, the closed-form covariance matrix and mean of p θ ( s | z , y ) are given by

with ( · ) † being the Moore-Penrose inverse [19].

D Closed-Form Solution of the CSVAE Reconstruction Loss

E Closed-Form Solution of the CSVAE KL Divergences

By inserting the closed form of the KL divergence between two Gaussians (cf. [49]), we get

where µ s | y i , ˜ z i θ (˜ z i ) and C s | y i , ˜ z i θ (˜ z i ) are the mean and covariance matrix of p θ ( s | ˜ z i , y i ) . Moreover [47],

with q ϕ ( z | y i ) = N ( z ; µ ϕ ( y i ) , diag ( σ 2 ϕ ( y i ))) and µ j, ϕ ( y i ) and σ 2 j, ϕ ( y i )) being the j th entry of µ ϕ ( y i ) and σ 2 ϕ ( y i ) , respectively. Additionally, N L denotes the CSVAE's latent dimension.

F Proof of Lemma 3.2

The optimization problem we aim to solve is given by

First, we reformulate the objective as

Moreover, we observe that log p ( y i | s ) does not depend on { ρ k , γ k } and we can leave it out from the optimization problem. Additionally,

where we inserted p ( s | k ) = N ( s ; 0 , diag ( γ k )) (cf. (7)) and γ k,j denotes the j th entry of γ k . In the next step, let E p t ( s | y i ,k ) [ | s j | 2 ] = | µ s | y i ,k k,j | 2 + C s | y i ,k k,j,j , where µ s | y i ,k k,j and C s | y i ,k k,j,j denote the j th entry of µ s | y i ,k k,t and the diagonal of C s | y i ,k k,t in the t th iteration, respectively. Thus, our optimization problem of interest can be restated as

(35)

with Lagrangian

and Lagrangian multiplier ν . Taking the derivative of L with respect to γ m,q and ρ m and setting it to zero leads to

where (41) comes from the normalization of ∑ k ρ k = 1 .

G Estimators Based on the CSVAE and CSGMM

CSVAE. The CME approximation in (9) with the proposed CSVAE in Section 3.2 is given by

with Z containing samples ˜ z i ∼ q ϕ ( z | y ) . Alternatively, one can estimate x ∗ based on a newly observed y in the following way. We use the mean µ ϕ ( y ) as maximum a posteriori (MAP) estimate based on q ϕ ( z | y ) and, then, estimate x ∗ by

This method is applied in [33] to reduce computational complexity, but also deviates from the CME approximation and potentially improves the perceptual quality of the reconstruction.

CSGMM. In case of the proposed CSGMM in Section 3.3 the CME approximation in (9) exhibits a closed form, i.e.,

An alternative estimator of x ∗ is given by

with ˆ k MAP = argmax p ( k | y ) [23]. A pseudo-code for all estimators is provided in Appendix N (cf. algorithm 2-6).

H CSVAE and CSGMM Training on Ground-Truth Data

The CSGMM as well as the CSVAE can both be trained given ground-truth datasets. For training the CSGMM, this dataset either contains the compressible ground-truth signals s i , i.e., S , or the signals x i themself, i.e., X . On the other hand, for training the CSV AE, both datasets must also contain their corresponding observations y i . Thus, the training dataset must be G or W (cf. Section 2.1).

CSGMM. To train the CSGMM, the training goal is to maximize the log-evidence of the training dataset. In case of having access to S , the vanilla EM algorithm is employed with the modification of enforcing the GMM's means to be zero and covariance matrices to be diagonal in every update and, thus,

step. More precisely, after computing the posteriors p t ( k | s i ) by the Bayes rule in the t th iteration, the m-step is given by

In case of having access to X , the same modified EM algorithm from Section 3.3 is employed by exchanging AD with D and setting the noise variance σ 2 to some small value.

CSVAE. To train the CSVAE using the dataset G , we also aim optimize the log-evidence of the training dataset

By considering that log p θ ( y i | s i ) does not depend on the CSVAE's parameters (see (7)), introducing additional variational parameters ϕ and approximating p θ ( z | y , s ) with q ϕ ( z | y ) , we apply the standard reformulations of VAEs, which ends up in an ELBO resembling the standard ELBO in (6), given by

with ˜ z i ∼ q ϕ ( z | y i ) and p θ ( s i ) being defined in (10) with p ( z ) = N ( z ; 0 , I ) . In the case of training the CSVAE with W , the same training procedure from Section 3.2 is applied, resulting in the modified ELBO ˜ L ( θ , ϕ ) approximated by

where we set the variational distribution q ϕ ( s , z | x i , y i ) of the latent variables given the observations to q ϕ ( z | y i ) p θ ( s | z , x i ) . Consequently, instead of computing the posterior p θ ( s | z , y i ) , we use p θ ( s | z , x i ) , which is given by (8) with D instead of AD and setting the noise variance σ 2 to some small value. Additionally, to compute E p θ ( s | ˜ z i , x i ) [log p ( x i | s )] we replace y i , M and AD in (29) with x i , N and D , respectively.

I Implementation Aspects for Reducing the Computational Overhead

In a first step towards a computationally more efficient implementation for training the CSVAE and CSGMM, we reformulate the expressions of the conditional mean µ s | y , z θ ( z ) and conditional covariance matrix C s | y , z θ ( z ) from [18, 19] in (8). To do so, we observe that conditioned on z , s and y in (7) are jointly Gaussian. In consequence, we can alternatively apply the standard formulas for computing the moments of a conditional distribution for a jointly Gaussian setup [50], i.e.,

with and C s , y | z θ ( z ) = diag ( γ θ ( z )) D T A T . Importantly, we do not need to explicitly compute C s | y , z θ ( z ) according to (51). The subsequent reformulations differ between CSGMM and CSVAE.

CSGMM. For the CSGMM, only the diagonal entries of C s | y , z θ ( z ) must be explicitly computed, i.e.,

The matrix C s , y | z θ ( z ) ( C y | z θ ( z ) ) -1 (i.e., C s , y | k k ( C y | k k ) -1 6 ) has been precomputed for the conditional mean in (50) and subsequently determining diag ( C s , y | k k ( C y | k k ) -1 ( C s , y | k k ) T ) only requires O ( SM ) operations. Moreover, ( C y | k k ) -1 has already been determined for evaluating the posterior distributions p ( k | y ) in the preceding e-step (cf. Section 3.3). The closed-form m-step in (18) only takes diag ( C s | y ,k k ) for which we can directly use (53). In this way, we circumvent explicitly computing the full posterior covariance matrices in (8), rendering training the CSGMM more efficient.

CSVAE. The objective for training the CSVAE is given in (15), which consists of the modified reconstruction loss in (29) as well as the KL divergences in (30) and (31). While µ s | y , z θ ( z ) in (29) can be directly computed using (50), we reformulate the trace-term in (29) to circumvent the explicit computation of C s | y , z θ ( z ) . To do so, we apply the following steps

with C s | y i , ˜ z i θ ,j,j (˜ z i ) being the j th diagonal entry of C s | y i , ˜ z i θ (˜ z i ) and γ θ ,j (˜ z i ) being the j th entry of γ θ (˜ z i ) . For the derivation, we mainly apply the formula of C s | y i , ˜ z i θ (˜ z i ) in (8). By observing that the σ 2 in (55) cancels out with the 1 /σ 2 in (29), the trace-term in (29) cancels out with -S + tr ( diag ( ( γ θ (˜ z i ) -1 ) C s | y i , ˜ z i θ (˜ z i ) ) in (30). We also reformulate log det ( C s | y i , ˜ z i θ (˜ z i ) ) in (30), rendering the explicit computation of C s | y i , ˜ z i θ (˜ z i ) obsolete, i.e.,

where we use the formula of C s | y i , ˜ z i θ (˜ z i ) in (8), C y | ˜ z i θ (˜ z i ) = AD diag ( γ θ (˜ z i )) D T A T + σ 2 I as well as Sylvester's determinant theorem for the reformulation from (59) to (60).

In general, equivalent reformulations can also be applied when the CSV AE and CSGMM are trained on ground-truth data (cf. Appendix H) by replacing C s | y i , ˜ z i θ (˜ z i ) with C s | x i , ˜ z i θ (˜ z i ) .

0

100

200

0

0

.

6

0

100

200

-

0

.

5

0

0

.

5

0

100

200

0

1

0

100

200

0

1

2

0

100

200

-

1

.

5

0

1

.

5

0

100

200

0

3

J Implementation of the 1D Dataset of Piecewise Smooth Functions

The artificial 1D dataset of piecewise smooth functions is generated in the following way:

where h ( j ) m,i ∼ Bernoulli (0 . 5) and either takes -0 . 4 or 0 . 4 , a ( m ) i ∼ N (0 , 0 . 1 2 ) , η ( m ) i ∼ U (0 , 2 π ) , g 1 ∼ U (0 , 2) and g 2 ∼ U (2 , 4) . After generation, each x i ( t ) is sampled equidistantly N = 256 times, and its samples are stored in vectors x i , which then constitute the ground-truth dataset X . Exemplary signals and their wavelet decomposition are given in Fig. 4 and 5, respectively.

K Detailed Overview of Hyperparameter Configurations

MNIST & FashionMNIST. The non-learnable baselines for the simulations on MNIST are Lasso as well SBL. We apply Lasso directly in the pixel domain with its shrinkage parameter λ set to 0 . 1 in line with [4]. For SBL, we use an overcomplete db4 dictionary with symmetric extension [48]. Moreover, although we do not include noise in the simulations for MNIST, we set σ 2 in (2) to an increment corresponding to 40 dB SNR to be able to apply the computationally efficient reformulations from Appendix I. For CSGAN, we use the exact hyperparameter configurations specified in [30]. For CSGMM, we set the number K of components to 32 and iterate until the increments of the training dataset's log-evidence reach the tolerance parameter of 10 -3 , a standard stopping criterion for GMMs [51]. The CSVAE encoders and decoders contain two fully connected layers with ReLU activation and one following linear layer, respectively. The widths of the layers are set in a way such that for the first two layers, the width increases linearly from the input dimension to 256 , while the final linear layer maps from 256 to the desired dimension (i.e., either S for the decoder or twice the latent dimension for the encoder). The latent dimension is set to 16 , the learning rate is set to 2 · 10 -5 , and the batch size is set to 64 . We use the Adam optimizer for optimization [47]. We once reduce the learning rate by a factor of 2 during training and stop the training, when the modified ELBO in (15) for a validation set of 5000 samples does not increase. For CSGMM as well as CSVAE we set σ 2 in (7) to an increment corresponding to 40 dB SNR to be able to apply the training reformulation from Appendix I. We also use the same overcomplete db4 dictionary as for SBL. Moreover, we use N s = 64 samples to approximate the outer expectation in (9).

Piecewise smooth function. For the simulations on the set of piecewise smooth functions, we adjust the shrinkage parameter of Lasso based on a ground-truth validation dataset of 5000 samples once for every M . Moreover, we also use the overcomplete db4 dictionary for Lasso as for all other dictionary-based estimators. Instead of choosing 256 , we choose 128 as the maximum width of the en- and decoder layers. Otherwise, the hyperparameters remain the same as for the simulations on MNIST.

CelebA. For the simulations on celebA, we set the shrinkage parameter λ of Lasso to 0 . 00001 (cf. [4]). As celebA contains colored images, we choose a block-diagonal dictionary with the overcomplete db4 dictionary three times along the diagonal and zero matrices in all off-diagonals.

1

4

16

64

0

.

01

0

.

02

0

.

03

0

.

04

N

s

nMSE

80

120

160

200

0

.

01

0

.

03

0

.

05

M

2

·

10

2

2

·

10

3

2

·

10

4

0

.

01

0

.

03

0

.

05

N

t

80

120

160

200

0

.

01

0

.

05

0

.

09

0

.

13

M

1

4

16

64

0

.

5

0

.

6

0

.

7

0

.

8

N

s

SSIM

80

120

160

200

0

.

2

0

.

4

0

.

6

0

.

8

M

2

·

10

2

2

·

10

3

2

·

10

4

0

.

5

0

.

6

0

.

7

0

.

8

N

80

120

160

200

0

.

2

0

.

4

0

.

6

0

.

8

M

a) MNIST

c) MNIST

e) MNIST

g) FashionMNIST

b) MNIST

d) MNIST

f) MNIST

h) FashionMNIST

CSGAN [30]

(trained on

Y

)

CSVAE

(trained on

Y

, ours)

CSGMM

(trained on

Y

, ours)

SBL [19]

CSGAN [30]

(trained on

ground truth

)

CSVAE

(trained on

ground truth

, ours)

CSGMM

(trained on

ground truth

, ours)

t

Each block corresponds to one color channel. We use this dictionary for all estimators on celebA. . Otherwise, all hyperparameters are chosen in the same way as for MNIST.

The batch size to set to 32 Generally, we did no detailed network architecture search for the proposed CSV AE since we observed no considerable change in performance by testing out different architectures.

L Additional Results

Robustness comparison. The proposed CSVAE relies on approximating the outer expectation in (9) by Monte-Carlo sampling. Similar to this approximation, the baseline CSGAN also applies several random restarts, i.e., it estimates the ground-truth sample several times for a single observation and chooses the best-performing estimation by comparing their tractable measurement errors [30, 4]. In this way, both methods can be compared in terms of the number N s of repeated estimations they perform for a single observation. In Fig. 6 a) and b), we compare the nMSE and the SSIM for both with respect to N s . We evaluate their performance on the MNIST dataset with M = 200 and N t = 20000 . It can be seen that the proposed CSV AE achieves already good performance for a single Monte-Carlo sample, i.e., N s = 1 , while CSGAN is significantly worse when only using one restart (i.e., N s = 1 ) compared to having many restarts.

MNIST with training on ground-truth data. In Fig. 6 c)-f), the results from Fig. 2 a)-d) are extended with the performance of the corresponding models trained on ground-truth data, which are represented by the dashed curves. It should be noted that ground-truth information here refers to the MNIST images x i themself. In consequence, CSGAN as well as the proposed CSGMM are trained on X , while CSVAE is trained on W (cf. Section 2.1 and Appendix H). For the sake of readability, we leave out the error bars in these plots. Generally, the CSGAN, as well as the proposed CSV AE and CSGMM, benefit from the additional information during training in terms of the distortion metrics nMSE and SSIM . For the nMSE , the overall performance comparison remains the same, while for SSIM CSGAN trained on X outperforms CSVAE and CSGMM trained on W and X , respectively. In Fig. 7, additional exemplary reconstructed MNIST images are shown for all models trained on compressed as well as ground-truth data X in case of the CSGAN and CSGMM and W in case of the CSVAE. It can be seen that perceptually, the CSGAN significantly benefits from the ground-truth

Original

SBL

CSGAN

CSVAE

(ours)

CSGMM

(ours)

a) reconstructions from models solely trained on compressed data

Y

without any ground-truth information

CSGAN

CSVAE

(ours)

CSGMM

(ours)

b) reconstructions from models trained on ground-truth data

X

or

W

2

2

5

3

2

information during training, while the proposed CSV AE and CSGMM perform similarly in perception for a training set of compressed or ground-truth samples. This highlights their effective regularization effect explained in Section 3.4.

FashionMNIST. In Fig. 6 g) and h), the performance of SBL, CSGAN, and the proposed CSVAE and CSGMM for varying observation dimensions M and fixed number N t = 20000 of training samples is shown. While for small M , CSGAN outperforms the proposed CSVAE and CSGMM, its performance saturates for increasing M , and it performs worse than CSVAE and CSGMM. In this case, CSGAN's regularization to enforce the reconstruction to be in the generator's domain is beneficial for strongly compressed observations (i.e., for small M ). In Fig. 8, exemplary reconstructed FashionMNIST images are shown.

Additional exemplary reconstructions. In Fig. 9, 10 and 11, additional exemplary reconstructions for the piecewise smooth functions, MNIST as well as celebA are shown.

M Overview of Compute Resources

All models have been simulated on an NVIDIA A40 GPU except for the proposed CSGMM, whose experiments have been conducted on an Intel(R) Xeon(R) Gold 6134 CPU @ 3.20GHz . We report the number of learnable parameters, the time used for training as well as the average reconstruction time after training for simulations with piecewise smooth functions, MNIST and celebA in Table 1, 2 and 3, respectively. The average reconstruction time of estimating x ∗ from a newly observed y has been measured for the MAP-based estimators in Appendix G. While the reported numbers give an overview of the comparison between the different tested models, it is important to note that we did not aim to fully optimize our simulations for computational efficiency.

N Pseudo-Code for the Training and Inference of the CSVAE and CSGMM

Algorithm 1 summarizes one iteration of the training procedure for the CSV AE. Algorithm 4 does the same for the CSGMM. In algorithm 2 and 3, the pseudo-code of the CME approximation and the MAP-based estimator using the CSVAE is presented, respectively (cf. (42) and (43)). In algorithm 5 and 6, the same is given for the CSGMM (cf. (44) and (45))

Original

SBL

CSGAN

CSVAE

(ours)

CSGMM

(ours)

Original

SBL

CKSVD

CSVAE

(ours)

Original

SBL

CSGAN

CSVAE

(ours)

CSGMM

(ours)

4

G

3

5

5

Original

SBL

CSVAE

(ours)

CSGMM

(ours)

Algorithm 1 Update Step in the Training Phase of the CSVAE

Input: parameters in the t th iteration θ ( t ) (i.e., the decoder) and ϕ ( t ) (i.e., the encoder), batch Y batch , meas. matrix A (or corresponding batch meas. matrices { A i } i ), dict. D , noise σ 2 , optimizer Adam t in the t th iteration, learning rate λ

( t +1) ( t +1)

Algorithm 2 CME Approximation with the CSVAE in the Inference Phase (cf. (42))

Input: observation y , encoder ( µ ϕ ( · ) , σ ϕ ( · )) , decoder γ θ ( · ) , meas. matrix A , dict. D , noise σ 2 , cardinality |Z| in (42)

Output: CME approximation ˆ x ∗ CME

end for

Algorithm 3 MAP-based Estimator with the CSVAE in the Inference Phase (cf. (43))

Input: observation y , encoder ( µ ϕ ( · ) , σ ϕ ( · )) , decoder γ θ ( · ) , meas. matrix A , dict. D , noise σ 2 Output: MAP-based estimator ˆ x ∗ MAP

Algorithm 4 One EM Step in the Training Phase of the CSGMM with one fixed A

Input: parameters in the t th iteration { γ ( t ) k , ρ ( t ) k } K k =1 , training set Y , meas. matrix A , dict. D , noise σ 2

end for

Algorithm 5 CME Approximation with the CSGMM in the Inference Phase (cf. (44))

Input: observation y ,GMM { ρ k , γ k } K k =1 , meas. matrix A , dict. D , noise σ 2 Output: CME approximation ˆ x ∗

CME

Algorithm 6 MAP-based Estimator with the CSGMM in the Inference Phase (cf. (45))

Input: observation y ,GMM { ρ k , γ k } K k =1 , meas. matrix A , dict. D , noise σ 2 Output: MAP-based estimation ˆ x ∗

end for

MAP

Output: ( t +1) { γ ( t +1) , ρ ( t +1) } K

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: All claims made in the abstract and introduction are discussed in Section 2.4, 3.1, 3.2 ,3.3 and 4.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Limitations are discussed in Section 3.4.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Assumptions of Theorem 3.1 and Lemma 3.2 are stated in the theorem and lemma themself, while rigorous proofs of Theorem 3.1 and Lemma 3.2 are provided in Appendix A and F, respectively.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: All necessary information to reproduce the results is stated in Section 4.1 and Appendix K. Appendix N provides additional pseudo-code to facilitate the reproduction. Moreover, a link to a GitHub repository with our source code is provided in Section 1.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: Three of the four used datasets (MNIST, FashionMNIST, celebA) are freely accessible, while a detailed description of generating the fourth dataset (piecewise smooth functions) is given in Appendix J. Moreover, a link to our GitHub repository containing the source code is provided in Section 1.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: All necessary information is stated in Section 4.1. Moreover, we refer to the link to a GitHub repository with source code in Section 1.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We used a reasonable high number of test samples, see Section 4.1. Additionally, we provide error bars representing the results' standard deviations.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: The compute resources are provided in Appendix M.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have carefully read the NeurIPS Code of Ethics and confirm that our research conforms with it in every respect.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: There are no potential societal impacts of our work, which we think must be specifically highlighted.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: We do not see any potential risk for misuse of our proposed model.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We cite and explicitly state the licenses and terms of use of all used datasets in Section 4.1.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: All details are outlined in Section 4 and N.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: Our work does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: Our work does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
comments: SUMMARY:
This paper introduces a new type of sparsity inducing generative prior for the inverse problem. The authors theoretically underpin our approach by proving that its training maximizes a variational lower bound of a sparsity inducing log-evidence.

STRENGTHS:
This work can learn from a few corrupted data samples and, thus, requires no ground-truth information in its training phase.

WEAKNESSES:
1. The comparison methods include Lasso, CKSVD, and CSGAN, all of which are presented before 2018. The comparison methods are too old.
2. I think the author should clarify the specific application scenarios for this research direction.
3. In Figure 7, compared with CSGAN, the results of CSVAE and CSGMM are not very clear.

QUESTIONS:
1. In Table 2: Resources for simulations on MNIST (M = 200, Nt = 20000, Fig. 2 a)), CSGAN has much less parameters than the proposed CSVAE , why does CSGAN take much more training time?
2. Could the reconstruction results be influenced by the amount of noise?

LIMITATIONS:
I think the application scenes of this work is not very clear. The datasets used for comparison are very naive, colorful images with more textures should be used. And the authors should include more recent comparison methods, existing comparison methods are all presented before 2018.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-06-17

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors present an elegant new approach for dictionary based compressive sensing wherein the sparsity inducing prior is tuned from data by maximizing a lower bound on the evidence. This is a new paradigm for compressive sensing which appears to improve on reconstruction error over standard approaches and does not require solving an optimization problem at inference time.\n\nSTRENGTHS:\n- Approaching dictionary based compressive sensing in this way is novel as far as I'm aware. Not having to solve an optimization problem at inference time has the potential to be impactful.\n- I found the paper to be very well-written. The authors did a good job of motivating their approach by relating it back to relevant background literature and discussing necessary theory.\n- Their results appear to be promising and should hopefully motivate future work in this paradigm.\n\nWEAKNESSES:\n- Since their approach offers a new perspective on sparse Bayesian learning, it would have been helpful to compare the sparsity of predictions in addition to the nMSE.\n- I understand that space is limited, but I found the description of the experiments to be a bit terse. \n- It wasn't clear to me when a practitioner should choose CSVAE vs. CSGMM\n\nQUESTIONS:\n- How does your approach compare to methods for sparse Bayesian learning with proper priors; for example Louizos et. al. \"Bayesian Compression for Deep Learning.\" NeurIPS 2017?\n- Have you tried evaluating the quality of error bars from the predictive posterior so that your approach might be used in the context of UQ?\n- Can you explain the reason for the numerical instabilities in the noise free simulations for some of the image datasets?\n- Have you analyzed the histogram of the posterior over $s$? Can you suggest how the practitioner who is interested in a sparse $s$ might $0$ entries of the vector?\n\nLIMITATIONS:\nI think the authors have done a good job addressing the potential limitations of their approach in section 3.4. It would be interesting to analyze how well generative priors trained on a particular dataset transfer to similar datasets.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a novel training algorithm for generative models used as priors in linear inverse problems, with a specific focus on compressed sensing. The authors propose a training principle that regularizes the prior to learn a sparse representation of the signal of interest, implemented in Variational Autoencoders (VAEs) and Gaussian Mixture Models (GMMs). This is achieved by learning two posterior distributions p(y∣s) and p(s∣z) over a set of parameters through optimization of the Evidence Lower Bound (ELBO) derived in Equation 15.\n\nDuring the inversion phase, the true signal is estimated by sampling from the posterior distribution E[x∣y] as given in Equation 9. Unlike many contemporary generative priors, this method does not require solving an optimization algorithm to achieve the estimate. This approach simplifies the inversion process, making it more computationally efficient.\n\nThe performance of these implementations is validated on datasets containing different types of compressible signals. The results demonstrate the effectiveness of the proposed method in accurately reconstructing signals. Additionally, the paper provides theoretical support on the tightness of the lower bound of the estimate, further validating the robustness and reliability of the proposed approach.\n\nSTRENGTHS:\nThe proposed method appears to be novel and aligns with an intriguing line of work that focuses on training generative priors with the intention of using them for inverse problems.\n\nSection 2 of the paper is well-crafted, effectively elucidating the connection between the proposed method and existing literature.\n\nFor the given baselines, the proposed method yields better reconstructions of the true signal, as verified by commonly used metrics and visual inspection.\n\nWEAKNESSES:\nThe authors provide theoretical motivation; however, they do not address how the potential estimation of the measurement y impacts the estimation of the true signal x. Additionally, building on this point, the CSVAE may experience representation error, as noted in [5], which is not mentioned as a limitation. While the proposed method does improve upon [5], this remains a source of error.\n\n\nSection 4 is limited in the following ways:\n\n* The only metric reported for natural images is the normalized mean square error, while other commonly used metrics, such as PSNR and SSIM, are not included.\n\n* Given the computational focus of this paper, it is concerning that all experiments were conducted on low-resolution grayscale images, especially considering the A40 GPU budget.\n\n* The baselines appear to be less contemporary compared to current literature on compressed sensing inversion algorithms. [Normalizing Flow](https://proceedings.mlr.press/v119/asim20a/asim20a.pdf) and [Diffusion Models](https://openreview.net/pdf/4f6f0e2347a3d6f9a88b39e445f77c1e7503064e.pdf) have largely replaced GANs due to issues with representation error. Similar to the proposed method, Diffusion Generative Priors do not require gradient-based approaches to estimate the posterior. There are hyperlinks to the work in the responses. \n\n* Considering this paper is training generative before be used downstream, there seems to be no reported metrics about training other than computational resources.\n\nQUESTIONS:\n* Could the authors please provide perceptual metrics (i.e. SSIM, LPIPS) for the reconstruction comparison experiments in Figures 2 and 3? \n* Could the authors elaborate on the computational overhead provided in footnote 4 and explain the reason for utilizing only grayscale images when reporting having a computational budget of NVIDIA A40 GPU for CSVAE? \n* Could the authors please explain the advantages the proposed method has over contemporary generative prior such as normalizing flow and diffusion-based priors? \n* One concept I am unclear about is how do you check if the proposed method is overly biased towards the measurements y? This issue might arise because the conditional distribution P(y|s)  has equally likely probabilities when Y=AS, but does not guarantee S will be consistent with training samples depending on how sparse s is in practice.\n\nLIMITATIONS:\nPlease refer to the weakness portion of the review.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-15"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a set of methods to learn a generative model over the sparse representations of  a signal. The dictionary basis is fixed and given, and the signal of interest is assumed to be sparse in this basis. Therefore, the generative model learns to provide such sparse representations. The method learns the generative prior from the noisy and compressed observations. The authors leverage the notion of conditional Gaussianity and propose VAE and GMM based models for learning such priors. The choice enables the inference with only forward operation.\n\nSTRENGTHS:\nThe idea of using purely forward operations for inference is quite nice, since most of the solutions based on generative priors require gradient descent on the latent space and suffer from latency among other problems. Furthermore, learning the prior from compressed observations makes the approach more practical as it does not need ground truth signal.\n\nWEAKNESSES:\nOverall, the paper combines some of the existing ideas from prior works (for example posterior updates from SBL, VAE ELBO derivations), and therefore, the novelty seems marginal. To be more concrete, it *seems* straightforward that if one wants to do variational Bayes, then the VAE style decomposition is the standard approach. If there are any additional challenges, the authors should clarify.\n\nThe idea of forward pass only inference has been tried before (see for example the paper “Solving Linear Inverse Problems Provably via Posterior Sampling with Latent Diffusion Models” NeurIPS 2023).  \n\nLearning from the compressed samples can be compelling although not new as mentioned by the authors themselves (for example CSGAN). The proposed method requires further elaboration on the loss with respect to learning from uncompressed measurements substantiated with more experiments.\n\nThe experiment section is a bit thin given the limited technical novelty, particularly with respect to the baselines. Given a lot of prior works in the literature on learning based solvers for inverse problems (for example, all variations of generative priors based on flows and diffusions, unfolded algorithms like lista and its variations), the authors have a more difficult task of placing their contribution within the prior work. The paper is not very convincing in showing what is the unique direction in which the field is moved forward.\n\nQUESTIONS:\nSome of my questions about the novelty was implicit in my comments above.\n-\tCan authors provide some comment on the performance loss with respect to uncompressed generative prior?\n-\tHow many MC samples are used at inference for (9)?\n-\tGenerative priors generally suffer from convergence issues and require multiple restarts. Could the authors comment if their forward-only operation has a similar issue? \n-\tIt would be good to incorporate error bars in the plots and numerical results. Given the MC step, I expect a higher variance for the proposed approach.\n\nLIMITATIONS:\nSee comments above.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-26"}