Variational Learning ISTA
Authors: Fabio Valerio Massoli
Christos Louizos
Arash Behboodi
Venue: Submitted to ICLR 2023 (2023)
Keywords: compressed sensing
LISTA
variational models
inverse problems

Abstract: Compressed sensing combines the power of convex optimization techniques with a sparsity inducing prior on the signal space to solve an underdetermined system of equations. For many problems, the sparsifying dictionary is not directly given, nor its existence can be assumed. Besides, the sensing matrix can change across different scenarios. Addressing these issues requires solving a sparse representation learning problem, namely dictionary learning, taking into account the epistemic uncertainty on the learned dictionaries and, finally, jointly learning sparse representations and reconstructions under varying sensing matrix conditions.
We propose a variant of the LISTA architecture that incorporates the sensing matrix into the architecture. In particular, we propose to learn a distribution over dictionaries via a variational approach, dubbed \ac{VLISTA}, which approximates a posterior distribution over the dictionaries as part of an unfolded LISTA-based recovery network. Such a variational posterior distribution is updated after each iteration, and thereby adapts the dictionary according to the optimization dynamics. As a result, \ac{VLISTA} provides a probabilistic way to jointly learn the dictionary distribution and the reconstruction algorithm with varying sensing matrices. We provide theoretical and experimental support for our architecture and show that it learns calibrated uncertainties.

Variational Learning ISTA
Abstract
Compressed sensing combines the power of convex optimization techniques with a sparsity inducing prior on the signal space to solve an underdetermined system of equations. For many problems, the sparsifying dictionary is not directly given, nor its existence can be assumed. Besides, the sensing matrix can change across different scenarios. Addressing these issues requires solving a sparse representation learning problem, namely dictionary learning, taking into account the epistemic uncertainty the learned dictionaries and, finally, jointly learning sparse represen- tations and reconstructions under varying sensing matrix conditions. We propose variant of the LISTA architecture that incorporates the sensing matrix into the architecture. In particular, we propose to learn a distribution over dictionaries via variational approach, dubbed Variational Learning ISTA (VLISTA), which approximates a posterior distribution over the dictionaries as part of an unfolded LISTA-based recovery network. Such a variational posterior distribution is updated after each iteration, and thereby adapts the dictionary according to the optimiza- tion dynamies. As result, VLISTA provides probabilistic way to jointly learn the dictionary distribution and the reconstruction algorithm with varying sensing matrices. We provide theoretical and experimental support for our architecture and show that it learns calibrated uncertainties.
Introduction
Compressed sensing methods aim at solving under-determined inverse problems imposing prior about signal structure. Sparsity and linear inverse problems were canonical examples of the signal structure and sensing mediums (modelled with a linear transformation +). Many works during recent years focused on improving the performance and complexity of compressed sensing solvers for given dataset. typical approach is based on unfolding iterative algorithms as layers of neural networks and learning the parameters end-to-end starting from learning iterative soft thresholding algorithm (LISTA) Gregor LeCun (2010) with many follow-ups works. Varying sensing matrices and unknown sparsifying dictionaries some of the main challenges of data-driven approaches. The works in Aberdam et al. (202T); [cchnoor et al. 2022) address these issues by learning a dictionary and include it in the optimization iteration. However, the data samples might not have any exact sparse representations, which means that there is no ground truth dictionary. The issue can be more severe for heterogeneous datasets where the choice of the dictionary might vary from one sample another. A principled approach to this problem would be to take Bayesian approach and define a distribution over the learned dictionaries with proper uncertainty quantification.
In this work, first, we formulate an augmented LISTA-like model, termed Augmented Dictionary Learning ISTA (A-DLISTA), that can adapt its parameters to the current data instance. We theoreti- cally motivate such a design and empirically prove that it can outperform other LISTA- models in non-static measurement scenario, i.c., considering varying sensing matrices across data samples. We are aware that an augmented version of LISTA, named Neurally Augmented ALISTA (NALISTA), was already proposed in Behrens al. (2021), however, there some fundamental differences between NALISTA and A-DLISTA. First, our model takes as input the per-sample sensing matrix and the dictionary the current layer. This means that A-DLISTA adapts the parameters to the
Preprint. Under review.

---

current measurement setup as well as to the learned dictionaries. In contrast, NALISTA assumes to have a fixed sensing matrix to analytically evaluate its weight matrix, W. Hypothetically. NALISTA could handle varying sensing matrices, however, that comes at the price of having to solve for each data sample the inner optimization step to evaluate the W matrix. Moreover, the architectures the augmentation networks are profoundly different. Indeed, while NALISTA uses an LSTM, A-DLISTA employ a convolutional neural network, shared across all layers. Such a different choice reflects the different types of dependencies between layers and input data that the networks try to model. We report in Subsection 3.3 a detailed discussion about the theoretical motivation and architectural design for A-DLISTA. Moreover, the detailed architecture is described in Appendix A.
Finally, we introduce Variational Learning ISTA (VLISTA) where we learn distribution over dictionaries and update it after each iteration based on the outcome of the previous layer. In this sense, our model learns an adaptive iterative optimization algorithm where the dictionary is iteratively refined for the best performance. Besides, the uncertainties estimation provides an indicator for detecting Out- OfDDrisriibttinnn (OOD) samples. Intuitively, our model can be understood as form of recurrent variational autoencoder, e.g., ss.[Chung al. (2015), where on each iteration of the optimization algorithm, we have an approximate posterior distribution over the dictionaries, conditioned on the outcome of the last iteration. The main contributions of our work are as follows.
We design an augmented version of LISTA, dubbed A-DLISTA, that can handle non-static measurement setups, i.e., per-sample sensing matrices, and that can adapt parameters the current data instance.
We propose Variational Learning ISTA (VLISTA) that learns a distribution over sparsifying dictionaries. The model can be interpreted as Bayesian LISTA model that leverage A-DLISTA as the likelihood model.
VLJITA adapts the dictionary to optimization dynamics and therefore can be interpreted as a hierarchical representation learning approach, where the dictionary atoms gradually permit more refined signal recovery.
The dictionary distributions can be used for -fittr-diitribution detection.
The remaining part of the paper is organized as follows. In Section 2 we briefly report related works that are relevant to the current research, while in sectioon 3 the model formulation is detailed. The datasets description, as well as the experimental results, are reported in ssectionnnn 4] Finally, we report our conclusion (section
2 Related Works
Compressed sensing field is abound with works on theoretical and numerical analysis of recovery algorithms (see Foucart & Rauhut (2013) with iterative algorithms one of the central approaches like Iterative Soft- Soft-hhresholdig Algorithm (ISTA) Daubechies et al. (2004), Approximate message passing (AMP) Donoho et al. (2009) Orthogonal Matching Pursuit (MMP)Pai et al. (1993); Davis
et al. (194) and Iterative Hard-Threste Threshhhhsoo Algorithm (IHTA) Blumensath & Davies 2009). The mentioned algorithms characterized by specific set of hyperparameters, e.g., number of iterations and soft thresholds, that can be tuned to obtain a better trade-off between performance and complexity. be learned
With unfolding iterative algorithms as layers of neural networks, these parameters in an end-to-end fashion from a dataset, see for instance some variants Zhang & Ghanem (2018): Metzler al. (2017); yyng et al. (2016); Borgerding al. 2017) Sprechmann et al. (2015).
Bayesian Compressed Sensing (BCS) and Dictionary learning. A non-parametric Bayesian approach to dictionary learning has been introduced in Zhou et al. (2009 2012). where the authors consider a fully Bayesian joint compressed sensing inversion and dictionary learning. Besides, their atoms are drawn and fixed priori. Bayesian compressed sensing Ji et al. (2008) leverages relevance vector machines (RVMs) Tipping (2001) and uses a hierarchical prior to model distributions of each entry. This line of work quantifies uncertainty of recovered entries while assuming fixed dictionary. In contrast, in our work, the source of uncertainty is the unknown dictionary over which we define a distribution.
LISTA models. Learning ISTA was first introduced in Gregor & LeCun (0100) with many follow-up variations. The follow-up works in Behrens et al. (202 Liu et al. (2019); Chen al. (202I); Wu

---

et a.. (2020) provides various guidelines for architecture change to improve LISTA for example convergence, parameter efficiency, step size and threshold adaptation, and overshooting. The common assumptions of these works are fixed and known sparsifying dictionary and fixed sensing matrix. Steps toward relaxing these assumptions were taken in Aberdam et al. (202T); Behboodi et al.) (Scnoor
(2022); Schnoor et al. (2022). In AAberdam et al. (20TT) the authors propose model to deal with varying sensing matrix (dictionary). The authors in Schnoor et al. (2022); Behboodi et al. (2022) provide an architecture that can both incorporate varying sensing matrices and learn dictionaries, although their focus is on learning theoretical analysis of the model. There are theoretical studies on the convergence and generalization of unfolded networks, see for example Giryes et al. (2018);
Pu et al. (2022); Aberdam et al. (202]); Pu et al. 2022);; Chen et al. (0018); Behboodi et al. (0022) ;
Schnoor et al. (2022). In our paper, not only we consider varying sensing matrix and dictionary, but we also model distribution over dictionaries and thereby characterizing the epistemic uncertainty.
Recurrent Variational models. Variational autoencoders (VAFs) is framework, that learns generative model over the data through latent variables Kaingbles/Kingma Welling (2013); Rezende et al.) (2014). When there are data-sample specific dictionaries in proposed model, it reminisces extensions VAEs to the recurrent setting Chung et al. (2015 2016). which assumes a sequential structure in the data and imposes temporal correlations between the latent variables. There are also connections and similarities to Markov state-space models, such as the ones described at Krishnan ct al. (0017).
Bayesian Deep Learning. When we employ global dictionaries in VLISTA, the model essentially becomes variational Bayesian Recurrent Neural Network. Variational Bayesian neural networks have been introduced at Bllundell al. (2015). with independent priors and variational posteriors for each layer. This work has been further extended to recurrent settings at Foottunnt et al. (2019). The main difference between these works with our setting is the prior and variational posterior; in our case where the prior and variational posterior for each step is conditioned on previous steps, instead of being fixed across steps.
3 Variational Learning ISTA
In this section, we briefly report on the ISTA and LISTA models to solve linear inverse problems. Then, we introduce our first model, A-DLISTA, capable of learning the sparsifying dictionary and adapting to different sensing matrices. Finally, we focus on the VISTA model, variational framework for solving linear inverse problems that leverages A-DLISTA as the likelihood model and achieves high power to reject OOD samples.
3.1 Linear inverse problems
We consider the following linear inverse problem: Tx.
The matrix + is called the sensing matrix. If the vector is sparse in a dictionary basis V the problem can be cast as a sparse recovery problem = z with z given as a sparse vector. A proximal gradient descent-based approach to this problem yields ISTA iterations:
2, (z,- + The 4
1))2
(1)
where oz, 7 > 0 are hyper-parameters of the model meaning that the algorithm does not possess any trainable parameters. Generally speaking, % is called the step size and its value given as the inverse of the spectral norm of the matrix A, where A Φ The function given by: termed threshold and is the value characterizing the so-called soft-threshold (x())) sign(x)(e ([=c 0)+. In the ISTA formulation, those two parameters are shared across all the iterations. Therefore, we have Te, 02 > 7,
0 is
3.2 LISTA
LISTA Gregor& & LeCun 2010) reparametrized unfolded version of the ISTA algorithm which each iteration, or layer, is characterized by learnable matrices. Specifically, LISTA reinterpret Equa tion as defining the layer of a feed-forward neural network implemented as So, (Va,,1 + W.y where V,, W. are learnt from dataset. In that way, those weights implicitly contain information
3

---

about T and L. However, in many problems, the dictionary L is not given, and the sensing ma- trix can change for each sample the dataset. As LISTA, also its variations, e.g., Analytic LISTA (ALISTA)Lis et al. 2019), NALISTA Behrens et al. (202T) and HyperI Chen et al. (2021, require similar constraints such fix dictionary and sensing matrix. Thus, making those algorithms fail in situations where either 4 is not fixed or T is not known.
3.3
Augmented Dictionary Learning ISTA
To deal with situations where the underlying dictionary is not known, and moreover the sensing matrix is changing across samples, one can use an unfolded version of ISTA in which the dictionary is considered as learnable matrix, termed Dictionary Learning ISTA (DLISTA), for which each layer is given as follows:
70, (z--1 + ( (y----v 3
with one last linear layer mapping z to reconstructed input. The model can be trained end to end to learn all Or, Yr, ,, The base model is very similar Behhboodi et al. (2022) Aberdam et al. (2021) but as we will see further, it requires additional changes. To see this, consider the basic scenario where the sensing matrix is fixed to & there a ground-truth (unknown) dictionary V, such that I. ozz. with sparse 2, having support S, i.e., supp( 2.) =.
Consider the layer of DLISTA with fixed sensing matrix 4, and define:
p = max
(()))))) (D)))s)
(3)
ISHJSN
j1 max (( :)) ( v.));
(4)
151JSN
8(7) max - (VV
The term j is called the mutual coherence of the matrix L The term jūg is closely connected to generalized mutual coherence, however it differs in that unlike generalized mutual coherence, it includes the diagonal inner product for i = j. Finally, the term 8(7) is the reminiscent of restricted isometry property (RIP) constant Foucart & Rauhut (2013), key condition for many recovery guarantees in compressed sensing. Note that there a dependency on y. For simplicity, we only kept the dependence on y in the notation and dropped the dependence of j., Ao and 6 on & and L from the notation.
The following theorem provides conditions on each layer improving the reconstruction error.
Theorem 3.1. Consider the layer of DLISTA given by equation and suppose that y = DL. with supp( 2.) = S. We have
1. Suppose 21-1 has the same support 2n, i.e., supp(zz) supp(zz- (ji ||z 21- + Jia 1|==111) O.,
(6)
then supp(zz) C supp(z.).
2. Assuming that the conditions of the last step hold, then we get the following bound on the error:
1 - 2+11 S (5(m) + ni(SS) 1)) [1= 2011, /1 +S
We provide the derivations in the supplementary materials. Theorem 3.T provides insights about the choice of and 0,, and also suggests that (Ó(7) + S 1)) needs to be smaller than one to reduce the еттоr at each step. Similar to many existing works in the literature, Theorem 3.1 emphasizes the role of small mutual coherence, equation for good convergence.
Looking at the theorem, it can be seen that 7: and 0, play crucial role for the convergence. However, there is trade-off underlying these choices. Let's fix 0,. Decreasing "i can guarantee good support selection but can increase 8(yx). When the sensing matrix is fixed, the network can hopefully find good choices by end-to-end training. However, when the sensing matrix & changes across different data samples, i.c., & + &', ,' is not guaranteed anymore that there is a unique choice of 7 and Or for all different Φ'. Since these parameters can be determined for fixed & and V,, we propose using an augmentation network that determines 7t and 0, from each pair & and L. Following from theory, we show in Figure I the resulting model named A-DLISTA.

---

A-DLISTA relies on two basic operations at each layer, namely, soft threshhrehold (blue blocks in Figure 1) and augmentation (red blocks in Figure lp. The former represents an ISTA-like iteration parametrized by the set of learnable weights: { 0,, }t whilst the latter is im- plemented using encoder-deecder-like type of network. As shown the figure, the aug- mentation network takes input the sensing matrix for the given data sample, s together with the dictionary learned at the layer for which the augmentation model will generate the 0 and y parameters. Through such operation, the A- A-DLISTA adapts those last two parameters to the current data sample. We report more details about the augmentation network in Appendix A
Figure 1: A-DLISTA architecture. The blue blocks represent a single soft-thresholding oper- ation parametrized by the dictionary V, together with threshold and step size Ye) layer t. The red blocks represent the augmentation net- work (with shared parameters across layers) that adapts {0,, 7+ } for layer / based on the dictionary V, and the current measurement setup c for the i-th data sample. The dashed arrows connecting each blue block with a red one mean that, each iteration, the augmentation newtork receives the learned dictionary at the current iteration as input (together with the sensing matrix).
3.4 VLISTA
Although A-DLISTA possesses adaptivity to data samples, it is still based on the assump-
tion that ground truth dictionary exists. We relax that hypothesis by defining a probability distribution over the sparsifying dictionary and formulate a variational approach, titled VLİSTA, to jointly solve the dictionary learning and the sparse recovery problems. To forge variational framework whilst retaining the helpful adaptivity property of A-DLISTA, we re-interpret the soft- thresholding layers of the latter as part of likelihood model that defines the output mean for the reconstructed signal. Given its recurrent-like structure Chung et al. (2015), we equip VLISTA with a conditional trainable prior where the condition is given by the dictionary sampled at the previous iteration. Therefore, the full model comprises three components, namely, the conditional prior pe (-). the variational posterior (-). and the likelihood model, pe (.).). All components are parametrized by neural networks whose outputs represent the parameters for the underlying probability distribution. In what follows, we describe more in detail the various building blocks of the VLISTA model.
3.4.1 Prior distribution over dictionaries
The conditional prior, P5 ( 1).. is modelled as Gaussian distribution with parameters con- ditioned on the previously sampled dictionary. We parametrize Pę(-) using a neural network,
fe(-) Ifa o 95. (.), o 9E. (-)], with trainable parameters & = {Eo. }2, The model's ar- chitecture comprises a shared convolutional block followed by two different branches generating the mean and the standard deviation, respectively, of the Gaussian distribution. Therefore, at layer 1, the prior conditional distribution is given by: P: ( =
[,,, = -2_ (9Ea ( (vr-))) where the indices i, run over the rows and
S (9Ea (1) (Vb-1))a.si columns of Ue. In order to simplify our expressions, we will abuse notation and refer to distri- butions like the former as Pę ( .--)) ) = (N = f (gea (.-1)));'²
fg. (oto (.))))). We will use the same type of notation throughout the rest of the manuscript to simplify formulas. The prior's 's design allows for enforcing dependence of the dictionary at iteration t to the sampled the previous iteration. Thus, allowing us to refine the A as the iterations proceed. The only exception such process is the prior imposed over the dictionary sampled dictionary in this case. We handle such an exception by assuming a standard Gaussian distributed 1 Finally, the joint prior distribution over the dictionaries for VLISTA is given by:
= 1, since there is no a previously
T
Pe 1 = N 0:
NW (05 (1. (1-1))); o², = -fe (95 ( (V-1))))
2=
5

---

3.4.2 Posterior distribution over dictionaries
Similarly to the prior model, the variational posterior too is modeled as Gaussian distribution parametrized by neural network fo(-) [f héw (-), hdp (-)] which outputs the mean and variance for the underlying probability distribution: qo qo 1; 40) N = f (hoo (#:-1: y. F))); o² JØz (Nep (.. Φ²))). The posterior distribution is conditioned on the data, {y", ). well the reconstructed signal at the previous layer, #;-1- Therefore, the joint posterior probability over the dictionaries at each layer is given by:
go V 1:T W1T: a.r..y.... v')
II qo v 1t- ,
(8)
3.4.3 Likelihood model
At the heart the reconstruction module there is the soft-thresholding block of A-DLISTA. Similarly to the prior and posterior, the likelihood distribution is modelled as a Gaussian parametrized by the output of A-DLISTA block. Specifically, the likelihood network generates the mean vector only for the Gaussian distribution since we treat the standard deviation as tunable hyper-parameter. Therefore, interpret the reconstructed sparse vector given layer the mean of the likelihood distribution. The joint log-likelihood distribution can then be formulated as:
T
log pe ( V Wa., y' 4¹)
Σ
log N(m A-DLISTA ( v...y (";"))) = =6)
(9)
where 8 is hyper-parameter of the network.
We train all the different components of VLISTA, in an end-to-end fashion by the maximization of the Evidence Lower Bound (ELBO). The full objective function is given by:
T
Eg E (1 2o.--1 [log pe - adjx M la, y', FD)
ELBO
(10)
E E
DKL gol (', T6, .1 (C(
t=2
- DKL. (1 (V, ((0o) || Pe (VVi)
The first term in Equation 10 represents the likelihood contribution whilst the second and third terms account for the KL divergence. We report more details about models' architecture and the objective function in Appendix A and Appendix B respectively.
Experimental Results
To assess the performance of the proposed approach, employ three datasets, namely, MNIST, CIFARIO 10, and a synthetic one. We compare our models' performance against ISTA, LISTA Gregor & LeCun (2010). and BCS Di et al. (2008). However, do not consider other LISTA variations such as ALISTA Liu et al. (219) or NALISTA Behrens et al. (202T) since assuming varying measurement setup across the dataset requires solving an inner optimization problem to evaluate the W matrix for each data sample. As a result, training such models is extremely slow. Moreover, to prove the benefit of adaptivity, we conduct an ablation study on A-DLISTA by removing augmentation network and making the parameters 0,, 7i learnable through backprop. We refer to the non-augmented version of A-DLISTA as DLISTA (see/subsection 3.3 for more details). Hence, for DLISTA, O, and 7, cannot be adapted anymore to the specific input sensing matrix. For all models that train, consider

---

three layers. However, being ISTA classical method with no learning properties, we also considered the results obtained using 1000 iterations. To consider scenario with varying sensing matrices, we adopt the following procedure. For each data sample in the training and test sets, æ²' we generate sensing matrix, 4,, by randomly sampling its entries from standard distribution. Subsequently, for each pair of sensing matrix and ground truth signal, we generate the corresponding observations as y' & x². We report more details about the training of the models in Appendix
4.1 MNIST & CIFARIO
The first task we test our models against is image reconstruction considering the MNIST and CIFAR1O datasets. We report the results in terms of the Structural Similarity Index Measure (SSIM) considering the following setups. We fix the number of layers, iterations, for all models three and then measured SSIM varying the number of measurements. To compute the observation vector (y') generate a different sensing matrix, for each digit, by sampling its entries from standard Gaussian distribution (more details about the data generation can be found in Appendix B The results are reported in Table and Table for MNIST abd CIFARIO, respectively.
Table MNIST SSIM (the higher the better) for different number of measurements. Top three rows concern non- non- oon-aayesian models whilst the bottom report results for Bayesian approaches. For each of the two set of results, we highlight bold the best performance.
SSIM 1
number of measurements
I0
I00
300
500
ISTA
0.10
0.07
0.06
0.20
0.34
ISTA (1000 iterations)
0.10
0.07
0.12
0.46
0.76
LISTA
0.21
0.54
0.56
0.63
0.75
DLISTA
0.27
0.60
0.64
0.64
0.76
A-DLISTA
0.33
0.66
0.81
0.84
0.88
BCS
0.09
0.12
0.19
0.45
0.73
VLISTA
0.22
0.45
0.56
0.65
0.78
Table CIFARIO SSIM (the higher the better) for different number of measurements. Top three rows concern non- Bayesian models whilst the bottom two report results for Bayesian approaches. For each of the two set of results, we highlight in bold the best performance.
SSIM 1
number of measurements
I0
50
100
300
500
700
850
ISTA
0.001
0.004
0.009
0.025
0.035
0.040
0.041
ISTA (1000 iterations)
0.002
0,009
0.015
0.038
0.051
0.057
0.055
LISTA
0.008
0,227
0.271
0.597
0.658
0.716
0.779
DLISTA
0.348
0,458
0.472
0.581
0.647
0.713
0.778
A-DLISTA
0.581
0.684
0.717
0.776
0.799
0.812
0.852
BCS
0.005
0.028
0.046
0.114
0.194
0,286
0.359
VLISTA
0.265
0.286
0.464
0.617
0.692
0.731
0.774
From Table and Table 2] we can draw the following conclusions. Concerning the non non-Bayesian models, our A-DLISTA model outperforms the others. Moreover, by comparing of A-DLISTA with its non-augmented version, i.e., DLISTA, we see the benefits of using aug- mentation network to make the model adaptive. Instead, concerning the Bayesian approaches, our VLISTA model outperforms BCS. Especially, that our models outperform others considering
performance

---

low number of measurements. Concerning the lower performance of VLISTA compared to A- DLISTA, we can mention few reasons that could explain such behaviour. One contribution such difference might come from the noise that is naturally injected at training time due to the random sampling procedure to generate the dictionaries. Also, another contribution is represented by the amortization gap that affects all models based on amortized variational inference. However, although VLISTA shows lower performance than A-DLISTA, it is important to notice that it still performs better than BCS. Moreover, it can detect OODs, characteristic that Bayesian models only possess.
4.2 Synthetic Dataset
To generate the synthetic dataset, we follow a similar protocol as in & Chen (2019). First, we generate sensing matrix and sparsifying dictionary, for each data sample, by sampling their entries from a standard Gaussian distribution. Then, the components of the ground truth sparse signal, =", sampled from standard Gaussian distribution well. Finally, some of the components of =" set to zero as dictated by a Bernoulli distribution with p = 0.1. The overall dataset accounts for 1000 samples shared across the train and test sets. To compare the performance of different models, first, we draw the c.d.f of the Normalized Mean Square Error (NMSE) on the test set and then we compute the 40% quantile. Results are reported Table 3 Similarly to the setup we used in the previous section, also for the synthetic dataset we fix the number of layers, or iterations, for each model to three and then we varied the number of measurements.
Table NMSE's quantile (the lower the better) for different number of measurements. Top three rows concern non- -on-Bayesian models whilst the bottom two report results for Bayesian approaches. For each of the two set of results, we highlight in bold the best performance.
Q=0.4 4
number of measurements
10
100
300
500
ISTA
+0.38
+0.05
-0.13
-1.90
-3.28
ISTA (1000 iterations)
+0.01
-0.04
-1.62
-8.30
-15.76
LISTA
-0.01
-1.60
-1.28
-2.82
-4.37
DLISTA
-3.12
-5.22
-7.18
-10.62
-17.92
A-DLISTA
-5.45
-14.82
-20.91
-19.47
-22.35
BCS
+0.16
+0.17
+1.91
+2.34
+2.55
VLISTA
-3.67
-8.20
-13.94
-15.31
-14.02
By looking at Table we can draw similar conclusion as for the MNIST and CIFARIO datasets.
4.3 Out Of Distribution detection
In this section, we focus on one of the most important differences among non- on-Bayesiaa models for solving inverse linear problems and VLISTA. Indeed, differently from any non-Bayesian approach to compressed sensing, VLISTA allows for quantifying uncertainties on the reconstructed signals which, in turn, enables OOD detection without the need access ground truth data at inference time. Moreover, whilst other Bayesian approaches [Ti et al. (2008): Zhou et al. (2014) usually focus on designing specific priors to satisfy the sparsity constraint on the reconstructed signal after marginalization, VLISTA completely overcomes such an issue as the thresholding operations is not affected by the the marginalization over dictionaries. To prove that VLISTA can detect OOD samples, we employ the MNIST dataset. First, we split the full dataset into two subsets named "Train". or In- D--Ditributtion (ID), and OOD. The subset contains images from three digits only, namely, 0, and 7 (randomly chosen). Instead, the OOD subset contains images from all the other digits. Then, we split the partition into training and test sets and train VLISTA on former one. Once trained, we evaluate the model performance by considering reconstructions from the test set (ID) and the OOD partition. We reconstruct 100 times every single image, sampling every time a new dictionary. Subsequently, summarizing statistics compute the variance's 's c.d.f. of the per-pixel standard

---

deviation (varo,,, across reconstructions. Subsequently, to assess whether a given digit belongs to the ID or OOD distribution, we compute the p-value for var opp by employing the two-sample t-test. Moreover, to assess whether the OOD detec- tion robust to measurement noise, we repeat the same test for different levels of noise. As baseline for the current task, we consider BCS, Due to the different nature of the BCS frame- work, we employ a slightly different procedure to evaluate the p- p-values for it. Specifically, use the same ID and OOD splits as for VLISTA. However, for BCS, consider the c.d.f. of the reconstruction error that is evaluated by the model itself. The of the procedure is the same as for VLISTA. We report the results for from
Figure 2: p-value for OOD rejection function of the noise level. The green line represents a reference p-value equal to 0.05.
109
10
10-1
p-value
OOD detection in Figure As the figure, VLISTA outperforms BCS for each level of noise showing lower p-value than BCS which corresponds to a higher rejection power. As expected, by increasing the level of noise we observe a larger p-value meaning that the OOD rejection becomes harder for more noisy data. However, we can see that whilst VLJSTA is still capable of detecting OOD samples, BCS fails in doing so when the Signal-to-Noise Ratio (SNR), expressed in decibels, is greater than 10. As reference point to define whether the model is correctly
10-1
10-8
BCS
VLISTA
10
p-value
0.05
16-1
10
10a
SNR (dB)
rejecting OOD samples or not, we report in Figuree 2 the 5% line for the p-value. Such value is typically used as a reference in hypothesis testing to decide whether not reject the null hypothesis.
Conclusion
We report about variational approach, dubbed VLISTA, to solve the dictionary learning and the sparse recovery problems jointly. Typically, compressed sensing frameworks assume the existence of ground truth dictionary used reconstruct the signal. Furthermore, in state-of-the-art LISTA-like models, a stationary measurement setup is usually considered. In our work, we relax both assumptions. First, show that it is possible to design soft-thresholding algorithm, termed A-DLISTA, that can handle different sensing matrices and that can adapt its parameters to the given data instance. We theoretically justifly the use of an augmentation network which adapts the threshold and step size for each layer based on the current input and the learned dictionary. Finally, we also relax the hypothesis concerning the existence of ground truth dictionary by introducing probability distribution for it. Given such an assumption, we formulate the VLISTA variational framework to solve the compressed sensing task. We report results for both our models, A-DLISTA and VLISTA, concerning non- Bon-Baasi and Bayesian approaches to solve the sparse recovery and dictionary learning problems jointly. We empirically show that the adaptation capability of A-DLISTA results in boost in performance compared to ISTA and LISTA models, in a non-static measurements scenario. Although we observe that in terms of reconstruction, VLISTA does not outperform A-DLISTA, the variational framework enables to evaluate uncertainties over the reconstructed signals useful to detect OOD. On the other hand, none of the LISTA-like models allows for such task. Moreover, differently from other Bayesian approaches to compressed sensing, VLISTA does not need to design specific priors to retain sparsity after marginalization of the reconstructed sparse signal; the averaging operation concerns the sparsifying dictionary instead of the sparse signal itself.

---

References
Aviad Aberdam, Alona Golts, and Michael Elad. Ada-LISTA: Learned Solvers Adaptive to Varying Models. IEEE Transactions on Pattern Analysis and Machine Intelligence, pp. 1-1, 2021.
Arash Behboodi, Holger Rauhut, and Ekkehard Schnoor. Compressive Sensing and Neural Networks from Statistical Learning Perspective. In Gitta Kutyniok, Holger Rauhut, and Robert Kunsch (eds.). Compressed Sensing in Information Processing, Applied and Numerical Harmonic Analysis, pp. 247-277. Springer International Publishing, Cham, 2022.
Freya Behrens, Jonathan Sauder, and Peter Jung. Neurally Augmented ALISTA. In International Conference on Learning Representations, 2021.
Thomas Blumensath and Mike E. Davies. Iterative hard thresholding for compressed sensing. Applied and Computational Harmonic Analysis, 27(3):265- 274, November 2009.
Charles Blundell, Julien Comebise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty in
neural network. In International conference on machine learning, pp. 1613-1622. PMLR, 2015.
Mark Borgerding, Philip Schniter, and Sundeep Rangan. AMP-inspired deep networks for sparse linear inverse problems. IEEE Transactions on Signal Processing, 65(16):4293-4308, 2017. Publisher: IEEE.
Xiaohan Chen, Jialin Liu, Zhangyang Wang, and Wotao Yin. Theoretical linear convergence of unfolded ISTA and its practical weights and thresholds. Advances Neural Information Processing Systems, 31, 2018.
Xiaohan Chen, Jialin Liu, Zhangyang Wang, and Wotao Yin. Hyperparameter tuning is all you need for lista. Advances in Neural Information Processing Systems, 34, 2021.
Junyoung Chung, Kyle Kastner, Laurent Dinh, Kratarth Goel, Aaron Courville, and Yoshua Bengio. A recurrent latent variable model for sequential data. Advances in neural information processing systems, 28, 2015.
Junyoung Chung, Sungjin Ahn, and Yoshua Bengio. Hierarchical multiscale recurrent neural networks arXiv preprint arviv: 1609.0/91000 00900//74, 2016.
Ingrid Daubechies, Michel Defrise, and Christine De Mol. An iterative thresholding algorithm for linear inverse problems with a sparsity constraint. Communications on Pure and Applied Mathematics: A Journal Issued by the Courant Institute of Mathematical Sciences, 57(11):1413 1457, 2004.
Geoffrey M. Davis, Stephane G. Mallat, and Zhifeng Zhang. Adaptive time-frequency decompositions. Optical Engineering, 33(7):2183-2191, July 1994.
David L. Donoho, Arian Maleki, and Andrea Montanari. Message-passing algorithms for compressed sensing. Proceedings of the National Academy of Sciences, 1894499,,, November 2009.
Meire Fortunato, Charles Blundell, and Oriol Vinyals. Bayesian Recurrent Neural Networks. arXiv: [cs, stat), May 2019. URL URL/http::// htttp:///axiit http://aaxxvv..ggaaaa///asa 1704. 02798 1704.02798.
arXiv:
Simon Foucart and Holger Rauhut. A Mathematical Introduction to Compressive Sensing. Applied and Numerical Harmonic Analysis. Springer New York, New York, NY. 2013.
Raja Giryes, Yonina C. Eldar, Alex M. Bronstein, and Guillermo Sapiro. Tradeoffss between con- vergence speed and reconstruction accuracy in inverse problems. IEEE Transactions on Signal Processing, 1667))1166-1699 2018. Publisher: IEEF.
Karol Gregor and Yann LeCun. Learning fast approximations of sparse coding. In 27th International Conference Machine Learning, ICML 2010, 2010.
10

---

Shihao Ji, Ya Xue, and Lawrence Carin. Bayesian Compressive Sensing, JEEE Transactions on Signal Processing. 56(6):2346-23555 June 2008. Conference Name: IEEE Transactions on Signal Processing.
Diederik P Kingma and Max Welling. arXiv:1312.61 14, 2013.
Auto-encoding variational bayes. arXiv preprint
Rahul Krishnan, Uri Shalit, and David Sontag. Structured inference networks for nonlinear state space models. Proceedings of the AAAI Conference Artificial Intelligence, 31(1,, Feb. 2017.
Jialin Liu and Xiaohan Chen. Alista: Analytic weights are as good as learned weights in lista. In International Conference Learning Representations (ICIR), 2019.
Jialin Liu, Xiaohan Chen, Zhangyang Wang, and Wotao Yin. ALISTA: Analytic Weights Are As Good As Learned Weights in LISTA. In International Conference on Learning Representations, 2019.
Chris Metzler, Ali Mousavi, and Richard Baraniuk. Learned D-AMP: Principled Neural Network based Compressive Image Recovery. In I. Guyon, U. V. Luxburg, S. Bengio. H. Wallach, R. Fergus, S. Vishwanathan, and R. Gareett Advances in Neural Information Processing Systems 30,
pp. 1770-1781. Curran Associates, Inc., 2017.
Y.C. Pati, R. Rezaïffar and P.S. Krishnaprasad. Orthogonal matching pursuit: recursive function approximation with upplications to wavelet decomposition. In Proceedings of 27th Asilomar Conference on Signals, Systems and Computers, pp. 40 44 vol.1, November 1993. doi: 10.1109/ ACSSC.1993.3424655
Wei Pu, Yonina C. Eldar, and Miguel R. D. Rodrigues. Optimization Guarantees for ISTA and ADMM Based Unfolded Networks. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 8687-8691, May 2022.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In International conference on machine learning, pp. 1278-1286. PMLR, 2014.
Ekkehard Schnoor, Arash Behboodi, and Holger Rauhut. Generalization Error Bounds for Iterative Recovery Algorithms Unfolded as Neural Networks. arXiv:2112.04364, January 2022.
Sprechmann, A. M. Bronstein, and G. Sapiro. Learning Efficient Sparse and Low Rank Models. IEEE Transactions on Pattern Analysis and Machine Intelligence, 18799)182111888 September 2015. ISSN 0162-8828.
Michael E. Tipping. Sparse Bayesian learning and the relevance vector machine. Journal of machine learning research, (JJun):211-24: 2001.
Kailun Wu, Yiwen Guo, Ziang Li, and Changshui Zhang. Sparse coding with gated learned ista. In International Conference on Learning Representations, 2020.
yan yang, Jian Sun, Huibin Li, and Zongben Xu. Deep ADMM-Net for Compressive Sensing MRI. In D. Lee, M. Sugiyama, U. Luxburg. I. Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 29, Curran Associates, Inc., 2016.
Jian Zhang and Bernard Ghanem. ISTA-Net: Interpretable
cetiiiizaatinnnnsiiieee deep network for image compressive sensing. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1828 1037-8877,27 2018.
Mingyuan Zhou, Haojun Chen, Lu Ren, Guillermo Sapiro, Lawrence Carin, and John Paisley. Non- Parametric Bayesian Dictionary Learning for Sparse Image Representations, In Advances in Neural Information Processing Systems, volume 22. Curran Associates, Inc., 2009.
Mingyuan Zhou, Haojun Chen, John Paisley, Lu Ren, Lingbo Li, Zhengming Xing, David Dunson, Guillermo Sapiro, and Lawrence Carin. Nonparametrie Bayesian Dictionary Learning for Analysis of Noisy and Incomplete Images. IEEE Transactions on Image Processing, 21(1):130-144 January 2012.
Zhou Zhou, Kaihui Liu, and Jun Fang. Bayesian compressive sensing using normal product priors. IEEE Signal Processing Letters, 5285)5883-587 2014.

---

A Architecture Details
In this section we report the details of the architecture for our proposed models.
Concerning A-DLISTA, as we show in Figure 1 the reconstruction network, i.e., blue blocks, is an unfolded ISTA-like model with parametrized dictionary V. Each layer is characterized by own dictionary which used to both reconstruct the sparse vector and as an input for the augmentation network. As mentioned in (suusecction 3.3] the augmentation model, red block in Figure takes as input the measurement matrix, 4., and the dictionary at given reconstruction layer I, Vr, and generates the adaptive parameters (7e, 0t) for the 1-th layer. We show the architecture for the augmentation network in Figure 3
AdeplirekuPPoa22 (oul size-200)
Conv2D kr3,
LeakyReLUU
Conv2D (i_c=1, c=1, k=3, s=2)
LeakyRolug)
Liner
toatt 121, feat-25)
LeakyRelUU
Linser (n_.eet=25,, featt11
Lineer
(in, feel-25, out, feat-1)
Softllas)
Softlus)
0
Figure 3: Augmentation network for A-DLISTA. Each red block in Figure corresponds to the shown model.
Concerning VLISTA, as introduced in subsection 3.4, it comprises three different models: prior, posterior, and likelihood. Concerning the likelihood model, it is assumed represent gaussian distribution whose mean is parametrized by means of the A-DLISTA model (subsubsection 3.4.3). Instead, the prior (subsubsection 3.4.I) and posterior (subsubsection 3.4.2) models are implemented using an encoder-decoder scheme based on convolutional layoo... We report in Figure the architecture for the prior and posterior mdoels.
Finally, we report in Figure the graphical model for the posterior and conditional prior in the left and right plot, respectively.
B Implementation and Training Details
We report in this section few details about the implementation and training the A-DLISTA and VLISTA models. We implemented both using the Lightning framework. As we mentioned in the main body of the manuscript, ISTA and LISTA require known dictionary order reconstruct the non non-sparse signal. Concerning the three datasets that we consider, we define the dictionaries in the following way: canonical for MNIST (since MNIST is already sparse) with 784 atoms and wavelet for CIFARIO with 1024 atoms. Regarding the synthetic dataset, we randomly generated the dictionary from a standard distribution and we consider 765 atoms. Concerning A-DLISTA, we trained both the reconstruction and the augmentation network, blue and red blocks in Figuure respectively, end-to-end using the Adam optimizer. We set the initial learning rate 1.e. - and T.e T.e-3 for the reconstruction and augmentation network respectively, and we dropped its value by factor 10
12

---

Canid
Corvip
1.00..1
pwdd rass odl 1013 hw194)
Canon
Frort oA led, and)
Taxly
(8. 100, 11
ord4 our ondo andy, entty
onli, endi, sndl 677)
Uiwar
Thano
Gael 23
and)
Dawris Turlei
Teano
Turki
(n ort, dut and only
(n out en knf, andt
(In onl, dut eni and and)
(n. out en pnriinn,, ien, endy
Dani 120 (hucr), and an2)
Ganyaa
Tur
Turld)
Figure 4: Left: prior network architecture. Right: posterior network architecture. For the posterior model we show in the figure the output shape from each of the three head. Such a structure is necessary since the posterior model accepts as input three quantities, namely, the observations, the sensing matrix, and the reconstruction from the previous layer which are characterized by different shapes. The term "B" indicates the batch size.
Figure 5: Graphical model of the Variational LISTA model dependencies on y 4b factored for simplicity. The sampling is done only based on the posterior g0 4 (i,-1, y. D²). The dashed line shows variational approximations
a ( (4 ,()
Φ
Ps (4p1))
g
W
4
4
4t
X
01 Yt-1
every time the loss stopped to improve for more than 30 training steps. Moreover, we set the weight decay to 5.e and the batch size to 128. We applied the same scheme across all the datacts we used. As the objective function, used the MSE between the ground truth signal and the reconstructed one for image datasets and the NMSE for the synthetic dataset, respectively.
Concerning the VLISTA training, similarly to what we did for A-DLISTA, we train the full model end-to-end. However, compared to A-DLISTA, in this case the hyperparameter space has much higher dimension. Therefore, we employed the Tune library for hyperparameter search. Specifically, we used HyperOht and ASHAScheduler as searcher and scheduler, respectively. We set the learning rates to 7.e -3, 5. and 1.e-1 1 for the likelihood, posterior, and prior models, respectively. Also in this case, we use scheduler for reducing the learning rate similarly to what we did for A- A-DLISTA. Concerning the objective function, maximize the ELBO typically done for such
13

---

type of models. Moreover, we set the weight for the KL divergence to 1.e.3, We report in Equation 11 details about the obejctive.
log (((11r = " Φ01) = log / p(a11 = miee v 1.T, y ppl V F d 1:T
(11)
p(Z11: Tig. V.r. y¹, + 4 1:
1: y Tb 01rr)
log
d 1:T
1 1:7 y T #1.T)
p = gt v 1T vi s )p( 11T)
q( (WWirll 16, .1.T) log
di L:T
a ((VI. 1:T y T1:T)
a( ( (Pilly 4. .I.T) log p
#ge M 1:T, y & 1")d
p v 1.T))
+
q (1rl'y' Φ xIT) log q 1.7 y ob "2,21r))
1:T
T
Σ
Ev Sv ao. 11) log p(x, = x 40 11 V. x0)
t=1
T
100( ilvi 1) [KL (q Φ 2', pl U p
Evi.
DEL
(g(
(
|| p(V)))
Finally, to give an overall overview of the training scheme for VLISTA, Figure shows a diagram of the full training pipeline.
Posterior network
Prior network
N
Sample
N 19
LISTA network
Entropy loss
LISTA 0, Tc)
y. db
(leamable
Da (R0(V) Po(V)))
parameters)
- || x
Reconstruction loss
Figure 6: Variational LISTA oveall architecture and training pipeline. Note that to simplify the figure, did not report the index for the data sample (i) but only for the iteration (f).
14

---

C Derivation for Theorem 3.1
Convergence proofs of ISTA type models involve steps in general. First, it is investigated how the support is found and locked in, and second how the error shrinks at each step. We focus on these two steps, which matter mainly for our architecture design. Our analysis similar in nature to Chen et al. (2018); Aberdam et al. (2021), however it differs from Aberdam et al. (2021) in considering unknown dictionaries and from Chen al. (2018) in both considered architecture and varying sensing matrix. In what follows. we consider noiseless setting. However, the results can be extended to noisy setups by adding additional terms containing noise norm similar to Chen et al. (2018). We make following assumptions:
1. There is a ground-trute truth (unknown) dictionary L. such that I., L 2...
2. As a consequence, y = T 2
3. We assume that 2, sparse with its support contained in S. In other words: 2,,+ 0 for i E S
As first step, fix the sensing matrix o and conduct the analysis. First define the following:
A max (( (++ (b)))))
(12)
ISI$IN
jo = max
(( :): ( {)))s
(13)
ISiJEN
8(7) max - [| (
(14)
The main step of soft-thresholding algorithm is given as follows:
2, = 170 (z--1 + 76 (
TT (y $v)2(-1)),
(15)
with entry-wise relation given by
žt,i = 1o, (#1-1,4 + 7x T TV (2,1))))))
(16)
Using the assumptions we have:
(Dv)) ( (D) ( cz. 44
(17)
C.1 Locking the support
First, we show under what conditions, the algorithm locks on the support. Suppose that the support of x is already the same as z., namely (upp(z:-1) = supp(z.) = Consider i E S We have
žt,i no, (+.((*v,..)) (y -1,,,,111)
(18)
To lock the support, we need to guarantee that:
ln(((v))))) (y - 11)==11) s 0.
(19)
We have:
(())))) (p...=
(T - 1( = (( x)) 2 - (1-1111
+ ((DV,)) 1 V... v.z..)
(20)
Σ )3) (4V)) (, - Zz 2z-1,))
(21)
JES
+ (( ()) 1 (VV. - 122))
(22)
We bound the norm by:
Σ (( (DV ((V.)) (
21-15)
(( (( [()))))((en->>))) 21117)
(23)
EES
jES
S fi ||,, 2
(24)
15

---

where we use the definition of mutual coherence for the upper bound.
The other norm is bounded by
())) (V.. - -1(22)]
(
-
- -V,))saj
(25)
jES
(()
( v,))s] 13,,1
(26)
jES
2z l
(27)
Therefore, obtain the following sufficient condition for locking the support:
Vt (ī || - 1211 + #2 ||=,|11) S 0.
(28)
If this condition is satisfied, get lock the support.
C.2 Controlling the errors
For E S, we have:
2 z.,il S 11-1, +
0v,,=-11 - za,1| + 0g.
(29)
We start again with equation 16 but with S:
2t- + ( (y 21)(=(-1)=
2- + 7ol Σ ((( (
( 21-1,) + T 122)))
jES
For the first part, we get:
21-14 + 7/ Σ (Qv)))
(DV.)) ( - zz-1,))
JES
(I 1
(v)))))) (dW.)))), + Σ ((V))) ((W:)))))WW)) - z1-1))
JES,j4
Therefore:
|2--1 + (eV(( .1),,1-1) - 20,1 S (b(
( 2.,1
+ Σ ((b)))) (DD,)))E - -z) + 7r (()b)))) p(V.z. v,2.))
JESJ
S 5(1)) ((--)))()11--))) 2, + 7t
Σ
7122 || مlzns 21
$ES,j4i
Therefore, have:
||5S, z=111 = Σ |1, 22,1 S
iES
< (6(7)) + 7t - 1)) || - 2=11, + S || + |S|0..
Note that this analysis shows that strive for having smaller Ø2, namely being close to the dictionary, small enough 9 as iterations grow, and small ğ and 5(y)).
In general, if we have
Yt (f 1|2. 21-1/12 fiz 2z 111) S O..
(30)
Then we get the final result:
2.111
21,s 5..155 (0(7)) + 1x 1)) ||=-11 2x11, + 71 ||z + +S|00.
16


Reviews:
[{"review_id": "nHWEu8KKSZq", "reviewer": "Reviewer_uswp", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a version of LISTA where the update parameters can be made specific to each data sample even when the sensing matrix is not the same among samples. The paper then also proposes a variational method to dictionary learning to be integrated with the former approach.\n", "summary_of_the_review": "I have several questions regarding clarity on the details of the algorithm and the significance of the experimental results that prevent me from recommending acceptance. Given the wealth of approaches based on data-driven models and compressed sensing, the numerical results need to be more compelling.\n\nPost-rebuttal: The authors have addressed the majority of my questions satisfactorily ", "strength_and_weaknesses": "The paper considers a novel setting that is more flexible than the common assumption in the literature. \n\nI am personally not convinced regarding the impact of an approach that tolerates variable sensing matrices. When compressed sensing architectures exhibit different sensing operators for different data, it is usually because there is little control over the behavior of the operator, so it would be difficult to accurately obtain the operators involved. Perhaps the authors can elaborate in Section 3.2 regarding the problems where the matrix $\\Phi$ can change for each sample but is known in each case. \n\nThe approach that includes variational dictionary learning, described in Sections 3.4.2-3 apparently assumes that the sensing matrix $\\Phi$ is known, which counters the setup where LISTA is applied (in addition to the point above) and the matrices involved in LISTA are \"learned\". It is not clear how one would infer both $\\Phi$ and the dictionary $\\Psi$ simultaneously. The authors should also clarify whether the sensing matrix needs to be known as it varies in each case.\n\nThe experimental results are limited - for such a flexible prior more challenging datasets should be attempted. Other papers that use data-driven priors in compressed sensing, such as GANs, do so. Furthermore, the only baseline used that employs a data-driven model is LISTA, a method that has driven significant research into improved variations; thus, the comparison against older algorithms BCS (2008) and ISTA (2010) is not informative. Finally, if the authors take into account the computation time for ALISTA and NALISTA in evaluating the matrix $W$ for each sample to discount them from the comparison, they should compare this against the training time for the deep learning networks used in this and competing papers. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "lMoBeR0fglW", "reviewer": "Reviewer_3LdD", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors present two unrolled ISTA-type algorithms for compressed sensing. The first approach is amenable to adaptive measurements matrix while learning the sparsifying dictionary per layer. The second approach called as VLISTA places a probability distribution the dictionaries. This provides a probabilistic way to learn dictionaries which could be useful in out of distribution (OOD) detection problems. ", "summary_of_the_review": "The authors present two approaches for learning sparse representations and dictionaries under the unrolled optimization framework.\nThe ideas are simple but interesting. The authors provide a sufficient condition for the algorithm to converge to the correct support and provide an error bound. Experimental results demonstrate the efficiency of the proposed algorithms on simulated data and on MNIST dataset. The paper could be significantly improved if the authors could better motivate the selection of the Gaussian distributions for the priors and posteriors and provide more insight into how restrictive these assumptions are in practical settings. The experimental section could also be improved by providing experimental results on datasets such as CIFAR dataset and more extensive results of the probabilistic method (VLISTA). \n\n-----------------------------\nPost-rebuttal update:\nI wanted to thank the authors for their time and effort in addressing the reviewers' comments. I am pretty satisfied with the responses, and I  appreciate the changes they made to the paper. I  thus raise my score to 6.", "strength_and_weaknesses": "Strengths\n\n- The paper is well-written and easy to follow. The main ideas are well explained.\n- Allowing to learn the dictionary in a measurement adaptive environment is quite interesting. Also, the probabilistic formulation of the problem, which leads to VLISTA, uses ideas of Bayesian deep learning in the unrolled optimization framework, and thus it has its own interest.\n\nWeaknesses\n- I understand that assigning Gaussian distributions in the prior and posterior simplifies things a lot regarding the implementation of the algorithm. However, isn't quite restrictive in practice? I wonder if other distributions can be modeled in that framework. E.g., using hierarchical models that encode heavy-tailed distributions.\n- In the experiments,  Section 4.1, it seems that A-DLISTA outperforms ISTA and LISTA. However, it is not clear what is the measurement matrix that is given as input to these algorithms. Does LISTA know that the measurement matrix changes over time?\n- It is not well explained why VLISTA works so much worse than A-DLISTA.\n- In the OOD experiment, there is no comparison with any other competing method. Why BSC algorithm is not included in the experiments?\n- In Theorem 1, since $\\theta_t,\\gamma_t$ are learned by the algorithm. Hence for the sufficient condition given in (7)  to be satisfied, it is important $\\theta_t$, and $\\gamma_t$ converge as t ( here denotes the number of layers) grows. Can the authors provide some insight regarding the convergence of these parameters? Moreover, it would be interesting for the authors to show experimentally that this condition is satisfied at a certain number of layers T.\n\nMinor\n- There seems to be an issue with statement 1 in Theorem 1. The conclusion should be $\\mathrm{supp}(z_{t}) \\subseteq \\mathrm{supp}(z^\\ast)$.  It seems there is a typo there.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "ji2l4rCevPV", "reviewer": "Reviewer_2dyB", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper considers the problem of sparse vector recovery with unknown dictionary and time varying sensing matrix, and proposes a data-driven approach using variational learning. The signal recovery performance of the proposed method as well as the possibility of out of distribution detection are demonstrated via numerical experiments.", "summary_of_the_review": "The paper considers a timely and important problem, and work in the paper is solid and interesting, while the performance gain shown in the numerical results might be rather marginal.", "strength_and_weaknesses": "The paper is generally interesting, and enough contribution can be recognized in the proposed approach. However, the signal recovery performance of the proposed method is worse than that of existing method (i.e., A-DLISTA) in the experimental results. Also, the organization of Sect.3 might be somewhat misleading, because it contains both the existing methods and the proposed method while it is titled \"variational learning ISTA\", which is the name of the proposed method.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject