Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors present two unrolled ISTA-type algorithms for compressed sensing. The first approach is amenable to adaptive measurements matrix while learning the sparsifying dictionary per layer. The second approach called as VLISTA places a probability distribution the dictionaries. This provides a probabilistic way to learn dictionaries which could be useful in out of distribution (OOD) detection problems. 
strength_and_weaknesses: Strengths

- The paper is well-written and easy to follow. The main ideas are well explained.
- Allowing to learn the dictionary in a measurement adaptive environment is quite interesting. Also, the probabilistic formulation of the problem, which leads to VLISTA, uses ideas of Bayesian deep learning in the unrolled optimization framework, and thus it has its own interest.

Weaknesses
- I understand that assigning Gaussian distributions in the prior and posterior simplifies things a lot regarding the implementation of the algorithm. However, isn't quite restrictive in practice? I wonder if other distributions can be modeled in that framework. E.g., using hierarchical models that encode heavy-tailed distributions.
- In the experiments,  Section 4.1, it seems that A-DLISTA outperforms ISTA and LISTA. However, it is not clear what is the measurement matrix that is given as input to these algorithms. Does LISTA know that the measurement matrix changes over time?
- It is not well explained why VLISTA works so much worse than A-DLISTA.
- In the OOD experiment, there is no comparison with any other competing method. Why BSC algorithm is not included in the experiments?
- In Theorem 1, since $\theta_t,\gamma_t$ are learned by the algorithm. Hence for the sufficient condition given in (7)  to be satisfied, it is important $\theta_t$, and $\gamma_t$ converge as t ( here denotes the number of layers) grows. Can the authors provide some insight regarding the convergence of these parameters? Moreover, it would be interesting for the authors to show experimentally that this condition is satisfied at a certain number of layers T.

Minor
- There seems to be an issue with statement 1 in Theorem 1. The conclusion should be $\mathrm{supp}(z_{t}) \subseteq \mathrm{supp}(z^\ast)$.  It seems there is a typo there.

clarity,_quality,_novelty_and_reproducibility: The main ideas are clearly presented in the paper. Learning dictionaries in an adaptive measurements environment seems to be novel (though I might miss works that might have recently been published on that topic).
summary_of_the_review: The authors present two approaches for learning sparse representations and dictionaries under the unrolled optimization framework.
The ideas are simple but interesting. The authors provide a sufficient condition for the algorithm to converge to the correct support and provide an error bound. Experimental results demonstrate the efficiency of the proposed algorithms on simulated data and on MNIST dataset. The paper could be significantly improved if the authors could better motivate the selection of the Gaussian distributions for the priors and posteriors and provide more insight into how restrictive these assumptions are in practical settings. The experimental section could also be improved by providing experimental results on datasets such as CIFAR dataset and more extensive results of the probabilistic method (VLISTA). 

-----------------------------
Post-rebuttal update:
I wanted to thank the authors for their time and effort in addressing the reviewers' comments. I am pretty satisfied with the responses, and I  appreciate the changes they made to the paper. I  thus raise my score to 6.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper considers the problem of sparse vector recovery with unknown dictionary and time varying sensing matrix, and proposes a data-driven approach using variational learning. The signal recovery performance of the proposed method as well as the possibility of out of distribution detection are demonstrated via numerical experiments.
strength_and_weaknesses: The paper is generally interesting, and enough contribution can be recognized in the proposed approach. However, the signal recovery performance of the proposed method is worse than that of existing method (i.e., A-DLISTA) in the experimental results. Also, the organization of Sect.3 might be somewhat misleading, because it contains both the existing methods and the proposed method while it is titled "variational learning ISTA", which is the name of the proposed method.
clarity,_quality,_novelty_and_reproducibility: I don't have any difficulty in understanding the proposed method.
Also, a certain novelty can be found in the methodology itself.
summary_of_the_review: The paper considers a timely and important problem, and work in the paper is solid and interesting, while the performance gain shown in the numerical results might be rather marginal.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a version of LISTA where the update parameters can be made specific to each data sample even when the sensing matrix is not the same among samples. The paper then also proposes a variational method to dictionary learning to be integrated with the former approach.

strength_and_weaknesses: The paper considers a novel setting that is more flexible than the common assumption in the literature. 

I am personally not convinced regarding the impact of an approach that tolerates variable sensing matrices. When compressed sensing architectures exhibit different sensing operators for different data, it is usually because there is little control over the behavior of the operator, so it would be difficult to accurately obtain the operators involved. Perhaps the authors can elaborate in Section 3.2 regarding the problems where the matrix $\Phi$ can change for each sample but is known in each case. 

The approach that includes variational dictionary learning, described in Sections 3.4.2-3 apparently assumes that the sensing matrix $\Phi$ is known, which counters the setup where LISTA is applied (in addition to the point above) and the matrices involved in LISTA are "learned". It is not clear how one would infer both $\Phi$ and the dictionary $\Psi$ simultaneously. The authors should also clarify whether the sensing matrix needs to be known as it varies in each case.

The experimental results are limited - for such a flexible prior more challenging datasets should be attempted. Other papers that use data-driven priors in compressed sensing, such as GANs, do so. Furthermore, the only baseline used that employs a data-driven model is LISTA, a method that has driven significant research into improved variations; thus, the comparison against older algorithms BCS (2008) and ISTA (2010) is not informative. Finally, if the authors take into account the computation time for ALISTA and NALISTA in evaluating the matrix $W$ for each sample to discount them from the comparison, they should compare this against the training time for the deep learning networks used in this and competing papers. 

clarity,_quality,_novelty_and_reproducibility: Some specific points in the paper are not clear:

I do not see how LISTA models assume that the learned dictionary is known a priori - my understanding is that this is part of what LISTA learns. It seems at times that the authors are combining the concept of "fixed dictionary" with "known dictionary", e.g., the last sentence of Section 3.2. Perhaps the authors mean to say that the sparse representation of the signal obtained by LISTA is not useful on its own without knowing the dictionary that can translate it into the actual signal?

In page 3, a matrix A is described such that $A^TA = \Phi\Psi$, but I do not believe that $\Phi\Psi$ would be a square matrix when compressed sensing is applied.

The performance characterization metrics in (4-6) appear to depend on the iteration count $t$, but the dependence is not referenced in the notation.

In eq. (5), it appears $\Psi$ should be $\Psi_t$ or $\Psi_o$ in the first part of the right hand side of the equation, given that no single dictionary is assumed?

In Theorem 3.1, the assumption includes "$\mathrm{supp}(z^*)=\mathrm{supp}(z_{t-1})$" and the conclusion says "$\mathrm{supp}(z_{t-1}) \subseteq \mathrm{supp}(z^*)$", which appears obvious to me. Is there a typo?

In Fig. 1, the dictionary $\Psi^t$ appears as an output of each blue block, and it is not clear how that is the case. There is also little detail on the augmentation network (red blocks) that produces the parameters. Note also the discrepancy in notation $\Psi_t$ vs. $\Psi^t$.

Some of the notation in Section 3.4 describes distribution for matrices and vectors in terms of scalar PDFs - if this is an abuse of notation, its meaning should be stated explicitly.

The training of the augmentation networks that need to take into account the sparsity matrices and dictionaries seems to be a significant undertaking. It is not clear how these networks are trained for the experiments.
summary_of_the_review: I have several questions regarding clarity on the details of the algorithm and the significance of the experimental results that prevent me from recommending acceptance. Given the wealth of approaches based on data-driven models and compressed sensing, the numerical results need to be more compelling.

Post-rebuttal: The authors have addressed the majority of my questions satisfactorily
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposes a variational-learning-based ISTA for solving compressed sensing problems with an unknown dictionary. As an intermediate, it also proposes A-DLISTA which, unlike DLISTA (3) that learns $\theta_t,\gamma_t,\Psi_t$ in an end-to-end manner, uses an augmentation network that outputs $\theta_t,\gamma_t$ from $\Phi,\Psi_t$ and is trained in an end-to-end manner, in order to allow $\theta_t,\gamma_t$ to be determined adaptively rather than being fixed as a result of the end-to-end training. The main proposal, VLISTA, is built on A-DLISTA by further adding to it a prior model for the dictionaries $\Psi_{1:T}$ and a set of variational distributions to be optimized by maximizing the ELBO (10). The setting discussed in this paper where the sensing matrix $\Phi$ is known but varies from sample to sample and where the sparsifying dictionary $\Psi_o$ is unknown and to be estimated would appear in applications such as MRI reconstruction and MIMO channel estimation. The proposed methods were shown to perform well in such a scenario.

Of the three reviewers, one initially rated well below the acceptance threshold, and it appears that it is mainly due to rather unclear exposition of the original manuscript and limited experimental comparison with baselines, which constituted the main weaknesses of the initial version. Upon reading the revised version of this paper by myself, I think that clarity of the presentation is not enough even after the revision. For example, regarding the ELBO objective (10), the authors used an undefined symbol $x_{gt}$ (I guess that it is for "$x$ ground truth" or something like that), and considered the likelihood given $\hat x_t=x_{gt}$ for all $t\in[1:T]$ in the first term of the ELBO, which would need further justification as it should be inconsistent with their use of not $x_{gt}$ but $\hat x_{t-1}$ in the second term of the ELBO. I would like to mention that on the right-hand side of (9) $\hat x_{1:T}$ should appear whereas it does not.

I would also like to add that a more detailed explanation on why different dictionaries $\Psi_{1:T}$ and their distributions are assumed in the proposal of VLISTA, from the viewpoint of Bayesian inference. Although from the deep-unfolding viewpoint it would be seen natural to consider different dictionaries for different layers, one can argue that it would conceptually be more natural to assume a single dictionary $\Psi$ as well as a single prior for it, independent of $t$.
justification_for_why_not_higher_score: Although the exposition in this paper has been improved via the revision and the added experimental results are fine in demonstrating efficiency of the proposed methods, I feel that there are still several points that require further clarification, as detailed above.
Decision: Reject