Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper considers learning priors in the case when one only has access to corrupted measurements of signals, rather than clean signals themselves. The authors first provide an analysis of learning a sparsifying dictionary when only given compressed measurements. It is shown that if only one measurement operator is used, then information in the kernel of the measurement operator cannot be recovered. The situation changes, however, when multiple measurement matrices are used which are “diverse” enough for information to be learned. The authors then experimentally demonstrate that recovery with only corrupted data is possible when convolutional structure is used as a prior in certain problems, namely inpainting. This is shown to not be the case, however, in deconvolution.
strength_and_weaknesses: 
**Strengths:**
- The paper studies an important and challenging problem, which is to identify when and why its possible to learn priors directly from corrupted data.
- The experimental make some interesting comments on when priors can be learned, which can happen when information is introduced into the prior model that can compensate for the type of degradation in the forward model.

**Weaknesses:** 
- There is a lack of novelty in the theory presented in this paper. The results presented here are very similar in spirit to previous work. Please see the Novelty and significance section below.

clarity,_quality,_novelty_and_reproducibility: 
**Clarity:** Overall, the paper was easy to read. There were some notational issues and questions regarding derivations that I was looking to clarify.

- I am confused by the notation at the bottom of page 3. Should the objective functional be denoted by $F(Z_A(D), D)$ to be consistent with the gradient derivations on the following page? 
- Also, is equation (3) the true “gradient” of the objective? We should have sub gradients due to the ell1 norm penalty on $Z_{A_i}(D)$. Moreover, this quantity itself depends on $D$, so I think the full sub gradient should be more involved.
- In equation (9), should each term in the loss be $||Af_{\theta}(Y_i’) - Y_i||_2^2$? Otherwise, I am not sure I understand why one should apply the degradation operator $A$ to the already degraded measurements $Y_i = AX_i + \epsilon_i$.

**Novelty and significance:** 

I think that while the topic tackled is timely, the results are not very significant. The results are very similar to previous results in a slightly different setting. The experiments make some interesting comments on when priors can be learned (namely, when the structure of the priors compensate for the type of information lost in the degradation operator). However, I feel as though there’s a disconnect between the presented theory and the experiments. In particular, the experiments seem to make a novel claim about what situations allow for prior learning, but the theory does not tackle these cases and, instead, shows results similar to previous results but in a particular setting (namely, dictionary learning). The particular results I am referencing are those of Tachella et al. (2022), that show recovery is not possible in the kernel of the measurement operator (but is possible with access to examples from several forward models).

Additionally, some of the experimental comments being made in this paper have been made before in previous works. For example, the Deep Image Prior paper effectively shows that convolutional structure aids in recovering signals from noisy measurements with only access to an implicit prior. This paper shows this phenomenon also holds in models based on Convolutional Dictionary Learning and other frameworks, but the punchline is effectively unchanged. I think further theory along these lines would be needed to elevate the contributions of this paper.

**General comments:**
- One question I had while reading the main text was what the role or influence the number of examples $N$ played in the recovery error/success rate? This information can be found in the appendix (namely, that the number of samples was kept high so that this was not a factor in the results), but it would be good to note this in the main text.

summary_of_the_review: I think that the paper is tackling an interesting problem and showcases some promising/interesting interesting experimental observations. However, in its current form, I think that it lacks novelty/significance compared to prior work in both the theory and experiments.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper tries to answer when prior learning can help in inverse problems using unsupervised ML methods. I don’t think this is an easy question to answer yet and understandably the paper considers a few simpler cases.  The paper starts off with traditional dictionary learning methods and shows that one can only learn atoms in the range of the operator. Next, they discuss convolutional dictionary learning and finally provide some empirical observations about unsupervised learning methods.
strength_and_weaknesses: -  I am not quite sure what the contributions of the paper are. The implications drawn from Prop 2.1 and 2.2 have been known for a while in inverse problems. 
- Similarly in the experiments, I do not think that SNR dropping in the kernel space while mostly stable in the range space is a novel observation; it is expected behaviour.
- I don’t quite understand Eq 9, shouldn’t A only be applied to f_\theta(Y_i)? In Eq 8, f_theta was taking in terms from the space of images (i.e. X space) while in 9 f_theta takes in measurements (Y-space)? Can the authors clarify how 8 leads to 9?
- I do not find the WSS justification for convolutions convincing? Why is it ok for X  (an image) to be considered a WSS process?  I think the reasoning could have been given without any WSS assumption. Consider any image x, if we apply a random mask with masking probability p, the expected norm of the masked image should automatically be rho*\|x\|. What am I missing here?
- “Natural images are stable enough to allow convolution-based algorithms to learn from all frequencies” – what does it mean for natural images to be stable? Also DIP or any other network has more than convolutions. In fact non-linearities are essential for a network’s performance. What is the claim here?
- In Fig 3 and 6, please also include the image over which the experiment was run.
- I also don’t understand what the authors mean to say when they claim that convolutions won’t work on deblurring. We do have sharpening filters that work to sharpen slightly blurred images. Is the claim that CNNs when trained DIP-style can’t deblur? 

clarity,_quality,_novelty_and_reproducibility: I am quite confused about what exactly the authors are trying to claim in this paper. They analyze some cases but never really connect the conclusions together in a convincing and straightforward way. I would say the novelty of the analysis presented is modest at best. They do provide code for reproducibility.


summary_of_the_review: I think the paper lacks few clear novel ideas that it wants to bring to the community. A lot of known conclusions are presented but unfortunately do not lead to something concrete or novel that the community can benefit from. I cannot recommend acceptance at this stage.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper focuses on unsupervised inverse problems, and studies under what conditions recovery is possible. Their contribution involves a series of characterization and analyses of the problem, explained below.

- They focus on the compressive dictionary learning problem and provide the conditions under which dictionary recovery is possible given one or multiple compression operators (measurement matrix).
- They focus on convolutional structure as a weak prior, and show how unsupervised CDL is able to perform similarly to supervised CDL when compression is not high.
- They show that for deblurring, where high-frequency information is completely lost, supervised CDL, unsupervised CDL, and all the hand-crafted features fail to recover.
- Given all these analyses and characterization of unsupervised inverse problems, they phrase the ultimate goal as "the key to success is finding a prior that can predict missing high frequencies from observed low frequencies." They show that the structure of DnCNN is indeed able to do that by capturing spectral information.

strength_and_weaknesses: The paper is well organized. I enjoyed reading the paper. It provides very good analysis, nice explanations to give intuitions (e.g., paragraph after (4)), and clear experiments on when and where supervised/unsupervised inverse problems may fail.

- The paper points out interesting insights on why dictionary learning with one operator may fail. It would be nice to indeed cite works that use several (instead of one) random measurement matrices for compressive dictionary learning (here are three examples [1,2,3]) when you introduce compressive dictionary learning (1).

- My main concern is that it is not clear which result is known from prior work and which is theirs. Hence, please discuss the novelty of the work and how it is different from prior work. I find this paper more of a review paper and summarizing what is known in the literature. Please make a clear discussion on the new finding of this paper compared to prior work. For example, some of the findings of this paper are based on Tachella et al. (2022).

- The paper considers only dictionary learning and linear inverse problem. How generalizable is this to other generative models and for example non-linear inverse problems?

minor comments:
- Before (4), "if this is not feasible, then (2) is not feasible". Please explain why?
- It would be nice to visualize some of the dictionaries learned in this work.
- Although very well-known, please cite lasso before (1) [4].
- Move figures close to their explanation.
- please use the same y-axis in sub-figures (e.g., page 16).

[1] Anaraki, Farhad Pourkamali, and Shannon M. Hughes. "Compressive k-svd." 2013 IEEE International Conference on Acoustics, Speech and Signal Processing. IEEE, 2013.

[2] Pourkamali-Anaraki, Farhad, Stephen Becker, and Shannon M. Hughes. "Efficient dictionary learning via very sparse random projections." 2015 International Conference on Sampling Theory and Applications (SampTA). IEEE, 2015.

[3] Chang, Thomas, Bahareh Tolooshams, and Demba Ba. "Randnet: Deep learning with compressed measurements of images." 2019 IEEE 29th International Workshop on Machine Learning for Signal Processing (MLSP). IEEE, 2019.

[4] Tibshirani, Robert. "Regression shrinkage and selection via the lasso." Journal of the Royal Statistical Society: Series B (Methodological) 58.1 (1996): 267-288.

clarity,_quality,_novelty_and_reproducibility: Clear and written well. It has nice visualization and explanation of their method. Some of the discussed properties/characterizations are known in the literature, and not clear what is new (see my comment about novelty and a need for discussion). They provided detailed info and code on the results, hence it seems reproducible.
summary_of_the_review: I enjoyed reading the paper. It is an interesting work on analyzing why unsupervised inverse problems may fail, how to bake weak priors to alleviate that, how supervised inverse problems may fail, and how to apply methods to capture spectral information to recover information that seems to be lost from the perspective of classical optimization. However, it is not clear if these results are already known in prior works. A discussion is needed here. Given that, I rate this paper as marginal acceptance. I may change my rating depending on the requested discussion.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper studies where (weak) prior learning can and can’t work in unsupervised inverse problems, by considering dictionary learning in a conventional model as well as convolutional dictionary learning. Inpainting and deblurring asks are addressed in this work.

strength_and_weaknesses: Weakness: this paper is roughly an extension of the work conducted by Tachella et al. (2022)

Strengths:
The paper is well written, clearly presenting the claims and discussions. The paper presents many interesting results, within different tasks such as inpainting and deblurring. See other comments for details
clarity,_quality,_novelty_and_reproducibility: The presentation is clear. The questions are well posed and clearly answered, providing clear claims and discussions for the readers.
summary_of_the_review: Essentially, this work extends the study conducted by Tachella et al. (2022) in order to provide a practical understanding of when one can and can’t learn a prior for unsupervised inverse problems. In the submitted paper, the authors focus on the case where the prior is not constrained, with the analysis of dictionary learning, using a single measurement operator first and then extending these results to multiple operators. Finally, they address prior knowledge within convolutional dictionary learning, and demonstrated several claims, such as why convolutions are likely to work on tasks like inpainting, including learning convolutional dictionaries from incomplete data as well as the unsupervised reconstruction. The deblurring task was also considered in the paper.
The claims are clearly presented and going to the main claims, such as the prior knowledge should compensate for the lack of information in deblurring tasks.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper studies the problem of dictionary learning in the setting where the sampled training data is only accessible via incomplete linear measurement operators. In the single-operator setting, the authors show that learning the 'true' dictionary is impossible. However, if multiple measurement operators are available, the problem becomes less challenging (but still, difficult in the presence of noise). To remedy this, the authors suggest using regularization schemes (or 'priors') such as plug-n-play, or deep-image priors, or convolutions; these improve performance in some cases. 

The paper received split scores. Some reviewers appreciated the insights provided by the suggested regularization schemes. Other reviewers pointed out issues with writing clarity, justification for certain assumptions/claims, and a lack of sufficient explanation of how the results would generalize to other, more practically relevant settings. Importantly, all reviewers raised a common concern about novelty; similar observations have been made several times in the past (and most recently, in the exact same setting of dictionary learning from incomplete observations, by Tachella et al. '22). Therefore, the contributions of this paper -- as presented -- are modestly incremental, and an addendum to this earlier paper at best.

Recommendation: reject.
justification_for_why_not_higher_score: As mentioned in several reviews and the summary above, the contribution of this paper is somewhat too incremental to merit acceptance.
Decision: Reject