Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper researches the lottery ticket hypothesis for networks as a deep image prior or deep generative prior. The specific approach is to (1) train deep networks to reconstruct multiple images for DIP (Ticket ﬁnding objectives), (2) conduct iterative magnitude pruning to the trained network, (3) obtain the pruned mask and reset the model parameter to initialization weight and (4) perform deep image prior to new (or training) images. 
The proposed objective is an important design for the system to work, i.e., the author optimizes the network by minimizing the expected error on multiple images.
The experimental results are somehow interesting that (1) different pruning methods actually performs differently and (2) there are winning tickets in the studied problem. As for GAN compressive sensing task, there are also winning tickets that exists. The point I am more interested in is that some networks are more suitable to be used as deep image prior than others. 
main_review: As I have mentioned in the above summary, the conclusion (some networks are more suitable to be used as deep image prior than others) is very interesting. It seems that the author only discussed the existence of these networks, but did not discuss the possible reasons why these structures is superior to other structures in the deep image prior task. Is this because in these networks, the important information (such as low-frequency parts, etc.) of the image used for recovery is easier to learn from the obtained network structure, while the high-frequency, harmful information is more difficult to capture by these networks? The lack of this part of the discussion is regrettable. It is possible that for DIP, the preferred network structure is different from discriminative tasks and GANs. I hope the author can introduce relevant discussions. If possible, some experiments can also be conducted. For example, maybe the convergence curves of DIP are different when different networks are used, and the separation of high frequency and low frequency is more obvious for winning network architectures?

Another regret is that GAN compressed sensing and DIP seem to be separated. It is not clear whether the cause of the effect on GAN CS and the result on DIP are the same. It is very likely that the reason for the effective pruning on GAN CS is different from DIP. Maybe the former is because some knowledge in GAN has no or negative influence on the CS task, while the latter is because different structures express different components of the image differently. After reading this paper, I still lack knowledge of these issues.

Another minor problem is that this paper is difficult to follow. The writing needs improvement. Especially in terms of specific methods, I think it is difficult to quickly understand the author's approach. Moreover, the practices of GAN CS and DIP are very different, and it is difficult to put them together clearly. 
summary_of_the_review: Overall, this is an interesting paper. This paper reveals some interesting results, but there are still some questions (see above). If the author can discuss these issues and improve writing, the paper can be accepted.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper presents the application of the Lottery Ticket Hypothesis (LTH) to the problem of the deep image prior in order to solve image restoration tasks. The paper shows the effectiveness of such a lottery image prior which trains subnets in isolation.
main_review: Positives

+ The paper addresses an important problem of image restoration using deep prior which has been quite effective in the absence of labelled examples. 

+ The work deserves merit due to the novel application of the lottery ticket hypothesis to this problem and has been well presented.

+ The paper conveys several new ideas and is shown to induce sparsity on the network while achieving impressive results.

+ The paper is well written with several questions justified and analysed thoroughly.

Concerns

- The paper does have a lot of typos such as the line - "that can be training in isolation" in two different places.

- Experimental evaluation and comparison for deep image prior tasks lacking rigour.

- The paper does not cite several works on line of the deep image prior such as those listed below. they have not been compared with.

1. Gandelsman, Y., Shocher, A., & Irani, M. (2019). " Double-DIP": Unsupervised Image Decomposition via Coupled Deep-Image-Priors. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 11026-11035).
2. I. D. Mastan and S. Raman, "DCIL: Deep Contextual Internal Learning for Image Restoration and Image Retargeting," 2020 IEEE Winter Conference on Applications of Computer Vision (WACV), 2020, pp. 2355-2364
3. Mastan, I. D., & Raman, S. (2021). DeepCFL: Deep contextual features learning from a single image. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (pp. 2897-2906).


summary_of_the_review: The paper has several positives, the concerns include missing references and comparisons. Hence, it is just below acceptance threshold.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: Summary
This submission studies lottery ticket hypothesis for deep neural network based inverse imaging. It considers two scenarios: 1-inference for compressed sensing based on pre-trained deep generative models, 2-deep image prior where all network parameters are fit to a single image. Both scenarios are dealing with a single image training task, or, an overfitting task, which makes this work different from the past work using LTH. The empirical results suggest that with a high level of sparsity not only the model size can be made much smaller, but also the generalization can improve which suggests pruning as a regularization. They also suggest good transferability from one image restoration task to another.


main_review: Originality
This work studies the LTH for low-sample deep inverse imaging for the first time

Significance
Strong points
- for the first time studies LTH for deep image prior and pre-trained deep generative compressed sensing
- provides empirical results for several low-level image restoration (as well as image classification) tasks and datasets
- the results are interesting suggesting LTH exists with high degree of sparsity that not only makes the models much smaller but also improves the generalization performance as a result of sparsity regularization
- the results also suggest transferability of LIP subnetworks across data and image restoration tasks

Weak points
- The paper presentation is poor. There are several typos throughout the text. The organization is poor and confusing. There are several scenarios studied in this paper with regards to the datasets and tasks and settings, but the link among them is missing and confusing from the text. 
- This work is purely empirical study and it's not clear for example why LIP can transfer across image restoration tasks, and dataset? Providing insights would be very useful. It is also not quite clear how the LIP pruned for denoising is transferred to image classification. Are only the intermediate representations transferred as in auto-encoders?


More comments and suggestions
- In the modified IMP, when using clean labels for training, where is the randomness coming from? How is the expectation defined?
- The presentation of the paper can benefit from more visuals such as diagrams/graphs to connect the tasks and datasets. 
- The experiments only report PSNR. It would be better to report SSIM and FID to see if the pruning can hurt the high frequency perceptual details or not? This is more important for the pretrained GAN model. 
- It is not clear how the sparsity pattern looks like for the considered hour-glass network. Isn’t this one of the important points of pruning as advocated in the introduction? A visualization of the sparsity pattern per network layers would be very insightful. It would also be helpful to see if pruning creates a bottleneck in the network and where that bottleneck is. 

summary_of_the_review: This is empirical work studying an important problem. The results are also interesting which suggest a high degree of sparsity for certain learable inverse imaging problems, that closes the gap with the traditional sparsity based inverse imaging algorithms. The presentation of the paper however needs significant improvement to connect the conclusions toward a coherent story.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper exploits the LTH in the low-level vision tasks, including image restoration and compressive sensing image reconstruction. Through some experiments, it identifies the existence of LIP in low-level vision and also gives some interesting conclusions on the transferability of the proposed LIP.
main_review: My questions mainly concentrate on the setting 1 of the paper, i.e., the DIP related tasks.

1. This work shows that there exists a very sparse subnetwork whose performance can surpass the original dense network. To find such sparse network, it directly minimizes the MSE relying on the ground-truth image. In practice, however, the ground-truth image is always not accessible. How to obtain such sparse subnetwork in real applications?

2. In my opinion, the goal of the research on LIP is to simplify the DNN architecture in DIP based on the pre-known knowledge on the sparse subnetwork. For example, given a noisy image without the latent clean image, if we know the sparse subnetwork in advance, we can directly optimize it to obtain the denoising result under DIP.  How to achieve this goal? If LIP cannot embed into current DIP related works like this, it is useless in practice.

3. LIP is implemented by the network weight pruning, and all the experiments depends on a current existing weight pruning method. I think  the contribution of this paper is incremental and not enough.

4. The mathematic notions in algorithm1 and algorithm2 are confused. In algorithm 1, what does $f_{\mu}(x;\theta_0 \odot m_{\mu})$ mean? In DIP, the network $f$ takes the latent code $z$ as input instead of the image $x$. As for the loss function of algorithm 2, whom does $E(f(x_a;\theta \odot m))$ denotes the MSE distance between the network output $f(x_a;\theta \odot m)$ and?
summary_of_the_review: Even though this paper is interesting, I still argue the usefulness of the proposed LIP. This paper only attempts to do some exploration based current existing LTH related works, which weakens  the novelties and the contributions of this work.
review_title: null

Meta-review: The paper studies the lottery ticket hypothesis in the context of deep image priors. Deep image priors are convolutional neural networks that are imposed as a prior for image reconstruction problems. A deep image prior can be an un-trained convolutional network, or it can be a trained generator, and the paper considers both types of priors. Deep image priors are often highly over-parameterized and thus the paper under review asks the question on whether the networks really have to be heavily parameterized or whether a small subnetwork will also do. The paper performs experiments on image restoration with an entirely un-trained DNN (this constitutes the larges part of the paper) and on image restoration with a pre-trained network. 

The paper received four reviews out of which three recommend weak acceptance and one strong acceptance.
- Reviewer wMsj finds it interesting that the paper shows that some networks are more suitable as deep image priors than others, but finds that the paper lacks evidence on why some structures are better than others.
- Reviewer Kjf3 strongly recommends acceptance (8) in a relatively generic review. The reviewer finds the paper is interesting as it provides a novel application of the lottery ticket hypothesis. However, the review also criticizes that the experimental evaluating lacks rigor.
- Reviewer jFko appreciates that the paper studies the lottery ticket hypothesis for the first time for un-trained and pre-trained image prior, and that the results are interesting as they suggest that the models can be made smaller and that this can even improve performance. The reviewer criticizes that the paper's presentation is confusing, and points out a few weaknesses in the empirical evaluation. The authors responded and after a brief discussion, the reviewer raised their score. 
- Reviewer vf8k provides a relatively brief review and criticizes that to find sparse networks, one requires the ground truth image, which is not accessible. The authors clarify that the method is transferable in that the network identified can be used for other images and is thus transferable. The reviewer was satisfied with this response. 

The score of this paper 6.5 after the discussion period. Three of the reviewers are on the fence about the paper, one reviewer is not, and that reviewer significantly impacted the score. This reviewer, however, did not provide convincing arguments about the merits of the paper. All reviewers find that `3: Some of the paper’s claims have minor issues.', and I agree with that statement.

I do not recommend acceptance of the paper in its current form, because of insufficiently rigorous experiments to justify the claims:
- Specifically, the paper's goal is to address the research question 'do they [neural-network based priors for image reconstruction] really have to be heavily parameterized?' The literature already answers this question since as the paper under review reads on page 3, the literature found that an under-parameterized non-convolutional model can function as an un-trained image prior. Those underparameterized networks perform well and have fewer parameters than the best-performing networks found in the paper under review.
- The paper argues that a sparse network can give better performance. This claim is based on an at most 0.1dB difference, which can only be achieved when choosing the optimal sparsity level, which is not clear to do without knowing the ground truth image.
Decision: Reject