Lottery Image Prior
Authors: Qiming Wu
Xiaohan Chen
Yifan Jiang
Pan Zhou
Zhangyang Wang
Venue: ICLR 2022 Submitted (2022)

Abstract: Deep Neural Networks (DNNs), either pre-trained (e.g., GAN generator) or untrained (e.g., deep image prior), could act as overparameterized image priors that help solve various image inverse problems. Since traditional image priors have much fewer parameters, those DNN-based priors naturally invite the curious question: do they really have to be heavily parameterized? Drawing inspirations from the recently prosperous research on lottery ticket hypothesis (LTH), we conjecture and study a novel “lottery image prior” (LIP), stated as: given an (untrained or trained) DNN-based image prior, it will have a sparse subnetwork that can be training in isolation, to match the original DNN’s performance when being applied as a prior to various image inverse problems. We conduct extensive experiments in two representative settings: (i) image restoration with the deep image prior, using an untrained DNN; and (ii) compressive sensing image reconstruction, using a pre-trained GAN generator. Our results validate the prevailing existence of LIP, and that it can be found by iterative magnitude pruning (IMP) with surrogate tasks. Specifically, we can successfully locate the LIP subnetworks at the sparsity range of 20%-86.58% in setting i; and those at sparsity range of 5%-36% in setting ii. Those LIP subnetworks also possess high transferrability. To our best knowledge, this is the first time that LTH is demonstrated to be relevant in the context of inverse problems or image priors, and such compact DNN-based priors may potentially contribute to practical efficiency. Code will be publicly available.

Under review as a conference paper at ICLR 2022
LOTTERY IMAGE PRIOR
Anonymous authors
Paper under double-blind review
ABSTRACT
Deep Neural Networks (DNNs), either pre-trained (c.g., GAN generator) or un- trained (e.g., deep image prior), could act as overparameterized image priors that help solve various image inverse problems. Since traditional image priors have much fewer parameters, those DNN-based priors naturally invite the curious ques- tion: do they really have to be heavily purameterized? Drawing inspirations from the recently prosperous research on lottery ticket hypothesis (L.HH), we conjec- ture and study a novel "lottery image prior" (LIP), stated as: given an (untrained or trained) DNN-based image prior, it will have a sparse subnetwork that can be trained in isolation, to match the original DNN's performance when being applied as a prior to various image inverse problems. We conduct extensive experiments in two representative settings: (i) image restoration with the deep image prior, using an untrained DNN; and (ii) compressive sensing image reconstruction, us- ing a pre-trained GAN generator. Our results validate the prevailing existence of LIP, and that it can be found by iterative magnitude pruning (IMP) with surrogate tasks. Specifically. we can successfully locate the LIP subnetworks at the spar- sity range of 20%-86.5844 in setting i; and those at sparsity range of 5%-36% in setting ii. Those LIP subnetworks also possess high transferability. To our best knowledge, this is the first time that LTH is demonstrated to be relevant in the context of inverse problems or image priors, and such compact DNN-based priors may potentially contribute to practical efficiency. Code will be publicly available.
INTRODUCTION
Background Deep neural networks (DNNs), in particular convolutional neural networks (CNNs), have been powerful tools for solving various image inverse problems such as denoising (Zhang et al.)
2017 Guo et al. 2019 Lehtinen et al. 2018), inpainting (Pathak et al. 2016: Yu et al. 2018 2019b),
and super resolution (Ledig et al. 2017 [im et al.) 2017] Zhang et al. 2018) Conventional wisdom
believes that is owing to DNNs' universal approximation ability and learning from massive training data. Yet, recent studies have revealed the specific architectures of CNNs have the inductive bias to represent and generate natural images well, and such favorable architecture inductive bias can work independently from fitting specific training sets (Ulyanov et al. 2018 Cheng et al. 2019: Bora al.) 2017 Heckel & Hand 2019 Jalal et al. 2020).
For example, deep image prior (DIP) UUyanov et al. 2018) shows untrained neural network can be used as a handerafted prior that transfers well across multiple inverse problems. The authors attributed the success to the CNN architecture itself, that appeared to possess high noise impedance even with only random initializations. As another example, in compressive sensing, Bora et al. (2017) Dalal et al. (2020) replaced the common structural assumptions such as sparsity with a pre- trained generative adversarial networks (GAN). The underlying rationale lies in that pre- pre-trained generator should (approximately) represent the notion of vector being or more likely in the target domain such as natural images; in other words, sample more like a natural image will be closer to the output range of the pre-trained generator. In this paper, we refer to such purpose image prior parameterized by (cither untrained pre-trained) DNNs DNN-based image priors.
Recall that, classical image regularizers in the spatial or frequency domains are often not learning- based (Tomasi & Manduchi 1998. Sardy et al.) 2001 Dabov et al. (007). or rely on compact learning models (Cao et al. 2008 Elad & Aharon 2006 He et al. 2015). On contrary, DNN-based image priors have massive number of parameters (we compare the parameter numbers between the full model and the sparse subnetwork in Table. ( typically magnitudes more than the image (even the image size) size. The two extremes invite the natural question: Can identify highly compact

---

Under review as a conference paper at ICLR 2022
Denoising
Iopainting
LTH IMP
Image Restoration or
Compressed Sensing with GANs
Super- Super-Resolution
Compressed Sensing Ax" 1. Reconstract
Danse Model (untrained NN or pretrained Generator)
Pruting
Tickets with LIP
Figure Overview of our work. In both untrained and pretrained DNN-based image priors, we study the existence of Lottery Image Prior (LIP) that could transfer to various image inverse prob- lems such as denoising, inpainting, super-resolution, and/or compressive sensing restoration.
DNN-based image priors, that are same effective? We note that, diving into this question has two- fold appeals. On the algorithmic side, that could help us understand further how the topology and connectivity of CNN architecture itself will affect the effectiveness of those priors, and to what extent sparsity could be relevant. On the practical side, if we could provide an affirmative answer to this question, that would potentially lead to more computationally savings when applying those DNN-based priors in practice, leading faster restoration or computational imaging with them.
Towards the above question, the tool we refer to in this paper is the recently emerged Lottery Ticket Hypothesis (LTH) (Frankle & Carbin 2018: Frankle et al., 2020a). LTH suggests that every dense DNN has an extremely sparse "matching subnetwork" that can be trained in isolation to match the original dense DNN's accuracy. While the vanilla LTH studies training from random scratch, the latest works also extend similar findings to fine-tuning the pre-trained models (Chen et al. 2020a 2021ap. LTH has widespread success in image classification. language modeling. reinforcement
learning and multi-modal learning, e.g., Yu al. 2019a; Renda et al. 2020 Chen al. 2020a
Gan et al. (022))
Our Contributions Drawing inspirations from the LTH literature, conjecture and empirically study a novel "lottery image prior" (LIP), stated as:
Given an (untrained or trained) DNN-based image prior, will have sparse subnetwork that can be trained in isolation, to match the original DNN's performance when being applied prior to regularizing various image inverse problems.
Studying this new problem is, however, NOT a naive extension from the existing LTH methods, owing several technical barriers: (a) till now, LTH has been demonstrated for image inverse problem or DNN-based priors, to our best knowledge. Most LTH works studied discriminative tasks, with one exception (Chen al. 2021e). It is therefore uncertain whether high-sparsity DNN still viable for reconstruction-oriee orreente tasks; (b) existing LTH works typically require a full training set to locate the sparse subnetwork mask, whereas our LIP settings are only not data-rich. For example, DIP needs the DNN to be trained to overfit one specific image, making it drastically different from previous problems; (c) the objectives between finding the sparse mask (e.g., learning the prior) and fitting the sparse subnetwork (e.g., using the prior) often unaligned in LIP problems. For example, DIP will overfit a corrupted input image (during which the sparse mask will be found) in order to reconstruct clean output image (when the found sparse subnetwork will be used); the pre-trained generator will also be used towards a different goal (regularizing compressive sensing) from their original pre-training task (generating realistic images).
Our extensive experimental study confirms the existence of LIP in two representative settings: (i) image restoration with the deep image prior, using untrained DNN (as shown in Fig. 2 and (ii) compressive sensing image reconstruction, using pre-trained GAN generator. Using iterative magnitude pruning (IMP) with surrogate tasks (the overview of our work paradigm is Fig. [ we can successfully locate the LIP subnetworks at the sparsity range of 20%-88.55%% in setting and those the sparsity range of 5%-36% in setting Those LIPP subnetworks also possess high transferability. For example, the LIP ticket found in the setting transfer well across not only different images, but also different tasks such as denoising, inpainting and super-resolution. Our contributions are summarized below:
The first comprehensive study on LTH in DNN-based image priors and inverse problems, establishing the "lottery image prior" (LIP) and demonstrating the prevailing relevance of LTH more broadly than previously typical settings.
2

---

Under review as a conference paper at ICLR 2022
Impxintting
(Brund-Tath
Moded
Full Minid PSNR -20.22
Spanity - 79% PSNR 331.22
Sparity - 88% -310
Sporsiy - 91.4% -SNR-IU1
Spariity =99 50% 1SNit=200
Supar Resoltton Fartow =4
Ground-ruuh
Resolation
Spaniiy -8%% PSNR
Spreaty 8%.9%
Spansity -91.%%
Spasiiy - 98.53%
PSNR 255.0
PSNU 22.77
PSNR 25433
PSNR =2172
Super- Resolution Farlor =B
Troun--rrth
Low Resolution Facoo
Model PSNR 28.73
Speeiity 67% PSNR
Spaesit 85%% MSNR -2115
Dennising
GoundFrat
Noee
Mode
Spasily 50%
Spanily - 67%
Spesity PSNR - 31.13
Speesity 590.5%%
PSNR -31.07
PSNR 33.17
PSNR - 31.24
FSNR= 2200
Figure 2: LIP visual results: inpainting (row 1), super-resolution (rows 2/3) and denoising (row 4).
The last column (in blue) intends to display the results with the most extremely sparse subnetwork.
The investigation of both untrained and pre-trained DNNs as image priors, verifying that LIP subnetworks can be found in both settings, by overcoming several impediments such as severely limited training data and task objective mismatch.
High transferability of LIP subnetworks across data/datasets, and various inverse problem tasks. Our finding reflects the underlying common image prior that agnostic to specific data task, through the lens of the CNN architecture together with sparsity.
2 BACKGROUND WORK
Lottery Ticket Hypothesis LTH (Frankle & Carbin 2018) states that the dense, randomly ini- tialized DNN contains sparse matching subnetwork. which could reach the comparable or even better performance
independently being trained for the same epoch number the full network do. Since then, the statement has been verified a variety of fields, such as image classification
(Frankle & Carbin 2018, Liu et al., 2018 Wang et al. 2020 Evei al. 2019 Frankle et al. 2020b Savarese et al.) 2019 Yin et al.] [2099] You et al. 2019 Ma et al.] 2021 Chen et al. 202 la), nat-
ural language processing (Gale et al.
2019 Chen al. 2020a), reinforcement learning Yuu al.
2019a), lifelong learning (Chen et al.
2020b), graph neural networks (Chen et al. 202 lb), and ad-
versarial robustness (Cosentino et al.
2019). Rewinding was proposed by (Frankle al. 2019) to
scale up the LTH to large models and datasets. The found matching subnetworks also demonstrate
transferability across datasets and tasks (Morcos al. 2019
Desai et al. 2019).
Deep Image Prior and Its Variants Despite CNN's tremendous success various imaging tasks, their outstanding performance is often attributed to massive data-driven learning. DIP (Ulyanov al. 2018) pioneered to show that CNN architecture alone has captured important natural image priors: by over-fitting a randomly initialized untrained CNN to a single degraded image (plus some early stopping), can restore the clean output without accessing ground truth. Follow-up work (Mataev et al. 2019) strengths DIP performance by incorporating it with the regularization by de- noising (RED) framework and a series of works (Mastan & Raman 2020 2021 use the contextual feature learning method to achieve the same goal of DIP. Besides natural image restoration, DIP was successfully applied to PET image reconstruction (Gong et al. 2018), dynamic magnetic reso- nance imaging Jin et al. 2019), unsupervised image decomposition (Gandelsman al. 2019) and quantitative phase imaging (Yang et al. 2021). Heckel Hand (2018) further demonstrated that pardeeeerreete non-convolutional model, named "Deep Decoder". over-fit sin-
3

---

Under review as a conference paper at ICLR 2022
gle degraded image like DIP did, which does not critically rely on early stopping or Chen et al. (202cc) was the first to study the possibility to optimize CNN architectures for capturing stronger image priors in DIP, via leveraging Neural Architecture Search (NAS).
Compressive Sensing using Generative Models Compressive Sensing (CS) reconstructs an un- known vector from samel-d linear measurements of its entries (Foucart & Rauhut 2013). by assuming the unknown vector to admit certain structural priors. The most common structural prior to assume that the vector is k-sparse in some known bases (Candes et al. 2006, Donoho 2006), and more sophisticated statistical assumptions were also considered (Baraniuk al. (0110). How- ever, those priors are inevitably oversimplified to depict the high-dimensional manifold of natural images. [Bora al. (2017) presented the first algorithm that used pre-trained generative models such as GANs, as the prior for compressed sensing. As a prior, the pre-trained generator encourages CS to produce vectors close its output distribution, which approximates its training image distribution. Significant research has since followed to better understand the behaviours and theoretical limits of CS using generative priors, e.g.. (Hand & Voroninski 2018 Bora et al. 2018 Hand et al. 2018
Kamath et al. 2019 Liu & Scarlett 2020 Jalal et al. 2020).
3 PRELIMINARIES AND APPROACH
3.1 NEURAL NETWORK AS PRIORS: Two SETTINGS
Although the implementation process of the GAN Compressed Sensing (CS) task and the DIP restoration task are different, yet we combine the two studies in this one paper for two reasons: 1) we are motivated by two current LTH streams: networks with weights trained from scratch (Frankle & Carbin 2018) and networks with pre-trained weights (Chen et al. 2021a). 2) Network structures with random weights or pre-trained weights can be priors, which is the commodity we view as the two could be unified. In setting i, we use an untrained CNN the dense model, and solve the DIP
optimization (Ulyanov et al. 2018)
6" = arg min F(fo()); x² = x== fo
(1)
8
where E(:) denotes the Mean Square Error (MSE), 3 is the corrupted version of the image R3xH*W fo represents the dense model f with initial parameter 0 and = is the random tensor that RC We choose the same hourglass architecture with skip connections in (Ulyanov et al. 2018), to be the dense model by default.
In setting we follow [ora et (2117) to use pre-trained GAN generator, to reconstruct the unknown vector x² E R2,, after observing m < linear measurements of its entries with noise: y Axx + 17, where A m is the measurement matrix and r/ E R" is the noise. Within the range of the pre- pre-trained generator prior, GANs could reconstruct the vector x with high perceptual quality, Following the recommendation of Dalal et al. (2020). we use the official pre-trained model
PGAN (Karras et al. 2017).
3.2 FINDING LOTTERY TICKETS
Datasets In setting i, we use the popular Set5 (Bevilacqua al. 2012) and Set14 (Zeyde al.,
2010) datasets. Besides, we evaluate the transferability of subnetworks on image classification datasets such as MNIST (LeCun al. 2010) and CIFARIO (Krizhevsky al. 2009). In setting ii our evaluation dataset is CelebA-HQ (020).
Lee et al. 2020 Karras et al. 201 7), following Jalal et al.
Subnetworks Consider a network fz; 0) parametered by 0 with input x, then subnetwork is defined as [ar; m @ f), where @ E (0. 1) d d 1101| and @ is the element-wise product. Let AT (f 0)) to be the training algorithm, that is, training model f(r; 0) on the specific task T with t iterations. We also denote the random initialization weight as fg and the pre-trained weight as 8,;
0. as weight at the i-th training iteration and ET (f( 0)) the model performance evaluation.
Finding Subnetworks Following the definitions of |Fannlle et al. (2020a), we define that if the subnetworks is matching, satisfies the following conditions (we use 0, for example, to denote a pre-trained lottery ticket (Chen et al. 2020a 2021a;; ; O, can be defined likewise):
ET (A (f(z; E (AT (f (A
2 0)).
(2)
4

---

Under review as a conference paper at ICLR 2022
That is matching subnetwork that performs no worse than the dense model under the same training algorithm AAT and the evaluation metric E². Similarly, we define the winning tickets: a matching subnetwork (x; m 8) has 0 = Zz. then it is the winning tickets under the training algorithm A
In practice, we find the matching subnetworks in the following ways: 1) IMP on the dense model with only one image (applicable to the image restoration task on the same domain of images); 2) IMP on the dense model with various kinds of images through weight-sharing (applicable to the image restoration tasks on images from different domains); 3) LTH IMP on the dense model with the sub-dataset (applicable to the CS task with pre-trained GANs).
Ticket Finding Objectives When finding LIP subnetwork in untrained DNN, i.e., DIP, there is problem that may be easily overlooked: what is the target during IMP training? Specifically, since the training target in Eq. [ for DIP is ā (the corrupted image), the model parameters may easily overfit the corrupted image during IMP training if we do not fine-tune the training epochs. Then the obtained mask will contain the information of corrupted images instead of the desired image prior (but we can also find the effective subnetworks by noisy targets, experiments are summarized in Table. 3 and 4 in Supplementary Materials). Therefore, we modify the optimization objectives of DIP during IMP training: arg ming ((fo (=); ar) to ensure that model parameters learn the clean image prior.
Evaluations of Subnetworks After obtaining the matching subnetwork f(z; m O 0). we evaluate their performances by 1) resetting the model parameter 9 to initialization weight 0g" 2) adding the mask to the model; 3) training the model to another N iterations. We evaluate the DIP subnet- works performances mainly through Peak Signal to Noise Ratio (PSNR) values, plus reconstruction errors (defined in (Jalal al. 2020) for compressive sensing with GANs.
Pruning Methods We use the standard pruning method (Han et al. 2015), which iteratively prune the 20% of the model weight each time. For setting i, our basic algorithm performs IMP over just one single image (i.e.. DIP's default setting), and the algorithm is summarized in Algorithm [ (Appendix). We further design an extended algorithm, that can perform IMP for DIP over multiple images, through backbone weight sharing: the algorithm is outlined in Algorithm 2 (Appendix). We
will discuss the algorithm variants in Section 4. For setting ii, we use IMP find winning tickets in pre-trained GANs, following the routine in (Chen et al.] 2021c). In each iteration of IMP, we will prune 20% of the remaining weights, and reset to the pre-trained weights. Note that we also include other pruning methods to compare the effectiveness of the matching subnetworks such as random pruning (randomly generate the sparsity mask m'), and pruning- saainng-t--inn at-initializattinn methods, e.g.. SNIP (Lee et al. 2018).
first fine-tune the pre-trained GANs on (sub-)datase)
4 LIP FOR DEEP IMAGE PRIOR WITH UNTRAINED DNNS
In this section, we will investigate the lottery image prior (LIP) property under setting i, for deep image prior (DIP) models. Model performance is measured by PSNR (Peak Signal to Noise Ratio) between the restored images and the clean ground truths (SSIM (Structural Similarity) results are in supplementary materials). We run all experiments with three different random seeds. All images used for plotting results are summarized in Fig. 14
In DIP, for each degraded image to be restored, an untrained DNN will be over-fitted over that single image (with proper early stopping). This casts crucial difference from typical LTH settings where winning tickets are identified using IMP over a distribution of data and are verified to be effective when trained data from that specific distribution. Therefore, besides the usual properties in winning tickets such as the existence, the superiority over other pruning methods, and the effect of rewinding, we will strive answer two more questions about LIP in DIP. The first question is: can we extend the IMP algorithm for dingle-image DIP to multi-image and thus improve the performance of winning tickets hopefully? The second question is to what extend the LJP found in DIP setting can be transferred? Answering these two questions not only provides a clearer picture of LIP, but also hints new perspective for deeper understanding of LTH.
Existence of LIPs In DIP setting, we first find the winning tickets with LIP property by imple- menting the single-image IMP on the hourglass model (i.e., the DIP model used in Ulyanov et al. (2018). We adopt the modified objectives using clean images as labels as we discussed in Sec. 3.2 The algorithm is described in detail in Algorithm I We apply the implemented algorithm on Set5
5

---

Under review as a conference paper at ICLR 2022
35.0
355
31.0
Deti
Derid
Dena
Hardin
SHP
UP
300
(a) IMP-Baby
(b) -MP-Woman
IMP-Facel
(d) IMP-Face2
31.4
21.
-01
213
0,3
31.3
Rosion
304
(e) Eval-Baby
(I) Eval- -omaa
(g) Eval-Facel
(b) Eval-Face2
Figure 3: Experimental results of finding LIP in setting i (i.e., DIP). The first row of the figure
summarizes the LTH IMP training loops and the second row denotes the evaluation of found LIP. Note that we compare the LTH IMP with Random Prune (Random) and SNIP (Lee al. 2018) prune methods, on images from different (Baby and Woman) or same domains (Facel and Face2).
29
30
28.
Unme
304
Rowidd -3
R
244
Rowird
Reaind
Rewest
Rewed, 10
Rewied_
364
Rodid
Rowiral
Rewist
Rewel 20
(a) Face2
(b) Face4
(c) Head
(d) Butterfly
Figure 4: Experiments of the rewind strategy (background task: denoising). Note that we train the model with N epochs in IMP and Rewind.j means rewinding the ticket parameter to fj. the weights
after 1% x N steps of training.
(evilaaquaa et al. 2012) and Set14 (Zeyde et al. 2010) datasets to obtain the sparse subnetworks and evaluate these subnetworks on the denoising task.
Results of image IMP in Fig. (curves in red colors) verify that the LIP exists in the DIP set- ting. To be specific, during the IMP finding process when we use clean images in DIP objectives, we are able to find the winning tickets with LIP on untrained DNN at sparsity as high as 86.58% While at the evaluation stage, when only have access to the degraded images, we find that the winning tickets found with the modified objectives are still applicable, matching the dense performance at sparsity high 83.23%.
Besides the dense model baselines, we also compare the single-image IMP with Random Pruning, and aaaaiigga--initializatiio method SNIP (Lee al., (2018), whose results also presented in Fig. 3 in other colors. Specifically, we clearly observe from the first row in Fig. 3 that the single- image IMP for DIP outperforms random pruning and SNIP at a wide sparsity range [20%, 96%). Interestingly, we find random prune is a good competitor to SNIP sometimes, but both of them suffer significant performance decrease at extreme sparsities (over 80%), where IMP still persists.
The Effect of Weight Rewinding In this part, we study the effect of weight rewinding (Frankle) al. 2019) when applied the single-image IMP for DIP models. Weight rewinding proposed to scale LTH up to large models and datasets. Specifically, we say we use p% weight rewinding if we reset the model weights at the end of each IMP iteration to the weights the dense model after p% ratio of training steps within a standard full training process, instead of the model's random initialization. For the single-image IMP in DIP, we consider 5%, 10% and 20% weight rewinding schemes. The resulting models are denoted Rewind_5, Rewind_10 and Rewind_20. respectively. The results of different weight rewinding schemes are summarized in Fig. We can see that weight
We prune 20% of the remaining weights in each IMP iteration, resulting in sparsity ratios 8, 1 =1- 0.8".

---

Under review as a conference paper at ICLR 2022
214
322
Nanden
(a) Baby
(b) Woman
(c) Facel
(d) Face2
Figure 6: Experimental Results of Multi LIP on images from same/different domains. We compare
Multi LIP with LIP and random prune methods. The background task is denoising in setting i.
rewinding is not beneficial for identifying LIP in the DIP setting. Too much rewinding (10% and 20%) even hurts performance or fails it completely. We conjecture that this is due to the extremely low data complexity DIP (single image).
Is Multi-image IMP A Good Extension in DIP? DIP over-fits an untrained DNN on a single image. reasonable conjecture is that the DNN will be dedicated to the features in that specific image and so should the resulting winning ticket be if we apply single-image IMP to the DIP model. We then ask that whether we can co-consider multiple images during IMP for DIP models to find winning tickets that are suitable for more general features and thus yield better image restoration performance? To verify our idea, we propose new multi-imant -multi-image IMP for DIP where we replace the single-image DIP objective with the average of mul- tiple images during the IMP process. Note that all images will share the same fixed random code during IMP. Otherwise, differ- random codes will intervene with one another because they lead to different winning tickets, resulting in a share subnetwork with inferior performance. The algorithm is formally described in Algorithm@
PBNR
Darce
UP
Woman,
Bid LIF
Butertyy_
Mul LP
46
160
Spenay(S)
Figure 5: We evaluate differ- ent single-image LIPs and the multi-image LIP on Baby image measure LIP's transferability (cross images).
We evaluate the multi-image IMP two different settings: (i) cross- setting where we apply the multi-image IMP to the five images from Set5 (Bevilacqua et al.) 2012); (2) single-domain setting where we apply the multi-image IMP five images of human faces with glasses. We think images from Set 5 are from more diversified domains because they include bird, butterfly and human face contents. We compare single-image IMP winning tickets found on the Baby and the Woman images from Set5 with the cross-domain ticket, and the single-image IMP winning tickets found on Face-1 and Face-2 with the single-doman ticket. Results presented in Fig. shoow that imati IMP significantly improves the quality of the winning tickets in the cross-domain setting, which is aligned with our previous hypothesis.
To what extent can LIP tickets be transferred? Transferability is an important metric to measure the usefulness of LIP in the DIP setting. It would significantly undermine the practical application of LIP if the winning tickets can only be dedicatedly found for each image. In this part, we evaluate the transferability of LIP for DIP models from three perspectives, i.c., the transfcrability between images, between image restoration tasks and between high-level tasks.
Observation 1: LIP can transfer across images. We identify the single-image LIPs for all image from Set5 and the multi-image LIP on Set5 using the new multi-image IMP proposed in the last part. Then we evaluate all the above LIPs on the Baby image, results being presented in Fig. 5 We can see that the LIPs found on other images, even those from the Bird and the Butterfly images, can perform comparably well with LIP dedicatedly found on the Baby image. This shows that LIP has reasonable transferability across images, even for those coming from slightly different domains (for example, the Baby and the Butterfly). We can also observe the fact that the multi-image LIP outperforms all single-image LIPs, as side support for the superiority of malti-image LIP in the cross-domain settings.
Observation 2: LIP can transfer across image restoration tasks but not to other high-level tasks. Ulyanov et al. (2018) showed the effectiveness of DIP in different image restoration tasks including denoising, inpainting and super-resolution. We conduct experiments to verify if a LIP winning ticket identified on one image restoration task can be re-used in another because they share some common
7

---

Under review as a conference paper at ICLR 2022
24:
340
291
Donoo
Derso
260
TpRILLPP
LP
Dasto
265
Damu
313
312
(a) Denoising-Hoon Head
(b) Denoising- Woman
(c) Inpainting- Kate
(d) Inpainting-Vee Vase
215
200
10
$224
222
21.
314
UP
UP
Denobe
Durese LP
Denone
Spant//
Sperttti
(e) SR-factor---Baoon
(f) SR-faccor-4-Pepper
(g) SR-factor-8 (h) SR-factor---Pepper
Figure 7: Transferability (cross tasks) experimental results. We study the transferability of denois- ing LIP on the restoration tasks such as inpainting and super-resolution (SR); we also study the inpainting and SR LIP on the denoising task. We consider two SR scale factors 4, 8.
Derza
Rander
Derme
Bazy
Hardimm
MJl
Denoze_1P UP
CIPARL MP
OPARLEE LP
Senvac
(a) MNIST
(b) CIFARIO
(c) CIFAR-LIP-nn-DIE
Figure 8: Transferability experiments. We test the denoising LIP on MNIST and CIFARIO datasets (left figures). Note that we replace the last convolutional layer of DIP models with the linear layer and load the same initial weights. We evaluate the CIFAR LIP on the denoising task.
interests? Furthermore, if the above question has affirmative answer, such transferability sus- tainable when transferring to or from some high-level tasks such as classification?
To evaluate the transferability of LIP between image restoration tasks, first find three LIPs for the denoising, inpainting and super-resolution tasks respectively and then transfer between them, as shown in Fig. 7 We observe that a LIP winning ticket transferred from another image restoration task always yields restoration performance comparable with the single-image LIP found on the original task, sometimes even better, for examples in Fig. ( and 7(g)
We then evaluate the transferability of LIP between the denoising task and image classification task on CIFAR-10 10. We apply the standard IMP on the DIP model, i.e., the hourglass architecture in (llannov et al.) 2018). to obtain winning tickets on MNIST and CIFAR-10 datasets. We show results of transferring the denoising LIP to MNIST in Fig. ((a) and CIFAR-10 in [Bb) and the CIFAR-10 winning tickets to denoising task in Fig. 8(c) Although transferring denoising LIP to MNIST seems to yield winning tickets due to MNIST's s low complexity, its transfer to CIFAR-10 unsuccessful. Interestingly, the multi-image LIP becomes worse ticket CIFAR-10 10 than the moagle-image LIP. Moreover, transferring winning tickets on CIFAR-10 10 back to the denoising DIP task also fails to generate winning tickets that are comparable with denoising LIPs.
Based on those observations, we conjecture that the architecture priors needed to "win the lottery ficket" may substantially differ between low-level (reconstruction, restoration....) and high-level (classification, recognition....) vision tasks, but seems to be quite overiapped/shareable among dif- ferent tasks that are all low-level (or high-level). The high-level task transferability of winning tickets has been found in prior works (Morcos et al., 2019 Desai et al. 2019 Chen et al., (021a). We believe this interesting "incompatibility" between low-level and leve- tasks is unstudied new direction, and leave this for our future work.
LIP FOR COMPRESSIVE SENSING WITH GENERATIVE MODELS
Finding LIP in GAN priors for compressive sensing task is natural than in the setting i. The training process of GANs is more similar to a typical IMP setting where the models are trained on distribution of data for image classification. And the existence of winning tickets in GANs for
8

---

Under review as a conference paper at ICLR 2022
Full Model
S =020
S -330
-449%
S -59%
S-77%
7478%
Figure 9:
Visual results of compressed sensing using LIPs found in pre-trained PGAN. Full Moddi
Masked
$=20%
S
S=3%
S=67%
Figure 10: Visual results of inpainting using LIPs found in a pre-trained PGAN.
the generative task has been shown in (Chen et al. 2021c). But has not been shown for GANs image priors for the compressive sensing task. In this section, first describe how identify LIP in pre-trained GANs using IMP and show the existence of LIP for the compressive sensing task. Then show that the LIPs found be transferred other image restoration tasks such as denoising and inpainting.
Existence of LIP in GANs for Compressive Sensing We use PGAN (Karras al., 2017) pre- trained on CelebA-HQ dataset (Lee et al. (020)) as the model in this section. To obtain LIP tickets in pre-trained GANs, on 40% of images in celebA-HQ for 30 epochs, has 20% of its remaining weights pruned, and then reset to the pre-trained weights. We only prune the generator it is found in Chen et al. (202Tc) that pruning discriminator or not only has marginal influence on the quality of the winning tickets. We then evaluate the tickets on the compressive sensing task following the setting in Jalal et al. (2020): we fix the number of measurements to 1,000 with 20 corrupted measurements, and minimize the MOM objective for 1,500 iterations to recover the images. We compare the performance (measured in per-pixel reconstruction error) of LIP with the dense baselines in the first row of Table. Tand provide a visual example in Fig. Tickets with higher sparsities match the reconstruction performance of the dense model, confirming the existence of the winning tickets.
apply the IMP algorithm. In each IMP iteration, PGAN first fine-tuned
the IMP process because
Sparsity 0%
20%
36%
49%
59%
67%
74%
Random-CS 0.0725
0.0963
0.1165
0.1276
0.2184
0.2086
0.3655
LIP-CS
0.0725
0.0744
0.0732
0.0737
0.0711
0.0728
0,0728
Random-I 0.0541
0.0682
0.0748
0.08101
0.1142
0.1904
0.2195
LIP-I
0.0541
0.0542
0.0504
0.0514
0.0506
0.0524
0.0509
Table 1: Experimental results of GAN LIP. We evaluate the LIPs found in PGAN on the compressed sensing (CS) and the inpainting (I) tasks. The results are based on celebA-HQ dataset (Lee et al.) 2020). Note that use the MSE (per pixel) to evaluate the LIP effectiveness and compare the LIP with random pruning results.
Transfer to other image restoration tasks inpainting Besides the experiments on the com- pressed sensing restoration tasks, we also evaluate the effectiveness of GAN LIP on the inpainting task: masking the image and then optimize the input tensor of generator in the GAN LIP range to reconstruct the pristine image. More formally, consider the input tensor z' E RIx112 the pristine image æ sampled from celebA-HQ, inpainting mask (binary mask), masked image y = Az and generator G, then the optimization loss function is: ||AG(=) w112. The results are summarized in Table. and Fig. 10 demonstrating the transferability of GAN LIP.
CONCLUSION
In this paper, we successfully find the lottery image prior (LIP) via lottery ticket hypothesis and we have also empirically demonstrated the prevailing existence of LIP in image inverse problems such as denoising, inpainting. super-resolution and compressed sensing restoration. Specifically, show that subnetworks with high sparsity still retain the beneficial image prior properties in both settings. We also prove the powerful transferability of LIP across these tasks, reflecting its promising application potential.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Richard G Baraniuk, Volkan Cevher, Marco F Duarte, and Chinmay Hegde. Model-based compres- sive sensing, IEEE Transactions on information theory, 19842:198,-0001, 2010.
Marco Bevilacqua, Aline Roumy, Christine Guillemot, and Marie Line Alreri-Morel. Low- complexity single-image super-resolution based nonnegative neighbor embedding. 2012.
Ashish Bora, Ajil Jalal, Eric Price, and Alexandros G Dimakis. Compressed sensing using genera- tive models. In International Conference on Machine Learning, pp. 537-546. PMLR, 2017.
Ashish Bora, Eric Price, and Alexandros G Dimakis. Ambientgan: Generative models from lossy measurements. International Conference on Learning Representations, 2018.
Emmanuel J Candes, Justin K Romberg, and Terence Tao. Stable signal recovery from incomplete and inaccurate measurements. Communications Pure and Applied Mathematics: A Journal Issued by the Courant Institute of Mathematical Sciences, 19(8):1207-122) 2006.
Yang Cao, Yupin Luo, and Shiyuan Yang. Image denoising with gaussian mixture model. In 2008 Congress on Image and Signal Processing, volume 3. pp. 339 339-343. IEEE, 2008.
Tianlong Chen, Jonathan Frankle, Shiyu Chang, Sijia Liu, Yang Zhang, Zhangyang Wang, and Michael Carbin. The lottery ticket hypothesis for pre-trained bert networks. arXiv preprint arXiv:2007. 12223, 2020a.
Tianlong Chen, Zhenyu Zhang, Sijia Liu. Shiyu Chang, and Zhangyang Wang. Long live the lottery: The existence of winning tickets lifelong learning. In International Conference on Learning Representations, 2020b.
Tianlong Chen, Jonathan Frankle, Shiyu Chang, Sijia Liu, Yang Zhang, Michael Carbin, and Zhangyang Wang. The lottery tickets hypothesis for supervised and self-supervised pre-training in computer vision models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16306-16316, 2021a
Tianlong Chen, Yongduo Sui, Xuxi Chen, Aston Zhang, and Zhangyang Wang. A unified lottery ticket hypothesis for graph neural networks. In International Conference on Machine Learning. pp. 1695-1706. PMLR, 2021b.
Xuxi Chen, Zhenyu Zhang, Yongduo Sui, and Tianlong Chen. Gans play lottery tickets too. arXiv preprint arXiv:2106.0013: 2021c
Yun-Chun Chen, Chen Gao, Esther Robb, and Jia-Bin Huang. Nas-dip: Learning deep image prior with neural architecture search. In Computer Ecion-ECCC 2020: 16th European Conference, Glasgow, UK. August 23-28, 2020, Proceedings, Part XVIII 16, pp. 442-459. Springer, 2020c.
Zezhou Cheng, Matheus Gadelha, Subhransu Maji, and Daniel Sheldon. A bayesian perspective on the deep image prior. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5443-5451, 2019.
Justin Cosentino, Federico Zaiter, Dan Pei, and Jun Zhu. The search for sparse, robust neural networks. arXiv preprint 19/2.039/ 2019.
Kostadin Dabov, Alessandro Foi, Vladimir Katkovnik, and Karen Egiazarian. Image denoising by sparse 3-d transform-domain collaborative filtering. IEEE Transactions on image processing, 16 (8):2080-209) 2095, 2007.
Shrey Desai, Hongyuan Zhan, and Ahmed Aly. Evaluating lottery tickets under distributional shifts. arXiv preprint arXiv: 12708, 2019.
David I. Donoho. Compressed sensing. IEEE Transactions on information theory, 52(4): 1289- 1306, 2006.
Michael Elad and Michal Aharon. Image denoising via sparse and redundant representations over learned dictionaries. IEEE Transactions Image processing, 15(12):3736-3745 2006.
10

---

Under review as a conference paper at ICLR 2022
Utku Evci, Fabian Pedregosa, Aidan Gomez, and Erich Elsen. The difficulty of training sparse neural networks. arXiv preprint arXiv:1906.10732. 2019.
Simon Foucart and Holger Rauhut. An invitation to compressive sensing. In A mathematical intro- duction to compressive sensing, pp. 1-39. Springer, 2013.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprint arXiv: 1803.3363, 2018.
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M Roy, and Michael Carbin. Stabilizing the lottery ticket hypothesis. arXiv preprint arXiv: 01611, 2019.
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin. Linear mode con- nectivity and the lottery ticket hypothesis. In International Conference on Machine Learning, pp. 2259-3269 PMLR, 2020a.
Jonathan Frankle, David Schwab, and Ari S Morcos. The early phase of neural network training. arXiv preprint 200. 10365, 2020b.
Trevor Gale, Erich Elsen, and Sara Hooker. The state of sparsity in deep neural networks. arXiv preprint arXiv:1902.09574 2019.
Zhe Gan, Yen-Chun Chen, Linjie Li, Tianlong Chen, Yu Cheng, Shuohang Wang, and Jingjing Liu. Playing lottery tickets with vision and language. arXiv preprint wrXiv:210..11832, 2021.
Yosef Gandelsman, Assaf Shocher, and Michal Irani. "dubbleddip": Unsupervised image decompo- sition via coupled deep-image-priors In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11026-11035, 2019.
Kuang Gong, Ciprian Catana, Jinyi Qi, and Quanzheng Li. Pet image reconstruction using deep 2018.
image prior. IEEE transactions medical imaging,
Shi Guo, Zifei Yan, Kai Zhang, Wangmeng Zuo, and Lei Zhang. Toward convolutional blind de- noising of real photographs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1722-1722, 2019.
Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint 1rXiv:1510.0014 2015.
Paul Hand and Vladislav Voroninski. Global guarantees for enforcing deep generative priors by empirical risk. In Conference On Learning Theory, pp. 970-978. PMLR, 2018.
Paul Hand, Oscar Leong, and Vladislav Voroninski. Phase retricval under a generative prior. arXiv preprint arXiv: 1807.0:800.042250 2018
Wei He, Hongyan Zhang, Liangpei Zhang, and Huanfeng Shen. Hyperspectral image denoising via ndise-adjusted iterative low-rank matrix approximation. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 8(6):30503361,, 2015.
Reinhard Heckel and Paul Hand. Deep decoder: Concise image representations from untrained non-convolutional networks. arXiv preprint arXiv:1810.03992 2018
Reinhard Heckel and Paul Hand. Deep decoder: Concise image representations from untrained non-convolutional networks, 2019.
Ajil Jalal, Liu Liu, Alexandros G Dimakis, and Constantine Caramanis. Robust compressed sensing using generative models. Advances in Neural Information Processing Systems, 2020.
Kyong Hwan Jin, Harshit Gupta, Jerome Yerly, Matthias Stuber, and Michael Unser. Time- dependent deep image prior for dynamic mri. arXiv e-prints, pp. arXiv-1910, 2019.
Akshay Kamath, Sushrut Karmalkar, and Eric Price. Lower bounds for compressed sensing with generative models. arXiv preprint
2019.

---

Under review as a conference paper at ICLR 2022
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for im- proved quality, stability, and variation. arXiv preprint 1xX1:11110.1116,,2000 2017.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Yann LeCun, Corinna Cortes, and Chris Burges, Mnist handwritten digit database, 2010.
Christian Ledig, Lucas Theis, Ferenc Huszár, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang. et al. Photo-realistic sin- gle image super-resolution using a generative adversarial network. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4681 16990, 2017.
Cheng-Han Lee, Ziwei Liu, Lingyun Wu, and Ping Luo. Maskgan: Towards diverse and interactive facial image manipulation. In IEEE Conference on Computer Vision and Puttern Recognition (CVPR), 2020.
Namhoon Lee. Thalaiyasingam Ajanthan, and Philip HS Torr. Snip: Single-shot network pruning based on connection sensitivity. arXiv preprint arXiv: 2018.
Jaakko Lehtinen, Jacob Munkberg, Jon Hasselgren, Samuli Laine, Tero Karras, Miia Aittala, and Timo Aila. Noise2noise: Learning image restoration without clean data. arXiv preprint 18X3v:8003044: 2018.
Bee Lim, Sanghyun Son, Heewon Kim, Seungjun Nah, and Kyoung Mu Lee, Enhanced deep resid- ual networks for single image super-resolution. In Proceedings of the IEEE conference com- puter vision and pattern recognition workshops, pp. 136-144, 2017.
Zhaoqiang Liu and Jonathan Scarlett. Information-theorete lower bounds for compressive sensing with generative models. IEEE Journal on Selected Areas Information Theory, 1(1):292-303, 2020.
Zhuang Liu, Mingjic Sun, Tinghui Zhou, Gao Huang, and Trevor Darrell. Rethinking the value of network pruning. arXiv preprint arXiv:1810.05270 2018.
Haoyu Ma, Tianlong Chen, Ting- Kuei Hu, Chenyu You, Xiaohui Xie, and Zhangyang Wang. Good students play big lottery better. arXiv preprint arXiv:2101.03255 2021.
Indra Deep Mastan and Shanmuganathan Raman. Deil: Deep contextual internal learning for im- age restoration and image retargeting. In Proceedings of the IEEF/CVE Winter Conference on Applications of Computer Vision, pp. 2366-2375 2020.
Indra Deep Mastan and Shanmuganathan Raman. Deepefl: Deep contextual features learning from single image. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 2897-2906 2021.
Gary Mataev, Peyman Milanfar, and Michael Elad. Deepred: Deep image prior powered by red. In Proceedings of the IEEF/CVF International Conference on Computer Vision Workshops, pp. 0-0, 2019.
Ari S Morcos, Haonan Yu, Michela Paganini, and Yuandong Tian. One ticket to win them all: generalizing lottery ticket initializations across datasets and optimizers. arXiv preprint arXiv: 00006.02773.. 2019.
Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros. Context encoders: Feature learning by inpainting. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 2536-2544, 2016.
Alex Renda, Jonathan Frankle, and Michael Carbin. Comparing rewinding and fine-tuning in neural network pruning. arXiv preprint arXiv:2003. 02389, 2020.
Sylvain Sardy, Paul Tseng, and Andrew Bruce. Robust wavelet denoising. IEEE Transactions on Signal Processing, 49(6): 146-1161116 152, 2001.
12

---

Under review as a conference paper at ICLR 2022
Pedro Savarese, Hugo Silva, and Michael Maire. Winning the lottery with continuous sparsification. arXiv preprint arXiv: 2019.
Carlo Tomasi and Roberto Manduchi. Bilateral filtering for gray and color images. In Sixth interna- tional conference on computer vision (IEEE Cat. No. 98C1136271), pp. 839-846. IEEE, 1998.
Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Deep image prior. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 9446-9458 2018.
Chaoqi Wang. Guodong Zhang, and Roger Grosse. Picking winning tickets before training by preserving gradient flow. arXiv preprint arXiv:2002.07331 2020.
Fangshu Yang, Thanh-An Pham, Nathalie Brandenberg, Matthias P Lütolf, Jianwei Ma, and Michael Unser. Robust phase unwrapping via deep image prior for quantitative phase imaging. IEEE Transactions on Image Processing. 30:7025-7037, 2021.
Shihui Yin, Kyu-Hyoun Kim, Jinwook Oh, Naigang Wang, Mauricio Serrano, Jae-Sun Seo, and Jungwook Choi. The sooner the better: Investigating structure of early winning lottery tickets. 2019.
Haoran You, Chaojian Li, Pengfei Xu, Yonggan Fu, Yue Wang, Xiaohan Chen, Richard G Baraniuk, Zhangyang Wang, and Yingyan Lin. Drawing early- bird tickets: Towards more efficient training of deep networks. arXiv preprint arXiv:
2019.
Haonan Yu, Sergey Edunov, Yuandong Tian, and Ari S Morcos. Playing the lottery with rewards and multiple languages: lottery tickets in rl and nlp. arXiv preprint (606:1900007666 2019a.
Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas S Huang. Generative image inpainting with contextual attention. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5505-5514, 2018.
Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas S HHang. Free-form image inpainting with gated convolution. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4471-4480, 2019b.
Roman Zeyde. Michael Elad, and Matan Protter. On single image scale-up using sparse- representations. In International conference on curves and surfaces, pp. 711-730. Springer, 2010.
Kai Zhang, Wangmeng Zuo, Yunjin Chen, Deyu Meng, and Zhang. Beyond gaussian denoiser: Residual learning of deep enn for image denoising. IEEE transactions on image processing, 26 (7):3142-3155, 2017.
Yulun Zhang, Yapeng Tian, Yu Kong, Bineng Zhong, and Yun Fu. Residual dense network for image super-resolution. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2472-2481, 2018
13

---

Under review as a conference paper at ICLR 2022
A SUPPLEMENTARY MATERIALS
In this section, will present more experimental results of LIP subnetworks (e.g., the layer-wish sparsity ratio, discussions about the training targets, learning curves of different training targets, etc.) to better understand the Lottery Image Prior.
Images Used in Experiments
Fig. 14 we organize and present the images used this paper with their names. Note that these images sampled from Set5 (Bevilacqua et al. 20122) and Set14 Zeyde et al. 2010) datasets, and we use their default names.
Algorithm To begin with, there exist two ways of finding LIP: 1) Single-image based IMP. 2) Multi-image based IMP (weight-sharing). We have formulated the method in Algorithm. [ and 2] respectively. The major difference lies in the loss function of IMP training. Specifically, for the denoising task, the loss function of single-image IMP is (z; @ m), r), where Z is randomly initialized tensor, is the groundtruth image and E(;; represents the MSE distance. In parallel, the weight-sharing IMP loss function is Σ0= ((f 0 @ m), za); a E [1, n). where n represents the 5.). Note that method to find the LIP with the powerful transferability on images from different domains and use the single-image IMP on the images from the same domains (such as human face images).
number of shared weight images (e.g., in Set5 dataset,
the weight-sharing
Parameter Redundancy Problem We discover that the com- monly used hourglass model in DIP (Ulyanov et al. 2018) is high parameter complex. The statistic results of parameter numbers is summarized in Table. 2 We compare the parameter numbers of dense DIP models with the found winning tickets and surpris- ingly discover that the winning tickets could perform better than the full model while containing 2 million parameters fewer. This phenomenon motivates us to suspect that there is high possibil- ity of finding the matching subnetworks of the pristine dense DIP model, which indicates that the subnetworks may also contain the outstanding image prior property as the dense one does.
Dense Tickets
PSNR
30.35
30.61
Parameters (non-zero) 2.2M
0.2M
Table 2: The comparison of pa- rameter numbers full model and the winning tickets. Note that we evaluate the PSNR val- ues on the image Bird.
Training Target of Finding LIP When making the IMP training loop, one question arises: How to it- eratively prune the model while maintaining the im- age prior property in DNNs? In this paper, we pro- pose to set the training label of the DIP model to clean image Since the DIP model restore the cor- rupted image (e.g., the noisy image, masked image and low-resolution image) by training itself with the corrupted target ã, and the best restoration image usu- ally occurs some points of the process not always the training ending (DIP model training often requires the researcher design the training settings by hand). Therefore, if we set the IMP training target to ã, the model parameters may overfit the corrupted image, which will degrade the desired image prior prop- erty. In Table. 3 we compare the ticket performances between the noisy target IMP trained (Noisy M) and clean target IMP trained (Clean M) at the same sparsity s 50%. This experiment is done on the Baby image and the background task is denoising. In order to test the effect of overfitting, we set the DIP training epochs to 6000 (3000 is usually used for the Baby image). We observe that the Clean M reaches 0.45 higher PSNR value than Noisy M and the Clean M performance is higher than the Dense model, which demonstrates the importance of choosing the training target during IMP.
Noisy M
Clean M
Dense
(s = 50%)
(s = 50%)
PSNR 31.06
30.76
31.21
Table 3: Comparison results of different training targets. We train the DIP model on one image (Baby.png) for 6000 epochs and separately set the training target to (clean target) and i (noisy target). Then we evaluate the masks on the same image for 3000-epoch denoising task.
However, the above observations do not indicate that our proposed method is applicable only to the ground- truth image settings. We used clean images as the training targets since this made it easy to control the learning curves and avoid overfitting. Even using one clean image target, the ob- tained LIP is practical due to the powerful transferability to other image with no clean groundtruth. Moreover, we can use the noisy target in finding the same effective LIPSs as long we perform
14

---

Under review as a conference paper at ICLR 2022
"early-stopping" in the optimization process. Note that it was the same trick mentioned when train- ing the original DIP for denoising tasks (Ulyanov et al. 2018) and was identified to be essential to the original DIP's success in (Heckel & Hand 2019). In Table. I we conduct the comparison experiments of clean and noisy image targets (experimental settings are summarized in the caption of the table) and we find that there no large differences between the results of noisy image targets and clean image targets.
ShaP
SAEP
Randion
Reskors
WWW
LP
SNP
SHBP
Faaddoo
(a) Sparsity-3655 36%%
(b) Sparsity-59r, 55%
(c) Sparsity-899 899%
(d) Sparsity- 95%
Figure 11: Layer-wise sparsity ratio results of LIP, SNIP and randomly pruned tickets. Note that summarize the sparsity ratio of each layer: the ratio of the number of parameters whose values are equal to zero to the number of total parameters of the layer. And the x-axis of these figures composed of the serial numbers of model layers. We sampled subnetworks with four different sparsities (sparsity = 36%, 59%, 89%, 95%) to observe.
Closer Look into the Lottery Image Prior Structure As shown in Fig. IT the structure the LIP subnetwork is drastically different from those found by SNIP and random pruning, in particular the distribution of layerwise sparsity ratios. LIP tends to preserve weights of the carlier layers (closer to the input), while pruning the latter layers more aggressively (e.g, Fig. II(a)). In contrast, SNIP tends to prune much more of the earlier layers compared to the latter ones. Random pruning by default prunes each layer at approximately the same ratio.
Comparing the three methods seem to suggest that for finding effective and transferable LIP subnet- works, to specifically keep more weights earlier layers more important. That is explainable finding, since for image restoration tasks, the low-level features (color, texture, shape, etc.) presum- ably matter more and are more transferable, than the high-level features (object categories, etc.). The earlier layers are known to capture more of low-level image features, hence contributing more to retraining the image restoration performance with DIP.
0.
03
UP SUP Fondon
LL
08
SNP
SHP
12
Randon
Fandon
05-
Rondan
1000
(a) PSNR-S 89%
(b) PSNR-S = 95%
(c) SSIM-S = 89%
(d) SSIM-S = 95%
The learning curve plots when using different subnetworks towards the DIP task. In the figure, S denotes the sparsity of the model. We compare both PSNR and SSIM values. For fair comparisons. we trained these subnetworks on the denoising task on the Baby image with 3000 iterations. then trained isolation (the iteration number recommended by (Chen al. 2020c) to capture the "carly-stopping" phenomenon of DIP), and summarized their performances.
Figure 12:
Learning Curve Comparison of Using Various Subnetworks for the Restoration Task We further compare the training convergence curves of different subnetworks on the restoration task. In Fig. 12 we summarize the convergence of LIP, SNIP and randomly pruned subnetworks on the denoising task, and the experimental details are included in the caption. We use the PSNR and SSIM metrics to measure the quality of the generated images: SSIM is often considered better "perceptually aligned",, by attending more the contrast in high-frequency regions than PSNR.
15

---

Under review as a conference paper at ICLR 2022
At the early stage of optimization, observe that the learning curves of LIP and SNIP subnetworks are almost overlapped (either PSNR or SSIM curves), while the randomly pruned subnetworks failed to perform comparably with them. Yet when the iterations increase, the SNIP subnetworks start to lag behind the LIP subnetworks (e.g., the largest PSNR gap between the two can reach 3dB and the largest SSIM gap can be 0.7). Only the LIP subnetworks can match the comparable performances of the full model when reaching the 3000-th iteration. Lastly, the SSIM gap is noticeably enlarged higher sparsity levels (95%) when comparing LIP and SNIP, which implies LIP to be better at capturing perceptually aligned details.
Irage
Image
tmage
600
shuffee
3,000
9 cc0
firste
Ug
image
206
Claan Irage
mole
1.00
200
200
Seatick
8006
1906
2006
(a) LIP
(b) Random Pruning
(c) SNIP
(d) Dense
Figure 13: Learning curves using four different training targets: clean image (Baby.png) the same image added with noises, the same randomly scrambled and white noise. Note that we use four different models: the LIP subnetwork (S 89%), randomly pruned subnetwork (S 89%), SNIP subnetwork (S = 89%) and the dense model (S = 0%). And we trained them in isolation in the same experimental settings for 10000 iterations.
Learning Curves of Subnetworks with Different Training Targets To better explain the success of the LIP subnetworks, inspired by the Figure 2 of (Ulyanov et al. (018), we further train the obtained subnetworks (LIP, SNIP and random pruning) with four different targets: 1) natural image, 2) the same added with noise, 3) the same after randomly permuting the pixels and 4) the white noise, We also train the dense model as the baseline results. The experimental details are summarized in the caption of Fig. 13 We first observe that for LIP, SNIP and dense models, the optimization converges much faster in case 1) and 2) than case 3) and 4). But the randomly pruned subnetworks have failed in all cases. Interestingly, perform similarly with the dense model. Mcanwhile, the parametrization of LIP subnetworks offers the higher impedance noise and the lower impedance to signal than the dense model, which indicates that the separation of high frequency and low frequency is more obvious for winning network architectures. Also, in order to observe whether the high-frequency information will be lost during the pruning, we apply Fourier Transformation to the Baby figure (described in Fig. (4] and visualize the frequency intensity of the ground-truse -ruth image and the reconstructions from three different subnetworks (LIP, SNIP and random pruning). The results are summarized in Fig. [5 We found that compared with random pruning, LIP and SNIP can maintain most of the high frequency information in the ground-truth (e.g., in Fig 15, SNIP and LIP can both maintain the high-frequency information the sparsity of 79%; however, SNIP could lose more high-frequency information than LIP at lower sparsity ratios.).
also find that SNIP subnetworks
Sparsity %)
0
36
59
67
79
89
95
Clean Image
32.60
32.51
32.45
32.40
32.21
32.10
31.99
Noisy Image
32.60
32.56
32.46
32.41
32.22
32.13
31.94
Table 4: We compare the results of clean and noisy image targets in setting i. Note that the used image is F16.png, the evaluation metric is PSNR and the training iteration number is 3000 to capture the "carly-stopping" phenomenon. The results suggest that there are no large differences (with PSNR value smaller than 0.05) between the performances of subnetworks with clean and noisy image targets (the sparsity ranges from 0% to 95%).
16

---

Under review as a conference paper at ICLR 2022
Face 1
Face 2
Face 3
Face 4
Face
Bird
Baby
Butterfly
Woman
Head
Vase
Kate
F16
Baboon
Pepper
Figure 14: Images used in plotting the curves of experiments.
Algorithm 1: Single Image-based IMP
Input:
The desired sparsity s, the random code as the model input, the untrained model fx and the image x.
Output: A sparse DIP model {2; @ m) with image prior property
Set m. = 1 RIl") cnnd 0g are the initial weights of the model fa. Iteration = 0, training epochs N j E [0, N]:
while the sparsity of m, do
Train the fu (2; 0p & m.) with the objective ((f(2; @ m);x) for N epochs to reach the parameter 0x
Create the mask ml,:
Update the mask m. -m,;;
Set the model parameters to 0y: ((2;0,)) create the sparse model: f(2;0 02 ④ m..);
i i + 1;
10 end
17

---

Under review as a conference paper at ICLR 2022
Algorithm 2: Weight-sharing IMP with Various Domain Images
Input: The desired sparsity 8, the random code z as the model input, the untrained model fu and images from domains
{21, (21,222..... F...
Output: A sparse DIP model f(z; O m) with image prior property
Set my = 1 RIl0 annd 0g are the initial weights of the model fa. Iteration 0, training epochs N and j E [O,
while the sparsity of mh < 8 do
loss - a= [2; @ ma) E [1;
Train the (z; 0o @ m.) ) by Backpropagation (loss) for N epochs to reach the parameter
0'x:
Create the mask ml and update the mask my m'y
Set the model parameters (2;0;);
create the sparse model [(2; ( mu);
= + 1;
10 end
Ground-truit
LIP S = 59%
LIP S = 67%
LIP S = 79%
Ground-trite truth
SNIP 59%
SNIP = 67%
SNIP S = 79%
Ground-truth
Random S 59%
Random 67%
Random S 79%
Figure 15: Evaluating the reconstruction images of different subnetworks (LIP, SNIP and random pruning) by FFT (Fast Fourier Transformation) to check whether the high frequency information has been lost during pruning. Note that we experimented on the Baby.png. We found that com- pared with random pruning, LIP and SNIP can both maintain the high frequency information ground-truth. For example, the LIP and SNIP subnetworks both maintain mostof the high- high-frequency information of the ground-trutt at the sparsity 79%, but the LIP could also performs well the spar- sity 59% where the SNIP loses more figh-freeuency information.
18


Reviews:
[{"review_id": "pOVmujQ3qK", "reviewer": "Reviewer_wMsj", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "As I have mentioned in the above summary, the conclusion (some networks are more suitable to be used as deep image prior than others) is very interesting. It seems that the author only discussed the existence of these networks, but did not discuss the possible reasons why these structures is superior to other structures in the deep image prior task. Is this because in these networks, the important information (such as low-frequency parts, etc.) of the image used for recovery is easier to learn from the obtained network structure, while the high-frequency, harmful information is more difficult to capture by these networks? The lack of this part of the discussion is regrettable. It is possible that for DIP, the preferred network structure is different from discriminative tasks and GANs. I hope the author can introduce relevant discussions. If possible, some experiments can also be conducted. For example, maybe the convergence curves of DIP are different when different networks are used, and the separation of high frequency and low frequency is more obvious for winning network architectures?\n\nAnother regret is that GAN compressed sensing and DIP seem to be separated. It is not clear whether the cause of the effect on GAN CS and the result on DIP are the same. It is very likely that the reason for the effective pruning on GAN CS is different from DIP. Maybe the former is because some knowledge in GAN has no or negative influence on the CS task, while the latter is because different structures express different components of the image differently. After reading this paper, I still lack knowledge of these issues.\n\nAnother minor problem is that this paper is difficult to follow. The writing needs improvement. Especially in terms of specific methods, I think it is difficult to quickly understand the author's approach. Moreover, the practices of GAN CS and DIP are very different, and it is difficult to put them together clearly. ", "summary_of_the_paper": "This paper researches the lottery ticket hypothesis for networks as a deep image prior or deep generative prior. The specific approach is to (1) train deep networks to reconstruct multiple images for DIP (Ticket ﬁnding objectives), (2) conduct iterative magnitude pruning to the trained network, (3) obtain the pruned mask and reset the model parameter to initialization weight and (4) perform deep image prior to new (or training) images. \nThe proposed objective is an important design for the system to work, i.e., the author optimizes the network by minimizing the expected error on multiple images.\nThe experimental results are somehow interesting that (1) different pruning methods actually performs differently and (2) there are winning tickets in the studied problem. As for GAN compressive sensing task, there are also winning tickets that exists. The point I am more interested in is that some networks are more suitable to be used as deep image prior than others. ", "summary_of_the_review": "Overall, this is an interesting paper. This paper reveals some interesting results, but there are still some questions (see above). If the author can discuss these issues and improve writing, the paper can be accepted.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "7i69Xnj7fZk", "reviewer": "Reviewer_Kjf3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Positives\n\n+ The paper addresses an important problem of image restoration using deep prior which has been quite effective in the absence of labelled examples. \n\n+ The work deserves merit due to the novel application of the lottery ticket hypothesis to this problem and has been well presented.\n\n+ The paper conveys several new ideas and is shown to induce sparsity on the network while achieving impressive results.\n\n+ The paper is well written with several questions justified and analysed thoroughly.\n\nConcerns\n\n- The paper does have a lot of typos such as the line - \"that can be training in isolation\" in two different places.\n\n- Experimental evaluation and comparison for deep image prior tasks lacking rigour.\n\n- The paper does not cite several works on line of the deep image prior such as those listed below. they have not been compared with.\n\n1. Gandelsman, Y., Shocher, A., & Irani, M. (2019). \" Double-DIP\": Unsupervised Image Decomposition via Coupled Deep-Image-Priors. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 11026-11035).\n2. I. D. Mastan and S. Raman, \"DCIL: Deep Contextual Internal Learning for Image Restoration and Image Retargeting,\" 2020 IEEE Winter Conference on Applications of Computer Vision (WACV), 2020, pp. 2355-2364\n3. Mastan, I. D., & Raman, S. (2021). DeepCFL: Deep contextual features learning from a single image. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (pp. 2897-2906).\n\n", "summary_of_the_paper": "The paper presents the application of the Lottery Ticket Hypothesis (LTH) to the problem of the deep image prior in order to solve image restoration tasks. The paper shows the effectiveness of such a lottery image prior which trains subnets in isolation.", "summary_of_the_review": "The paper has several positives, the concerns include missing references and comparisons. Hence, it is just below acceptance threshold.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "5uXXjikU6hL", "reviewer": "Reviewer_vf8k", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "My questions mainly concentrate on the setting 1 of the paper, i.e., the DIP related tasks.\n\n1. This work shows that there exists a very sparse subnetwork whose performance can surpass the original dense network. To find such sparse network, it directly minimizes the MSE relying on the ground-truth image. In practice, however, the ground-truth image is always not accessible. How to obtain such sparse subnetwork in real applications?\n\n2. In my opinion, the goal of the research on LIP is to simplify the DNN architecture in DIP based on the pre-known knowledge on the sparse subnetwork. For example, given a noisy image without the latent clean image, if we know the sparse subnetwork in advance, we can directly optimize it to obtain the denoising result under DIP.  How to achieve this goal? If LIP cannot embed into current DIP related works like this, it is useless in practice.\n\n3. LIP is implemented by the network weight pruning, and all the experiments depends on a current existing weight pruning method. I think  the contribution of this paper is incremental and not enough.\n\n4. The mathematic notions in algorithm1 and algorithm2 are confused. In algorithm 1, what does $f_{\\mu}(x;\\theta_0 \\odot m_{\\mu})$ mean? In DIP, the network $f$ takes the latent code $z$ as input instead of the image $x$. As for the loss function of algorithm 2, whom does $E(f(x_a;\\theta \\odot m))$ denotes the MSE distance between the network output $f(x_a;\\theta \\odot m)$ and?", "summary_of_the_paper": "This paper exploits the LTH in the low-level vision tasks, including image restoration and compressive sensing image reconstruction. Through some experiments, it identifies the existence of LIP in low-level vision and also gives some interesting conclusions on the transferability of the proposed LIP.", "summary_of_the_review": "Even though this paper is interesting, I still argue the usefulness of the proposed LIP. This paper only attempts to do some exploration based current existing LTH related works, which weakens  the novelties and the contributions of this work.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "5FDvke7U-yx", "reviewer": "Reviewer_jFko", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality\nThis work studies the LTH for low-sample deep inverse imaging for the first time\n\nSignificance\nStrong points\n- for the first time studies LTH for deep image prior and pre-trained deep generative compressed sensing\n- provides empirical results for several low-level image restoration (as well as image classification) tasks and datasets\n- the results are interesting suggesting LTH exists with high degree of sparsity that not only makes the models much smaller but also improves the generalization performance as a result of sparsity regularization\n- the results also suggest transferability of LIP subnetworks across data and image restoration tasks\n\nWeak points\n- The paper presentation is poor. There are several typos throughout the text. The organization is poor and confusing. There are several scenarios studied in this paper with regards to the datasets and tasks and settings, but the link among them is missing and confusing from the text. \n- This work is purely empirical study and it's not clear for example why LIP can transfer across image restoration tasks, and dataset? Providing insights would be very useful. It is also not quite clear how the LIP pruned for denoising is transferred to image classification. Are only the intermediate representations transferred as in auto-encoders?\n\n\nMore comments and suggestions\n- In the modified IMP, when using clean labels for training, where is the randomness coming from? How is the expectation defined?\n- The presentation of the paper can benefit from more visuals such as diagrams/graphs to connect the tasks and datasets. \n- The experiments only report PSNR. It would be better to report SSIM and FID to see if the pruning can hurt the high frequency perceptual details or not? This is more important for the pretrained GAN model. \n- It is not clear how the sparsity pattern looks like for the considered hour-glass network. Isn’t this one of the important points of pruning as advocated in the introduction? A visualization of the sparsity pattern per network layers would be very insightful. It would also be helpful to see if pruning creates a bottleneck in the network and where that bottleneck is. \n", "summary_of_the_paper": "Summary\nThis submission studies lottery ticket hypothesis for deep neural network based inverse imaging. It considers two scenarios: 1-inference for compressed sensing based on pre-trained deep generative models, 2-deep image prior where all network parameters are fit to a single image. Both scenarios are dealing with a single image training task, or, an overfitting task, which makes this work different from the past work using LTH. The empirical results suggest that with a high level of sparsity not only the model size can be made much smaller, but also the generalization can improve which suggests pruning as a regularization. They also suggest good transferability from one image restoration task to another.\n\n", "summary_of_the_review": "This is empirical work studying an important problem. The results are also interesting which suggest a high degree of sparsity for certain learable inverse imaging problems, that closes the gap with the traditional sparsity based inverse imaging algorithms. The presentation of the paper however needs significant improvement to connect the conclusions toward a coherent story. \n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper studies the lottery ticket hypothesis in the context of deep image priors. Deep image priors are convolutional neural networks that are imposed as a prior for image reconstruction problems. A deep image prior can be an un-trained convolutional network, or it can be a trained generator, and the paper considers both types of priors. Deep image priors are often highly over-parameterized and thus the paper under review asks the question on whether the networks really have to be heavily parameterized or whether a small subnetwork will also do. The paper performs experiments on image restoration with an entirely un-trained DNN (this constitutes the larges part of the paper) and on image restoration with a pre-trained network. 

The paper received four reviews out of which three recommend weak acceptance and one strong acceptance.
- Reviewer wMsj finds it interesting that the paper shows that some networks are more suitable as deep image priors than others, but finds that the paper lacks evidence on why some structures are better than others.
- Reviewer Kjf3 strongly recommends acceptance (8) in a relatively generic review. The reviewer finds the paper is interesting as it provides a novel application of the lottery ticket hypothesis. However, the review also criticizes that the experimental evaluating lacks rigor.
- Reviewer jFko appreciates that the paper studies the lottery ticket hypothesis for the first time for un-trained and pre-trained image prior, and that the results are interesting as they suggest that the models can be made smaller and that this can even improve performance. The reviewer criticizes that the paper's presentation is confusing, and points out a few weaknesses in the empirical evaluation. The authors responded and after a brief discussion, the reviewer raised their score. 
- Reviewer vf8k provides a relatively brief review and criticizes that to find sparse networks, one requires the ground truth image, which is not accessible. The authors clarify that the method is transferable in that the network identified can be used for other images and is thus transferable. The reviewer was satisfied with this response. 

The score of this paper 6.5 after the discussion period. Three of the reviewers are on the fence about the paper, one reviewer is not, and that reviewer significantly impacted the score. This reviewer, however, did not provide convincing arguments about the merits of the paper. All reviewers find that `3: Some of the paper’s claims have minor issues.', and I agree with that statement.

I do not recommend acceptance of the paper in its current form, because of insufficiently rigorous experiments to justify the claims:
- Specifically, the paper's goal is to address the research question 'do they [neural-network based priors for image reconstruction] really have to be heavily parameterized?' The literature already answers this question since as the paper under review reads on page 3, the literature found that an under-parameterized non-convolutional model can function as an un-trained image prior. Those underparameterized networks perform well and have fewer parameters than the best-performing networks found in the paper under review.
- The paper argues that a sparse network can give better performance. This claim is based on an at most 0.1dB difference, which can only be achieved when choosing the optimal sparsity level, which is not clear to do without knowing the ground truth image.
Decision: Reject