Chasing Better Deep Image Priors between Over- and Under-parameterization
Authors: Qiming Wu
Xiaohan Chen
Yifan Jiang
Zhangyang Wang
Venue: Accepted by TMLR ()

Abstract: Deep Neural Networks (DNNs) are well-known to act as \textbf{over-parameterized} deep image priors (DIP) that regularize various image inverse problems. Meanwhile, researchers also proposed extremely compact, \textbf{under-parameterized} image priors (e.g., deep decoder) that are strikingly competent for image restoration too, despite a loss of accuracy. These two extremes push us to think whether there exists a better solution in the middle: \textit{between over- and under-parameterized image priors, can one identify ``intermediate" parameterized image priors that achieve better trade-offs between performance, efficiency, and even preserving strong transferability?} Drawing inspirations from the lottery ticket hypothesis (LTH), we conjecture and study a novel ``lottery image prior" (\textbf{LIP}) by exploiting DNN inherent sparsity, stated as: \textit{given an over-parameterized DNN-based image prior, it will contain a sparse subnetwork that can be trained in isolation, to match the original DNN's performance when being applied as a prior to various image inverse problems}. Our results validate the superiority of LIPs:  we can successfully locate the LIP subnetworks from over-parameterized DIPs at substantial sparsity ranges. Those LIP subnetworks significantly outperform deep decoders under comparably compact model sizes (by often fully preserving the effectiveness of their over-parameterized counterparts), and they also possess high transferability across different images as well as restoration task types. Besides, we also extend LIP to compressive sensing image reconstruction, where a \textit{pre-trained} GAN generator is used as the prior (in contrast to \textit{untrained} DIP or deep decoder), and confirm its validity in this setting too.  To our best knowledge, this is the first time that LTH is demonstrated to be relevant in the context of inverse problems or image priors. Codes are available at https://github.com/VITA-Group/Chasing-Better-DIPs.

Yifan Jiang Published in Transactions on Machine Learning Research (08/2023)
Chasing Better Deep
Image Priors
between
Over- and
Under-paramettriiation
Qiming Wu
gimingundcs. uesb.
University of California, Santo. Barbara
Xiaohan Chen
Siachan.chen@aalllie
Decision Intelligence Lab, Damo Academy, Alibaba Group (U.S.)
y
University of Texos at Austin
Zhangyang Wang
ataswang@uueras.ccm edu
University of Teros Austón
Reviewed on OpenReview: https:// openreviee. net/ forum? id- BuJJks2cSa
Abstract
Deep Neural Networks (DNNs) are well-known to act as over-parameterizet deep image priors (DIP) that regularize various image inverse problems. Meanwhile, researchers also proposed extremely compact, under-parameterized image priors (e.g,, deep decoder) that are strikingly competent for image restoration too, despite loss accuracy. These two extremes push us to think whether there exists a better solution in the middle: between over- and mnder-ppramelerizel image priors, con one identify "intermediale" porameter- ized image priors that achieve better trade-offs between performance, efficiency, and even preserving strong transferability? Drawing inspirations from the lottery ticket hypothesis (LTH), we conjecture and study a novel "lottery image prior" (LIP) by exploiting DNN inherent sparsity, stated as: given an over-paramelerized DNN-based image prior, il will contain a sparse subnetwork that can be trained in isolation, to match the original DNN's performance when being applied as prior various image inverse problems. Our results validate the superiority LIPs: we can successfully locate the LIP subnetworks from over- parameterized DIPs at substantial sparsity ranges. Those LIP subnetworks significantly outperform deep decoders under comparably compact model sizes (by often fully preserving the effectiveness of their over preepaaameterized counterparts), and they also possess high trans- ferability across different images as well as restoration task types. Besides, we also extend LIP to compressive sensing image reconstruction, where a pre-trained GAN generator is used as the prior (in contrast to untrained DIP or deep decoder), and confirm its validity in this setting too. To our best knowledge, this the first time that LTH is demonstrated to be relevant in the context of inverse problems image priors. Codes available at Better-DIPs
http:://ggg.tttb.o.oo/////-oorr DI
1 Introduction
Deep neural networks (DNNs) have boen powerful tools for solving various image inverse problems such denoising Zhang et. al. (2017) Guo el al. 22019 Lehtinen et al. (2018); Jiang et al. (2022), inpainting Pathak
et al.
2016); Yu et al. (2018 2019b). and super-resolution Ledig et al. (0017): Lim et al. (2017) Zhang 2018 Conventional wisdom believes that is owing DNNs universal approximation ability and
et al.
learning from massive training data. However, a recent study Ulyanov al. (0018) discovered that degraded
images be restored independently using randomly initialized and untrained convolutional neural networks

---

Published in Transactions on Machine Learning Research (08/2023)
(CNNs) without the supervised training phase, which is called Deep Image Prior (DIP). A series of works Cheng et al. (2019); Gandelsman et. al. (2019) have been proposed to improve CNN-based DIPs, which indicates that specific architectures of CNNs have the inductive bias to represent and generate natural images well.
Despite the advantageous performance, most DIP methods use highly over- weerparamettrized CNNs with a massive number of parameters (we show this in Table g in appendices). Over- vver-prrameterization causes computational inefficiency and overfitting (and thus proneness) to noises. This trend has naturally invited the curious question: does DIP have to be heavily parameterized? That question is partially answered by Heckel & Hand (2018) by proposing the first under-parameterizedd non-convolutional neural network for DIP named "deep decoder". Deep decoder shares across all pixels a linear combination over feature channels in each layer and thus has an extremely compact parameterization. Thanks to its under- parammttrtttii,,nnnnnn deep decoder alleviates the "noise overfitting noise' in DIP. However, the empirical performance of deep decoder is often not on par with overparameterized DIP models, especially on tasks like inpainting and super-resolution. This is potentially altributed to the extremely restricted capacity of deep decoder from the very beginning. The dissatisfaction on both extremes, that is using either over-parameterized CNN DIPs or ander-paramcterized deep decoders, pushes us to think if there is a better "middle ground": Between over- and under-pone parameterized image priors, can one identify "intermediate" parameterization better trade-offs between performance, efficiency, as well as even preserving strong transferability?
In this work, we adopt sparsity as our main tool for the above question. We seek more compactly pa- rameterized subnetworks by pruning superfluous parameters from the dense over- wee-paraaettriized CNN DIPs (overview of our work paradigm is shown in Figure D essentially viewing pruning as a way to smoothly and flexibly "interpolate' between over- and audder-parameterizatoon We have several motivations for choosing sparsity and pruning from preeparramttiiiitionn. On one hand, compared with dense and overparameter- ized DIP models, sparsity saves computations during inference and DIP fitting. Moreover, sparsity can serve as an effective prior that regularizes DNNs to have better robustness to noise Chen ct al. (202)), that is important for DIP which is tasked to distinguish image content. from noise. These two blessings combined make sparsity a promising "win-win" for not only DIP efficiency but also performance. On the other hand, unlike deep decoder which sticks to a compact design, exploiting sparsity follows a different "first redundant then compact" training route, which is widely found to enhance performance compared to training com- pact model directly from scratch Zhou et al. (2020). Since overparameterized (especially wide) DNNs have smoother loss surfaces while smaller ones have more rugged landscapes, starting from overparameterizztiin can case the training difficulty Saffran et al. 2021 and may particularly help the "chaotic" carly stage of training Frankle et al. 20000c
The recently emerged Lottery Ticket Hypothesis (LTH) Frankle & Carbin (0018): Frankle et al. (2020a)
suggests that every dense DNN has extremely sparse "matching subnetwork", that can be trained in isolation to match the original dense DNN's accuracy. While the vanilla LTH studies training from random scratch, the latest works also extend similar findings to fine-tune the pre-trained models Chen et al. 20200a
2021b). LTH has widespread success in image classification, language modeling, reinforcement learning and multi-modal learning, e.g., Yu et al. (2019a): Renda et al. (2020) Chen et al. (2020a)) Gan et al. (022 Drawing inspirations from the LTH literature, we conjecture and empirically study a novel "lottery image prior' (LIP), stated as:
Given on (untrained trained) over-parameterized DIP, it will have a sparse subnetwork that can be trained in isolation, to match the original DIP's performance when being applied as a prior to regularizing various image inverse problems. Moreover, ils performance shall surpass under- parameterized priors of similar parameter counts.
Diving into this question has two-fold appeals. On the algorithmic side, as the first attempt to investigate LTH in the DIP scenario, il could help us understand how the topology and connectivity of CNN archilectures encode natural image priors, and whether "overparameterization + sparsity' make the best DIP recipe. On the practical side, the affirmative answer to this question can lead to finding compact DIP models that are more performant than the deep decoder, hence yielding more computationally efficient solutions for image inverse problems without sacrificing performance.
2

---

Published in Transactions on Machine Learning Research (08/2023)
Over- Over-parameterizel Image Priors
Lottery Image Priors
Downsampling
Upsampling
Downsampling
Upsampling
Prune
Inference
Skip Connections
Skip Connections
Redundant parameters;
Overfit noises
Denoising
Inpainting
Under-parameterized Image Priors
Super Resolution
Compressed Sensing
B1
B2
Bd
Loss of accuracy
k,
y = Ax' T
ka
Figure 1: Overview of our work. Between the over- and under-pareee ander-parameterized image priors, aim to find the sparse matching networks with better performances, efficiency and strong transferability.
1.1 Summary of Contributions
We now state our intended contributions from two angles: how our work might affect the deep image prior research (application), and the lottery ticket hypothesis research (methodology). To avoid misunderstand- ings, we want to state again: efficiency is definitely not the key target. motivation. Our key target instead is to verify the existence of 'compact' DIP model that is free from severe overfitting but at the time different from the deep decoder. We want it partially because the deep decoder is bad in terms of performance (e.g., impainting and super-resolution.. Efficiency is also a natural byproduct that results from pruning and sparsity. Improving efficiency could reduce the single-image inference time of DIP to accelerate restoration and reconstruction, which could be helpful latency applications.
For DIP research community, In between over- and under-parane padde--parmeterieeed image prior models, aim to use sparsity to find an internediate parameterized network. Specifically, compared to over-parameterized CNN DIPs:
We successfully locate the "matching subnetworks' from over-parameterized DIP models by itera- live magnitude pruming, Their prevailing existence demonstrates that sparsity, arguably the most classical natural image prior, remains relevant a component. in DIPs.
Beyond "matching', our found LIP subnetworks can even outperform DIP models on several image restoration tasks (shown Figure 2), which is line with the previous findings that the sparsity can enhance DNN robustness to noisc and dogradations Chen et al. (2022).
Also, the sparse LIP subnetworks are found with powerful transfcrability across both test images and restoration tasks. Importantly, that amortizes the extra cost of finding the subnetwork per DIP network, by extensively reusing the found sparse mask. It is also neatly aligned with the recent study of LTH transferability Redman et al. (2022).
Mcanwhile, regarding under-parameteriien image priors such as the deep decoder:
AA matching subnetwork Frankle & Carbin 2018 defined as pruned subnetwork whose performance match the original dense one, when separately trained from scratch

---

Published in Transactions on Machine Learning Research (08/2023)
Our found LIP subnetworks perform remarkably better than the deep decoder of comparable pa- rameter counts, regardless of test. images restoration tasks. It strongly signifies that pruning from overtparrmettrizzaionn is more promising than sticking to under- cnder-parameterizaton: perhaps the first time indicated in the DIP field up to our best knowledge.
Besides untrained DIP scenarios (over- and mnder-parameterizzl), we extend method to the image prior used in compressive sensing, which further validates
pre-trained GAN generator the prevalence of LIPs. It is beyond the existing scope of deep decoder.
For the LTH research community, studying this new LIP problem is also NOT naive extension from the existing LTH methods, owing to several technical barriers that we have to cross: We summarize the contributions as follows:
Till now, LTH has not been demonstrated for image inverse problems or DNN-based priors, to our best knowledge. Most LTH works studied discriminative tasks, with few exceptions Chen el al. (2021d). It is therefore uncertain whether high-sparsity DNN is still viable for low-level vision tasks such as DIPs. Interestingly though, despite that we demonstrate the existence of LIP, we also find that the LIP matching subnetworks cannot directly transfer to high-level vision tasks such as classification.
Existing LTH works typically require a training set to locate the sparse subnetwork. In stark contrast, owing to the nature of DIP, we train a DNN to overfit one specific image. While a handful of papers studied LTH in "data diet' settings Chen et al. 2021ap; Paul et al. (2022). this paper pushes the extreme to "one-shot lottery ticket finding' for the first time. We also develop a multi-image variant of LIP which boosts performance in the cross-domain fitting.
DIP fitting practically relies on early-stopping to avoid over-fitting noise in images. The same risk exists in our iterative pruning, and we demonstrate how to robustly find the sparse matching subnetwork from one noisy image, without needing a clean reference image.
2 Background Work
2.1 Over-parameterized Image Priors
Despite CNNs' tremendous success various imaging tasks, their outstanding performance is often at- tributed to massive data-driven learning. DIP et al. 2018) pioneered to show that CNN architecture alone has captured important natural image priors: by over-fitting randomly initialized untrained CNN to a single degraded image (plus some early stopping), it can restore the clean output without accessing ground truth. Follow-up work Matacv et al. (2019) strengths DIP performance by incorporating it with the regularization by denoising (RED) Tramework and a series of works Mastan Raman (2020 2021) use the contextual feature learning method achieve the same goal of DIP. Besides natural image restoration, DIP was successfully applied to PET image reconstruction Gong et al. (2018), dynamic magnetic resonance imaging Jin et al. (2019). insupervised image decomposition Gandelsman al. (2019) and quantitative phase imaging Yang et al. (2021).
There have been several efforts toward customizing DIP network architectures. Liu et al. 2019 extends the DIP framework with total variation (TV) and this combination leads to considerable performance gains. Jo et al. p further propose the "stochastic temporal ensemble (STE) method to prevent DIP models from overfitting noises and thus improving performances. Chen et al. (020cc) proposed a neural architect ure search (NAS) algorithm, which searches for an optimal DIP neural architecture from search space of upsampling cell and residual connections. The searched 'NAS-DIP" model can be reused across images and tasks. Arican ct al. (2022) observe that different images and restoration tasks often prefer different architectures, and hence design the image-specific NAS to find an optimal DIP network architecture for each specific image. While our work also pursues better DIPs, substantially differs from those prior arts in terms of model compactness (approaching the under-parameterization end) and reusability. The simple pruning-based recipe also overcomes any extra design hassle (search space or algorithm) caused by NAS. We compare the results with them in the paragraph "Comparisons with NAS-DIP and ISNAS-DIP models" in Appendix B
4

---

Published in Transactions on Machine Learning Research (08/2023)
Inpainting
to Cho
S
B
Ground-Truth
Masadd
Full
Spansity 79% PSNR =3152
Sparsity -33% PSNR -3122
Sparsity -91-2% PSNR -38
Sparsit 99%5% P
PSNR =3.32
Super- Resolution
Factor =4
Ground-rruth
Low Resolution Factor=s =4
Full Model PSNR = =25.99
Sparsity =8% PSNR 25.75
Sparsity =86.7% PSNR 25.73
Spansity =914%% PSNR 25.6
Sparsiyy 909.3% PSNR -372
Super- Resolution Fadtor =8
Ground-Trthh
Low Resolution Facto=8 =8
Rull Model PSNR 228.3
Sparsity PSNR 28.8
Sparsity 83% PSNR 2899
Spansity 899 PSNR 28811
Sparsity 99.53% P
Denoising
Ground Truth
Natsy
Full PSNR 33107
Sparsity 55%
Sparsity = 67%
Spunstyy - 74%
Spansity = 99.53% PSNR 29.00
PSNR 33.17
PSNR 31.21
PSNR =3113
Figure 2: LIP visual results: inpainting (row 1), super-resolution (rows 2/3) and denoising (row 4). The last column (in blue) intends to display the results with the most extremely sparse subnetwork.

---

Published in Transactions on Machine Learning Research (08/2023)
2.2 Under-parameterizer Image Priors
Recall that, classical image regularizers in the spatial or frequency domains often rely no or little learning
component. For example, Tomasi & Manduchi (1998) first proposed bilateral filtering method to smooth images while preserving edges by combining image values in a nonlinear way. After that, (ardyy et al. (2001) proposed a wavelet-based cstimator with a robust loss function to recover signals from noises. Later, Dabov et al. (2007)) proposed a strategy based on enhancing the sparsity through grouping similar 2-D image fragments into 3-D data arrays, followed by a collaborative filtering procedure. And this algorithm achieves state-of-the-art performances in terms of both peak signal-to-noise ratio and subjective visual quality. In addition to these traditional methods, [ao et al. (2008) proposed Gaussian mixture model for image denois- ing that partitions image into overlapping patches and estimates the noise-free image patch parameters using expectation maximization (EM) and Minimum mean square error (MMSE) techniques. Elad & Aharon (2006) also presented an approach to remove Gaussian noises based on sparse and redundant representalions using trained dictionaries obtained with the K-SVD algorithm.
As another important concern, while paempparamteerieed DIPs are prone to overfitting the corrupted image, they can generate almost, uncorrupted images surprisingly after few iterations of gradient descent. Inspired by those, Heckel & Hand (2018) pioneered to design concise and non-convolution neural network as the "prior for image restoration tasks. Such under- parametreeddee DIPs not only improve the efficiency of DIP- based restoration but also enable researchers to theoretically analyze the signal process. Besides, they find the architecture's 's simplicity could effectively avoid overfitting in the restoration process. Heckel & Soltanolkotabi (2019) further analyzed the dynamics fitting a two-layer convolutional generator to a noisy signal and prove that carly-stopped gradient descent denoises denoises/rogularize rrguularzzss..
2.3 Lottery Ticket Hypothesis
The lottery ticket hypothesis (LTH) Frankle & Carbin 2018 states that the dense, randomly initialized DNN contains sparse matching subnetwork, which could reach the comparable or even better performance by independently being trained for the same epoch number as the full network do. Since then, the statement has been verified in a varicty of fields. In natural language processing (NLP), Gale et al. (2019) established the first rigorous baselines for model compression by evaluating stale-of-the-art methods on transformer Vaswani et al. 2017) and ResNet He et al. (2016). Later, Chen et al. (2020a) successfully found trainable, transferrable subnetworks in the pre-trained BERT model. In reinforcement learning (RL), Yu et al. (2)19a) confirmed the hypothesis both in NLP and RL tasks, showing that 'winning ticket" initializations generally outperform randomly initialized networks, even with extreme pruning rates. In lifelong learning, [Cenn et al. (2020b) first demonstrated the existence of winning tickets via bottom-up pruning. In graph neural networks ks (GNNs), Chen et al.) (2021c) first defined the graph lottery ticket and developed an unified GNN sparsification (UGS) framework to prune graph adjacency matrices and model weights. And in adversarial robustness research, Cosentino et al. (2019) evaluated the LTH with adversarial training and they successfully proved this approach can find sparse, robust neural networks.
Simultaneously, many researchers started to rethink the network pruning techniques and build a rigorous bascline benchmark Liu et al.] (2018): Evei et al. (2019). Some rescarchers explore the sparse subnetwork at the initialization stage (e-g., Wang
al. (2020) proposed the "GraSP" algorithm, which leverages the gradient flow.). And some try to find the winning tickets at early iterations Frankle et al. 2020cl) Savarese et al. 2019). For example, You et al. (019)) found the "early-bird' tickets via low-cost training schemes with a large learning ratc. However, as the original LTH paper Frankle & Carbin (2018) said, the proposed method camnot scale to large model and datasets. Rewinding was proposed by Frankle et al. (2019) to solve this dilemma. The found matching subnetworks also demonstrate transferability across datasets and tasks Morcos et al.] (2019); Desai et al.] (2019). Ma et al. (2021) thought the rewinding stage is not the only way to solve this problem and proposed a simpler yet more powerful approach called "Knowledge Distillation ticket..
6

---

Published in Transactions on Machine Learning Research (08/2023)
3 Preliminaries and Approach
3.1 Finding Lottery Tickets
Networks For simplicity, we formulate the dense network output as f(2, 0). where is the input tensor and 0 E R4 is the model parameters. In the same way, a subnetwork is defined as m(z,m00) with the binary mask m E {0,1}*, where @ means element-wise product.
A Algorithm 2 Weight-sharing IMP
Algorithm Single bmage-based IMP
Input: The desired sparsity s, the random code 2, the untrained model fus denotes the degraded image and images from domains
The desired sparsity s, the random code 2, the untrained model Ju-
Input:
Fa E {a1, #2, Fn).
Output: A sparse DIP model f(z; (2;;00 @ m) with image prior property.
Output: A sparse DIP model f(;00 m) with image prior property.
Initialization: Set my = e Rlll Set itera-
Initialization:
Set m, = 1 E RR R1 Set itera- tion i = O, training epochs N and j NJ. while the sparsity of my < 8 do
tion i = 0 training epochs N and j E N].
while the sparsity of my < s do
Train the fu(2; 0 @ m.,) for N epochs; Create the mask m/:
1. loss === [({(2 @ m)
2. Train the (= m.) by Backpropaga-
3. Update the mask m. m,,;
tion (loss) for N epochs;
4. Set the model parameters: [(z; 0z);
3. Update the mask mh = m',;
5. create the sparse model: f(;; 0g @ m.);
4. Set the model parameters f(z; 0z);
6. i++;;
3. create the sparse model f(2; 03 O mu):
end while
6. i++;
end while
Pruning Methods We use the classic iterative magnitude-basee pruning (IMP) method Frankle & Carbin (2018), which iteratively prunes the 20% of the model weight each time. In cach IMP iteration, models are trained towards the standard DIP objective to fit the degraded observations for a certain number of training steps following the original DIP Ulyanov et al. (2018). Our basic algorithm performs IMP over just one degraded image, and the algorithm is summarized in Algorithm D We further design extended algorithm, that can perform IMP for DIP over multiple degraded images, through backbone weight sharing: (neither algorithm requires clean images). To show the non-trivialily of the identified matching subnetworks, we also compare LIP with random pruning and SNIP et al. (2018). a pruning-at-iniee inittiallaaaiiiitiiinnn mothod. We also derive new method based on empirical obscrvations to decide when stop IMP iterations to find matching networks with maximal sparsities without any reference to the clean ground truths. More details are in Appendix B
the algorithu is outlined in Algorithm 2
For evaluation models, use hourglass architecture Ulyanov et al. 2018 and deep decoder Heckel & Hand (2018), as two representative untrained DNN image priors in the over- and under- parameterization ends, respectively. For evaluation datasets, we use the popular Set.5 [evilacqua et al. (2012) and Set14 Zeyde et al. (2100). We also evaluate the transferability of subnetworks on image classification datasets such as ImageNet-20 Deng et al. (2009) and CIFAR10 Krizhevsky et al.] (2009). For metrics, we mainly compare the PSNR and/or SSIM results between the restorod image and the ground truth, as Fig.
Experimental Setup
The parameter count of the original DIP model is 2.2 million (M); and that of the deep decoder is 0.1 M for denoising and super-resolution experiments, and 0.6 M for inpainting experiments, all following the original settings of Heckel & Hand (2018). The model sizes are plotted as horizontal coordinates in the figures. We run all experiments with 10 different random seeds: every solid curve is plotted over the 10-time average, and the accompanying shadow regions indicate the 10-time variance. Most plots see consistent results across random seed experiments and hence small variances. All images used are summarized in Fig. 17
7

---

Published in Transactions on Machine Learning Research (08/2023)
31.
36.0
360
31.8
913
*
Dersse DIP
922
Derise DIP
*
Darie DIP
Dersse DIP
fandor
Random
32
Random
330
Random
215
Deep Dacoder
22.
Deep Decoder
Deep Derader
32.3
Deep Dacoder
04
22
Panameer
(a) IMP-F'1
(b) IMP-Woman
(c) IMP-Pace4
(d) IMP-Facc2
31.5
33
3600
31.0
0
305
11.
*
Dense DI
Dense DIP
32
*
Dense DIP
012
*
Densa DIP
Random
Random
Random
Random
30.
32
330
20.
UP
S24
LI
Deep Decoder
Deep Decoder
31
Deep Decader
225
Deep Decoder
22
08
0.1
22
Coum
Co
Parariste (VI
Pantale
(e) Eval-F16
(f) Eval- Woman
(g) Eval-Faced
(h) Eval-Face2
Figure 3: Experimental results finding LIP subnetworks. The first row of the figure summarizes the LTH IMP training loops and the second row denotes the evaluation of found LIP. Note that we compare the LTH IMP with Random Prune (Random) and SNIP Lee al. (2018) prune methods, on images from different (F16 and Woman) same domains (Face2 and Face4). Background task is denoising.
34.5
20.3
3005
Derse
Random
Random
Random
Random
LP
LIP
LIP
LP
Mull
Doep
Deep
Deep Decoder
Doop
Povtrater CounaM)
Presiee courd))
13
98
04
0.1
32.5
60
(a) F16
(b) Woman
(c) Face4
(d) Face2
Figure 4: Experimental Results of Multi LIP on images from same/different domains. We compare Multi LIP with LIP and random prune methods. The background task is denoising.
4 LIP in Untrained Image Priors
Existence of LIPs In prramearamttrrize image priors, first find the matching subnetworks with LIP property by implementing the single-image IMP on the DIP model. We apply the implemented Algorithm [] on Set5 and Set14 to obtain the sparse subnetworks and evaluate these subnetworks on the denoising task. Results of single-image IMP in Fig. 3 (curves in gray colors) verify the existence of LIPs. To be specific, during the IMP finding process, we can find the LIP subnetworks on untrained DIPs at sparsity as high as 86.58%. While at the evaluation stage, the found LIP subnetworks with the modified objectives are still applicable, matching the dense performance at sparsity as high as 83.23%. We also compare the single- image IMP with Random Pruning, and SNIP Lee et al. (2018). We observe from the first row in Fig. 3 that single-image IMP outperforms them at wide sparsity range [20%,96% 96%].
Is Multi-image IMP over-fits an untrained DNN on single image and learns the features in that specific image during iterative magnitude pruning loops. To obtain the LIP subnets with more general features, we propose a new multi-
Good Extension for Finding LIPs? Like the original DIP, single-image IMP
We prune 20% of the remaining weights each TMP iteration, resulting sparsity ratios s; 0.8*

---

Published in Transactions on Machine Learning Research (08/2023)
image IMP (Algorithm 2 for DIP where we replace the DIP objective with the average of multiple images. Note that all images will share the same fixed random code during IMP.
We evaluate the multi-image IMP in two different settings: (i) cross-domain setting where we apply the multi-image IMP to the five images from Set5 Bevilacqua et al. 2012]; (2) single-domain setting where we apply the multi-image TMP to five images of human faces with glasses. We think images from Set 5 are more diversified because they include bird, butterfly and human face contents. We compare single-image IMP winning tickets found on the F16 and the Woman images from Set5 with the cross-domain ticket, and the single-image IMP winning tickets found on Face-4 and Face-2 images with the single-domain ticket. Results presented in Fig. 4 show that multi-image LIP subnetworks can achieve better performances than dense DIP models and randomly pruned ones in the cross-domain setting.
To What Extent Can LIP tickets Be Transferred? In this part, we evaluate the transferability of LIP for DIP models from three perspectives, i.e, across images, across image restoration task types, and from low-level to high-level tasks.
Observation 1: LIP can trunsfer across images. As we obtained the multi-image LIPs from Set5, we evaluate them on different images in 4 For instance, Fig. [[]] shows the evaluation results on F16.png and we found the multi LIPs perform comparably well (better at extreme sparsities) with LIP dodicatedly found on the png (the same phenomenon is reflected on the face-4.png (Fig [c) and face-2.png (Fig [dd]. ). This shows that LIP has reasonable transferability across images, even for those coming from slightly different domains.
Observation 2: LIP can transfer across image restoration tasks but not to other high-level tasks. We con- duct experiments to verify if a LIP matching subnet identified on one restoration task can be re-used in another. Furthermore, the above question has an affirmative answer, is such transferability sustainable when transferring to or from some high-level tasks such as classification?
To evaluate the transferability of LIP between image restoration tasks, we first find three LIPs for the denoising, inpainting and super-resolution tasks respectively and then transfer between them, as shown in Fig. 5 We observe that a LIP winning licket transferred from another image restoration task always yields restoration performance comparable with the single-image LIP found on the original task, sometimes even better, for examples in Fig. 5(a) and
g
We then evaluate the transferability of LIP between the denoising task and image classifications on CIFAR-10 and ImageNet-220 datasets. We show the results of transferring the denoising LIP ImageNet-20 in Fig. [(c) and CIFAR-10 in [[a] the CIFAR-10 LIPs to denoising task in Fig. 6(b) and ImageNet LIPs to denoising in Fig 6(d) We find transferring denoising LIP to classification is unsuccessful. Moreover, transferring winning tickets on CIFAR-10 back to the denoising DIP task also fails to generate winning tickets that are comparable
with denoising LIPs.
Why is the LIP Subnetwork Special and Good? To answer this, we investigate the inner structures of different subnetworks in Fig. 7 The structure of the LIP subnetwork is drastically different from those found by SNTP and random pruning, in particular the distribution of layer-wise sparsity ratios. LIP tends to preserve weights of the earlier layers (closer to the input), while pruning the later layers more aggressively (e.g, Fig. 7(a)). In contrast, SNIP tends to prune much more of the earlier layers compared to the later ones. Random pruning by default prunes each layer at approximately the same ratio. Comparing the three mothods soem to suggest that for finding effective and transfcrable LIP subnetworks, specifically keeping more weights at earlier layers is important. That is an explainable finding, since for image restoration tasks, the low-level features (color, texture, shape, etc.) presumably matter more and are more transferable, than the high-level features (object categories, etc.). The carlier layers are known to capture more low-level image features, hence contributing more to retraining the image restoration performance with DIP.
We summarize and plot the lawyer-wizt wise sparsity ratio result of classification tickets in Figure 7(c) and Figure 7(f) Note that these 6 tickets are directly pruned by LTH on CIFAR10 datasct with the sparsity ratio of 36%, 59% 89% and 95%. In contrast to the inner architecture of LIP subnetworks, we have observed an interesting trend in LTH that it tends to preserve more weights in the middle layers and prune weights in the
9

---

Published in Transactions on Machine Learning Research (08/2023)
$1.0
912
20.5
313
934
915
26.0
g22
*
Up
Dersee DIP
532
* Derie DIP
275
Dense DEP
Denee DVP
213
918
214
Denciee
Duriiee
340
Deep Decoder
215
Deep Decoder
Deep Derader
Deep Dacoder
22
(oung
64
Paransis
04
22
3.6
Cownt
(a) Denoising-FWl
(b) Demoising- Woman
(c) Impainting-
(d) Inpainting- Butteerll
37-1
200
222
91.5
203
213
3112
2.20
200
Dense DP
305
Dense DIP
21
Dense DEP
Dense DP
214
274
Denoise_ LP Deep Decoder
Denotse LUP
211
Denoito LIP Deep Decader
Denolss, LIP
214
245
Deep Decoder
PSWR:
20.57
27.
Deep Decoder
22.02
22
13 Court
Pawrrr
Count Wy;
Paraniate IM
(e) SR- Factor-4- Baboon
(f) SR- factor-4 Pepper
(8) 3R-factor-88 Bahoon
SR- PR-factor-88 Seaerr---Peeepe 8-Pepper
Figure 5: Transferability (cross tasks) experimental results. We study the transferability of denoising LIP on the restoration tasks such as inpainting and super-resolution (SR); we also study the inpainting and SR
LIP on the denoising task. We consider two SR scale factors 4, We evaluate Multi-LIP subnetworks here.
330
12.3
0.7
077
12.4
Denge
323
Randons
328
100
Dense
ImageNac
MUIL UP
315
LF
0.4 Nut LI
31.8
LP
CIFAR IM
CFAR_LUP
Donse
31.5
ImgeNeer_F
80
22
Naurster Coud 201
Panmee Doue 201
Payanaber Courk
(a) CIFAR1O
(b) CIFAR-LIP-nn-DIP
(c) ImageNet
(d) ImageNet-LLIPon--DDD
Figure 6: High- and low-level task transfcrability experiments. We test the denoising LIP on ImageNet-20 (a subset of ImageNet with images resized to 256*256) and CIFAR10 datasets. Note that we replace the last convolutional layer of DIP models with the linear layer and load the same initial weights. We also evaluate the classification LIP on the denoising task.
earlier and later layers (closer to input and output). This phenomenon may partially explain the challenges in transferring LIP subnetworks from restoration tasks to classifications, and vice versa.
The failure of transferring LIP subnetworks to image classification could also be partially explained by the aforementioned sparsity distribution diserepancy: LIP subnetwork tends to prune more weights from later layers, which might damage the network's 's capability in capturing semantic features (usually needing mid- to-high network layers). However, wruning/sparsity will carry more contextual information and would lead to better downstream classification. Also, we could finetune on top of the intermediate layers of the model when more level information is needed. We assume that if we want the DIP model to transfer better to the image classification task, adding and finetuning linear layers on top of the intermediate layers (i.e., transferring only the winning tickets of the encoder) will help.
observe that the bottleneck layers with presumably higher levels of
10

---

Published in Transactions on Machine Learning Research (08/2023)
60
LIP
LIP
SNIP
SNIP
Sparsty(K)
Random
Sparsiyy
Random
Sparsty(S))
40
sparsity_365 36%
30
sparsity_499 49%
sparsity 59%
(a) Sparsity-36%-LLD
(b) Sparsity-59%--ID
(c) Classification LTH-1
96
Eparsiy(()
Spansty(S))
Sparsty
85
80
LIP
LIP
sparsity_se 83%
SNIP
SNIP
sparsity_889
Random
Random
70
sparsity 95%
Layer
Layar
Layer
(d) Sparsity-89%-LIP
(c) Sparsity-95%-LIP
(f) ClassificationnLL1122
Figure 7: Layer-wise sparsity ratio results LIP, SNIP, randomly pruned tickets, and classification tickets directly pruned by LTH. Note that we summarize the sparsity ratio of cach layer: the ratio the number of parameters whose values are equal to zero the number of total parameters of the layer. And the x-axis of these figures composed of the serial numbers of model layers. We sampled subnetworks with four different sparsities in LIP (sparsity 36%, 59%, 89%, 95%), and six different sparsities in classification tickets (sparsity 36%, 49%, 59% 83%, 89%, 95%) to observe.
Why do the LIP Subnetworks Outperform the Dense Model with fewer Parameters at the Beginning? Our initial conjecture is that pruning at low sparsities does not hurt the model capacity only at a small level but rather enhances it with stronger robustness against noisy signals, a phenomenon that has been observed in Ulyanov et al. (2118). In fact, it has been widely observed that LTH-pruned subnetworks can outporform dense models in the low sparsity range, such when 10% 50% of paramcters are prunod. This phenomenon various machine learning tasks, spanning vision Chen et. al. (2021b), languages Chen et al. (2020a), and reinforcement learning Yu et al. (2019a).
first observed in Frankle & Carbin (2018) and has been consistently observed in
5 Extending LIP to Pre-trained Neural Network Priors
Although the implementation process of the GAN CS task and the DIP restoration task different, we still find them related and could be unified for reasons: 1) current research LTH consists of two main streams, networks with weights with randomly initialized weights Frankle & Carbin (018)) and networks with pre-trained weights Chen et al. (2021b); 2) network structures with random weights pre-trained weights can be priors, which is the commodity we view as the two could be unified. However, we do not intend to put the GAN part in a parallel position to the DIP part. We just want to demonstrate the effectiveness of our pruning method as an "extension' from frouinggfrom-ssracchh deep image prior to using pre-trained neural networks as image priors.
Compressive Sensing using Generative Models Compressive Sensing (CS) reconstructs an unknown vector from under-sampled linear measurements of its entries Foucart & Rauhut, (2013). by assuming the
11

---

Published in Transactions on Machine Learning Research (08/2023)
unknown vector to admit certain structural priors. The most common structural prior is to assume that the vector is k-sparse in some known bases Candes et al. (2006); Donoho (2006), and more sophisticated statistical assumptions were also considered Baraniuk et al. (2010). However, those priors are inevitably oversimplified to depict the high-dimensional manifold natural images. Bora et al. (2017) presented the first algorithm that. used pre-trained generalive models such as GANs, as the prior for compressed sensing. As a prior, the pre-trained generator encourages CS to produce vectors close to its output distribution, which approximates its training image distribution. Signilicant research has since followed to better understand the behaviors and theoretical limits of CS using generative priors, e.g., Hand & Voroninski (2018) Bora
ct al. 2018); Hand et al. 2018); Kamath et al. (2019): Liu & Scarlett (0220) Jalal et al. (2020)
Table 1: Results of GAN LIP. We evaluate the LIPs found in PGGAN on the compressed sensing (CS) and the inpainting (I) restoration tasks. The results are based on celebA-HQ dataset [Le et al. (2020). Note that
we use the MSE (Mean Squared Error, per pixel) to evaluate the LIP effectiveness and we also compare the performance LIP with random pruning results.
Sparsity
0%
20%
36%
49%
59%
67%
74%
Random-CS
0.0725
0.0963
0.1165
0.1276
0.2184
0.2086
0.3655
LIP-CS
0.0725
0.0744
0.0732
0.0737
0.0711
0.0728
0.0728
Random-I
0.0541 0.0541
0.0682
0.0748
0.08101
0.1142
0.1904
0.2195
LIP-1
0.0542
0.0504
0.0514
0.0506
0.0524
0.0509
Full Model
=020
S636
S 4449%
S=59%
S=67%
S474%
(a) PGGAN- - Sensing
Masked
Full Model
$=20%
=
S
559
$=67%
(b) PGGAN-Inpainting
Figure 8: Visual results of compressed sensing and inpainting using LIPs. The images framed in red are the reconstruction results of sparse subnetworks, which are better than those of the full model.
Existence of LIP in GANs for Compressive Sensing We use PGGAN Karras al. (2017 pre-trained Cel dataset Lee al. (2020) as the model in this section. To obtain LIP tickets in pre-trained GANs, apply the IMP algorithm. In each IMP iteration, PGGAN is first fine-tuned
40% of images in CelebA-HQ for 30 epochs, has 20% of its remaining weights pruned, and then reset to the pre-trained weights. We only prune the generator in the IMP process because it is found in Chen et al. (2021d) that pruning discrininator only has a marginal influence on the quality of the winning lickets. We then evaluate the tickets on the compressive sensing task following the setting in Jalal et al. (2020): we fix the number of measurements to 1,000 with 20 corrupted measurements, and minimize the MOM objective (Median-of- Means, an algorithm proposed by Jalal al. (2020)) for 1,500 iterations recover the images. We compare the performance (measured in per-pixel reconstruction error) of LIP with the dense baselines in the first.
12

---

Published in Transactions on Machine Learning Research (08/2023)
row of Table. | and provide visual example in Fig. 8(a) Tickets with higher sparsities can match the reconstruction performance of the dense model, confirming the existence of the winning tickets.
Transfer to other image restoration tasks inpainting Besides the experiments the compressed sensing restoration tasks, we also evaluate the effectiveness of GAN LIP on the inpainting task: masking the image and then optimizing the input tensor of the generator in the GAN LIP range to reconstruct the pristine image. More formally, consider the input lensor z' E R1x512 the pristine image x sampled from CelebA- HQ, impainting mask (binary mask), masked image = Ax and a generator G, then the optimization loss function [|AG(z) yll2. The results are summarized Table. and Fig. 8(b) demonstrating the transferability of GAN LIP.
6 Limitations and Future Work
Although the LIP subnetworks outperform the dense DIP model and Deep Decoder many image restora- tion tasks, we also find some limitations of LIP. Firstly, the LIP subnetworks cannot directly transfer from low-level (image restorations) to high-level vision tasks (image classifications) and we conjecturally attribute this the sparsity distribution discrepancy in the inner structure of LIP subnetworks. Secondly, LIP sub- networks fail to achieve comparable performances of the dense DIP model when at extreme sparsities (as high 99.53%). Thirdly; we observe that the LIP subnetworks do not perform very well on specifie restoration tasks such super resolution because of the inherent difficulty for the deep image prior models to generate high-quality textures. In the future, we will keep exploring more efficient ways to construct more sparse and trainable image prior models with powerful transferability.
7 Conclusion
This paper identifies the new intermediate solution between the under- and peraparrmeterieed DIP regimes. Specifically, we validate the lottery image prior (LIP), novel, non-trivial extension of the lottery ticket hypothesis to the new domain. We empirically demonstrate the superiority of LIP in image inverse problems: those LIP subnetworks often compare favorably with overppaameeterized DIPs, and significantly outperform deep decoders under comparably compact model sizes. They also possess high transferability across different images as well as restoration lask types.
References
Metin Ersin Arican, Ozgur Kara, Gustav Bredell, and Ender Konukoglu. Isnas-dip: Image-specific neural architecture search for deep image prior. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1960-1968, 2022.
Richard G Baraniuk, Volkan Cevher, Marco F Duarte, and Chinmay Hegde. Model-based compressive sensing. IEEE Transactions on information theory, 56(4):1982-20) 2001, 2010.
Marco Bevilacqua, Aline Rouny, Christine Guillemot, and Marie Line Alberi-Morel. Low-complexity single- image super-resolution based on nonnegative neighbor embedding. Proceedings of the British Machine Vision Conference, 2012.
Ashish Bora, Ajil Jalal, Eric Price, and Alexandros G Dimakis. Compressed sensing using generative models. In International Conference on Machine Learning, pp. 537-546. PMLR, 2017.
Ashish Bora, Eric Price, and Alexandros G Dimakis. Ambientgan: Generative models from lossy measure- ments. In International Conference on Learning Representations, 2018.
Emmanuel J Candes, Justin K Romberg, and Terence Tao. Stable signal recovery from incomplete and inaccurate measurements. Communications on Pure and Applied Mathematics: A Journal Issued by the Courant. Institude of Mathematical Sciences, 1907 2006.
13

---

Published in Transactions on Machine Learning Research (08/2023)
Yang Cao, Yupin Luo, and Shiyuan Yang. Image denoising with gaussian mixture model. In 2008 Congress on Image and Signal Processing, volume 3, pp. 339-343. IEEE, 2008.
Tianlong Chen, Jonathan Frankle, Shiyu Chang, Sijia Liu, Yang Zhang, Zhangyang Wang, and Michael Carbin. The lottery ticket hypothesis for pre-trained bert networks. arXiv preprint arXin:2077. 12223, 2020a.
Tianlong Chen, Zhenyu Zhang, Sijia Liu, Shiyu Chang, and Zhangyang Wang. Long live the lottery: The existence of winning tickets in lifelong learning. In International Conference on Learning Representations, 2020b.
Tianlong Chen, Yu Cheng, Zhe Gan, Jingjing Liu, and Zhangyang Wang. Data-efficient gan training beyond (just) augmentations: A lottery ticket perspective. Advances in Neural Information Processing Systems, 34:20941-20955, 2021a.
Tianlong Chen, Jonathan Frankle, Shiyu Chang, Sijia Liu, Yang Zhang, Michael Carbin, and Zhangyang Wang. The lottery tickets hypothesis for supervised and self-supervised pro-training in computer vision models. In Proceedings of the IEEE/CVF Conference on Compuler Vision and Paltern Recognition, pp. 16306-16316, 2021b.
Tianlong Chen, Yongduo Sui, Xuxi Chen, Aston Zhang, and Zhangyang Wang. A unified lottery licket hypothesis for graph neural networks. In International Conference on Machine Learning, pp. 1695-1706. PMLR, 2021c.
Tianlong Chen, Zhenyu Zhang, Santosh Balachandra, Haoyu Ma, Zehao Wang, Zhangyang Wang, et al. Spar- sity wimning twice: Better robust generalization from more efficient training. International Conference on Learning Representations, 2022.
Xuxi Chen, Zhenyu Zhang, Yongduo Sui, and Tianlong Chen. Gans can play lottery lickets too. arXiv preprint XrXi::2106. 00134, 2021d.
Yun-Chun Chen, Chen Gao, Esther Robb, and Jia-Bin Huang. Nas-dip: Learning deep image prior with neural architecture search. In Computer Aision-ECCA 2020: 16th European Conference, Glasgow, UK, August 28-28, 2020, Proceedings, Part XVIII 16, pp. 442-459. Springer, 2020c.
Zezhou Cheng, Matheus Gadelha, Subhransu Maji, and Daniel Sheldon. A bayesian perspective the deep image prior. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 5443-50 5451, 2019.
Justin Cosentino, Federico Zaiter, Dan Pei, and Jun Zhu. The search for sparse, robust neural networks. arXiv preprint Xr 1912. 02386, 2019.
Kostadin Dabov, Alessandro Foi, and Karen Egiazarian. Video denoising by sparse 3d censsform-dmmain collaborative filtering. In 2007 15th Furopeon Signal Processing Conference, pp. 145-149. TEEE, 2007a.
Kostadin Dabov, Alessandro Foi, Vladimir Katkovník, and Karen Egiazarian. Image denoising by sparse 3-d transform-domain collaborative filtering. IEEE Transactions on image processing, 16(8):208022)) 2095, 2007b.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hicrarchical image database. 2009 IEEE conference on computer vision and paltern recognition, pp. 248 255. leee, 2009.
Shrey Desai, Hongyuan Zhan, and Ahmed Aly. Evaluating lottery tickets under distributional shifts. arXiv preprint xrXiv:1910.122700 2019.
David I Donoho. Compressed sensing. IEEE Transactions on information theory, 52(())1289- 1306, 2006.
Michael Elad and Michal Aharon. Image denoising via sparse and redundant representations over learned dietionaries. IEFE Transactions Image processing, 15(12):3736 3774373533375 2006.
14

---

Published in Transactions on Machine Learning Research (08/2023)
Utku Evci, Fabian Pedregosa, Aidan Gomez, and Erich Elsen. The difficulty of training sparse neural networks. arviv preprint arXin: 10732, 2019.
Simon Foucart and Holger Rauhut. An invitation to compressive sensing. In A mathematical introduction to compressive sensing, pp. 1 1-39. Springer, 2013.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprint Xrx:::1803.0000 03635, 2018.
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M Roy, and Michael Carbin. Stabilizing the lottery ticket hepot arXiv preprint ar Kiiv1903.011 2019.
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin. Linear mode connectivity and the lottery ticket hypothesis. In International Conference on Machine Learning, pp. 3259 3269. PMLR, 2020a.
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M Roy, and Michael Carbin. Pruning neural networks at initialization: Why are we missing the mark? orXiv preprint arXiv:2009.0833 08556, 2020b.
Jonathan Frankle, David J Schwab, and Ari Morcos. The early phase of neural network training. Tn International Conference on Learning Representations, 2020c.
Jonathan Frankle, David J Schwab, and Ari S Morcos. The early phase of neural network training. anXiv preprinl Xrx:22002. 10365, 2020d.
Trevor Gale, Erich Elsen, and Sara Hooker. The state of sparsity in deep neural networks. arXiv preprint arXiv:1902.09577, 2019.
Zhe Gan, Yen- Chun Chen, Linjie Li, Tianlong Chen, Yu Cheng, Shuohang Wang, and Jingjing Liu. Playing lottery tickets with vision and language. arXiv preprint orXiv:2104. 11832, 2021.
Yosef Gandelsman, Assaf Shocher, and Michal Irani. doublc-dip': Unsupervised image decomposition via coupled deep-inage-priors In Proceedings of the IEEE/CVP Conference on Compuler Vision and Pallern Recognition, pp. 11026 -1005,, 2019.
Kuang Gong, Ciprian Catana, Jinyi Qi, and Quanzheng Li. Pet image reconstruction using deep image prior. IEEE transactions on medical imaging, 387):1655-1165, 2018.
Shi Guo, Zifei Yan, Kai Zhang, Wangmeng Zuo, and Lei Zhang. Toward convolutional blind denoising of real photographs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pollern Recognition, pp. 1712-1722 2019.
Paul Hand and Vladislav Voroninski. Global guarantees for enforcing deep generative priors by empirical risk. In Conference On Learning Theory, pp. 970-978. PMLR, 2018.
Paul Hand, Oscar Leong, and Vladislav Voroninski. Phase retrieval under a generative prior. arXie preprint arXiv:1807.044 042261, 2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. Proceedings of the IEEE conference on computer vision and pattern recognilion, pp. 770 770-778, 2016.
Reinhard Heckel and Paul Hand. Deep decoder: Concise image representations from untrained non- convolutional networks. arXiv preprint arXic:1810.0395. 2018.
Reinhard Heckel and Mahdi Soltanolkotabi. Denoising and regularization via exploiting the structural bias convolutional generators. arXiv preprint 1eXio::910.1463. 2019.
Ajil Jalal, Lin Liu, Alexandros G Dimakis, and Constantine Caramanis. Robust compressed sensing using generative models. Advances in Neural Information Processing Systems, 2020.
15

---

Published in Transactions on Machine Learning Research (08/2023)
Yifan Jiang, Bart Wronski, Ben Mildenhall, Jon Barron, Zhangyang Wang, and Tianfan Xue. Fast and high-quality image denoising via malleable convolutions. arViv preprinl arXiv:2011.00 000092 2022.
Kyong Hwan Jin, Harshit Gupta, Jerome Yerly, Matthias Stuber, and Michael Unser. Time-dependent deep image prior for dynamic mri. arXiv e-prints, pp. arXiv-1910, 2019.
Yeonsik Jo, Se Young Chun, and Jonghyun Choi. Rethinking deep image prior for denoising. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 5087 5096, 2021.
Akshay Kamath, Sushrut Karmalkar, and Eric Price. Lower bounds for compressed sensing with generative models. arXiv preprint arXiv:1912.09953 2019.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation. arXiv preprint 0rXiv:1710.101900 2017.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
Christian Ledig, Lucas Theis, Ferenc Huszár, Jose Caballero, Andrew Cunningham, Alejandro Acosta, An- drew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, al. Photo-realistic single image super- resolution using generative adversarial network. In Proceedings of the IEFF conference on computer vision and pattern recognition, pp. 4681-4690, 2017.
Cheng-Han Han Lee, Ziwei Liu, Lingyun Wu, and Ping Luo. Maskgan: Towards diverse and interactive facial image manipulation. In JEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
Namhoon Lee, Thalalyasingam Ajanthan, and Philip HS Torr. Snip: Single-shot network pruning based on connection sensitivity. arXie preprint arXiv: Xio::8100002440 2018.
Jaakko Lehtinen, Jacob Munkberg, Jon Hasselgren, Samuli Laine, Tero Karras, Miik Aittala, and Timo Aila. Noisc2noisc: Learning image restoration without clean data. arXiv preprint arxiv: 803.04189. 2018.
Bee Lim, Sanghyun Son, Heewon Kim, Seungjun Nah, and Kyoung Mu Lec. Enhanced deep residual networks for single image super-resolution. In Proceedings of the IEEE conference computer vision and paltern recognition workshops, pp. 136-144, 2017.
Jiaming Liu, Yu Sun, Xiaojian Xu, and Ulugbek S Kamilov. Image restoration using total variation regu- larized deep image prior. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 7715 7719. Ieee, 2019.
Zhaoqiang Liu and Jonathan Scarlett. Information lower bounds for compressive sensing with generative models. IEEE Journal on Selected Areas in Information Theory, ((1):292-300)) 0009223--,,--0 2020.
Zhuang Liu, Mingjie Sun, Tinghui Zhou, Gao Huang, and Trevor Darrell. Rethinking the value of network pruning. arXiv preprint arXic:1810. 181.00520, 2018
Haoyu Ma, Tianlong Chen, Ting- Hu, Chenyu You, Xiaohui Xie, and Zhangyang Wang. Good students play big lottery better. arXiv preprint arXiv:2001.0322 2021.
Indra Deep Mastan and Shanmuganathan Raman. Deil: Deep contextual internal learning for image restora- tion and image retargeting. In Proceedings of the IEEE/CVF Winter Conference on Applications of Com- puter Vision, pp. 2366 2375, 2020.
Indra Deep Mastan and Shanmuganathan Raman. Deepefl: Deep contextual features learning from single image. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 2897-2906, 2021.
Gary Mataev, Peyman Milanfar, and Michael Elad. Deepred: Deep image prior powered by red. In Proceed- ings of the IFFE/CVF International Conference on Computer Vision Workshops, pp. 0 0-0, 2019.
16

---

Published in Transactions on Machine Learning Research (08/2023)
Ari S Morcos, Haonan Yu, Michela Paganini, and Yuandong Tian. One ticket to win them all: generalizing lottery ticket initializations across datasets and optimizers. arXiv preprint ar 190: 026600773, 2019.
Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros. Context encoders: Feature learning by inpainting. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2536 2544, 2016.
Mansheej Paul, Brett W Larsen, Surya Ganguli, Jonathan Frankle, and Gintare Karolina Dziugaite. Lot- tery tickets on data diet: Finding initializations with sparse trainable networks. arv preprint arXiv:2206. M.Xiv:2206.0127,, 2022.
William T Redman, Tianlong Chen, Zhangyang Wang, and Akshunna S Dogra. Universality of winning lickets: A renormalization group perspective. In International Conference on Machine Learning, pp- 18483-18448 18498. PMLR, 2022.
Alex Renda, Jonathan Frankle, and Michael Carbin. Comparing rewinding and fine-tuning in neural network pruning. arXiv preprint arXiv:2003. 0088080000008 2020.
Itay M Safran, Gilad Yehudai, and Ohad Shamir. The effects of mild over-paramenn wverpparameterization the opti- mization landscape of shallow relu neural networks. In Conference Learning Theory, pp. 3889 -9344. PMLR, 2021.
Sylvain Sardy, Paul Tseng, and Andrew Bruce. Robust wavelet denoising. IEEE Transactions on Signal Processing, 49(6):1146-11122 1152, 2001.
Pedro Savarese, Hugo Silva, and Michael Maire. Winning the lottery with continuous sparsification. arXiv preprint #xXi::912.04427, 2019.
Hidenori Tanaka, Daniel Kunin, Daniel L Yamins, and Surya Ganguli. Pruning neural networks without any data by iteratively conserving synaptic flow. Advances in Neural Information Processing Systems, 33: 6377 6389, 2020.
Carlo Tomasi and Roberto Manduchi. Bilateral filtering for gray and color images. In Sinth international conference on computer vision (IEEE Cot. No. 98CH36271), pp. 839-846. IEEE, 1998.
Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Deep image prior. In Proceedings of the 1EEE conference on computer vision and pattern recognition, pp. 1446-9454, 2018.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Chaoqi Wang, Guodong Zhang, and Roger Grosse. Picking winning tickets before training by preserving gradient flow. arXiv preprint arXi::2002.07370 2020.
Max Welling and Yee W Teh. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th international conference on machine learning (ICML-11), pp. 681-688, 2011.
Fangshu Yang, Thanh-An Pham, Nathalie Brandenberg, Matthias P Lütolf, Jianwei Ma, and Michael Unser. Robust phase unwrapping via deep image prior for quantitative phase imaging. IEEE Transactions Image Processing, 30:7025-70333 2021.
Haoran You, Chaojian Li, Pengfei Xu, Yonggan Fu, Yue Wang, Xiaohan Chen, Richard G Baraniuk, Zhangyang Wang, and Yingyan Lin. Drawing carly-bird tickets: Towards more efficient training of deep networks. Xiv preprint arxie: 1997,, 2019.
Haonan Yu, Sergey Edunov, Yuandong Tian, and Ari S Morcos. Playing the lottery with rewards and multiple languages: lottery tickets in rl and nlp. Xiv preprint arXiv 0276660000778 2019a.
17

---

Published in Transactions on Machine Learning Research (08/2023)
Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas S Huang. Generative image impaint- ing with contextual attention. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 5505 5514, 2018.
Jiahui Yu, The Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas S Huang. Free-form image inpainting with gated convolution. In Proceedings of the IEEE/CVF International Conference Computer Vision pp. 4471 4480, 2019b.
Roman Zeyde, Michael Elad, and Matan Protter. On single image scalc-up using sparsc-roprreentatiion.. International conference on curves and surfaces, pp. 711-730. Springer, 2010.
Kai Zhang, Wangmeng Zuo, Yunjin Chen, Deyu Meng, and Lei Zhang. Beyond a gaussian denoiser: Residual learning of deep enn for image denoising. IEEE transactions on image processing, 26(7):3142 -2153, 2017.
Yulun Zhang, Yapeng Tian, Yu Kong, Bineng Zhong, and Yun Fu. Residual dense network for image super-resolution. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2472 2481, 2018.
Denny Zhou, Mao Ye, Chen Chen, Tianjian Meng, Mingxing Tan, Xiaodan Song, Quoe Lc, Qiang Liu, and Dale Schuurmans. Go wide, then narrow: Elficient training of deep thin networks. In International Conference on Machine Learning, pp. 11546-11811 11555. PMLR, 2020.
18

---

Published in Transactions on Machine Learning Research (08/2023)
A More Discussions of the Experiments
Images Used Experiments In Fig. 17 we organize and present the images used in this paper with their names. Note that these images are sampled from Set5 Bevilacqua al (2012) and Set14 Zeyde et al.) (2010) datascts, and we their default names. Note that these names used in Section 4 (in curves and analyses). Also, in Table 5 conduct the evaluation experiments BM3D Dabov et al. (007a)) CBM3D Dabov et al. (2007a) and Set12 Zhang et al. (2017) datasets for fair comparisons with NAS-DIP and ISNAS-DIP models.
Parameter Redundancy Problem The commonly used hourglass model DIP Ulyanov et al. (2018) is highly complex in its parameters. The statistic results of parameter numbers summarized in Table 3 We compare the parameter numbers of dense DIP models with the identified winning tickets using LIP and discover that the winning tickets could perform better than the full model while containing 2 million parameters fewer. This phenomenon motivates us to suspect that there is a high possibility of finding the matching subnetworks of the pristine dense DIP model, which indicates that the subnetworks may also contain the outstanding image prior property as the dense one does.
As to the under-parameterriees image prior ner (e.g., deep decoder), we also count the number non- zero parameters in Table 2 for a fair comparison restoration tasks with LIP networks. As shown in the table, for denoising and super-resolution tasks, the number of parameters of the deep decoder is 0.1 million. For impainting tasks, the parameter number is 0.6 million. Therefore, for denoise and super-resolution tasks, we choose the sparsity 97.19% of LIP for comparisons; for inpainting tasks, we choose the sparsity 73.70% for comparisons.
Table 2: Compare the number of non-zero parameters in Deep Decoder (DD.Heckel Heckel & Hand (2018) and our LIP tickets. Note that we compare these models on three restoration tasks: Denoising (DN), Super- resolution (SR) and Inpainting (IP). And 'LIP- 97.15% means the sparsity of the LIP subnet is 97.15%.
Model Structure
DD 128
DD 320
LIP 97.15% (DN, SR)
LIP - 97.19% (DN, SR)
LIP 73.79%
LIP 79.03%
(DN, SR)
(IP)
(IP)
(IP)
Non-zero
100224 (0.1M)
619200
102746
90923
608298
494251
Parameter Numbers
(0.6M)
(0.IM)
(0.09M)
(0.6M)
(0.5M)
Table 3: Evaluation of the dense and LIP models on denoising task with the image bird. The LIP tickets achieve comparable results as the dense model when in extreme sparsity (parameter number: 2.2 million 0.2 million).
Dense Model
LIP Matching Subnetwork
Parameter Numbers (non-zero)
2.2 Millon
0.2 Millon
Performance (PSNR)
30.35
30.61
Definition of Subnetworks and Finding Them Consider nelwork f(x; 0) parametered by 0 with input then a subnetwork is defined as f(;; m@m) where @ E (0, 4 d = |10||o and @ is the element-wise product. Let A7 (f(2;; 0)) to be the training algorithm, that is, training model f(r; 0) on the specific task T with iterations. We also denote the random initialization weight as 9n and the pre-trained weight as 0p: 0; as weight at the i-th training iteration and ET (J(a;; 0)) the model performance evaluation. Following the definitions of Frankle et al. (2020a), we define that if the subnetworks is matching, it satisfies the following
conditions (we use O, for example, to denote pre-trained lottery ticket Chen et al. 2020a 2021bb; On can be defined likewise):
ET (A (f(z; < ET (f;
(1)
19

---

Published in Transactions on Machine Learning Research (08/2023)
34.7
32
230
24
Derue
Datie
Derne
Denae
Rewend_
Reer
Rewind_
Rewind_
24
Rewind
315
Rewind
Reernd,
Rewind
Ruwind
Rewind_2 20
Rew nd 20
Rewnd 20
Rewind 20
22
15
0.5
0.4
1.3 Parerater (M)
177 13
36
Con1)00)
(a) Face2
(b) Face4
(c) Head
(d) Butterfly
Figure 9: Experiments of the rewind strategy (background task: denoising). Note that we train the model with N epochs IMP and Rewind_ mcans rewinding the ticket parameter to fj, the weights after j% N steps of training.
0.10
0:12
0:10
0.15
Clcan Image
Clean image
0.08
Image shuffled
Image
U/C 13 nolss
UVB, 1) nolse
3.44
Clean Image
mikee
Clean Image
nolae
0.25
0.55
Imag
0.02
Image
Litt 1) moies
L8D, 7) nolee
0.0
2000
4000
5500
5000
12002
2000
4200
$300
5000
10000
2000
4022
5300
5000
10000
0
2000
4020
5000
0000
10000
(a) LIP
(b) Random Pruning
(c) SNIP
(d) Dense
Figure 10: Learning curves using four different training targets: a clean image (Baby.png), the same image added with noises, the same randomly scrambled and white noise. Note that we four different models: the LIP subnetwork (S 89%), randomly pruned subnetwork (S 89%), SNIP subnetwork (S 89%) and the dense model (S = 0%). And we trained them in isolation in the same experimental settings for 10000 iterations.
That is matching subnetwork that performs worse than the dense model under the same training algo- rithun AT and the evaluation metric 8T. Similarly, define the winning lickels: if a matching subnetwork f(x; m @ f) has 0 = 0p. then it is the winning tickets under the training algorithm AT.
B
Ablation Study of LIP Subnetworks
The Effect of Weight Rewinding In this part, we study the effect of weight rewinding Prankle et al. (0199h when applied to the single-image IMP for DIP models. Weight rewinding is proposed to scale LTH up to large models and datascts. Specifically, we say we use p% weight rewinding we reset the model weights the end of each IMP iteration to the weights in the dense model after a p% ratio of training steps within standard full training process, instead of the model's random initialization. For the single-image IMP in DIP, we consider 5%, 10% and 20% weight rewinding schemes. The resulting models are denoted as Rewind_5, 5, Rewind 10 and Rewind__2 20, respectively. The results of different weight rewinding schemes are summarized in Fig. g We can see that weight rewinding is not beneficial for identifying LIP in the DIP setting. Too much rewinding (10% and 20%) even hurts performance or fails it completely. We conjecture that this is due to the extremely low data complexity DIP (single image).
Learning Curves of Subnetworks with Different Training Targets To better explain the success of the LIP subnetworks, inspired by Figure 2 of Ulyanov et al. (2018), we further train the obtained subnet (LIP, SNIP and random pruning) with four different targets: I) a natural image, 2) the same added with noise, 3) the same after randomly permuting the pixels and 4) the white noise. We also train the dense model as the baseline results. The experimental details are summarized in the caption of Fig. 10 We first observe that for LIP, SNIP, and dense models, the optimization converges much faster in case 1) and 2)
20

---

Published in Transactions on Machine Learning Research (08/2023)
0.9
23
0.0
13
UP
UP
UP
0.6
UP
Random
Random
Random
Random
0
202
1092
1200
2003
2500
3002
0
530
10:30
1500
2003
2500
3038
500
1930
1500
2000
2530
3000
500
1053
1580
2000
2000
3300
(a) PSNR-S 89%
(b) PSNR-S 95%
(c) SSIM-S 89%
(d) SSIM-S = 95%
Figure 11:
The learning curve plots when using different subnetworks towards the DIP task. In the figure, S denotes the sparsity of the model. We compare both PSNR and SSIM values. For fair comparisons, we trained these subnetworks
the denoising task the Baby image with 3000 iterations, then trained in isolation (the iteration number is recommended by Chen et al. (020cc) to capture the "carly-stopping" phenomenon of DIP), and summarized their performances.
than in case 3) and 4). But the randomly pruned subnetworks have failed in all cases. Interestingly, we also find that SNIP subnetworks perform similarly to the dense model. Meanwhile, the parameterization of LIP subnetworks offers a higher impedance to noise and a lower impedance to signal than the dense model, which indicates that the scparation of high frequency and low frequency is more obvious for winning network architectures.
Does the Pruning Hurt the High-frequency Information of the Model? Also, in order to observe whether the high-frequency information will be lost during the pruning, we apply Fourier Transformation to the Baby figure (described in Fig. 17p and visualize the frequency intensity of the ground-truth image and the reconstructions from three different subnetworks (LIP, SNIP and random pruning). The results are summarized in Fig. [8 and Fig
19 We found that compared with random pruning, LIP and SNIP can maintain most of the high frequency information in the ground-truth (e.g., in Fig 18| SNIP and LIP can both maintain the high-frequency information at the sparsity of 79%; however, SNIP could lose more frgh-frequency information than LIP at lower sparsity ratios.).
Learning Curve Comparison of Using Various Subnetworks for the Restoration Task We further compare the training convergence curves of different subnetworks on the restoration task. In Fig. [1 we summarize the convergence of LIP, SNIP and randomly pruned subnetworks on the denoising task, and the experimental details are included in the caption. We use the PSNR and SSIM metrics to measure the quality of the generated images: SSIM is often considered botter "perceptually aligned', by attending more to the contrast in higu-freyuency regions than PSNR.
At the early stage of optimization, observe that the learning curves of LIP and SNIP subnetworks are almost overlapped (either PSNR or SSIM curves), while the randomly pruned subnetworks failed to perform comparably with them. Yet when the iterations increase, the SNIP subnetworks start to lag behind the LIP subnetworks (e.g., the largest PSNR gap between the two can reach 3dB and the largest SSIM gap can be 0.7). Only the LIP subnetworks can match the comparable performances of the full model when reaching the 3000-th iteration. Lastly, the SSIM gap noticeably enlarged al higher sparsity levels (95%) when comparing LIP and SNIP, which implies LIP is better at capturing perceptually aligned details.
Visualization Results of the LIP Subnetworks In Fig. 2 we present the restoration results of these subnetworks on tasks such as: denoising, inpainting and super-resolution (factor= 4, 8). We successfully find the LIP subnetworks with high sparsities (c.g., 91.4% for inpainting, 89% for super-resolution and 74% for denoising), they achieve the comparable or better performance than the dense model. Moreover, we show the LIP subnetworks in the extreme sparsity (e.g., sparsity = 99.53%) We find that the performances of the LIP subnetworks are not largely degraded (c.g., the PSNR value decreases by 1.37 for super-resolution with factor 8.), which demonstrates the LIP subnetworks retain the image prior property of the dense models. In Fig. 12 we visualize the DIP learning process the denoising task with the LIP subnetworks. And we also
21

---

Published in Transactions on Machine Learning Research (08/2023)
Full Model
Ground Truth
Iteration 0
Iter 500
Iter 1000
Iter 1500
Iter 2000
Iter 2500
Iter 3000
LIP Tickets
Sparsity = 50%
Q
Model
Iteration
Iter 400
Iter 1200
Iter 2000
Ier 2400
Ground-Trull
O
LIP Tickts
Sparsity = 55%
Figure 12: Visualize the learning process of the LIP subnetworks.
We compare the results the dense model
and LIP tickets to study their learning ability toward one image. We use the default optimization iteration numbers (F16.png: 3000 iterations; Snail.png: 2400 iterations)
Ulyanov al. (2018).
0.0084
314
0032
3.003
0.000
3.807
30.8
25006
3314
0.0055
3.009
211
Marching Subnstwcck
0.002
305
Marshing Subnetwork:
3.084
Derae
Derss Mudel
Subnwwwork
305
Suonsbwork
22
30.4
0.5.
22
1.7
1.3
04
0.
Paramarr Cour
fenc-ls-facee--oss
(b) Denoiso-facc3-IMMP
(c) Denoisc-face5-loss
(d) Denoiso-facco-IMMP
Lu99
0.002
315
0073
31.8
0.0022
31.0
0.800
Motchng Subnewwork
0.0011
30.5
Matching Sulmmtwack
Dense ooss
933
Dense Model
Donse loss
Dence Model
0.004
IMP
Subnetwork
IMP
Suznebwark
26.
Peurnte Good 30
Paromdter Gownt 201
Pammenrr Count (00
Parsmelrr Coun))))
Inpaint-face3-loss
(f) Inapint-face3-In
(g) SR-face3-loss
(h) ER-face3-IMM
Figure 13: Study of early-stopping in IMP loops. For comparisons, we study the IMP loss values and the model performances during single-image IMP loops. We experiment with face3 and face5 images on three restoration tasks (denoising, inpainting and super-resolution) For comparisons, we plot the IMP loss and model performances during TMP loops. Note that we use the yellow star to denote the matching subnetworks and the green star for the dense model.
compare the learning ability of the dense model and LIP tickets. Interestingly, we find these subnetworks learn the general outline of the image at the early stage of the optimization process (e.g., the outline of the snail's shell at the first 500 iterations), and then they learn the detailed features of the image objects at the late stages (e.g., the eye features of the snail the last 400 iterations). The outstanding learning ability also demonstrates their excellent image prior property.
22

---

Published in Transactions on Machine Learning Research (08/2023)
Comparisons with NAS-DIP Chen et al. (2020c) and ISNAS-DIP Arican et al. (2022) models. Chen et al. (022) first builds the searching space for upsampling and residual connections to capture better image priors for the dense DIP model. [Arican et al. (2022) first finds that there is a small overlap of the best DIP architectures for different images and restoration tasks (c.g., denoising, inpainting and super-resolution) Therefore, they propose to do the NAS algorithm on DIP frameworks on specific images for computat ional savings. These methods aim to optimize the architecture of over-parameterized models to achieve better performances, and therefore, the model size of these two networks is still comparable to that of the dense DIP model (they neither compact the model nor prune its layers). In Table 5 compare the performances of LIP subnetworks (the parameter number is 0.6 million) with DIP, NAS-DIP and ISNAS-DIP models. We find that LIPs can achieve comparable or even better performances than NAS-DIP models (especially on denoising and impainting (asks). The ISNAS-DIP model gains stale-of-hhe-rl performances on restoration tasks such as denoising and inpainting. But when it comes to the super-resolution, LIP subnetworks and the NAS-DIP model outperform the ISNAS-DIP network.
More discussions about LIP, NAS-DIP, and ISNAS-DIP. From an algorithmic perspective, we would like to humbly emphasize that this work is the first attempt to investigate LTH in the DIP scenario, which could help to understand how the topology and connectivity of CNN architectures encode natural image priors, and whether "overparameterizatoo angle from that of the NAS-based methods, which is about looking for the best "architecture (with dense weights)" for DIP.
sparsity' make the best DIP recipe. This is different
In terms of implementation, it is worth mentioning that the published implementations of both NAS-DIP and ISNAS-DIP do not offer an out-of-the-box mechanism for regularizing the network size inherently during the searching process. In consideration of the limited duration of the response period, we refrained from extending the implementations of NAS-DIP and ISNAS-DIP independently. Our concern is that the ensuing scarch outcomes might diverge significantly from the concepts initially proposed in the original NAS-based works.
Methodologically, our pruning-bases LIP is proposed to be orthogonal to NAS-DIP and ISNAS-DIP. More explicitly, the iterative pruning procedure can be equivalently applied to any network architecture discovered by the NAS-based methods, thus yielding a more compact network representation. We assume the feasibility of this combination to be grounded in the observed existence of the Lottery Ticket Hypothesis (LTH) across range of network architectures including (a) compact Multilayer Perceptrons (MLP) and smaller convolutional networks Frankle& & Carbin (2018): ; (b) over parammeeeette networks with residual connections, such as ResNet and VGG networks Frankle et al. (2019) and (c) U-Net-like hourglass convolutional networks, as evidenced in this study, based on which both NAS-basch methods search architectures.
Discussions about the "early-stopping" in the single-image based IMP loops. Similar to the DIP optimization, the single-image based IMP also needs "carly-stopping during the iterative pruming loops, or the final subnetwork will overfit the noises. In Figure 13] we summarize and plot the IMP loss and model performances during the itcrative magnitude pruning. Specifically, we experiment with face3 and face5 images on three restoration tasks (e.g., denoising, inpainting and super-resolution.). We find that the matching subnetwork (yellow stars in the figure) appears where the value of IMP loss continuously increases two or three times. Based on this observation, we summarize the approach as follows: To robustly find the matching subnetworks when the single noisy image IMP loops, one can stop the iterative magnitude pruming when observing the continuous increases in IMP loss values. However, there may be other ways (based on other signals) to identify the matching subnetworks in the single-image based IMP loops. For example, one can obscrve the gradient flow or the magnitude of gradients in the IMP loops to find the inner links between these signals and the appearance of matching subnetworks, which will be our future works. So far, the IMP loss value has been the most obvious signal in training loops.
Transferability study of the single-image based LIP subnetworks We evaluate several single-image basod LIP subnetworks (Baby-LIP, Bird-LIP, Woman-LIP and Butterfly-LIP) on the Baby.png and compare their performances with MUlti-LIP subnetworks. The experimental results are summarized in Fig [4 We found that when the sparsity level is low (0%-40%), the single-LIP subnetworks perform a little worse than
23

---

Published in Transactions on Machine Learning Research (08/2023)
31.4
31.4
31.4
31.2
31.2
312
31.0
$1.0
30.8
Dense
Dense
30.8
Dense
30.8
Baby_LIP
30.6
Baby_Lin
Baby LIP
Woman
Bird
Butterfly_
30.4
MultiLIP
30.4
LIP
1.7
1.3
13 Count (M)
Parameter Count (M)
Count (M)
(a) Woman LIP
(b) Bird LIP
(c) Butterfly LIP
Figure 14: Evaluate different LIPs on the baby.png. Note that the background task is denoising.
40
90
PONK
PONR
Rendom Prune (R_P) R.P. UPP Sparsty
Rendom Prane (R_ R.P. with UP. Sparsty
Rendom, Prana (RL P)
Random, Prune (R P)) R.P WPP Sparsty od
R_P _IP_
LIP Spansty
80
100
100
Sparthy
Sporthy
Sparthy
Spartyy
(a) Baby
(b) Bird
(c) Woman
(d) F16
Figure 15: Evaluate the random structure of the DIP models. We compare the performance of randomly pruned subnetworks with the ones randomly pruned with LIP sparsities. The background task is denoising.
the LIPs obtained on Baby.png. When the sparsity level increases (e.g., the number of remaining parameters is less than 0.6 million), they usually tend to perform comparably. Another intriguing phenomenon is that Multi-LIP performs comparably with the LIPs obtained on Baby-png but can perform better than the other LIPs when sparsity level is high, which also demonstrate that Multi-LIP could have better transferability than single-image based LIP subnetworks.
Ablation Study on Random Structures of original DIP models We have compared the randomly pruned subnetworks with LIP ones and have found these randomly pruned sparse networks perform worse, especially at high sparsity levels. This phenomenon indicates that the uniform randomness will lead to inferior performances. But what we provide the LIP sparsity to the random structure? In Fig. [5] we summarize the experimental results on Baby.png, Bird.png, Woman.png and F16.png. We observed that the randomly pruned subnetworks with LIP sparsity will collapse at much earlier stage than those that are uniformly randomly pruned. They may have comparable performances when at low sparsity levels (e.g, from 0% to 40%.). But the performance curve will quickly fall at high sparsity levels (c.g., from 50% to 90%.), which demonstrates that the inner structure of LIP subnetworks is unique.
Comparison with Other State-of-the-art Pruning Methods In the paper, compare the effec- tiveness of LIPs with SNIP and random pruning. To better explore the effectiveness of LIP subnetworks, further compare the sparse models with the "SynFlow' pruning methods Tanaka al. (202)). The ex- perimental results are summarized in Table 4 We observed that SynFlow subnetworks achieve comparable performances with the dense network and LIPs when al low sparsity levels (e.g., from 0% to 20%.). But LIP subnetworks start to win them out when the sparsity level increases (e.g., from 60% to 90%.). Also, the LIPs can achieve comparable performances when at extreme sparsitics (c.g., when sparsity level equals 80%.). But the "SynFlow' subnetworks have already failed. These phenomena indicate the effectiveness of proposed LIP methods, which also act side support to Frankle et al. (022bb)
24

---

Published in Transactions on Machine Learning Research (08/2023)
24.0
19.5
19.0
28
23.0
PONE
18.5
225
22.0
18.0
Dense
22
LIP
LIP
UP
17.5
Dense
21.0
2.2
0.6
0.4
0.1
1.3
0.8
18
1.3
0.8
0.4
0.1
Parameter Count
Parameber Count (M)
Parmmeter Count (M)
(a) SCLD-on-Babont
SCLD-on-Popper
(c) SCLD-on-Zebra.
Figure 16: Experiment method the SGLD-DIP model Cheng al. (2019 Note that run this experiment with 3 different random seeds and the evaluation images sampled from Set5 and Set14 datasets. Also, mark the performance of the original dense DIP model in these figures (shown as the green star). The code of the SGLD-DIP model is available at https://githin http///thuu..cm//ezouuutue com//eezhouCheng/GPP
How is the Performance of LIP Subnetworks on Other DIP-based Architectures? To demon- strate the effectiveness and versatility of the LTP method, we have implemented the proposed pruning algorithm on the SGLD-DIP model (heng al. (2019), which is state-of-the architecture based on the deep image prior. The SGLD-DIP model utilizes the stochastic gradient Langevin dynamics Welling Teh (2011), enabling the model to mitigate the overfitting of noise and generate improved restoration outcomes. By applying the single-image IMP to this model, we conducted experiments with three different random seeds, using the images 'Baboon.png png', "Pepper.png' and "Zebra.png' all sourced from the Set5 and Set14 datasets. The experimental results, depicted in Figure 16] provide a summary of our findings. Particularly, we compare the performance of LIP SGLD models with the original dense models. Our observations reveal that the LIP SGLD models consistently outperform the dense networks across sparsity levels ranging from 20% to 79,, thereby illustrating the generality of our proposed methods.
Table Comparison with synaptic-flow pruning method Tanaka al. (2020). We compare the performances of LIPs and "SynFlow' subnetworks at various sparsity levels. Notice that the background task is denoising and the experiment image is F16.png.
Sparsity (%):
0 (dense)
20
40
50
60
70
80
90
SynFlow
31.06
31.05
30.62
30.54
30.59
30.41
30.11
28.98
LIP
31.06
31.15
31.11
31.12
31.13
31.11
31.03
30.91
25

---

Published in Transactions on Machine Learning Research (08/2023)
Table 5: Comparison results between LIP (sparsity = 73.79%, parameter number = 0.6M), DIP Ulyanov et al. (2018), NAS-DIP Chen et (2020c) and ISNAS-DIP Arican et al. (2022) models. The evaluation (2017). CBM3D Dabov et al. (2007a), Set5
datasets are BM3D Dabov al. (200aa), Set12 Zhang Bevilacqua al. (2012) and S4t Zeyde et al. (2010). And the results of DIP, NAS-DIP and TSNAS-DIP models are directly picked from Table 2 in Arican et al. (022]. Since NAS-DIP and ISNAS-DIP both aim to find better architect ures to gain bel model performances via NAS, their model sizes are comparable to that of the dense DIP network (there are model compressions during the optimization). And we use wen to denote this in the table.
Datasets
DIP (2.2M)
LIP (0.6M)
NAS-DIP (*)
ISNAS-DIP (*)
Denoising
BM3D
27.87
28.27
27.44
28.39
Set12
27.92
28.12
26.88
28.06
CBM3D
28.93
29.45
29.13
30.36
Inpainting
BM3D
31.04
32.44
30.55
32.90
Sct12
31.00
31.78
30.86
32.22
Super-rosolution
Set5 x4
29.89
30.52
30.66
30.22
Set5 x8
25.88
26.17
25.88
25.94
Set14 x4
27.00
27.31
27.36
27.19
14 x8
24.15
24.31
23.96
24.03
Face 1
Face 2
Face 3
Face 4
Face 5
Bird
Baby
Butterfly
Woman
Head
Vase
Kate
F16
Baboon
Pepper
Figure 17: Images used in plotting the curves of experiments.
26

---

Published in Transactions on Machine Learning Research (08/2023)
Ground-truth
LIP S = 59%
LIP 67%
LIP S = 79%
Ground truth
SNIP S = 59%
SNIP S 67%
SNIP S = 79%
Ground- truth
Random S 59%
Random 67%
Random S 79%
Figure 18: Evaluating the reconstruction images of different subnetworks (LIP, SNIP and random pruning) by FFT (Fast Fourier Transformation) to check whether the high-frequency information has been lost during pruning. Note that experimented on the Baby-png. We found that compared with random pruning, LIP and SNIP can both maintain the high frequency information of the ground-truth. For example, the LIP and SNIP subnetworks both maintain most of the high-frequency information of the ground- truth at the sparsity 79%, but the LIP could also perform well at the sparsity 67% whore the SNIP loses more high-frequency information.
27

---

Published in Transactions on Machine Learning Research (08/2023)
Sparisty: 59.5%
Sparisty: 67.2%
Sparisty: 79%
LIP SNIP
LIP SNIP
LIP SNIP
LIP Random
LIP Random
LIP Random
Figure 19: we visualize the difference maps between the FFT (Fast Fourier Transformation) results of reconstruction images generated by differont methods (c.g., LIP, SNIP and random pruning). For example, 'LIP random' means the difference map between the FFT results restored images generated by LIP and randomly pruned subnetworks. The brighter color mcans larger difference in values. Also, the center of the visualization images represents the image areas with frequency equal to zero and the surrounding parts denote areas with high frequency. We observe that the reconstructions generated by LIP are consistent with those from other methods in the low-frequency domain (the center area of the FFT map) and mainly differ from other methods in the high-froquency domain.
28


Reviews:
[{"review_id": "OcrtpvBxQS", "reviewer": "Reviewer_wwSi", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n- **S1**: First showing the LTH is applicable in DIP domain. The subnetwork performs better than the Deep decoder and even the original full size network when pruned.\n- **S2**: Various detailed studies on transferability across tasks, sensitivity study by the layer-wise sparsity ratio and application to pre-trained neural network prior using GAN.\n\n**Weaknesses**\n\n- **W1**: The proposed idea is basically finding a better suited network architecture embedded in a large DIP network. The lottery ticket happens to be small in size but not guaranteed. The idea of finding better performing network architecture for DIP has been explored in NAS-DIP and ISNAS-DIP. Although these two methods did not try to reduce the size of the resulting networks, it is interesting to observe the performance when they try to reduce the size of the network.\n- **W2**: Finding optimal size is an open question. \n- **W3**: Lack of comparison to the state of the art DIP models. There are number of DIP models (Liu et al. 2019, Jo et al. 2021, Mastan & Raman 2020, 2021, Gandelsman et al. 2019, Yang et al. 2021) but there is no comparison to those models or at least try to use the best DIP model to prune the network. It is not clear why DIP method the method is mostly applied for the evaluation.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes to prune the over-parameterized image priors to find a better performing and efficient image prior based on the Lottery Ticket Hypothesis. The pruned image prior results in better quality solutions to various inverse problems addressed by the DIP, with the relatively small number of parameters compared to the full model, of course outperforming efficient counterpart proposed in DeepDecoder. The paper presents various detailed studies including transferability across tasks, sensitivity study by the layer-wise sparsity ratio and application to pre-trained neural network prior using GAN.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Clarify the based DIP method used for applying the LTH and show the effectiveness of the method in the state of the art DIP method or architecture.\n  - If possible, compare the effectiveness of the LTH in various DIP methods\n- Reduce the network size by the competing methods such as NAS-DIP or ISNAS-DIP and compare the quality by them and the vanilla IMP that the LIP is using\n\n", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "uaSMltPftI", "reviewer": "Reviewer_raKT", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros:\n\n+ First study on extending lottery ticket hypothesis (LTH) to address low-level vision tasks\n\n+ Extensive empirical results are provided for several low-level image restoration tasks\n\n+ Empirical results substantiate the existence of the proposed lottery image prior (LIP)\n\n\n\nCons:\n\n- It is very interesting to observe that LIP subnetworks can surpass DIP models in various image restoration tasks. However, the underlying reasons remain unclear. The authors briefly mention that sparsity can enhance DNN robustness to noise and degradations, but the article lacks rigorous validation to support this claim. \n\n- The relevance of GAN-based compressed sensing to Deep Image Prior (DIP) might appear unclear, raising the question of why Section 5 is necessary. While DIP and GAN-based compressed sensing follow distinct learning paradigms, employing the LIP for these two tasks may stem from different motivations. Thus, it is essential to explore and discuss these aspects in Section 5.\n\n- This is a purely empirical study that extends LTH to address low-level vision tasks. From a technical standpoint, the contribution is marginal. \n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this article, the authors introduce a new \"lottery image prior\" (LIP), which leverages the innate sparsity of deep neural networks (DNNs) to pinpoint an intermediate, parameterized image prior that achieves superior performance, efficiency, and transferability. Specifically, the authors expand upon the lottery ticket hypothesis (LTH) to address low-level vision tasks such as image restoration and compressive sensing image reconstruction. Through a series of experiments, they not only demonstrate the presence of LIP in low-level vision tasks but also provide some interesting conclusions on the transferability of the proposed LIP.\n\n", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I recommend that the authors enhance the article by addressing the concerns raised in the identified weaknesses:\n\n+ Provide more rigorous validation to demonstrate the reasons behind LIP subnetworks outperforming DIP full models in image restoration tasks.\n\n+ The connection between GAN-based compressed sensing and Deep Image Prior (DIP) may seem ambiguous, leading to questions about the necessity of Section 5. Please clarify the significance of the GAN CS portion and justify the inclusion of Section 5 in the article.\n\n+ Are there any failure cases or limitations associated with LIP? I would suggest the authors add a limitation analysis section in the revised article. \n\nMy current rating for the article is borderline. However, if the authors can address my concerns, I would be pleased to revise my rating upwards.\n\n***Updated Rating***\n\nI have read the authors' responses and comments from fellow reviewers. My major concerns can be successfully addressed. I am happy to recommend accepting this article.  \n\n\n\n\n\n", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "P83uYbdjRr", "reviewer": "Reviewer_RGN4", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. I think this paper is novel and high-quality.\n2. The idea is interesting, and the writing is clear. \n3. Experiment setup is nice - plotted on 10-time average with different random seeds and includes the corresponding variance.\n4. The results are impressive (Fig.3 - 6).\n5. Discussions on how LIP is different from SNIP (Fig. 7) is nice. \n\nWeaknesses:\n1. While I understand that the proposed LIP is closely connected to the over/under-parameterized image priors and LTH, and I appreciate the efforts that authors spent in listing a dense amount of references in the background work section, I don't find enough discussions on how those work connect to the proposed work. Especially in Section 2.3, where the authors simply list papers in LTH without any discussions on why those work belong to this paper.\n2. Discussions on some of the experiment results are lacking (see requested changes section).\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposed lottery image prior (LIP) that sits between over-parameterized deep image priors (DIP) and under-parameterized deep decoder. Analogous to the lottery ticket hypothesis (LTH), LIP aims to find a sparse sub-network from DIP that significantly out-performs under-parameterized image priors such as deep decoders while being compact in terms of model sizes. The key contribution of this paper is to verify the existence of LIP, which could be seen as a more compact DIP model (but not the same as deep decoders), not only out-performs deep encoders, but also free from severe over-fitting like DIP. By finding such LIP, better training efficiency could also be achieved.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. I think the background work section should be re-written, instead of listing a long list of papers that broadly belong to where this paper connects, the focus should focus more on how those work connect, influence or motivates the idea of LIP.\n2. While I am impressed by the LIP performance showed in Figure 3, 4 and 5, as listed in the Strengths section, I notice that LIP out-performs DIP with less parameter counts at the beginning, and I hope to see more discussions from the authors on why this is the case, especially when we don't see the same performance in image classification tasks, as shown in Fig. 6.", "broader_impact_concerns": "I don't find concerns on the ethical implications of the work that would require adding a Broader Impact Statement.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Please refer to the comments above.
Decision: Accept (poster)