Question: In my opinion, the goal of the research on LIP is to simplify the DNN architecture in DIP based on the pre-known knowledge on the sparse subnetwork. For example, given a noisy image without the latent clean image, if we know the sparse subnetwork in advance, we can directly optimize it to obtain the denoising result under DIP.  How to achieve this goal? If LIP cannot embed into current DIP related works like this, it is useless in practice.
Evidence: Background Deep neural networks (DNNs), in particular convolutional neural networks (CNNs), have been powerful tools for solving various image inverse problems such as denoising, inpainting, and super resolution. Conventional wisdom believes that is owing to DNNs' universal approximation ability and learning from massive training data. Yet, recent studies have revealed the specific architectures of CNNs have the inductive bias to represent and generate natural images well, and such favorable architecture inductive bias can work independently from fitting specific training sets.
For example, deep image prior (DIP) shows an untrained neural network can be used as a handcrafted prior that transfers well across multiple inverse problems. The authors attributed the success to the CNN architecture itself, that appeared to possess high noise impedance even with only random initializations. As another example, in compressive sensing,; replaced the common structural assumptions such as sparsity with a pretrained generative adversarial networks. The underlying rationale lies in that a pre-trained generator should (approximately) represent the notion of a vector being or more likely in the target domain such as natural images; in other words, a sample more like a natural image will be closer to the output range of the pre-trained generator. In this paper, we refer to such general-purpose image prior parameterized by (either untrained or pre-trained) DNNs as DNN-based image priors.
Recall that, classical image regularizers in the spatial or frequency domains are often not learningbased, or rely on compact learning models. On contrary, DNN-based image priors have a massive number of parameters (we compare the parameter numbers between the full model and the sparse subnetwork in Table 2), typically magnitudes more than the image (even the image size) size. The two extremes invite the natural question: Can we identify highly compact DNN-based image priors, that are same effective? We note that, diving into this question has twofold appeals. On the algorithmic side, that could help us understand further how the topology and connectivity of CNN architecture itself will affect the effectiveness of those priors, and to what extent sparsity could be relevant. On the practical side, if we could provide an affirmative answer to this question, that would potentially lead to more computationally savings when applying those DNN-based priors in practice, leading to faster restoration or computational imaging with them.
Towards the above question, the tool we refer to in this paper is the recently emerged Lottery Ticket Hypothesis (LTH). LTH suggests that every dense DNN has an extremely sparse "matching subnetwork", that can be trained in isolation to match the original dense DNN's accuracy. While the vanilla LTH studies training from random scratch, the latest works also extend similar findings to fine-tuning the pre-trained models. LTH has widespread success in image classification, language modeling, reinforcement learning and multi-modal learning, e.g.,.
Our Contributions Drawing inspirations from the LTH literature, we conjecture and empirically study a novel "lottery image prior" (LIP), stated as: Given an (untrained or trained) DNN-based image prior, it will have a sparse subnetwork that can be trained in isolation, to match the original DNN's performance when being applied as a prior to regularizing various image inverse problems.
Studying this new problem is, however, NOT a naive extension from the existing LTH methods, owing to several technical barriers: (a) till now, LTH has not been demonstrated for image inverse problem or DNN-based priors, to our best knowledge. Most LTH works studied discriminative tasks, with one exception. It is therefore uncertain whether high-sparsity DNN is still viable for reconstruction-oriented tasks; (b) existing LTH works typically require a full training set to locate the sparse subnetwork mask, whereas our LIP settings are only not data-rich. For example, DIP needs the DNN to be trained to overfit one specific image, making it drastically different from previous problems; (c) the objectives between finding the sparse mask (e.g., learning the prior) and fitting the sparse subnetwork (e.g., using the prior) are often unaligned in LIP problems. For example, DIP will overfit a corrupted input image (during which the sparse mask will be found) in order to reconstruct a clean output image (when the found sparse subnetwork will be used); the pre-trained generator will also be used towards a different goal (regularizing compressive sensing) from their original pre-training task (generating realistic images).
Our extensive experimental study confirms the existence of LIP in two representative settings: (i) image restoration with the deep image prior, using an untrained DNN (as shown in Fig.); and
(ii) compressive sensing image reconstruction, using a pre-trained GAN generator. Using iterative magnitude pruning (IMP) with surrogate tasks (the overview of our work paradigm is in Fig.), we can successfully locate the LIP subnetworks at the sparsity range of 20%-86.58% in setting i; and those at the sparsity range of 5%-36% in setting ii. Those LIP subnetworks also possess high transferability. For example, the LIP ticket found in the setting i transfer well across not only different images, but also different tasks such as denoising, inpainting and super-resolution. Our contributions are summarized below:
• The first comprehensive study on LTH in DNN-based image priors and inverse problems, establishing the "lottery image prior" (LIP) and demonstrating the prevailing relevance of LTH more broadly than previously typical settings. The last column (in blue) intends to display the results with the most extremely sparse subnetwork.
• The investigation of both untrained and pre-trained DNNs as image priors, verifying that LIP subnetworks can be found in both settings, by overcoming several impediments such as severely limited training data and task objective mismatch. • High transferability of LIP subnetworks across data/datasets, and various inverse problem tasks. Our finding reflects the underlying common image prior that is agnostic to specific data or task, through the lens of the CNN architecture together with sparsity.
In this section, we will investigate the lottery image prior (LIP) property under setting i, for deep image prior (DIP) models. Model performance is measured by PSNR (Peak Signal to Noise Ratio) between the restored images and the clean ground truths (SSIM (Structural Similarity) results are in supplementary materials). We run all experiments with three different random seeds. All images used for plotting results are summarized in Fig..
In DIP, for each degraded image to be restored, an untrained DNN will be over-fitted over that single image (with proper early stopping). This casts a crucial difference from typical LTH settings where winning tickets are identified using IMP over a distribution of data and are verified to be effective when trained on data from that specific distribution. Therefore, besides the usual properties in winning tickets such as the existence, the superiority over other pruning methods, and the effect of rewinding, we will strive to answer two more questions about LIP in DIP. The first question is: can we extend the IMP algorithm for single-image DIP to multi-image and thus improve the performance of winning tickets hopefully? The second question is to what extend the LIP found in DIP setting can be transferred? Answering these two questions not only provides a clearer picture of LIP, but also hints a new perspective for a deeper understanding of LTH.
In DIP setting, we first find the winning tickets with LIP property by implementing the single-image IMP on the hourglass model (i.e., the DIP model used in. We adopt the modified objectives using clean images as labels as we discussed in Sec. 3.2. The algorithm is described in detail in Algorithm 1. We apply the implemented algorithm on Set5  ( and Set14 datasets to obtain the sparse subnetworks 1 and evaluate these subnetworks on the denoising task.
Results of single-image IMP in Fig. (curves in red colors) verify that the LIP exists in the DIP setting. To be specific, during the IMP finding process when we use clean images in DIP objectives, we are able to find the winning tickets with LIP on untrained DNN at sparsity as high as 86.58%. While at the evaluation stage, when we only have access to the degraded images, we find that the winning tickets found with the modified objectives are still applicable, matching the dense performance at sparsity as high as 83.23%.
Besides the dense model baselines, we also compare the single-image IMP with Random Pruning, and a pruning-at-initialization method SNIP, whose results are also presented in Fig. in other colors. Specifically, we clearly observe from the first row in Fig. that the singleimage IMP for DIP outperforms random pruning and SNIP at a wide sparsity range [20%, 96%]. Interestingly, we find random prune is a good competitor to SNIP sometimes, but both of them suffer significant performance decrease at extreme sparsities (over 80%), where IMP still persists.
The Effect of Weight Rewinding In this part, we study the effect of weight rewinding Is Multi-image IMP A Good Extension in DIP? DIP over-fits an untrained DNN on a single image. A reasonable conjecture is that the DNN will be dedicated to the features in that specific image and so should the resulting winning ticket be if we apply single-image IMP to the DIP model. We then ask that whether we can co-consider multiple images during IMP for DIP models to find winning tickets that are suitable for more general features and thus yield better image restoration performance? To verify our idea, we propose a new multi-image IMP for DIP where we replace the single-image DIP objective with the average of multiple images during the IMP process. Note that all images will share the same fixed random code during IMP. Otherwise, different random codes will intervene with one another because they lead to different winning tickets, resulting in a share subnetwork with inferior performance. The algorithm is formally described in Algorithm 2.
We evaluate the multi-image IMP in two different settings: (i) cross-domain setting where we apply the multi-image IMP to the five images from Set5; (2) single-domain setting where we apply the multi-image IMP to five images of human faces with glasses. We think images from Set 5 are from more diversified domains because they include bird, butterfly and human face contents. We compare single-image IMP winning tickets found on the Baby and the Woman images from Set5 with the cross-domain ticket, and the single-image IMP winning tickets found on Face-1 and Face-2 with the single-domain ticket. Results presented in Fig. show that multi-image IMP significantly improves the quality of the winning tickets in the cross-domain setting, which is aligned with our previous hypothesis.
To what extent can LIP tickets be transferred? Transferability is an important metric to measure the usefulness of LIP in the DIP setting. It would significantly undermine the practical application of LIP if the winning tickets can only be dedicatedly found for each image. In this part, we evaluate the transferability of LIP for DIP models from three perspectives, i.e., the transferability between images, between image restoration tasks and between high-level tasks.
Observation 1: LIP can transfer across images. We identify the single-image LIPs for all image from Set5 and the multi-image LIP on Set5 using the new multi-image IMP proposed in the last part. Then we evaluate all the above LIPs on the Baby image, results being presented in Fig.. We can see that the LIPs found on other images, even those from the Bird and the Butterfly images, can perform comparably well with LIP dedicatedly found on the Baby image. This shows that LIP has reasonable transferability across images, even for those coming from slightly different domains (for example, the Baby and the Butterfly). We can also observe the fact that the multi-image LIP outperforms all single-image LIPs, as side support for the superiority of multi-image LIP in the cross-domain settings.
Observation 2: LIP can transfer across image restoration tasks but not to other high-level tasks.  We test the denoising LIP on MNIST and CIFAR10 datasets (left 2 figures). Note that we replace the last convolutional layer of DIP models with the linear layer and load the same initial weights. We evaluate the CIFAR LIP on the denoising task.
interests? Furthermore, if the above question has an affirmative answer, is such transferability sustainable when transferring to or from some high-level tasks such as classification?
To evaluate the transferability of LIP between image restoration tasks, we first find three LIPs for the denoising, inpainting and super-resolution tasks respectively and then transfer between them, as shown in Fig.. We observe that a LIP winning ticket transferred from another image restoration task always yields restoration performance comparable with the single-image LIP found on the original task, sometimes even better, for examples in Fig.(a) and 7(g).
We then evaluate the transferability of LIP between the denoising task and image classification task on CIFAR-10. We apply the standard IMP on the DIP model, i.e., the hourglass architecture in, to obtain winning tickets on MNIST and CIFAR-10 datasets. We show the results of transferring the denoising LIP to MNIST in Fig.(a) and CIFAR-10 in 8(b) and the CIFAR-10 winning tickets to denoising task in Fig.). Although transferring denoising LIP to MNIST seems to yield winning tickets due to MNIST's low complexity, its transfer to CIFAR-10 is unsuccessful. Interestingly, the multi-image LIP becomes a worse ticket on CIFAR-10 than the single-image LIP. Moreover, transferring winning tickets on CIFAR-10 back to the denoising DIP task also fails to generate winning tickets that are comparable with denoising LIPs.
Based on those observations, we conjecture that the architecture priors needed to "win the lottery ticket" may substantially differ between low-level (reconstruction, restoration,...) and high-level (classification, recognition,...) vision tasks, but seems to be quite overlapped/shareable among different tasks that are all low-level (or high-level). The high-level task transferability of winning tickets has been found in prior works. We believe this interesting "incompatibility" between low-level and high-level tasks is an unstudied new direction, and leave this for our future work.
Background Deep neural networks (DNNs), in particular convolutional neural networks (CNNs), have been powerful tools for solving various image inverse problems such as denoising, inpainting, and super resolution. Conventional wisdom believes that is owing to DNNs' universal approximation ability and learning from massive training data. Yet, recent studies have revealed the specific architectures of CNNs have the inductive bias to represent and generate natural images well, and such favorable architecture inductive bias can work independently from fitting specific training sets.
For example, deep image prior (DIP) shows an untrained neural network can be used as a handcrafted prior that transfers well across multiple inverse problems. The authors attributed the success to the CNN architecture itself, that appeared to possess high noise impedance even with only random initializations. As another example, in compressive sensing,; replaced the common structural assumptions such as sparsity with a pretrained generative adversarial networks. The underlying rationale lies in that a pre-trained generator should (approximately) represent the notion of a vector being or more likely in the target domain such as natural images; in other words, a sample more like a natural image will be closer to the output range of the pre-trained generator. In this paper, we refer to such general-purpose image prior parameterized by (either untrained or pre-trained) DNNs as DNN-based image priors.
Recall that, classical image regularizers in the spatial or frequency domains are often not learningbased, or rely on compact learning models. On contrary, DNN-based image priors have a massive number of parameters (we compare the parameter numbers between the full model and the sparse subnetwork in Table 2), typically magnitudes more than the image (even the image size) size. The two extremes invite the natural question: Can we identify highly compact DNN-based image priors, that are same effective? We note that, diving into this question has twofold appeals. On the algorithmic side, that could help us understand further how the topology and connectivity of CNN architecture itself will affect the effectiveness of those priors, and to what extent sparsity could be relevant. On the practical side, if we could provide an affirmative answer to this question, that would potentially lead to more computationally savings when applying those DNN-based priors in practice, leading to faster restoration or computational imaging with them.
Towards the above question, the tool we refer to in this paper is the recently emerged Lottery Ticket Hypothesis (LTH). LTH suggests that every dense DNN has an extremely sparse "matching subnetwork", that can be trained in isolation to match the original dense DNN's accuracy. While the vanilla LTH studies training from random scratch, the latest works also extend similar findings to fine-tuning the pre-trained models. LTH has widespread success in image classification, language modeling, reinforcement learning and multi-modal learning, e.g.,.
Our Contributions Drawing inspirations from the LTH literature, we conjecture and empirically study a novel "lottery image prior" (LIP), stated as: Given an (untrained or trained) DNN-based image prior, it will have a sparse subnetwork that can be trained in isolation, to match the original DNN's performance when being applied as a prior to regularizing various image inverse problems.
Studying this new problem is, however, NOT a naive extension from the existing LTH methods, owing to several technical barriers: (a) till now, LTH has not been demonstrated for image inverse problem or DNN-based priors, to our best knowledge. Most LTH works studied discriminative tasks, with one exception. It is therefore uncertain whether high-sparsity DNN is still viable for reconstruction-oriented tasks; (b) existing LTH works typically require a full training set to locate the sparse subnetwork mask, whereas our LIP settings are only not data-rich. For example, DIP needs the DNN to be trained to overfit one specific image, making it drastically different from previous problems; (c) the objectives between finding the sparse mask (e.g., learning the prior) and fitting the sparse subnetwork (e.g., using the prior) are often unaligned in LIP problems. For example, DIP will overfit a corrupted input image (during which the sparse mask will be found) in order to reconstruct a clean output image (when the found sparse subnetwork will be used); the pre-trained generator will also be used towards a different goal (regularizing compressive sensing) from their original pre-training task (generating realistic images).
Our extensive experimental study confirms the existence of LIP in two representative settings: (i) image restoration with the deep image prior, using an untrained DNN (as shown in Fig.); and
(ii) compressive sensing image reconstruction, using a pre-trained GAN generator. Using iterative magnitude pruning (IMP) with surrogate tasks (the overview of our work paradigm is in Fig.), we can successfully locate the LIP subnetworks at the sparsity range of 20%-86.58% in setting i; and those at the sparsity range of 5%-36% in setting ii. Those LIP subnetworks also possess high transferability. For example, the LIP ticket found in the setting i transfer well across not only different images, but also different tasks such as denoising, inpainting and super-resolution. Our contributions are summarized below:
• The first comprehensive study on LTH in DNN-based image priors and inverse problems, establishing the "lottery image prior" (LIP) and demonstrating the prevailing relevance of LTH more broadly than previously typical settings. The last column (in blue) intends to display the results with the most extremely sparse subnetwork.
• The investigation of both untrained and pre-trained DNNs as image priors, verifying that LIP subnetworks can be found in both settings, by overcoming several impediments such as severely limited training data and task objective mismatch. • High transferability of LIP subnetworks across data/datasets, and various inverse problem tasks. Our finding reflects the underlying common image prior that is agnostic to specific data or task, through the lens of the CNN architecture together with sparsity.
In this section, we will investigate the lottery image prior (LIP) property under setting i, for deep image prior (DIP) models. Model performance is measured by PSNR (Peak Signal to Noise Ratio) between the restored images and the clean ground truths (SSIM (Structural Similarity) results are in supplementary materials). We run all experiments with three different random seeds. All images used for plotting results are summarized in Fig..
In DIP, for each degraded image to be restored, an untrained DNN will be over-fitted over that single image (with proper early stopping). This casts a crucial difference from typical LTH settings where winning tickets are identified using IMP over a distribution of data and are verified to be effective when trained on data from that specific distribution. Therefore, besides the usual properties in winning tickets such as the existence, the superiority over other pruning methods, and the effect of rewinding, we will strive to answer two more questions about LIP in DIP. The first question is: can we extend the IMP algorithm for single-image DIP to multi-image and thus improve the performance of winning tickets hopefully? The second question is to what extend the LIP found in DIP setting can be transferred? Answering these two questions not only provides a clearer picture of LIP, but also hints a new perspective for a deeper understanding of LTH.
In DIP setting, we first find the winning tickets with LIP property by implementing the single-image IMP on the hourglass model (i.e., the DIP model used in. We adopt the modified objectives using clean images as labels as we discussed in Sec. 3.2. The algorithm is described in detail in Algorithm 1. We apply the implemented algorithm on Set5  ( and Set14 datasets to obtain the sparse subnetworks 1 and evaluate these subnetworks on the denoising task.
Results of single-image IMP in Fig. (curves in red colors) verify that the LIP exists in the DIP setting. To be specific, during the IMP finding process when we use clean images in DIP objectives, we are able to find the winning tickets with LIP on untrained DNN at sparsity as high as 86.58%. While at the evaluation stage, when we only have access to the degraded images, we find that the winning tickets found with the modified objectives are still applicable, matching the dense performance at sparsity as high as 83.23%.
Besides the dense model baselines, we also compare the single-image IMP with Random Pruning, and a pruning-at-initialization method SNIP, whose results are also presented in Fig. in other colors. Specifically, we clearly observe from the first row in Fig. that the singleimage IMP for DIP outperforms random pruning and SNIP at a wide sparsity range [20%, 96%]. Interestingly, we find random prune is a good competitor to SNIP sometimes, but both of them suffer significant performance decrease at extreme sparsities (over 80%), where IMP still persists.
The Effect of Weight Rewinding In this part, we study the effect of weight rewinding Is Multi-image IMP A Good Extension in DIP? DIP over-fits an untrained DNN on a single image. A reasonable conjecture is that the DNN will be dedicated to the features in that specific image and so should the resulting winning ticket be if we apply single-image IMP to the DIP model. We then ask that whether we can co-consider multiple images during IMP for DIP models to find winning tickets that are suitable for more general features and thus yield better image restoration performance? To verify our idea, we propose a new multi-image IMP for DIP where we replace the single-image DIP objective with the average of multiple images during the IMP process. Note that all images will share the same fixed random code during IMP. Otherwise, different random codes will intervene with one another because they lead to different winning tickets, resulting in a share subnetwork with inferior performance. The algorithm is formally described in Algorithm 2.
We evaluate the multi-image IMP in two different settings: (i) cross-domain setting where we apply the multi-image IMP to the five images from Set5; (2) single-domain setting where we apply the multi-image IMP to five images of human faces with glasses. We think images from Set 5 are from more diversified domains because they include bird, butterfly and human face contents. We compare single-image IMP winning tickets found on the Baby and the Woman images from Set5 with the cross-domain ticket, and the single-image IMP winning tickets found on Face-1 and Face-2 with the single-domain ticket. Results presented in Fig. show that multi-image IMP significantly improves the quality of the winning tickets in the cross-domain setting, which is aligned with our previous hypothesis.
To what extent can LIP tickets be transferred? Transferability is an important metric to measure the usefulness of LIP in the DIP setting. It would significantly undermine the practical application of LIP if the winning tickets can only be dedicatedly found for each image. In this part, we evaluate the transferability of LIP for DIP models from three perspectives, i.e., the transferability between images, between image restoration tasks and between high-level tasks.
Observation 1: LIP can transfer across images. We identify the single-image LIPs for all image from Set5 and the multi-image LIP on Set5 using the new multi-image IMP proposed in the last part. Then we evaluate all the above LIPs on the Baby image, results being presented in Fig.. We can see that the LIPs found on other images, even those from the Bird and the Butterfly images, can perform comparably well with LIP dedicatedly found on the Baby image. This shows that LIP has reasonable transferability across images, even for those coming from slightly different domains (for example, the Baby and the Butterfly). We can also observe the fact that the multi-image LIP outperforms all single-image LIPs, as side support for the superiority of multi-image LIP in the cross-domain settings.
Observation 2: LIP can transfer across image restoration tasks but not to other high-level tasks.  We test the denoising LIP on MNIST and CIFAR10 datasets (left 2 figures). Note that we replace the last convolutional layer of DIP models with the linear layer and load the same initial weights. We evaluate the CIFAR LIP on the denoising task.
interests? Furthermore, if the above question has an affirmative answer, is such transferability sustainable when transferring to or from some high-level tasks such as classification?
To evaluate the transferability of LIP between image restoration tasks, we first find three LIPs for the denoising, inpainting and super-resolution tasks respectively and then transfer between them, as shown in Fig.. We observe that a LIP winning ticket transferred from another image restoration task always yields restoration performance comparable with the single-image LIP found on the original task, sometimes even better, for examples in Fig.(a) and 7(g).
We then evaluate the transferability of LIP between the denoising task and image classification task on CIFAR-10. We apply the standard IMP on the DIP model, i.e., the hourglass architecture in, to obtain winning tickets on MNIST and CIFAR-10 datasets. We show the results of transferring the denoising LIP to MNIST in Fig.(a) and CIFAR-10 in 8(b) and the CIFAR-10 winning tickets to denoising task in Fig.). Although transferring denoising LIP to MNIST seems to yield winning tickets due to MNIST's low complexity, its transfer to CIFAR-10 is unsuccessful. Interestingly, the multi-image LIP becomes a worse ticket on CIFAR-10 than the single-image LIP. Moreover, transferring winning tickets on CIFAR-10 back to the denoising DIP task also fails to generate winning tickets that are comparable with denoising LIPs.
Based on those observations, we conjecture that the architecture priors needed to "win the lottery ticket" may substantially differ between low-level (reconstruction, restoration,...) and high-level (classification, recognition,...) vision tasks, but seems to be quite overlapped/shareable among different tasks that are all low-level (or high-level). The high-level task transferability of winning tickets has been found in prior works. We believe this interesting "incompatibility" between low-level and high-level tasks is an unstudied new direction, and leave this for our future work.
For example, the LIP ticket found in the setting i transfer well across not only different images, but also different tasks such as denoising, inpainting and super-resolution.****Figure 5: We evaluate different single-image LIPs and the multi-image LIP on Baby image to measure LIP’s transferability (cross images).****For example, the LIP ticket found in the setting i transfer well across not only different images, but also different tasks such as denoising, inpainting and super-resolution.****Figure 7: Transferability (cross tasks) experimental results. We study the transferability of denoising LIP on the restoration tasks such as inpainting and super-resolution (SR); we also study the inpainting and SR LIP on the denoising task. We consider two SR scale factors = 4, 8.
Answer: We respectfully disagree. The found LIP subnetworks by multi-image based IMP are practical because they have the powerful transferability: 1) They can transfer across different images (Figure 5 illustrates this point); 2) They can successfully transfer across different restoration tasks such as denoising, inpainting and super-resolution (Figure 7 illustrates this point). And therefore, there is no need to do the IMP process again and again for obtaining desired LIPs in different tasks. We hope the reviewer will capture this crucial point and give a more fair assessment, instead of rushing to an ungrounded accusation of “useless”.