Question: It is not clear how the sparsity pattern looks like for the considered hour-glass network. Isn’t this one of the important points of pruning as advocated in the introduction? A visualization of the sparsity pattern per network layers would be very insightful. It would also be helpful to see if pruning creates a bottleneck in the network and where that bottleneck is.
Evidence: In this section, we will investigate the lottery image prior (LIP) property under setting i, for deep image prior (DIP) models. Model performance is measured by PSNR (Peak Signal to Noise Ratio) between the restored images and the clean ground truths (SSIM (Structural Similarity) results are in supplementary materials). We run all experiments with three different random seeds. All images used for plotting results are summarized in Fig..
In DIP, for each degraded image to be restored, an untrained DNN will be over-fitted over that single image (with proper early stopping). This casts a crucial difference from typical LTH settings where winning tickets are identified using IMP over a distribution of data and are verified to be effective when trained on data from that specific distribution. Therefore, besides the usual properties in winning tickets such as the existence, the superiority over other pruning methods, and the effect of rewinding, we will strive to answer two more questions about LIP in DIP. The first question is: can we extend the IMP algorithm for single-image DIP to multi-image and thus improve the performance of winning tickets hopefully? The second question is to what extend the LIP found in DIP setting can be transferred? Answering these two questions not only provides a clearer picture of LIP, but also hints a new perspective for a deeper understanding of LTH.
In DIP setting, we first find the winning tickets with LIP property by implementing the single-image IMP on the hourglass model (i.e., the DIP model used in. We adopt the modified objectives using clean images as labels as we discussed in Sec. 3.2. The algorithm is described in detail in Algorithm 1. We apply the implemented algorithm on Set5  ( and Set14 datasets to obtain the sparse subnetworks 1 and evaluate these subnetworks on the denoising task.
Results of single-image IMP in Fig. (curves in red colors) verify that the LIP exists in the DIP setting. To be specific, during the IMP finding process when we use clean images in DIP objectives, we are able to find the winning tickets with LIP on untrained DNN at sparsity as high as 86.58%. While at the evaluation stage, when we only have access to the degraded images, we find that the winning tickets found with the modified objectives are still applicable, matching the dense performance at sparsity as high as 83.23%.
Besides the dense model baselines, we also compare the single-image IMP with Random Pruning, and a pruning-at-initialization method SNIP, whose results are also presented in Fig. in other colors. Specifically, we clearly observe from the first row in Fig. that the singleimage IMP for DIP outperforms random pruning and SNIP at a wide sparsity range [20%, 96%]. Interestingly, we find random prune is a good competitor to SNIP sometimes, but both of them suffer significant performance decrease at extreme sparsities (over 80%), where IMP still persists.
The Effect of Weight Rewinding In this part, we study the effect of weight rewinding Is Multi-image IMP A Good Extension in DIP? DIP over-fits an untrained DNN on a single image. A reasonable conjecture is that the DNN will be dedicated to the features in that specific image and so should the resulting winning ticket be if we apply single-image IMP to the DIP model. We then ask that whether we can co-consider multiple images during IMP for DIP models to find winning tickets that are suitable for more general features and thus yield better image restoration performance? To verify our idea, we propose a new multi-image IMP for DIP where we replace the single-image DIP objective with the average of multiple images during the IMP process. Note that all images will share the same fixed random code during IMP. Otherwise, different random codes will intervene with one another because they lead to different winning tickets, resulting in a share subnetwork with inferior performance. The algorithm is formally described in Algorithm 2.
We evaluate the multi-image IMP in two different settings: (i) cross-domain setting where we apply the multi-image IMP to the five images from Set5; (2) single-domain setting where we apply the multi-image IMP to five images of human faces with glasses. We think images from Set 5 are from more diversified domains because they include bird, butterfly and human face contents. We compare single-image IMP winning tickets found on the Baby and the Woman images from Set5 with the cross-domain ticket, and the single-image IMP winning tickets found on Face-1 and Face-2 with the single-domain ticket. Results presented in Fig. show that multi-image IMP significantly improves the quality of the winning tickets in the cross-domain setting, which is aligned with our previous hypothesis.
To what extent can LIP tickets be transferred? Transferability is an important metric to measure the usefulness of LIP in the DIP setting. It would significantly undermine the practical application of LIP if the winning tickets can only be dedicatedly found for each image. In this part, we evaluate the transferability of LIP for DIP models from three perspectives, i.e., the transferability between images, between image restoration tasks and between high-level tasks.
Observation 1: LIP can transfer across images. We identify the single-image LIPs for all image from Set5 and the multi-image LIP on Set5 using the new multi-image IMP proposed in the last part. Then we evaluate all the above LIPs on the Baby image, results being presented in Fig.. We can see that the LIPs found on other images, even those from the Bird and the Butterfly images, can perform comparably well with LIP dedicatedly found on the Baby image. This shows that LIP has reasonable transferability across images, even for those coming from slightly different domains (for example, the Baby and the Butterfly). We can also observe the fact that the multi-image LIP outperforms all single-image LIPs, as side support for the superiority of multi-image LIP in the cross-domain settings.
Observation 2: LIP can transfer across image restoration tasks but not to other high-level tasks.  We test the denoising LIP on MNIST and CIFAR10 datasets (left 2 figures). Note that we replace the last convolutional layer of DIP models with the linear layer and load the same initial weights. We evaluate the CIFAR LIP on the denoising task.
interests? Furthermore, if the above question has an affirmative answer, is such transferability sustainable when transferring to or from some high-level tasks such as classification?
To evaluate the transferability of LIP between image restoration tasks, we first find three LIPs for the denoising, inpainting and super-resolution tasks respectively and then transfer between them, as shown in Fig.. We observe that a LIP winning ticket transferred from another image restoration task always yields restoration performance comparable with the single-image LIP found on the original task, sometimes even better, for examples in Fig.(a) and 7(g).
We then evaluate the transferability of LIP between the denoising task and image classification task on CIFAR-10. We apply the standard IMP on the DIP model, i.e., the hourglass architecture in, to obtain winning tickets on MNIST and CIFAR-10 datasets. We show the results of transferring the denoising LIP to MNIST in Fig.(a) and CIFAR-10 in 8(b) and the CIFAR-10 winning tickets to denoising task in Fig.). Although transferring denoising LIP to MNIST seems to yield winning tickets due to MNIST's low complexity, its transfer to CIFAR-10 is unsuccessful. Interestingly, the multi-image LIP becomes a worse ticket on CIFAR-10 than the single-image LIP. Moreover, transferring winning tickets on CIFAR-10 back to the denoising DIP task also fails to generate winning tickets that are comparable with denoising LIPs.
Based on those observations, we conjecture that the architecture priors needed to "win the lottery ticket" may substantially differ between low-level (reconstruction, restoration,...) and high-level (classification, recognition,...) vision tasks, but seems to be quite overlapped/shareable among different tasks that are all low-level (or high-level). The high-level task transferability of winning tickets has been found in prior works. We believe this interesting "incompatibility" between low-level and high-level tasks is an unstudied new direction, and leave this for our future work.
Datasets In setting i, we use the popular Set5 and Set14 datasets. Besides, we evaluate the transferability of subnetworks on image classification datasets such as and CIFAR10. In setting ii, our evaluation dataset is CelebA-HQ, following.
Subnetworks Consider a network f (x; θ) parametered by θ with input x, then a subnetwork is defined as f (x; m θ), where ∈ {0, 1} d , d = ||θ|| 0 and is the element-wise product. Let A T t (f (x; θ)) to be the training algorithm, that is, training model f (x; θ) on the specific task T with t iterations. We also denote the random initialization weight as θ 0 and the pre-trained weight as θ p ; θ i as weight at the i-th training iteration and E T (f (x; θ)) the model performance evaluation.
Finding Subnetworks Following the definitions of, we define that if the subnetworks is matching, it satisfies the following conditions (we use θ p for example, to denote a pre-trained lottery ticket; θ 0 can be defined likewise):
That is a matching subnetwork that performs no worse than the dense model under the same training algorithm A T and the evaluation metric E T . Similarly, we define the winning tickets: if a matching subnetwork f (x; m θ) has θ = θ p , then it is the winning tickets under the training algorithm A T .
In practice, we find the matching subnetworks in the following ways: 1) IMP on the dense model with only one image (applicable to the image restoration task on the same domain of images); 2) IMP on the dense model with various kinds of images through weight-sharing (applicable to the image restoration tasks on images from different domains); 3) LTH IMP on the dense model with the sub-dataset (applicable to the CS task with pre-trained GANs).
Ticket Finding Objectives When finding LIP subnetwork in untrained DNN, i.e., DIP, there is a problem that may be easily overlooked: what is the target during IMP training? Specifically, since the training target in Eq. 1 for DIP is x (the corrupted image), the model parameters may easily overfit the corrupted image during IMP training if we do not fine-tune the training epochs. Then the obtained mask will contain the information of corrupted images instead of the desired image prior (but we can also find the effective subnetworks by noisy targets, experiments are summarized in Table 3 and 4 in Supplementary Materials). Therefore, we modify the optimization objectives of DIP during IMP training: arg min θ E(f θ (z); x) to ensure that model parameters learn the clean image prior.
Evaluations of Subnetworks After obtaining the matching subnetwork f (x; m θ), we evaluate their performances by 1) resetting the model parameter θ to initialization weight θ 0 ' 2) adding the mask m to the model; 3) training the model to another N iterations. We evaluate the DIP subnetworks performances mainly through Peak Signal to Noise Ratio (PSNR) values, plus reconstruction errors (defined in for compressive sensing with GANs.
Pruning Methods We use the standard pruning method, which iteratively prune the 20% of the model weight each time. For setting i, our basic algorithm performs IMP over just one single image (i.e., DIP's default setting), and the algorithm is summarized in Algorithm 1 (Appendix). We further design an extended algorithm, that can perform IMP for DIP over multiple images, through backbone weight sharing: the algorithm is outlined in Algorithm 2 (Appendix). We will discuss the algorithm variants in Section 4. For setting ii, we use IMP to find winning tickets in pre-trained GANs, following the routine in. In each iteration of IMP, we will first fine-tune the pre-trained GANs on a (sub-)dataset, prune 20% of the remaining weights, and reset to the pre-trained weights. Note that we also include other pruning methods to compare the effectiveness of the matching subnetworks such as random pruning (randomly generate the sparsity mask m ), and pruning-at-initialization methods, e.g., SNIP.
In DIP setting, we first find the winning tickets with LIP property by implementing the single-image IMP on the hourglass model (i.e., the DIP model used in Ulyanov et al. (2018). We adopt the modified objectives using clean images as labels as we discussed in Sec. 3.2. The algorithm is described in detail in Algorithm 1.****In setting i, we use the popular Set5 (Bevilacqua et al., 2012) and Set14 (Zeyde et al., 2010) datasets. Besides, we evaluate the transferability of subnetworks on image classification datasets such as MNIST (LeCun et al., 2010) and CIFAR10 (Krizhevsky et al., 2009).
Answer: We clarified the ambiguity in the original manuscript, such as the source of randomness when using clean labels for IMP and how to transfer to image classification. Thank you again for pointing them out, which definitely helped to make our write-up better.