Question: This work shows that there exists a very sparse subnetwork whose performance can surpass the original dense network. To find such sparse network, it directly minimizes the MSE relying on the ground-truth image. In practice, however, the ground-truth image is always not accessible. How to obtain such sparse subnetwork in real applications?
Evidence: Datasets In setting i, we use the popular Set5 and Set14 datasets. Besides, we evaluate the transferability of subnetworks on image classification datasets such as and CIFAR10. In setting ii, our evaluation dataset is CelebA-HQ, following.
Subnetworks Consider a network f (x; θ) parametered by θ with input x, then a subnetwork is defined as f (x; m θ), where ∈ {0, 1} d , d = ||θ|| 0 and is the element-wise product. Let A T t (f (x; θ)) to be the training algorithm, that is, training model f (x; θ) on the specific task T with t iterations. We also denote the random initialization weight as θ 0 and the pre-trained weight as θ p ; θ i as weight at the i-th training iteration and E T (f (x; θ)) the model performance evaluation.
Finding Subnetworks Following the definitions of, we define that if the subnetworks is matching, it satisfies the following conditions (we use θ p for example, to denote a pre-trained lottery ticket; θ 0 can be defined likewise):
That is a matching subnetwork that performs no worse than the dense model under the same training algorithm A T and the evaluation metric E T . Similarly, we define the winning tickets: if a matching subnetwork f (x; m θ) has θ = θ p , then it is the winning tickets under the training algorithm A T .
In practice, we find the matching subnetworks in the following ways: 1) IMP on the dense model with only one image (applicable to the image restoration task on the same domain of images); 2) IMP on the dense model with various kinds of images through weight-sharing (applicable to the image restoration tasks on images from different domains); 3) LTH IMP on the dense model with the sub-dataset (applicable to the CS task with pre-trained GANs).
Ticket Finding Objectives When finding LIP subnetwork in untrained DNN, i.e., DIP, there is a problem that may be easily overlooked: what is the target during IMP training? Specifically, since the training target in Eq. 1 for DIP is x (the corrupted image), the model parameters may easily overfit the corrupted image during IMP training if we do not fine-tune the training epochs. Then the obtained mask will contain the information of corrupted images instead of the desired image prior (but we can also find the effective subnetworks by noisy targets, experiments are summarized in Table 3 and 4 in Supplementary Materials). Therefore, we modify the optimization objectives of DIP during IMP training: arg min θ E(f θ (z); x) to ensure that model parameters learn the clean image prior.
Evaluations of Subnetworks After obtaining the matching subnetwork f (x; m θ), we evaluate their performances by 1) resetting the model parameter θ to initialization weight θ 0 ' 2) adding the mask m to the model; 3) training the model to another N iterations. We evaluate the DIP subnetworks performances mainly through Peak Signal to Noise Ratio (PSNR) values, plus reconstruction errors (defined in for compressive sensing with GANs.
Pruning Methods We use the standard pruning method, which iteratively prune the 20% of the model weight each time. For setting i, our basic algorithm performs IMP over just one single image (i.e., DIP's default setting), and the algorithm is summarized in Algorithm 1 (Appendix). We further design an extended algorithm, that can perform IMP for DIP over multiple images, through backbone weight sharing: the algorithm is outlined in Algorithm 2 (Appendix). We will discuss the algorithm variants in Section 4. For setting ii, we use IMP to find winning tickets in pre-trained GANs, following the routine in. In each iteration of IMP, we will first fine-tune the pre-trained GANs on a (sub-)dataset, prune 20% of the remaining weights, and reset to the pre-trained weights. Note that we also include other pruning methods to compare the effectiveness of the matching subnetworks such as random pruning (randomly generate the sparsity mask m ), and pruning-at-initialization methods, e.g., SNIP.
We used clean images as the training targets since this made it easy to control the learning curves and avoid overfitting. Even using one clean image target, the obtained LIP is practical due to the powerful transferability to other image with no clean groundtruth.Moreover, we can use the noisy target in finding the same effective LIPs as long as we perform “early-stopping” in the optimization process. Note that it was the same trick mentioned when training the original DIP for denoising tasks (Ulyanov et al., 2018) and was identified to be essential to the original DIP’s success in (Heckel & Hand, 2019). In Table. 4, we conduct the comparison experiments of clean and noisy image targets (experimental settings are summarized in the caption of the table) and we find that there are no large differences between the results of noisy image targets and clean image targets.****Note that the used image is F16.png, the evaluation metric is PSNR and the training iteration number is 3000 to capture the “early-stopping” phenomenon. The results suggest that there are no large differences (with PSNR value smaller than 0.05) between the performances of subnetworks with clean and noisy image targets (the sparsity ranges from 0% to 95%).****Figure 13: Learning curves using four different training targets: a clean image (Baby.png), the same image added with noises, the same randomly scrambled and white noise. Note that we use four different models: the LIP subnetwork (S = 89%), randomly pruned subnetwork (S = 89%), SNIP subnetwork (S = 89%) and the dense model (S = 0%). And we trained them in isolation in the same experimental settings for 10000 iterations.****We use the standard pruning method (Han et al., 2015), which iteratively prune the 20% of the model weight each time.****We further compare the training convergence curves of different subnetworks on the restoration task.
Answer: 1) We used clean images as the training targets since this made it easy to control the learning curves and avoid overfitting. Even using one clean image target, the obtained LIP is practical due to the powerful transferability to other images with no clean ground-truth (discussed in [Cons2.] in detail). 2) We can also use the noisy target in finding the same effective LIPs as long as we perform “early-stopping” in the optimization process. Note that it was the same trick mentioned when training the original DIP for denoising tasks [1] and was identified to be essential to the original DIP’s success in [2]. 3) In fact, our results support both using clean and noisy targets (we added the experiments in Table 4 in the section Supplementary Materials). We actually find there are no large differences between them when sticking to the same appropriate experimental setting. 4) Specifically in Table 4, we conducted the IMP experiments on the DIP model with clean and noisy targets on the same image (F16.png, presented in Figure 13). Note that both DIP optimization iteration numbers are 3000 (the number is recommended by the paper “Deep Image Prior” [1] as ``early stopping”) and prune 20% model parameters every IMP epochs. Interestingly, we find there are no large differences (with PSNR value smaller than 0.05) between the performances of subnetworks with clean and noisy image targets (the sparsity ranges from 0% to 95%). Moreover, we add the experiments to explore the convergence curves of subnetworks obtained from different methods. And we also investigate their sparsity distributions per layer. Details are in [Cons2.] Reviewer wMsj. [1] Ulyanov, Dmitry, Andrea Vedaldi, and Victor Lempitsky. "Deep Image Prior." CVPR, 2018. [2] Heckel, Reinhard, and Paul Hand. "Deep decoder: Concise Image Representations from Untrained Non-convolutional Networks." ICLR 2019.