Question: Why the specific lottery image prior structure is superior to other structures in DIP tasks?
Evidence: Background Deep neural networks (DNNs), in particular convolutional neural networks (CNNs), have been powerful tools for solving various image inverse problems such as denoising, inpainting, and super resolution. Conventional wisdom believes that is owing to DNNs' universal approximation ability and learning from massive training data. Yet, recent studies have revealed the specific architectures of CNNs have the inductive bias to represent and generate natural images well, and such favorable architecture inductive bias can work independently from fitting specific training sets.
For example, deep image prior (DIP) shows an untrained neural network can be used as a handcrafted prior that transfers well across multiple inverse problems. The authors attributed the success to the CNN architecture itself, that appeared to possess high noise impedance even with only random initializations. As another example, in compressive sensing,; replaced the common structural assumptions such as sparsity with a pretrained generative adversarial networks. The underlying rationale lies in that a pre-trained generator should (approximately) represent the notion of a vector being or more likely in the target domain such as natural images; in other words, a sample more like a natural image will be closer to the output range of the pre-trained generator. In this paper, we refer to such general-purpose image prior parameterized by (either untrained or pre-trained) DNNs as DNN-based image priors.
Recall that, classical image regularizers in the spatial or frequency domains are often not learningbased, or rely on compact learning models. On contrary, DNN-based image priors have a massive number of parameters (we compare the parameter numbers between the full model and the sparse subnetwork in Table 2), typically magnitudes more than the image (even the image size) size. The two extremes invite the natural question: Can we identify highly compact DNN-based image priors, that are same effective? We note that, diving into this question has twofold appeals. On the algorithmic side, that could help us understand further how the topology and connectivity of CNN architecture itself will affect the effectiveness of those priors, and to what extent sparsity could be relevant. On the practical side, if we could provide an affirmative answer to this question, that would potentially lead to more computationally savings when applying those DNN-based priors in practice, leading to faster restoration or computational imaging with them.
Towards the above question, the tool we refer to in this paper is the recently emerged Lottery Ticket Hypothesis (LTH). LTH suggests that every dense DNN has an extremely sparse "matching subnetwork", that can be trained in isolation to match the original dense DNN's accuracy. While the vanilla LTH studies training from random scratch, the latest works also extend similar findings to fine-tuning the pre-trained models. LTH has widespread success in image classification, language modeling, reinforcement learning and multi-modal learning, e.g.,.
Our Contributions Drawing inspirations from the LTH literature, we conjecture and empirically study a novel "lottery image prior" (LIP), stated as: Given an (untrained or trained) DNN-based image prior, it will have a sparse subnetwork that can be trained in isolation, to match the original DNN's performance when being applied as a prior to regularizing various image inverse problems.
Studying this new problem is, however, NOT a naive extension from the existing LTH methods, owing to several technical barriers: (a) till now, LTH has not been demonstrated for image inverse problem or DNN-based priors, to our best knowledge. Most LTH works studied discriminative tasks, with one exception. It is therefore uncertain whether high-sparsity DNN is still viable for reconstruction-oriented tasks; (b) existing LTH works typically require a full training set to locate the sparse subnetwork mask, whereas our LIP settings are only not data-rich. For example, DIP needs the DNN to be trained to overfit one specific image, making it drastically different from previous problems; (c) the objectives between finding the sparse mask (e.g., learning the prior) and fitting the sparse subnetwork (e.g., using the prior) are often unaligned in LIP problems. For example, DIP will overfit a corrupted input image (during which the sparse mask will be found) in order to reconstruct a clean output image (when the found sparse subnetwork will be used); the pre-trained generator will also be used towards a different goal (regularizing compressive sensing) from their original pre-training task (generating realistic images).
Our extensive experimental study confirms the existence of LIP in two representative settings: (i) image restoration with the deep image prior, using an untrained DNN (as shown in Fig.); and
(ii) compressive sensing image reconstruction, using a pre-trained GAN generator. Using iterative magnitude pruning (IMP) with surrogate tasks (the overview of our work paradigm is in Fig.), we can successfully locate the LIP subnetworks at the sparsity range of 20%-86.58% in setting i; and those at the sparsity range of 5%-36% in setting ii. Those LIP subnetworks also possess high transferability. For example, the LIP ticket found in the setting i transfer well across not only different images, but also different tasks such as denoising, inpainting and super-resolution. Our contributions are summarized below:
• The first comprehensive study on LTH in DNN-based image priors and inverse problems, establishing the "lottery image prior" (LIP) and demonstrating the prevailing relevance of LTH more broadly than previously typical settings. The last column (in blue) intends to display the results with the most extremely sparse subnetwork.
• The investigation of both untrained and pre-trained DNNs as image priors, verifying that LIP subnetworks can be found in both settings, by overcoming several impediments such as severely limited training data and task objective mismatch. • High transferability of LIP subnetworks across data/datasets, and various inverse problem tasks. Our finding reflects the underlying common image prior that is agnostic to specific data or task, through the lens of the CNN architecture together with sparsity.
Figure 11: Layer-wise sparsity ratio results of LIP, SNIP and randomly pruned tickets. Note that we summarize the sparsity ratio of each layer: the ratio of the number of parameters whose values are equal to zero to the number of total parameters of the layer. And the x-axis of these figures is composed of the serial numbers of model layers. We sampled subnetworks with four different sparsities (sparsity = 36%, 59%, 89%, 95%) to observe.****Figure 13: Learning curves using four different training targets: a clean image (Baby.png), the same image added with noises, the same randomly scrambled and white noise. Note that we use four different models: the LIP subnetwork (S = 89%), randomly pruned subnetwork (S = 89%), SNIP subnetwork (S = 89%) and the dense model (S = 0%). And we trained them in isolation in the same experimental settings for 10000 iterations.****Figure 2: LIP visual results: inpainting (row 1), super-resolution (rows 2/3) and denoising (row 4). The last column (in blue) intends to display the results with the most extremely sparse subnetwork.****To better explain the success of the LIP subnetworks, inspired by the Figure 2 of (Ulyanov et al., 2018), we further train the obtained subnetworks (LIP, SNIP and random pruning) with four different targets: 1) a natural image, 2) the same added with noise, 3) the same after randomly permuting the pixels and 4) the white noise.****Interestingly, we also find that SNIP subnetworks perform similarly with the dense model. Meanwhile, the parametrization of LIP subnetworks offers the higher impedance to noise and the lower impedance to signal than the dense model, which indicates that the separation of high frequency and low frequency is more obvious for winning network architectures.
Answer: Thanks for the advice. We add the experiments of studying the layer-wise sparsity ratio and the learning curves of different subnetworks, to add to the interpretability of what is special about LIP subnetworks. The results are summarized in Figure 11 and 13 in the section - Supplementary Materials of the paper. As shown in Figure 11, the structure of the LIP subnetwork is drastically different from those found by SNIP and random pruning, in particular the distribution of layerwise sparsity ratios. LIP tends to preserve weights of the earlier layers (closer to the input), while pruning the latter layers more aggressively (e.g, Figure 11(a).). In contrast, SNIP tends to prune much more of the earlier layers compared to the latter ones. Random pruning by default prunes each layer at approximately the same ratio. Comparing the three methods seems to suggest that for finding effective and transferable LIP subnetworks, specifically keeping more weights at earlier layers more is important. This is an explainable finding because, in image restoration tasks, the low-level features (color, texture, shape, etc.) presumably matter more and are more transferable,  than the high-level features (object categories, etc.). The earlier layers are known to capture more low-level image features, hence contributing more to retraining the image restoration performance with DIP. To better explain the success of the LIP subnetworks, inspired by the Figure 2 of the paper “Deep Image Prior”, we further train the obtained subnetworks (LIP, SNIP and random pruning) with four different targets: 1) a natural image, 2) the same added with noise, 3) the same after randomly permuting the pixels and 4) the white noise. We also train the dense model as the baseline. The experimental results are summarized in Figure 13. We first observe that for LIP, SNIP and dense models, the optimization converges much faster in case 1) and 2) than in case 3) and 4). But the randomly pruned subnetworks have failed in all cases. Interestingly, we also find that SNIP subnetworks perform similarly with the dense model. Meanwhile, the parametrization of LIP subnetworks offers a higher impedance to noise and a lower impedance to signal than the dense model, which indicates that the separation of high frequency and low frequency is more obvious for winning network architectures.