Question: What is the success of transferability of LIP (studying the sparsity pattern per network layer will help) and how to transfer DIP models to image classifications?
Evidence: Figure 11: Layer-wise sparsity ratio results of LIP, SNIP and randomly pruned tickets. Note that we summarize the sparsity ratio of each layer: the ratio of the number of parameters whose values are equal to zero to the number of total parameters of the layer. And the x-axis of these figures is composed of the serial numbers of model layers. We sampled subnetworks with four different sparsities (sparsity = 36%, 59%, 89%, 95%) to observe.****Figure 13: Learning curves using four different training targets: a clean image (Baby.png), the same image added with noises, the same randomly scrambled and white noise. Note that we use four different models: the LIP subnetwork (S = 89%), randomly pruned subnetwork (S = 89%), SNIP subnetwork (S = 89%) and the dense model (S = 0%). And we trained them in isolation in the same experimental settings for 10000 iterations.****As shown in Fig. 11, the structure of the LIP subnetwork is drastically different from those found by SNIP and random pruning, in particular the distribution of layerwise sparsity ratios. LIP tends to preserve weights of the earlier layers (closer to the input), while pruning the latter layers more aggressively (e.g, Fig. 11(a)). In contrast, SNIP tends to prune much more of the earlier layers compared to the latter ones. Random pruning by default prunes each layer at approximately the same ratio.Comparing the three methods seem to suggest that for finding effective and transferable LIP subnetworks, to specifically keep more weights at earlier layers more is important. That is an explainable finding, since for image restoration tasks, the low-level features (color, texture, shape, etc.) presumably matter more and are more transferable, than the high-level features (object categories, etc.). The earlier layers are known to capture more of low-level image features, hence contributing more to retraining the image restoration performance with DIP.
Answer: We have added the results of layer-wise sparsity ratio of different subnetworks (LIP, SNIP, Randomly Pruned) in Figure 11 and also plotted the convergence curves in Figure 13 of these subnetworks to better explore the reasons for the powerful transferability of LIP subnetworks. We have found that LIP tends to preserve weights of the earlier layers (closer to the input), while pruning the latter layers more aggressively (e.g, Figure 11(a).). In contrast, SNIP tends to prune much more of the earlier layers compared to the latter ones. Random pruning by default prunes each layer at approximately the same ratio. Comparing the three methods seems to suggest that to find effective and transferable LIP subnetworks, specifically keeping more weights at earlier layers more is important. That is an explainable finding, since for image restoration tasks, the low-level features (color, texture, shape, etc.) presumably matter more and are more transferable,  than the high-level features (object categories, etc.). The earlier layers are known to capture more low-level image features, hence contributing more to retraining the image restoration performance with DIP. The detailed discussions are included in [Cons1.] Reviewer wMsj.