Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes Prospect Pruning (ProsPr) to handle the problem of short-sightedness of existing methods. The main idea is to use meta-gradients through the first few steps of optimization to determine which weights to prune.
main_review: 1) Since the current methods are insufficient to enable this optimization and lead to a large degradation in model performance, the authors proposed a new method to identify a fundamental limitation, namely that their saliency criteria look at a single step at the start of training without considering the trainability of the network. The paper tackles an important problem. 
2) The paper is well organized and easy to follow. 
3) It would be better if the authors can validate their method on more tasks, such as human segmentation and image denoising, etc.
summary_of_the_review: See the main review.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work focuses on weight pruning at initialization. In this paper, the authors point out an important problem that the pruned subnetwork at initialization is going to be trained and previous prune-at-init methods ignore this fact. As a result, these prune-at-init methods ignore the trainability of weights. This paper proposes to use meta-gradients through the first few steps of optimization to determine which weights to prune. Experimental results show that ProsPr (this paper) achieves state-of-the-art pruning performance.
main_review: [Strengths]
- The proposed ProsPr is a simple and effective pruning method that prunes weights at initialization. The authors propose to use meta-gradients to compute saliency scores when pruning weights.

- The higher-order temrs in meta-gradients can be further dropped (Equ. 16) such that saving the initial weights $w_{init}$ is enough when computing meta-gradients.

- Experimental results show that ProsPr achieves state-of-the-art pruning performance.


[Weaknesses]
- Writing needs improvement. E.g., "Many methods attempts...", "...the original, unpruned, model" and "Previous works that prune at the start have training have not been...".

- Although experimental results show that using estimation over several steps of gradient descent improves pruning performance, the connection between meta-gradients and motivation (i.e., the trainability of weights) is not strong and convincing enough.

summary_of_the_review: It is a good technical paper, but it requires better writing. Although the proposed ProsPr, which is implemented by meta-gradients, is effective, I think the novelty is limited.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work studies the problem of pruning neural networks at initialization. It first identifies that the saliency score defined by the existing method SNIP has room for improvement. Specifically, the authors propose a method named prospect pruning to take into account the sequence of weight updates to determine the pruning mask. The experimental results on Tiny ImageNet and CIFAR show that the proposed method achieves better performance than existing methods of pruning at initialization.
main_review: Strengths:

+ This paper is well-written.

+ the proposed method is easy to implement.

+ The experimental results support the claim that the proposed method achieves higher accuracy compared to existing pruning-at-init methods.

Weaknesses:

1. According to the proposed first-order approximation, it seems that the proposed method is limited to apply with the optimization method SGD. There is a variety of models that are trained with other optimization methods, like Adam, Adamw, and Adabound. It would be helpful to provide a discussion on the direction of how to adapt the proposed method to work with generic (or specific) optimization methods.

2. The first-order approximation (Eq. (17)) may not be well studied. The following questions are not clear: Is it sensitive to how $W_{init}$ is initialized (e.g., xavier_uniform, kaiming_normal, etc)? Is it robust to the sequence data? How does the number $M$ of initial training steps affect the accuracy and sparsity?

3. An important step to the first-order approximation is to drop the higher-order terms of Eq. (15), i.e., $\prod_{m=1}^{M} I - \alpha \nabla_{w_{m−1}}^{2} \mathcal{L} (w_{m−1}; \mathcal{D})$. A follow-up question is, what is the error bound (w.r.t. M) between the approximated meta-gradient and the ground-truth meta-gradient? For example, assume that the loss function meets the second-order necessary condition $\nabla_{w_{m−1}}^{2} \mathcal{L} (w_{m−1}; \mathcal{D}) \succeq 0$, as $M$ gets large enough, the operation $\prod_{m=1}^{M} I - \alpha \nabla_{w_{m−1}}^{2} \mathcal{L} (w_{m−1}; \mathcal{D})$ could lead to vanishing and exploding gradients. However, the approximated gradient $\nabla_{w_{M}} \mathcal{L}( w_{m−1}; \mathcal{D}) w_{init}$ seems to be still robust. How to interpret the discrepancy?

4. Figure 2 shows that pruning at convergence achieves a better trade-off between accuracy and sparsity than pruning at initialization. So it is not clear that the advantage(s) of the techniques of pruning at initialization over the ones of pruning at initialization. I looked into the introduction and the related work of pruning at initialization. I didn't find the points related to this. It would be better to explicitly discuss the comparison to make the motivation clearer. If I missed something, please point it out.

5. The algorithm presumes that $w_{init}$ is randomly initialized. It would be interesting to verify if the proposed method behaves consistently with $w_{init}$ that is learned from unlabeled samples, which is a common practice in self-supervised learning. 

6. The related work of meta-gradients misses a related work, that is, meta-gradient in semi-supervised learning [r1].

7. This is not a critical comment, but I'd like to bring it to the discussion. Regarding the novelty, the proposed method relies on the saliency score (Eq. 1 and Eq. 11) defined in SNIP. This may make this work arguably look incremental. The authors present a discussion "our method, to be introduced in Section 3, also relies on computing the saliency scores for each element in the mask but uses a more sophisticated loss function to incorporate the notion of trainability into the objective." Still, I think the work is a bit weak in terms of the novelty of the methodology.

References:

[r1] Xiao, Taihong, Xin-Yu Zhang, Haolin Jia, Ming-Ming Cheng, and Ming-Hsuan Yang. "Semi-Supervised Learning with Meta-Gradient." In International Conference on Artificial Intelligence and Statistics, pp. 73-81. PMLR, 2021.
summary_of_the_review: This paper is well-written and provides detailed derivations. As a result, the proposed method has the potential to apply to a wide range of real-world applications. On the other hand, I think the proposed method may not be well studied and the experiment can be improved by adding the results of using $w_{init}$ learned from unlabeled data. Therefore, my initial recommendation is "marginally below the acceptance threshold". I will go over the review comments by other reviewers and the responses by the authors and adjust my recommendation accordingly.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work proposed a new efficient pruning method, by leveraging both loss sensitivity ("saliency score") during a few initial training steps. The authors leveraged meta-gradients with appropriate approximations to stabilize and speed up the pruning.
main_review: Strength:
The overall method is well explained and easy to follow.

Weakness:
The performance of this work can be decoupled into two aspects:
1) If the meta-gradients w.r.t. masks ($\textbf{c}$ is vital, or single-step gradients (i.e. $M=1$) may also work.
2) If dropping small terms (Eq. 15 to 16) is indeed negligible.

I did not see ablation studies on these two aspects. Therefore I propose some further studies:
1) If the single-step gradient is used (no meta-gradients) to update the mask with the same total number of iterations (e.g. 1024 steps with a batch size of 256 for ResNet-50), what will be the pruning results? This is actually more GPU-memory efficient since no extra computational graphs are stored. Both "pruning by optimizing the mask" (this work) and magnitude pruning could be studied.
2) If we do not drop small terms in Eq. 15, what will be the pruning results? Smaller $M$ would be acceptable if storing computational graph is GPU-memory consuming.

In short, I think it is important to understand which part is vital to pruning: 1) optimizing the mask or magnitude pruning; 2) mata-gradients or just single-step gradients with more steps; 3) dropping or keeping the small terms.
summary_of_the_review: I generally think the novelty of this work is a bit trivial. Leveraging information from early training steps is a reasonable strategy, but is still straightforward.
review_title: null

Meta-review: This paper is proposed to address neural network pruning at initialization with the help of meta-gradients considering the high-order relations between loss and optimization of trainable sub-network. The paper is well organized and written with the clear logic. The discussions of related works, as well as their limitations, are comprehensive. To verify the proposed method, the authors have tested it on various benchmarks with different settings. Overall, the meta-learning idea for model pruning is relatively new, which may bring more inspirations to the community.
Decision: Accept (Poster)