Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper attempts to explain why magnitude-based and gradient-based pruning is effective only after the network has converged, ie it has learned the given task. The aothors attempt to shed some light on this issue via Koopman theory, borrowing from the field of dynamic systems.
main_review: I guess the main strength of this paper is the perspective the authors use to look at this problem, which to me sounds new. Other merits of this paper is that it is readable and teh authors have made an effort to provide a tutorial on Koopman theory, depspite this subject being quite specialistic.
summary_of_the_review: While this paper has the merit of the originality of the approach, it is hard for me to provide a string feedback due to the unfamiliarity with this Koopman theory.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a new class of pruning algorithms based on Koopman operator theory and shows existing magnitude based pruning methods and gradient based pruning methods can be unified under the proposed framework. This is a very interesting approach to justify the success of magnitude based and gradient based methods in compressing neural networks by utilizing eigendecomposion, which has been successfully applied to understand the structure of RKHS and GPs.
main_review: This paper tries to explain the success of magnitude based and gradient based methods in compressing neural network models. Although these methods are naive, both of them are found to be robust and are easy to implement. 

strengths:
1. This paper is well-written and easy to follow.
2. The targeting problem of this work is very interesting and fundamental. 
3. The solution proposed in this work is easy to implement and is supported by solid theory. For practitioners who are familiar with eigen-decomposition, the theory part of this work may not be amazing, however this viewpoint on understanding pruning is very interesting and provides valuable insights into the success of pruning based methods.


questions:
1. This works presents results on GMP. I am wondering if the proposed framework can be applied to layerwise pruning. Although in [1], the layerwise pruning is outperformed by GMP. 
2. What are the  computational  requirements? For example, to compute the Koopman mode decomposition of ResNet-20 on CIFAR10, how much time is needed? How much memory is needed? How many snapshots are needed to obtain the eigenvector associated to the largest eigenvalue? 
3. Is the proposed approach be scalable to large networks? It seems there are some computational challenges since the dimension of the parameter space of a large network can be very high.  



[1] Blalock, Davis, Jose Javier Gonzalez Ortiz, Jonathan Frankle, and John Guttag. 2020. “What Is the State of Neural Network Pruning?” ArXiv:2003.03033 [Cs, Stat], March. http://arxiv.org/abs/2003.03033.

summary_of_the_review: This paper provides an interesting viewpoint in understanding the success of pruning based compression methods. The method is easy to implement and is very practical  (at least for small-scale networks).
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper uses a Koopman operator to provide a theoretical basis to explain the success of pruning deep neural networks.
main_review: Strength:
1. The paper is clearly written.
2. The experiments are motivated and executed well.
3. Provides a theoretical basis for pruning which is an intuitive and successful procedure.
4. The difference between the static and active regimes is interesting. While why this happens is not answered in the paper, it can be considered as a useful by product.

Weakness:

1. Is the message tautological? As in, DNN training is a hidden procedure and the Koopman operator again projects onto a high dimensional space so as to linearise the dynamics, and once this is done, it is sort of natural to expect that the top eigenvalue will have the most say. Now, the Koopman operator is also sort of hidden and does not provide further insights into the working of the DNN. Would any sort of linearisation of dynamics not lead to similar conclusions.

2. While Algorithm 1 is a "general form", it is only conceptual, in that, it does not directly yield a procedure that speeds up "wall clock time".




summary_of_the_review: The paper clearly presents interesting results, the experiments seem sound. However, the novelty/significance is not overwhelming and hence the borderline score.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors studied network pruning from the perspective of dynamical system theory. They show that a new type of pruning method, named Koopman pruning, unifies magnitude pruning and gradient-based pruning to a degree. It also clarifies aspects of magnitude-based pruning before convergence in training.
main_review: The paper is overall well written. There were several aspects of this paper that I liked a lot: 

- I think that the questions asked in the paper, e.g., why has magnitude-based pruning been successful and what is its relationship with gradient-based pruning, are highly sought-after. 

- I also appreciate the efforts that the authors made to ensure the reproducibility of the paper as mentioned in the discussion section.

Having said this, at a few places in the paper, I am slightly confused by the authors' interpretation of their results and feel that they might be somewhat overstated. Two such places a below -- I am happy to be corrected if wrong:

- The first place is Section 3, where the authors state that the global and magnitude pruning in the long training time limit is *equivalent*. Equivalence is a strong claim and seemingly this is unlike to hold without strong assumptions on the dynamics of neural network parameters. Indeed, to my understanding, the Koopman operator approximates the dynamic of neural network parameters using a linear transformation, as in Appendix A. This seems to suggest that (4) only holds in a very special regime of neural network training, where the parameter updates can be modeled by a linear update matrix. The assumption of "linear dynamics in parameters'', however, seems to be a big constraint -- it is known that even linear neural networks (without any nonlinear activation) have nonlinear dynamics in parameters [(Saxe et al. 2013)](https://arxiv.org/abs/1312.6120). To clarify this point, I suggest the authors either (i) state a theorem on the equivalence with all the necessary assumptions listed or (ii) slightly tone down the equivalence claim. 

- The second place that I got a bit confused about is how exactly does Koopman pruning provide "new insight on the success (or lack thereof) of magnitude pruning pre-convergence," which is one of the main contributions listed in the introduction. I believe that Section 4 was designated to explicate this insight, but I feel that there is no strong message in that section that I can take home. I feel that the text and experiments in Section 4 demonstrated that Koopman pruning and magnitude-based pruning are *not* equivalent in pre-convergence (by epoch 20 or so) -- but is that all the authors want to conclude for magnitude pruning at pre-convergence?

**Questions**:

- In Section 2.1, where the authors cast neural network training into Koopman decomposition, it will be great if the authors can explicitly write the interpretation of $\mathbf{U}$, $g$, and $\mathbf{T}$ in the context of neural network training. Based on my understanding, in neural network training, $g$ is the identity function, $\mathbf{T}$ is the parameter-update transformation modeled by a linear operator, and $\mathbf{U}$ is identical to $\mathbf{T}$. Is this interpretation correct? If so, perhaps it is worth mentioning already at this point that throughout this paper, we assume that the neural network update dynamics is modeled by a linear operator (a matrix) $\mathbf{T}$.

- In Section 2.2. and Algorithm 1, the authors assume that $\lambda_1 = 1$ and refer this to as "a reasonable assumption in the case of stable dynamical systems". I am wondering if this assumption is well-corroborated by experiments. That is, when we perform Koopman decomposition on snapshots of parameters, is $\lambda_1$ close to 1? I am not sure if I would expect it to be the case. I feel that the reason perhaps is not that the dynamical system of parameter evolution is "unstable'' but rather that it may not be well-approximated by a linear dynamical system.
summary_of_the_review: To summarize, I feel that the paper can be a meaningful contribution to the field. However, the "operator theoretic perspective" of this paper -- to my feeling as a non-expert -- is somehow overstated. I think that Koopman pruning is an interesting and useful pruning heuristic on its own, but I am not sure if it can genuinely explain magnitude-based pruning either before or after training convergence in a realistic setting, where network parameters cannot be modeled by a linear dynamical system.
review_title: null

Meta-review: The paper used the Koopman operator theory to explain and guide the DNN pruning. All the reviewers deemed that such a viewpoint is novel (but at different levels). However, the paper still had some issues, including unclear technical details, vague/overselling statements, being computation and memory expensive, etc. The paper finally got 4 "marginally above threshold" (one being of low confidence), making it on the borderline. The AC read through the paper and agreed that the Koopman operator theory brings new perspective to DNN pruning, with potential for other analysis of DNNs. Although the paper is imperfect and not strong, it does not have severe problems either and the issues pointed out by the reviewers could be easily fixed (except the scalability issue, which can be left as future work). In order to encourage new ideas, the AC recommended acceptance.
Decision: Accept (Poster)