Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposes a novel "Hyperpruning" method for RNNs, based on the Lyapunov spectrum. The idea is to select good pruned candidate networks after a few training steps based on their distance in the 'Lyapunov spectrum space' from the unpruned network.
The paper claims to achieve with this method state-of-the-art accuracy on the PTB benchmark with stacked LSTM and RHN,  while using fewer training epochs.

strength_and_weaknesses: Strength:
* The proposed hyperpruning method is novel.
* The submission bridges tools from different fields (network pruning, Lyapunov Spectrum, Language modelling)

Weakness:
1.) Many details are missing, which makes the submission hard to evaluate and impossible to reproduce. E.g. 
- How exactly are the variants generated? The relevant paragraph says "after the elimination, given the remaining n candidates in T, the knowledge of their configurations and distances s serves as an initial history/archive for Bayesian hyperparameter optimization methods such as Tree Parzen Estimators (TPE) and Adaptive TPE (APTE) to generate new candidates. For example, TPE uses this initial knowledge to build a surrogate and generate new configuration candidates via Expected Improvement." 
This is not very specific, what are the exact hyperparameter configurations that are being utilized? Important details seem to be missing here.
- Details are also missing for the LS Computation: which hidden states are being used? What is f respectively for LSTM and RHN, how is J obtained respectively? What was your choice of T and K? 
2.) No code is provided, which adds to the irreproducibility.
3.) No theory is provided and it doesn't become clear from the submission why the LS-based metric is working. There is an heuristic explanation in figure 1, but the claim that "While early perplexity is not indicative of the estimated accuracy, LS-based distance on the other hand appears to be consistent [...]", seems only anecdotal. It would be great to show that more systematic, e.g., a scatter plot of early vs late perplexity vs a scatter plot of LS-distance vs late perplexity and the corresponding correlations. 

4.) The paper claims that the hyperpruning methods would beat the state of the art, but the architectures being used (LSTM and LSH networks) seem not to be state of the art for language modeling, it would be interesting to see how LSH would perform on actual state of the art models. Moreover, the PTB benchmark is a very small data set, so it is unclear if the hyperpruning method would still perform on par when used on more recent benchmarks and architectures. Of course, the result is still of academic interest.
5.) The submission doesn't discuss the computational complexity. LS calculation involves QR-decomposion (which is cubic in Jacobian size), and also matrix-matrix multiplication (also cubic), so if I am not mistaken, this should become more expensive for larger problems, correct? Would the computational advantage still persist for state-of-the-art language models? 

Minor issues
* Figure 1 A: x-axes labels/ticks are missing, is that just an illustration or actual LS curves?
* Figure 1B: 
* How does the 2D LS-space look like? It would be good to add a (supplement) figure to plot embeddings in the LS space.
* Why is there an increase o the perplexity from the pruning ration of 0.6 to 0.5 in figure 3?
* It would be good to plot the test perplexity as a function of epochs
* Not clear what hyperparameters are actually used
* baseline fair? Maybe compare to something else than LS-based metric.
* Why increase perplexity from a pruning ration of 0.6 to 0.5?
* Shouldn't time efficiency (figure 3 right) be also compared with selfish RNN?
* (This might also just be a misunderstanding on my side.) It seems that the optimization of e_i and e_0 was only done for the LS-based metric and not for the loss-based metric. I was wondering what would happen if e_i and e_0 were optimized for the loss-based metric instead and then tested on the LS-based metric. 

clarity,_quality,_novelty_and_reproducibility: * Clarity: 
The introduction is very clear, the section on LS calculation and LS-based hyperpruning could use more details. For example, it doesn't become clear to me, how exactly new candidates are being generated. Which features exactly are being fed into TPE or ATPE? 

Quality:
The submission contains a good amount of work on comparisons with previous methods. (However, I am not an expert on efficient pruning, so I can't fully certify the quality of that part.
The Quality of the LS-based hyperpruning is hard to estimate, because no code is provided, so it is difficult to evaluate how the embedding and LS calculation were done exactly.
* For table 1, no confidence intervals are reported.

Novelty:
The proposed hyperpruning method seems novel. It seems an innovative addition to previous methods (e.g., Selfish RNN, APTE, GS) that seems to improve performance and reduce computational demand.

Reproducibility: 
Because details are missing, and no code is provided, the results are unfortunately not reproducible in the current form. 

summary_of_the_review: 
The submission suggests an innovative and novel hyperpruning method for RNNs based on Lyapunov exponents after a few training steps, resulting in higher performance and reduced computational cost.
The main issue of the submission are a lack of a theoretical understanding of how and why it works (including limitations/when it would break down) and a lack of scalability to real-world problems.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposed a new distance based on Lyapunov Spectrum (LS) for neural network hyperpruning. Here, the authors define ‘hyperpruning’ as finding a suitable pruning method with optimal hyperparameter configuration. The authors apply this method to RNN language model benchmarks and show good results on Penn Treebank Dataset.
strength_and_weaknesses: Using Lyapunov Spectrum for hyperparameter learning is an emerging method in the deep learning area. In this paper, the author proposed to use the Lyapunov Spectrum for channel pruning. The results show the proposed method achieves higher results than the SOTA.

I’m curious how is the LS distance compared with the L2 distance? What is the performance difference compared to using L2 distance?

clarity,_quality,_novelty_and_reproducibility: The paper is overall clear, but the reference format doesn’t align with the ICLR requirement. The authors can double-check how to make the reference format correct. Moreover, some references are missing, e.g., in the second paragraph of the introduction, when introducing Dynamic Sparse Training (DST), authors need to cite the relevant papers.  

The quality is moderate. I’m not very familiar with Lyapunov Spectrum (LS) or the language tasks. So it is hard to evaluate the novelty.

summary_of_the_review: Overall this paper proposed a plausible method that achieves good results on language tasks.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: - The motivation of this paper is that Lyapunov Spectrum (LS)-based distance in the early training stage can represent the final performance.
- Based on the above observation, this paper illustrates hyperparameter optimization method for RNN pruning using LS distance.
- Specifically, the candidate set of configurations (i.e., sparse RNN) are initialized and trained for E epochs.
- The LS is computed for each candidate and dense model through training: eq 1, 2, and 3.
- All the LS values of candidates and dense models over training epochs are then used to obtain embedding (e.g. PCA).
- Then the candidate is eliminated based on the L2 distance between candidate embedding and dense model embedding of the last epoch.
- The experiments are conducted on PTB dataset using stacked-LSTM and RHN.
- The proposed method outperforms baselines including dense model.
strength_and_weaknesses: Strength:
1. The main idea of this paper is intuitive: using LS-distance as a metric for eliminating unpromising sparse network candidates. It naturally supports the usage of reference (dense) model as a target, which is ambiguous for the original Loss-based metric.
2. The experimental results show the effectiveness of the proposed method.

Weakness:
1. The computational overhead of Lyapunov Spectrum is not discussed in the paper. In equation 2, QR decomposition and Jacobian matrix computation are required. This makes the proposed method more computation- and memory-intensive than Loss-based metric.
2. The reason why one should use 2D-PCA as a embedding function is rarely discussed. For instance, one can use either tsne embedding instead of 2D-PCA or the LS-distance itself without any modifications. At least, the reason why authors set 2D for PCA can be discussed. 
3. The writing can be improved, e.g., some paragraphs are too long, notations are overlapped (x is used for both input and embedding).
4. The experiment is too narrow to validate the general effectiveness of proposed method. RNNs are widely used for tons of NLP tasks (e.g. machine translation as explained in introduction), and the proposed method is not limited to any specific NLP tasks.
5. I am not expert of neither RNN pruning or LS-distance, but I think this method can be applied to Transformer-based neural networks. Transformer is recently widely used for any NLP tasks, and pruning is one of attractive research directions for these large-scale models. 
clarity,_quality,_novelty_and_reproducibility: Clarity:
The clarity of this paper can be improved: see weakness 3.

Quality:
The discussion of main technical components are missing: see weakness 1 and 2.

Novelty:
I am not expert of neither RNN pruning or LS literature, but I think this paper is novel in that it is the first one which introduces LS-distance for RNN pruning.

Reproducibility:
This paper provides all the experimental setups to reproduce results.



summary_of_the_review: This paper proposes a RNN pruning method, which eliminates sparse network candidates based on LS-distance metric. The idea is novel and motivating, but the discussion of main technical components are missing and experiments are too narrow to evaluate of general effectiveness of the proposed method.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: **Summary:** This paper proposes both a new research problem and a solution to that problem. Thew new problem is *hyperpruning* - that there are a huge variety of pruning techniques in the literature (and each pruning technique has lots of hyperparameters); determining which technique to use in which circumstance is a nightmare. I work in pruning myself, and I agree that this is a challenge and an interesting problem. The authors look into solving this problem in a style similar to hyperparameter search (although with slightly different knowledge/constraints, as the authors acknowledge).

**Strengths:** This is an interesting new problem, and one of the most valuable possible contributions that a paper can make is to propose a new problem in the literature.

**Weaknesses:**
* Strange choice of settings. The pruning literature typically focuses on computer vision (usually image classification for ResNets and vision transformers) or NLP using transformers. I won't argue that this is necessarily the right choice, but it is the basis for comparison with prior work. This paper is incomparable to that work.
* Small-scale settings. The paper does NLP work, but focuses on RNNs on PTB. RNNs are not a modern setting, and PTB is a very small-scale setting. Over the years, the pruning community has learned the hard lesson that results in niche and small-scale settings don't tend to generalize very well, and - at this point - the bar for publication is to look at some subset of ResNets on ImageNet, transformers for masked language modeling (e.g., BERT), autoregressive language modeling, translation, etc. These small-scale settings may not generalize, and the lyapunov metric may indeed become very expensive to compute on bigger networks when larger amounts of data may be required. 

**Other comments:**
* The choice of the lyapunov metric is a bit of an odd one. As far as I can tell, it looks at the underlying representation learned by the network as a means for comparing networks, which seems unreasonably restrictive. It's possible that two equally good networks may have very different representations - why would you want to rule out that possibility?
* There was some confusion over how new networks are generated during the search process. This is something that needs a significant amount of attention: there are a lot of possible subnetworks out there, and the promise of also picking among many pruning methods (as stated in the introduction) is especially tricky. This needs to be addressed in more detail. This is also where having a wide variety of different networks/datasets/tasks would be helpful: as framed in this paper, a good hyperpruning method should produce great results across a wide variety of different settings with minimal overhead, whereas any specific pruning method/hyperparameter choices should fall short in at least some settings. This is another reason the current evaluation is insufficient.
* I disregarded the review from reviewer cX2r - the review was brief and had no serious technical content.

**Recommendation:** For the above reasons, I recommend rejection.
justification_for_why_not_higher_score: The problem of "hyperpruning" is a good one, but the proposed method for solving it and the evaluation are insufficient. A good question is insufficient for publication alone without (at least) a simple, solid, reproducible baseline for others to build on.
Decision: Reject