HYPERPRUNING: EFFICIENT PRUNING THROUGH LYAPUNOV METRIC HYPERSEARCH
Authors: Yang Zheng
Eli Shlizerman
Venue: Submitted to ICLR 2023 (2023)
Keywords: Network Pruning
Efficient Hyperparameter Searching
Lyapunov Spectrum

Abstract: A variety of pruning methods have been introduced for over-parameterized recurrent neural networks to improve efficiency in terms of power and storage. With the advance in pruning methods and their variety, a new problem of ‘hyperpruning’ is becoming apparent: finding a suitable pruning method with optimal hyperparameter configuration for a particular task and network. Such search is different from the standard hyperparameter search, where the accuracy of the optimal configuration is unknown. In the context of network pruning, the accuracy of the non-pruned (dense) model sets the target for the accuracy of the pruned model. Thereby, the goal of hyperpruning is to reach or even surpass this target. It is critical to develop efficient strategies for hyperpruning since direct search through pruned variants would require time-consuming training without guarantees for improved performance. To address this problem, we introduce a novel distance based on Lyapunov Spectrum (LS) which provides means to compare pruned variants with the dense model and early in training to estimate the accuracy that pruned variants will achieve after extensive training. The ability to predict performance allows us to incorporate the LS-based distance with Bayesian hyperparameter optimization methods and to propose an efficient and first-of-its-kind hyperpruning approach called LS-based Hyperpruning (LSH) which can optimize the search time by an order of magnitude compared to standard full training search with the loss (or perplexity) being the accuracy metric. Our experiments on stacked LSTM and RHN language models trained with the Penn Treebank dataset show that with a given budget of training epochs and desired pruning ratio, LSH obtains more optimal variants than standard loss-based hyperparameter optimization methods. Furthermore, as a result of the search, LSH identifies pruned variants that outperform state-of-the-art pruning methods and surpass the accuracy of the dense model.

Under review as a conference paper at ICLR 2023
HYPERPRUNING: EFFICIENT PRUNING THROUGH LYA- PUNOV METRIC HYPERSEARCH
Anonymous authors Paper under double-bling review
ABSTRACT
Various pruning methods have been introduced for werrpaaametterized recurrent neural networks improve efficiency in terms of power and storage. With the advance in pruning methods and their variety, a new problem of 'hyperpruning' is becoming apparent: finding a suitable pruning method with optimal hyperparameter configuration for particular task and network. Such search is different from the standard hyperparameter search, where the accuracy of the optimal configuration is unknown. In the context of network pruning, the accuracy of the non-pruned (dense) model sets the target for the accuracy of the pruned model. Thereby, the goal of hyperpruning is to reach or even surpass this target. It is critical to develop efficient strategies for hyperpruning since direct search through pruned variants would require time-consuming training without guarantees for improved performance. To address this problem, we introduce a novel distance based on Lyapunov Spectrum (LS) which provides means compare pruned variants with the dense model and early in training to estimate the accuracy that pruned variants will achieve after extensive training. The ability predict performance allows us incorporate the LS LS-based distance with Bayesian hyperparameter optimization methods and to propose an efficient and first-of-its-kina hyperpruning approach called LS-based Hyperpruning (LSH) which can optimize the search time by an order of magnitude compared to standard full training search with the loss (or perplexity) being the accuracy metric. Our experiments on stacked LSTM and RHN language models trained with the Penn Treebank dataset show that with a given budget training epochs and desired pruning ratio, LSH obtains more optimal variants than standard loss-based hyperparameter optimization methods. Furthermore, as a result of the search, LSH identifies pruned variants that outperform state-of- the-art pruning methods and surpass the accuracy of the dense model.
INTRODUCTION
Over the last decade, the performance of sequence models, i.c. Recurrent Neural Networks (RNN),
has been significantly enhanced in various applications such action recognition (Su et al. 2020).
video summarization (Zhao et al. 2018) and voice conversion (Huang et al., 2021) In particular,
RNN variants such as LSTM (Hochreiter & Schmidhuber 1997 Zaremba et al. 201 Malhotra
et al. 2015) and RHN
(Zilly et al. (017) excel in various NLP application ranging from machine 2016) to language modeling (Irie al. 2019). However, the architectural
translation (Wu et al.
inherent computational demands of RNN, being linear to input sequence length and quadratic to model size, lead to a slowdown in training and inference. This hinders these models from being deployed on resource-limites devices such as mobile devices. Multiple methods have been proposed to alleviate this problem, including network quantization (Hernández et al. 2020 Han et al. 2015a) and weight sharing (Ullrich et al. 2017). Among these approaches, network pruning is advantageous since aims to achieve a sparse model which would require fewer computational resources. A particularly notable pruning approach is dense-to-sparss where the network is gradually being pruned starting from non non-pruned (dense) model (Han et al.) 2015b] Guo et al. 2016). While the inference time of the pruned network will eventually decrease, the training time remains similar to time training a dense model. Such training time typically extends to multiple days or weeks and poses challenges in achieving pruned models efficiently.
longer than the

---

Under review as a conference paper at ICLR 2023
A
Lyapunov Spectrum Curves
Dense
Pruned
Pruned 2
A
Magnitude
Pre-Training
Epoch=1
Epoch=3
Epoch=5
B
C
200
LS Distance
180
Pruned 2
Perplexity
160
160
140
120
120
Pruned 1
Pruned 1
100
80
Pruned 2
20
40
60
80
0
20
40
60
80
Epoch
Epoch
Figure 1: (A) Lyapunov Spectrum curves of dense (black), and two pruned variants (green and red) at pre-training, Epoch 3, and 5 (left to right). (B) L2 distance two variants the dense reference in LS space over training. (C) Perplexity curves for two pruned variants over training.
The Dynamic Sparse Training (DST) approach was introduced recently to meet the rising demand of optimizing computational costs for achieving pruned variants (Bellec al.) 2017). In contrast to dense-to-sparse approaches, DST a spurse-to-sparse pruning method that starts with a sparse model and maintains the number of non-zero parameters over the entire training to improve not only the training speed along with the inference speed. DST involves three main procedural steps of pruning: weight removal, weight growth, and weight redistribution. For each step, salience (controls) such as magnitude- no universal controls applicable all tasks and networks, as each has its intended scenario. This unique mapping between scenario and control in DST prevents generalizing standard rule over all scenarios. Therefore, for a certain scenario, the controls which characterize the pruning method become additional key hyperparameters that need to be set such that pruning is executed in the most optimal way. We term this type of hyperparameter search as 'Hyperpruning' Hyperpruning concerns with selecting both the pruning methodology with its controls along with other hyperparameters (related to training) for a particular scenario. Specifically, hyperpruning requires searching over methodological and non-methodological hyperparameters. Methodological hyperparameters define the pruning method, and non-methoooloogial hyperparameters are independent of the pruning method and are applicable across them.
gradient-based could be applied to decide how it is conducted. There are
A unique feature of hyperpruning that does not typically hold for other hyperparameter optimization problems the existence of estimated target accuracy since the optimal accuracy its non non-prume pruned (dense) counterpart is available and can serve as loose upper bound for the pruned network to guide the hyperpruning process. However, even with this knowledge, searching through all pruning methods and their hyperparameters is time-consuming task since each variant requires extensive training. Furthermore, there is no guarantee for achieving a more accurate configuration after investigating multiple unsuccessful configurations. Hyperparameter Optimization algorithms accelerate the search process by implementing a distance that could be optimal to either efficiently evaluate configuration variants or effectively generate reliable variants (Snoek et al., 2012 Hutter et al., 2011] Bergstra (et al. 2011). In particular, such a distance will aim to provide early estimation of the accuracy of considered configuration without proceeding with full training. This distance also targets to improve
2

---

Under review as a conference paper at ICLR 2023
the reliability of configurations by adaptively generating them according to the history of visited configurations and their corresponding distances. A commonly used distance for hyperparameter search is the loss curve from early training episodes, however, in many scenarios this distance is not effective since it is rarely guaranteed that models that perform well in the heginning of training will reach the desired accuracy at the end, illustrated in Fig. T.C Therefore, finding an efficient and reliable distance is essential to hyperpruning.
Some previous work has shown that sequence models (RNNs and their variations) can be treated as dynamical systems, and their dynamic flow can be analyzed using Lyapunov Spectrum (LS), which measures the contraction/expansion ratio of the hidden states over time. It was also proposed that LS could be a potential indicator of the trainability of the network and the accuracy that it could reach (Vogt et al. 2022a). Based on these results, a recent work suggested that an AutoEncoder learned from the LS of slightly trained-RNN samples with random hyperparameter configurations can provide a low-dimensional embedding space in which RNN variants are organized according to the accuracy they achieve after training (Vogt et al. 2022b). Such organization appears to persist in the pruning setup establishing the embedding directly from LS, as demonstrated in Fig. [TA The example compares two pruned variants (green and red) with the dense model (black). Pruned 1 variant (green) eventually after full training achieves more optimal/lower perplexity than Pruned variant (red) although its perplexity is higher for more than 60 epochs, see Fig. [ While early perplexity is not indicative of the estimated accuracy, LS-based distance appears to be consistent and indicates that Pruned I variant closer to the dense network for all epochs, as shown in Fig. B..
The consistency of distance between the dense and pruned variants over training leads us to introduce novel distance based on LS and its embedding (LS Space) to guide hyperpruning. Along with LS distance we propose a novel and efficient algorithm, called LS-based //yperpruning (LSH), which utilizes the distance in LS space (LS- distance) and adaptive generation to identify optimal pruning variants at fraction of the cost required for training them. We summarize the main contributions of our work below:
1. We propose a novel distance for hyperpruning. The distance is based on Lyapunov Spectrum (LS) and is capable of estimating the similarity between candidate pruned variants and the dense model.
2. We propose novel algorithm called LSH that utilizes the LS-based distance an early estimation criterion and allows sifting through and replenishing pruned variants with an order of magnitude smaller number of training epochs than loss- full training search.
3. Our experiments on two extensive RNN language model benchmarks including stacked LSTM and RHN language models trained with Penn Treebank Dataset, show that LSH achieves more accurate pruned models than loss- hyperparameter searches, pruning methods that achieved state-of-the-arr accuracy on benchmarks, and even more accurate than the dense network.
2 RELATED WORK
Network Pruning The desire of reducing networks complexity and running time fuels the investigation of network pruning. In early applications, pruning was proposed to improve the inference speed of pretrained dense networks by iteratively pruning and fine-tuning them until reaching the target sparsity (Janowsky] 1989; Mozer & Smolensky) 1989 1988; LeCun et al. 1989 Hassibi & Stork 1992). Such pipelines are called dense-to-sparse and were applied in various applications and many achieved only marginal drop in accuracy when compared to the dense models, e.g. magnitude- -based
pruning applied to convolutional networks (Han et al. 2015b Guo et al. 2016).
The requirement of a pre-trained model was mediated by Gradual Magnitude Pruning (GMP) methods which gradually train and prune the model over time and have been implemented on for broad range of network architectures (Narang et 2017 Zhuu & Gupta 2017 Gale et al., (2019). Recent approaches further proposed to generalize the pruning process such that the weights and the pruning mask are jointly learned through unified optimization or differential reparameterization function (Liu et al.) 2020 Kusupati et al. 2020; Lin et al. 2020). In addition to direct pruning on weights, indirect pruning methods have been proposed employing Lo (Louizos et al. 2017), L1 regularization (Wen) cet al. 2017). and Variational Dropout (Molchanov et al.) (017). However, for large-scale learning tasks these methods do not reach the accuracy of magnitude-based pruning (Gale et al. 2019).
3

---

Under review as a conference paper at ICLR 2023
Magnitude-basee -based pruning turned out to be effective in several aspects. While pruned networks optimize the inference time and save computational resources the pruning process, as being a generalized iterative regularization, could also achieve more optimal accuracy than the dense model, as already observed in Lottery Ticket Hypothesis (Frankle & Carbin 2018). In some cases the "lottery ticket" can be identified early in training and potentially save training cost (You al. 2019). While GMP succeeds in achieving faster inference time, their training time is similar to or even longer than training a dense network. To improve the training efficiency and avoid complex pruning schedules, has been proposed to prune at initialization according to particular salience (controls) and maintain the pruning mask throughout training (Lee al. 2018 2019 Wang et al. 2020 Tanaka et al. 2020). These methods are advantageous but fail to match the performance of dense-to-sparse methods (Wang) et al.
2020) and typically do not perform well in extreme sparsity cases
(Lee et al. 2019).
To address above challenges in expanding training demands, recent approaches proposed a novel type of pruning. Dynamic Sparse Training (DST). is a sparse-to-spare pruning concept which aims to optimize for both inference and training efficiency without sacrifice to accuracy. To keep being efficient, DST maintains the total number of non-zero weights during training and alters the position of non-zeros through prune-regrow process to make the model more flexible. This idea was first introduced in Deep-R (Bellec et al.) (017), which rewires the network from a posterior. Later, inspired by biological neural networks, SET (Mocanu et al.. 2018) simplified this process by replacing it with magnitude-based pruning and random weight growing to maintain the sparsity during training. Starting from a sparse seed network, NeST (Dai et al. 2019) prunes and grows both neurons and weights according to magnitudes and gradients, respectively. Consequently, DSR (Mostafa & Wang 2019) introduced non-uniform sparsity for different layers instead of uniform sparsity in SET. While showing promise, these sparse-to- -o-sarsse methods could not be on-par with dense-to- poose-tt-sppp to-ssarsee methods. For further enhancement, SNFS (Dettmers & Zettlemoyer 2019) proposed to use momentum for weight growing which achieves better accuracy than random growth and matches even outperforms dense-to-sparse methods but at the price of computational efficiency (Dettmers & Zettlemoyer 2019). To tackle the inefficiency. RigL (Evci al. (2020) proposed to adapt "lazy gradient" calculation into magnitude- baase pruming, and TAS-KAST (Jayakumar et al.] (2020) further optimized it by not requiring to calculate the dense gradients. Recently, DST pruning method called Selfish-RNN has been proposed specifically for RNN. It showed a significant improvement over the performance of sparse RNN via non- non-uniform weight redistribution across gates and SNT-ASGD (Liu et al.) (021). The method addresses parameter allocation among different layer types, which was also observed as key in other works (Kusupati al. 2020 Frankle et al. 2020). Additional comprehensive survey on network pruning is available in (Hoefler et al. [202]] Wa Wang et al. 202T) and references therein.
Hyperparaneter Search Multiple Hyperparameter Optimization (HPO) algorithms have been pro- posed to find the optimal hyperparameter configurations for a specific scenario. Many HPO algorithms propose configurations in an iterative fashion and then evaluate their performance. Therefore, HPO can be categorized the way they propose new configurations and how they evaluate configurations.
For configurations proposal. grid search and manual search are the most intuitive HPO, however are also inefficient. Instead, random search finds comparable or better models within limited time (Bergstra & Bengio 2012). contrast to these memory-less approaches, more advanced HPO algorithms include a memory of proposed configurations and their accuracy in an archive. They then decide between exploitation and exploration, i.c., exploiting the current best configuration (local search) exploring new random configuration (random search). Bayesian Optimization, such as TPE, uses the archive to fit a surrogate model and then adaptively finds configurations (Snoek let al.] 2012 Hutter et al.) 2011 Bergstra et al.) 2011). Given this extra information from the archive, Bayesian Optimization empirically outperforms a plain random search (Thornton et al. 2013 Eggensperger et al.
2013
Snoek et al. 2015).
Another variation of HPO is multifidelity search which focuses on accelerating configuration evalua- tion, i.c., fast evaluation is used to infer the configuration performance (Bischl et al. 2021). This approach aims to find promising candidates and allocate more resources to them. Typically, there is a trade-off between computational cost and evaluation accuracy, i.e., the more computation spent on evaluating configuration, the more accurate the evaluation is. Therefore, one common practice to remove less promising candidates. This approach can be incorporated into random search or other HPO to explore a more expansive searching space within the same resource budget (Li et al.) 2017]. Searching over a larger space is necessary for complex architectures, such as sparse stacked LSTM and RHN, due to many possible hyperparameter configurations. Therefore, fast and effective
4

---

Under review as a conference paper at ICLR 2023
configuration evaluation method becomes essential. Here show that Lyapunov Spectrum (LS) is candidate for such method since able to effectively perform early estimation in training.
Lyapunov Spectrum Treating sequence models, such as RNN dynamical systems and predicting
their long-term performance is an emerging topic in deep learning research (Chang et al. 2019 Zheng
& Shlizerman 2020 Vogt al. 2022a Ribeiro al. 2020). Lyapunov Spectrum (LS), formed by
Lyapunov Exponents, is targeted to identify nonlinear dynamical systems characteristics (Ruelle)
1979; Oseledets 2008). LS was initially used to provide insight into autonomous neural network models (Monteforte & Wolf 2010; Engelken et al. 2020), while RNNs are typically non-autonomous due to external inputs. However, as long as imput sequences are sampled from a stationary distribution, LS can be generalized to a non aun-oooonomous dynamical system such as RNN based on random dynamical theory (Arnold, 1995). Another property of LS is that if RNN sequence is long enough, the resulting LS converges to the same curve, proved by Oseledets theorem (Saitò & Ichimura 1979: Ochs, 1999). These results indicate that the information captured by LS could be indicative of
network dynamics. In (Legenstein & Maass 2007 Pennington et al. 2018 Laurent & von Brecht 2016), the correlation between the largest Lyapunov exponent sign and the existence of chaos in the network has been studied. In addition, research succeeded in connecting LS features, such as zeros, negative, mean, and variance, and concepts in dynamical systems, such as quasi-periodic orbits, fixed point attractor, heterogeneity, and rate of contraction (Dawson et al. 1994: Abarbanel al., 199T Shibata [001: (Brandsätter et al. 1983) Yamada & Ohkitani 1988). While each feature seems to extract knowledge about the network, their correlation with network accuracy is unclear (Vogt et al.. 2022a). With the efficient LS computing algorithm proposed in (Vogt et al. 2022a), the correlation between LS and network performance was found via an additional AutoEncoder network (Vogt et al.) 2022b) such that this AutoEncoder can be used to predict the performance of random hyperparameter configurations. Motivated by this work, we propose an LS-based distance that is used to eliminate unpromising candidates and generate new candidates. While in general applications an additional AutoEncoder network might be needed for organizing candidates into similarity groups, for pruning, the existence of dense model for which LS can be computed sets the similarity and we find that direct distance between LS of dense and pruned networks provides a robust indicator of their similarity.
METHODS
LS-based Hyperpruning (LSH) both eliminates and generates candidates. LSH effectively eliminates unpromising candidates from the candidate set based on their closeness to the dense reference in the LS space. It then generates new candidates given the remaining candidates in the set and inserts them into the set. Such elimination and insertion procedure is iterated for multiple times. At completion, the most promising candidates are selected with most of the training resources allocated to them for full extensive training. The LSH method is illustrated in Fig. 2
Lyapunov Spectrum Computation We adapt the algorithm from (Vogt et al. 2022b), which computes LS by monitoring the contraction/exxaasioo history of the network over the entire time sequence. Specifically, it first randomly selects a batch X' of K samples with sequence length T from the validation set Deat, i.c., X = (x* {a**]/ J[=1, x Duat, k 1, ..... K}. where zf is the t-th time step of the k-th sample in X. A vector of zeros O and an identity matrix I are initialized as the original hidden state he and orthogonal representation base 80. Hidden state h' tracks the evolution of network and orthogonal representation base B captures contraction/expaasion of each direction in the hidden state till time step L. At each time step, h and B are updated, i.e., h f ,x*.*) and B R = qr(Jt. B 1,,,, where f is the network function, J is the Jacobian matrix between hidden states, qr denotes QR decomposition, and denotes matrix multiplication. The contraction/expansioo ratio at time step i is incorporated into the i-th vector of B and measured by r.. the i-th diagonal element of Rº. The i-th LS,, X then calculated by taking the average over the entire time sequence T and K samples.
KKK T
A KT k=1t=1
log(r
We denote LS as A, i.e., A (A.)2 where L is the dimension of hidden states h'.
LS LS-based Hyperpruning (LSH) Starting with a random candidate set T where each candidate is hyperparameter configuration of pruned network, the goal of hyperpruning is to find the best
5

---

Under review as a conference paper at ICLR 2023
Reference
Space
Extensive Training
Final Selection
Initial Candidates
epochs
Elimination
Insertion
Generation
epochs
Figure 2: LS-based Hyperpruning: The process starts with an initial candidate set of hyperparameter configurations (gray circles) and dense reference model (black circle), and executes elimination and insertion turns. Elimination: The reference model and each candidate is trained for eo or e; their
epochs and projected to the LS space. Candidates are ranked and eliminated depending distance to the reference in the LS space. Insertion: New candidates are adaptively generated given the remaining candidate in the and inserted into the set. After several rounds of elimination and insertion, the remaining candidates in the set are extensively trained and the best one (green circle) is picked as the final selection.
configuration, see Fig. The procedure of LSH is guided by a dense network since it provides a target performance for the pruned model to reach or even surpass it. Candidate selection process includes LS- based elimination and adaptive insertion phases executed turns for several iterations. In the LS-based elimination step, each candidate t' TT is trained for E epochs, and after each epoch of training, their LS is computed according to Eq, and denoted as Aÿ where i and j stand for the i-th candidate and the j-th epoch, respectively. LS till the E-th epoch of t' are grouped as A i.e., A = {A Similarly, all LS from the dense model till epoch E are denoted as A. We then project and to an embedding space and use the distance between the last epoch of f and t' to indicate their closeness, i.e., ["o...., ......, " = embbedding(( [Å, A1) and = dislance( (Ün, w(g), where E can either be the starting epoch eo or the incremental epoch e, depending which time the LS-based elimination is being performed, see Fig. In our work, we choose PCA to construct the embedding space and project LS 2d PCA space, called L. Space. We use I.2 distance to measure the closeness of networks in this LS space. The distance s between t' and f is then used to rank the candidates in T. In our case, Y out of n candidates with longer distances are eliminated from T. Even though LS -baaaeed elimination can efficiently find the best configuration in T, is not guaranteed to find the best configuration for this task since best configuration may not necessarily exist initial set. Therefore, in addition to climination, LSH adaptively generates new candidates based on the remaining candidates in and inserts them to T. Generation is based on remaining promising candidates since as L.H is iterating through elimination, it refines the candidates such that promising candidate would move the distribution in T closer to the optimal distribution and would increase the chance of finding the best configuration. The new candidate generation and insertion happens after the L.Sbbaed climination. Specifically, after the elimination, given the remaining 7 candidates in T, the knowledge of their configurations and distances s serves an initial history/archive for Bayesian hyperparameter optimization methods such as Tree Parzen Estimators (TPE) and Adaptive TPE (APTE) to generate new candidates. For example, TPE uses this initial knowledge build surrogate model and generate new configuration candidates via Expected Improvement, In particular, 3 new candidates are generated and then added candidate set T. This uneven number of elimination( and insertion( ()) candidates from and T assymptticallly decreases the number of candidates in T so that eventually only the most promising candidates are left T. The climination/inscrtinn schedule is controlled by e and which decide when execute the first and future elimination, respectively. As result, they manipulate the required time budget for the candidate selection process.
After epochs of the candidate selection process, the remaining candidates in T are extensively trained for P epochs where P >> m until their losses/accuracies converge. Given the final performance of extensively trained candidates, the best one is selected as the final pruned network.

---

Under review as a conference paper at ICLR 2023
4 EXPERIMENTS
To demonstrate the effectiveness of LS-based distance (LSH), we perform multiple experiments comparing LS-based distance (LSH) and Loss-based distance as early estimation criteria for HPO algorithms in hyperpruning. The same initial candidates set size and elimination/inserton schedule are used for a fair comparison, i.e., the number of initial candidates n, the starting epoch cu, and incremental epoch are kept the same. This results in the same resource budget for both methods two
since it only depends on n, 6D, and 6ş. We show the generality our method by applying sequence models, stacked LSTM and RHN, trained with Penn Treebank dataset (Marcinkiewicz 1994) for a language modeling task. In each experiment, we follow the setup of SOTA RNN pruning (Liu et al. 2021). We also show the robustness of LSH by conducting experiments stacked LSTM under different pruning ratios. Furthermore, we demonstrate the time efficiency of LSH by comparing the time difference between LSH and the loss-based full training search to achieve larget accuracy. The training and LS computation are conducted on an NVIDIA RTX 2080 Ti. More hyperparameters of architectures be found in Supplementary Material.
Table I:
The validation and testing perplexity of stacked L.TTM and RHN on PTB dataset for language modeling task (lower perplexity means better performance). LSH applied with Grid Search (GS), TPE, and Adaptive TPE (ATPE). Vanilla GS, TPE, and ATPE use Loss-based distance. DSR, SNES, SET, RigL., GMP, Dense, and Selfish-RNV data are from (Liu al. 2021)
Stacked-LSTN
RHN
Method
Validation
Testing
Validation
Testing
DSR (Mostafa & Wang 2019)
90.0
88.2
65.4
63.2
SNFS (Dettmers & Zettlemoyer 2019)
88.3
86.3
74.0
71.0
SET (Mocanu et (0118)
87.3
85.8
63.7
61.1
RigL (Evci al. 20200
78.3
75.9
64.8
62.5
GMP (aae et al.
2019 hu & Gupta 2017
76.8
74.8
65.6
64.0
Dense (Liu et al. 2021
74.5
72.4
63.4
61.8
GS
74.2
72.5
62.1
60.4
APTE
74.3
72.7
63.4
61.6
TPE (Bergstra et al. 2011
74.4
72.7
62.1
60.2
Selfish RNN (Liu et al. 2021)
73.8
71.7
62.1
60.4
LSH-GS (Ours)
72.5
70.8
62.1
60.4
LSH-ATPE (Ours)
72.1
69.9
60.5
59.5
LSH-TPE (Ours)
72.0
69.9
60.2
59.0
Effectiveness. We first demonstrate the effectiveness of LSH for selecting hyperparameter config- urations including methodological and non-methodologica hyperparameters. We consider sparse initialization, death mode, and redistribution mode for the methodological hyperparameters. Sparse initialization determines the initial pruning mask. Death and redistribution modes determine how the weights in the network are pruned and redistributed before and after weight growth, respectively. There are two available initialization modes, four death modes, and three redistribution modes. For the non-methodological hyperparameters, we consider the death rate, which is the ratio of pruned weight and total nonzeros weight at each time. The death rate is generated from the range of [0.4, 0.9].
We evaluate LS-based and Loss-based distances on different HPO algorithms, such Grid Search (GS), TPE, and Adaptive TPE (APTE). The pruning ratio for Stacked LSTM and RHN is 0.67 and 0.53, respectively. eo and e are [3. and [6, 3] for stacked-LSTM and RHN, respectively. For GS. we fix the death rate of stacked LSTM to 0.8 and RHN to 0.5. Therefore, only methodological total of 24 configurations. For TPE and APTE in
hyperparameters are considered and result Table. I. is 40 for both stacked LSTM and RHN. As shown in Table LSH (bottom three rows) consistently outperforms by a margin dense models and SOTA pruning methods. Note that Table [] is that GS, TPE, and APTE row uses Loss-based distance in comparison with LS-based distance (bottom three rows). LSH with different HPO algorithms also achieves better performance than the boss-based counterparts in most cases (3/3 in stacked LSTM and 2/3 in RHN). The best result achieved by LSH-TPP both stacked LSTM and RNH cases which outperforms the previous best result (Selfish RNN), i.e., 69.9 vs. 71.7 1.7 for stacked LSTM and 59.0 0 60.4 for RHN. Another
7

---

Under review as a conference paper at ICLR 2023
Hours
76
200
Perplexity
75
Dense
150
Time
TPE
TPE
Selfish-RNN
Better
73
LSH (Ours)
Better
LSH (Ours)
72
0.5
Pruning Ratio
Achieved Perplexity
Figure 3: (Left) Pruning Ratio Perplexity: Comparison among LSH (blue), TPE/loss-based (brown), and selfish-RNN (gray) on different pruning ratios.; (Right) Achieved Perplexity vs Time Budget: Comparison between LSH (blue) and loss-based full training search (brown) on required time budget (hours) to reach a target perplexity.
important observation is that LSH with HPO algorithms, such TPE and ATPE, outperforms LSH with GS. Quantitatively, LSH obtains 0.9 and 1.4 perplexity improvement on stacked LSTM and RHN by switching from GS TPE, respectively. Notably, this improvement is found in the loss-based counterparts and indicates that LSH utilizes advanced hyperparameter optimization methods better for forming more optimal elimination and generation which leads to more optimal selection.
We also test LSH under different resource budgets, i.c, different sizes of the initial candidate set. We consider three resource budgets, low, moderate, and high which have 24, 30, and 40 candidates in the initial set, respectively. We demonstrate it on stacked LSTM with pruning ratio= 0.67. Each experiment repeated three times and the average perplexity with 95% confidence interval are reported. As shown in Table. 2 with increasing resource budget (Low - High), LSH can improve average performance (70.9 + 69.9 for TPE and ATPE) and achieve more reliable result as confidence interval drops (3.0 - 2.6 for TPE and 0.4 - 0.3 for ATPE), while this improvement does not necessarily exist in Loss based counterpart (mean: 72.3 + 72.7 for TPE 72.0 + 72.7 and ATPE; confidence interval: 0.8 - 1.1 for TPE and 1.9 0.1 for ATPE).
Table 2: Comparison between LS based (LSH) and Loss- loss-based distances applied with GS, TPE, and ATPE under different budgets. Stacked LSTTM with pruning ratio 0.67 trained PTB used here. The average perplexity with 95% confidence interval are reported.
Low Budget
Moderate Budget
High Budget
Methods
Validation
Testing
Validation
Testing
Validation
Testing
GS
74.2
72.5
74.2
72.5
74.2
72.5
АTPE
74.2 (1.2)
72.0
74.3 (0.7)
72.7 (0.1)
74.3 (0.7)
72.7 (0.1)
TPE
74.1 (1.3)
72.3 (0.8)
74.2 (0.6)
72.4 (1.0
74.4 (0.5)
72.7 (1.1)
LSH-ATPE (Ours)
73.3 (0.9)
70.9 (0.4)
72.1 (0.2)
(0.3)
72.1 (0.2)
69.9 (0.3)
LSH-TPE (Ours)
72.6 (1.9)
70.9 (3.0)
72.8 (1.1)
(2.6)
72.0 (0.2)
69.9 (2.6)
LSH-GS (Ours)
72.5
70.8
72.5
70.8
72.5
70.8
Robustness. We evaluate the robustness of LSH pruning ratios and show our results in Fig. Leftf. We test from moderate (0.5) to high (0.8) pruning ratios by step of 0.1. For each experiment, TPE used the HPO algorithm and is set to 20. The experiment is on Stacked LSTM and the same en and e; are used. As shown in Fig. 3 Left, LSH (blue) consistently outperforms TPE/loss-based (brown), Selfish-RNN (gray) methods, and dense model for pruning ratios.
Time Efficiency. We compare the time that LSH uses to reach a particular target perplexity with the baseline loss-based full training search, which selects the optimal configuration after finishing training all candidates. We use stacked LSTM with a fixed pruning ratio of 0.67 for these experiments. Fig. 3 Right shows the amount of time (y-axis) required for finding a configuration with particular target perplexity (x-axis). We observe that LSH is more efficient than the loss-based approach for all perplexities. Particularly, for finding the optimal configuration (72), LSH (ss hours) is an order of magnitude faster than the loss- based approach 150 hours, 6 days).
8

---

Under review as a conference paper at ICLR 2023
Ablation Study. For the completeness of LSH, perform ablation studies of the effect of elimina- tion/insertion schedule and LS computation efficiency. Experiments are on stacked LSTM.
We first studied the efficiency and effectiveness of climinationn/nssrtioon schedules set by eo and eq We fixed the hyperparameter optimization method to Grid Search with 24 configurations, and the rest of the setup as in previous experiments. Table. shows the number of epochs required for the candidate selection process under each schedule. Cells with green color denote that the optimal configuration was indeed selected. We find that with eo 3, e; 1, the minimum number of required epochs is achieved (130), the performance might not be robust across different experiments since, with small change in the eg direction. the selection might fail. Therefore, we set 6g = 6; which is robust is both eo and E, directions. While this configuration requires addition 116 epochs (an increase from 130 > 246) which is about 2 hours of additional running time, this time still significantly shorter compared to extensive training of the two remaining candidates (> 20 hours).
Table 3:
The efficiency and the effectiveness of scheduling. Each cell reports the number of required epochs for the candidate selection process. Green cells mean the best candidate indeed selected.
3
4
82
140
198
256
2
106
164
222
280
3
130
188
246 (*)
304
Notably, LS computation is an efficient process and is relatively faster than training the network itself, i.e., it takes 6s for the LS computation of each sample vs. 120s for one training epoch. However, with large LS computation batch sizes, the overall process could slow down hyperpruning. Here empirically show that only few validation samples are needed for the LS computation. We demonstrate by comparing the maximum, mean, and variance I.S computed from different numbers of validation samples across pruned models. As discussed in (Vogt al. 2022a), those are the main features that determine the characteristic of LS. As shown in Table. [4] maximum, mean and variance are similar for LS calculated from 2 and 10 validation samples. Their different A is one order of magnitude smaller compared to the difference between different pruned models.
Table 4: Effect of LS computation batch size: The maximum, mean, and variance of LS computed from 2 and 10 validation samples on two pruned models.
Pruned
Pruned
2
10
A
2
I0
maximum
-1.26
-1.20
0.06
-0.52
-0.46
0.06
mean
-4.00
-4.41
0.01
-3.97
-3.97
0
variance
2.59
2.67
0.08
1.92
1.98
0.06
5 CONCLUSION
In this work, we propose a novel hyperparameter search approach that deals with selecting the optimal pruning method with the corresponding hyperparameter configuration for a particular scenario. To resolve that, propose novel Lyapunov Sppectrum(SS)-basee hyperpruning algorithm, termed LSH, to effectively and efficiently select the optimal configuration. LSH iteratively eliminates unpromising pruned variants based their distance generates new variants that are enhanced iteratively. Eventually, most resources are allocated to the most promising selected candidates to fully train optimal pruned network. Our method can be incorporated into different existing hyperparameter optimization algorithms, such as TPE and ATPE, and achieves an order of magnitude boost compared the loss-based full training search in terms of efficiency. We conduct experiments on stacked LSTM and RHN models trained with the Penn Treebank Dataset on language modeling tasks. Our results show that LSH is robust different pruning ratios and given fixed pruning ratio, it outperforms its loss-based counterparts, other SOTA pruning method, and even the dense model.
dense reference model in LS space and adaptively

---

Under review as a conference paper at ICLR 2023
REFERENCES
Henry DI Abarbanel, Reggie Brown, and Matthew B Kennel. Variation of Iyapunov exponents on strange attractor. Journal of Nonlinear Science, 1(2):175-199, 1991.
Ludwig Amold. Random dynamical systems. Dynamical systems, pp. 1-43, 1995.
Guillaume Bellec, David Kappel, Wolfgang Maass, and Robert Legenstein. Deep rewiring: Training very sparse deep networks. arXiv preprint arXiv:1711.051336 2017.
James Bergstra and Yoshua Bengio. Random search for hyper-parameter optimization. Journal of machine learning research, 13(2), 2012.
James Bergstra, Rémi Bardenet, Yoshua Bengio, and Balázs Kégl. Algorithms for hyper-parameter optimization. Advances in neural information processing systems, 24, 2011.
Berdd Bischl, Martin Binder, Michel Lang, Tobias Pielok, Jakob Richter, Stefan Coors, Janek Thomas, Theresa Ullmann, Mare Becker, Anne-Laure Boulesteix, et al. Hyperparameter optimization: Foundations, algorithms, best practices and open challenges. arXiv preprint arXlv:2107.05847, 2021.
A Brandstäter, J Swift, Harry L Swinney, A Wolf, J Doyne Farmer, Erica Jen, and PJ Crutchfield. Low-dimensional chaos a hydrodynamic system. Physical Review Letters, 51(16):1442 1983.
Bo Chang, Minmin Chen, Eldad Haber, and Ed H Chi. Antisymmetricmnn: A dynamical system view recurrent neural networks. arXiv preprint arXiv:1902.09630 2019.
Xiaoliang Dai, Hongxu Yin, and Niraj K Jha. Nest: A neural network synthesis tool based on grow-and-prune paradigm. IEEE Transactions on Computers, 68(10):1487-1497 2019.
Silvina Dawson, Celso Grebogi, Tim Sauer, and James A Yorke. Obstructions to shadowing when Yyapunov exponent fluctuates about zero. Physical review letters, 73(14):1927, 1994.
Tim Dettmers and Luke Zettlemoyer. Sparse networks from scratch: Faster training without losing performance. arXiv preprint 190.:0/907.04400 2019.
Katharina Eggensperger, Matthias Feurer, Frank Hutter, James Bergstra, Jasper Snoek, Holger Hoos, Kevin Leyton-Brown, et al. Towards an empirical foundation for assessing bayesian optimization of hyperparameters. In NIPS workshop on Bayesian Optimization in Theory and Practice, volume 10, 2013.
Rainer Engelken, Fred Wolf, and Larry F Abbott. Lyapunov spectra of chaotic recurrent neural networks. arXiv preprint arXiv:2006.024 024277 2020.
Utku Evei, Trevor Gale, Jacob Menick, Pablo Samuel Castro, and Erich Elsen. Rigging the lottery: Making all tickets winners. In International Conference on Machine Learning, pp. 2943-2955 PMLR, 2020.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprint arXiv:1803.0348 2018.
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M Roy, and Michael Carbin. Pruning neural networks at initialization: Why are we missing the mark? arXiv preprint arXiv:2009.0857% 2020.
Trevor Gale, Erich Elsen, and Sara Hooker. The state of sparsity in deep neural networks. arXiv 2019.
preprint 1rXiv:1902.090
Yiwen Guo, Anbang Yao, and Yurong Chen. Dynamic network surgery for efficient dnns. Advances in neural information processing systems, 29, 2016.
Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffiman coding. arXiv preprint arXiv:1510.00/4 2015a.
Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. Advances in neural information processing systems, 28, 2015b.
10

---

Under review as a conference paper at ICLR 2023
Babak Hassibi and David Stork. Second order derivatives for network pruning: Optimal brain surgeon. Advances in neural information processing systems, 5, 1992.
Gerardo Hernández, Erik Zamora, Humberto Sossa, Germán Téllez, and Federico Furlán. Hybrid neural networks for big data classification. Neurocomputing, 390:327-340, 2020.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735-17801 1997.
Torsten Hoefler, Dan Alistarh, Tal Ben-Nun, Nikoli Dryden, and Alexandra Peste. Sparsity in deep learning: Pruning and growth for efficient inference and training in neural networks. Journal of Machine Learning Research, 22(241):1-124 2021.
Wen-Chin Huang, Tomoki Hayashi, Yi-Chiao Wu, Hirokazu Kameoka, and Tomoki Toda. Pretraining techniques for sequence-to- te-eneeeet--sseeeeeeeeee voice conversion. JEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:745-755, 2021.
Frank Hutter, Holger H Hoos, and Kevin Leyton-Brown. Sequential model-based optimization for general algorithm configuration. In International conference learning and intelligent optimization, pp. 507-523. Springer, 2011.
Kazuki Irie, Albert Zeyer, Ralf Schlüter, and Hermann Ney. Language modeling with deep trans- formers. arXiv preprint arXiv: 0902004 2019.
Steven A Janowsky. Pruning versus clipping in neural networks. Physical Review A, (9(12)66600, 1989.
Siddhant Jayakumar, Razvan Pascanu, Jack Rae, Simon Osindero, and Erich Elsen. Top-kast: Top-k always sparse training, Advances in Neural Information Processing Systems, 33:20744-20754, 2020.
Aditya Kusupati, Vivek Ramanujan, Raghav Somani, Mitchell Wortsman, Prateek Jain, Sham Kakade, and Ali Farhadi. Soft threshold weight reparameterization for learnable sparsity. In International Conference on Machine Learning, pp. 5544-5555. PMLR, 2020.
Thomas Laurent and James von Brecht. recurrent neural network without chaos. arXiv preprint arXivv1612.06621,, 2016.
Yann LeCun, John Denker, and Sara Solla. Optimal brain damage. Advances in neural information processing systems, 2, 1989.
Namhoon Lee, Thalaiyasingam Ajanthan, and Philip HS Torr. Snip: Single-shot network pruning based on connection sensitivity. arXiv preprint arXiv:1810.023400 2018.
Namhoon Lee, Thalaiyasingam Ajanthan, Stephen Gould, and Philip HS Tom. A signal propagation perspective for pruning neural networks at initialization. arXiv preprint arXiv: 10Xiv:99000 1906.063077 2019.
Robert Legenstein and Wolfgang Maass. Edge of chaos and prediction of computational performance for neural circuit models. Neural networks, 20(3):323-334, 2007.
Lisha Li, Kevin Jamieson, Giulia DeSalvo, Afshin Rostamizadeh, and Ameet Talwalkar. Hyperband: novel bandit- approach to hyperparameter optimization. The Journal of Machine Learning Research, (511)67665-8166 2017.
Tao Lin, Sebastian U Stich, Luis Barba, Daniil Dmitriev. and Martin Jaggi. Dynamic model pruning with feedback. arXiv preprint arXiv:2006.07253, 2020.
Junjie Liu, Zhe Xu, Runbin Shi, Ray CC Cheung, and Hayden KH So. Dynamic sparse train- ing: Find efficient sparse network from scratch with trainable masked layers. arXiv preprint arXiv:2005.066 06870, 2020.
Shiwei Liu, Decebal Constantin Mocanu, Yulong Pei, and Mykola Pechenizkiy. Selfish sparse mn training. In International Conference on Machine Learning, pp. 6893-6904. PMLR, 2021.
11

---

Under review as a conference paper at ICLR 2023
Christos Louizos, Max Welling, and Diederik P Kingma. Learning sparse neural networks through 1_0 regularization. arXiv preprint 1:X1::1112.03122,200 2017.
Pankaj Malhotra, Lovekesh Vig, Gautam Shroff, Puneet Agarwal, et al. Long short term memory networks for anomaly detection in time series. In Proceedings, volume 89, pp. 89-94, 2015.
Mary Ann Marcinkiewicz. Building large annotated corpus of english: The penn treebank. Using Large Corpora, pp. 273, 1994.
Decebal Constantin Mocanu, Elena Mocanu, Peter Stone, Phuong H Nguyen, Madeleine Gibescu, and Antonio Liotta. Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science. Nature communications, 9(1):1-12, 2018.
Dmitry Molchanov, Arsenii Ashukha, and Dmitry Vetrov. Variational dropout sparsifies deep neural networks. In International Conference on Machine Learning, pp. 2498-2507. PMLR, 2017.
Michael Monteforte and Fred Wolf. Dynamical entropy production in spiking neuron networks the
balanced state. Physical review letters, 105(26):268104 104, 2010.
Hesham Mostafa and Xin Wang. Parameter efficient training of deep convolutional neural networks by dynamic sparse reparameterization. In International Conference on Machine Learning, pp. 4646-4655. PMLR, 2019.
Michael C Mozer and Paul Smolensky, Skeletonization: A technique for trimming the fat from network via relevance assessment. Advances in neural information processing systems, 1, 1988.
Michael Mozer and Paul Smolensky. Using relevance to reduce network size automatically. Connection Science, 1(1):3-16, 1989.
Sharan Narang, Erich Elsen, Gregory Diamos, and Shubho Sengupta. Exploring sparsity in recurrent neural networks. arxiv e-prints, art. arXiv preprint arXiv: 2017.
Gunter Ochs. Stability of oseledets spaces equivalent to stability of Iyapunov exponents. Dynamics and Stability of Systems, 14(2): 181-001, 1999.
Valery Oseledets. Oseledets theorem. Scholarpedia, 3(1):1846, 2008
Jeffrey Pennington, Samuel Schoenholz, and Surya Ganguli. The emergence of spectral universality in deep networks. In International Conference on Artificial Intelligence and Statistics, pp. 1924-1932. PMLR, 2018.
Antônio H Ribeiro, Koen Tiels, Luis A Aguirre, and Thomas Schön. Beyond exploding and vanishing gradients: analysing rnn training using attractors and smoothness. In International Conference on Artificial Intelligence and Statistics, pp. 2370-2380. PMLR, 2020.
David Ruelle. Ergodic theory of differentiable dynamical systems. Publications Mathématiques de l'Institut des Hautes Études Scientifiques, 50(1):27-58, 1979.
Nobuhiko Saitó and Atsushi Ichimura. Ergodic components in the stochastic region in hamiltonian system. Stochastic Behavior Classical and Quantum Hamiltonian Systems, pp. 137-144. Springer, 1979.
Hiroshi Shibata. Ks entropy and mean іyapunov exponent for coupled map lattices. Physica A: Statistical Mechanics and its Applications, 192-1-9)182--19, 2001.
Jasper Snock, Hugo Larochelle, and Ryan P Adams. Practical bayesian optimization of machine learning algorithms. Advances in neural information processing systems, 25, 2012.
Jasper Snock, Oren Rippel, Kevin Swersky, Ryan Kiros, Nadathur Satish, Narayanan Sundaram, Mostofa Patwary, Mr Prabhat, and Ryan Adams. Scalable bayesian optimization using deep neural networks. In International conference on machine learning, pp. 1171-2180 PMLR, 2015.
Kun Su, Xiulong Liu, and Eli Shlizerman. Predict & cluster: Unsupervised skeleton based action recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9631-9640, 2020.
12

---

Under review as a conference paper at ICLR 2023
Hidenori Tanaka, Daniel Kunin, Daniel L Yamins, and Surya Ganguli. Pruning neural networks with- out any data by iteratively conserving synaptic flow. Advances in Neural Information Processing Systems, 33:6377-6388 2020.
Chris Thornton, Frank Hutter, Holger H Hoos, and Kevin Leyton-Brown. Auto-weka: Combined selection and hyperparameter optimization of classification algorithms. In Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 847-855, 2013.
Karen Ullrich, Edward Meeds, and Max Welling. Soft weight-sharing for neural network compression. arXiv preprint arXiv:/702.00 1et..74020000 2017.
Ryan Vogt, Maximilian Puelma Touzel, Eli Shlizerman, and Guillaume Lajoie. On Iyapunov expo- nents for rans: Understanding information propagation using dynamical systems tools. Frontiers in Applied Mathematics and Statistics, 8, 2022a. ISSN 2297-4687. doi: 2..3389/fams..022..81111
URL https: //s::///www www.forrrsstiioo org/ http://rtcceeee//o 3 fams. 2tt.ei....22111992 818799
Ryan Vogt, Yang Zheng, and Eli Shlizerman. syapunov-guides embedding for hyperparameter selection in recurrent neural networks. arXiv preprint arXiv:2204.04876 2022b.
Chaoqi Wang, Guodong Zhang, and Roger Grosse. Picking winning tickets before training by preserving gradient flow. arXiv preprint arXiv:2002.07376 2020.
Huan Wang, Can Qin, Yue Bai, Yulun Zhang, and Yun Fu. Recent advances on neural network pruning initialization. arXiv e-prints, pp. arXiv-2103, 2021.
Wei Wen, Yuxiong He, Samyam Rajbhandari, Minjia Zhang, Wenhan Wang, Fang Liu, Bin Hu, Yiran Chen, and Hai Li. Learning intrinsic sparse structures within long short-term memory. arXiv preprint arxiv: 7009 0700255 2017.
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, al. Google's neural machine translation sys- tem: Bridging the gap between human and machine translation. arXiv preprint 161v:660900814. 2016.
Michio Yamada and Koji Ohkitani. The inertial subrange and non-positive Iyapunov exponents in fully-developed turbulence. Progress of theoretical physics, 79(6):1265-122 1268, 1988.
Haoran You, Chaojian Li, Pengfei Xu, Yonggan Fu, Yue Wang, Xiaohan Chen, Richard G Baraniuk, Zhangyang Wang, and Yingyan Lin. Drawing early-bird tickets: Towards more efficient training of deep networks. arXiv preprint arXiv:1909.11 11957, 2019.
Wojeiech Zaremba, Ilya Sutskever, and Oriol Vinyals. Recurrent neural network regularization. arXiv preprint Xrv: 1409.2:440922299 2014.
Bin Zhao, Xuelong Li, and Xiaoqiang Lu. Hsa-rnn: Hierarchical structure-adaptive rnn for video summarization. Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 7405-7414, 2018.
Yang Zheng and Eli Shlizerman. R-force: Robust learning for random recurrent neural networks. arXiv preprint urXiv22003.11660, 2020.
Michael Zhu and Suyog Gupta. To prune, or not to prune: exploring the efficacy pruning for model compression. arXiv preprini 17Xiv:1710.01878. 2017.
Julian Georg Zilly, Rupesh Kumar Srivastava, Jan Koutnık, and Jürgen Schmidhuber. Recurrent highway networks. In International conference on machine learning, pp- 4189-4199 PMLR, 2017.
13


Reviews:
[{"review_id": "Y5UmF_GQhGn", "reviewer": "Reviewer_cX2r", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed a new distance based on Lyapunov Spectrum (LS) for neural network hyperpruning. Here, the authors define ‘hyperpruning’ as finding a suitable pruning method with optimal hyperparameter configuration. The authors apply this method to RNN language model benchmarks and show good results on Penn Treebank Dataset.", "summary_of_the_review": "Overall this paper proposed a plausible method that achieves good results on language tasks.", "strength_and_weaknesses": "Using Lyapunov Spectrum for hyperparameter learning is an emerging method in the deep learning area. In this paper, the author proposed to use the Lyapunov Spectrum for channel pruning. The results show the proposed method achieves higher results than the SOTA.\n\nI’m curious how is the LS distance compared with the L2 distance? What is the performance difference compared to using L2 distance?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "XxncKUuupLZ", "reviewer": "Reviewer_X75X", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "- The motivation of this paper is that Lyapunov Spectrum (LS)-based distance in the early training stage can represent the final performance.\n- Based on the above observation, this paper illustrates hyperparameter optimization method for RNN pruning using LS distance.\n- Specifically, the candidate set of configurations (i.e., sparse RNN) are initialized and trained for E epochs.\n- The LS is computed for each candidate and dense model through training: eq 1, 2, and 3.\n- All the LS values of candidates and dense models over training epochs are then used to obtain embedding (e.g. PCA).\n- Then the candidate is eliminated based on the L2 distance between candidate embedding and dense model embedding of the last epoch.\n- The experiments are conducted on PTB dataset using stacked-LSTM and RHN.\n- The proposed method outperforms baselines including dense model.", "summary_of_the_review": "This paper proposes a RNN pruning method, which eliminates sparse network candidates based on LS-distance metric. The idea is novel and motivating, but the discussion of main technical components are missing and experiments are too narrow to evaluate of general effectiveness of the proposed method.", "strength_and_weaknesses": "Strength:\n1. The main idea of this paper is intuitive: using LS-distance as a metric for eliminating unpromising sparse network candidates. It naturally supports the usage of reference (dense) model as a target, which is ambiguous for the original Loss-based metric.\n2. The experimental results show the effectiveness of the proposed method.\n\nWeakness:\n1. The computational overhead of Lyapunov Spectrum is not discussed in the paper. In equation 2, QR decomposition and Jacobian matrix computation are required. This makes the proposed method more computation- and memory-intensive than Loss-based metric.\n2. The reason why one should use 2D-PCA as a embedding function is rarely discussed. For instance, one can use either tsne embedding instead of 2D-PCA or the LS-distance itself without any modifications. At least, the reason why authors set 2D for PCA can be discussed. \n3. The writing can be improved, e.g., some paragraphs are too long, notations are overlapped (x is used for both input and embedding).\n4. The experiment is too narrow to validate the general effectiveness of proposed method. RNNs are widely used for tons of NLP tasks (e.g. machine translation as explained in introduction), and the proposed method is not limited to any specific NLP tasks.\n5. I am not expert of neither RNN pruning or LS-distance, but I think this method can be applied to Transformer-based neural networks. Transformer is recently widely used for any NLP tasks, and pruning is one of attractive research directions for these large-scale models. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "IIMfKw1hz3l", "reviewer": "Reviewer_mZ4i", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a novel \"Hyperpruning\" method for RNNs, based on the Lyapunov spectrum. The idea is to select good pruned candidate networks after a few training steps based on their distance in the 'Lyapunov spectrum space' from the unpruned network.\nThe paper claims to achieve with this method state-of-the-art accuracy on the PTB benchmark with stacked LSTM and RHN,  while using fewer training epochs.\n", "summary_of_the_review": "\nThe submission suggests an innovative and novel hyperpruning method for RNNs based on Lyapunov exponents after a few training steps, resulting in higher performance and reduced computational cost.\nThe main issue of the submission are a lack of a theoretical understanding of how and why it works (including limitations/when it would break down) and a lack of scalability to real-world problems.\n\n", "strength_and_weaknesses": "Strength:\n* The proposed hyperpruning method is novel.\n* The submission bridges tools from different fields (network pruning, Lyapunov Spectrum, Language modelling)\n\nWeakness:\n1.) Many details are missing, which makes the submission hard to evaluate and impossible to reproduce. E.g. \n- How exactly are the variants generated? The relevant paragraph says \"after the elimination, given the remaining n candidates in T, the knowledge of their configurations and distances s serves as an initial history/archive for Bayesian hyperparameter optimization methods such as Tree Parzen Estimators (TPE) and Adaptive TPE (APTE) to generate new candidates. For example, TPE uses this initial knowledge to build a surrogate and generate new configuration candidates via Expected Improvement.\" \nThis is not very specific, what are the exact hyperparameter configurations that are being utilized? Important details seem to be missing here.\n- Details are also missing for the LS Computation: which hidden states are being used? What is f respectively for LSTM and RHN, how is J obtained respectively? What was your choice of T and K? \n2.) No code is provided, which adds to the irreproducibility.\n3.) No theory is provided and it doesn't become clear from the submission why the LS-based metric is working. There is an heuristic explanation in figure 1, but the claim that \"While early perplexity is not indicative of the estimated accuracy, LS-based distance on the other hand appears to be consistent [...]\", seems only anecdotal. It would be great to show that more systematic, e.g., a scatter plot of early vs late perplexity vs a scatter plot of LS-distance vs late perplexity and the corresponding correlations. \n\n4.) The paper claims that the hyperpruning methods would beat the state of the art, but the architectures being used (LSTM and LSH networks) seem not to be state of the art for language modeling, it would be interesting to see how LSH would perform on actual state of the art models. Moreover, the PTB benchmark is a very small data set, so it is unclear if the hyperpruning method would still perform on par when used on more recent benchmarks and architectures. Of course, the result is still of academic interest.\n5.) The submission doesn't discuss the computational complexity. LS calculation involves QR-decomposion (which is cubic in Jacobian size), and also matrix-matrix multiplication (also cubic), so if I am not mistaken, this should become more expensive for larger problems, correct? Would the computational advantage still persist for state-of-the-art language models? \n\nMinor issues\n* Figure 1 A: x-axes labels/ticks are missing, is that just an illustration or actual LS curves?\n* Figure 1B: \n* How does the 2D LS-space look like? It would be good to add a (supplement) figure to plot embeddings in the LS space.\n* Why is there an increase o the perplexity from the pruning ration of 0.6 to 0.5 in figure 3?\n* It would be good to plot the test perplexity as a function of epochs\n* Not clear what hyperparameters are actually used\n* baseline fair? Maybe compare to something else than LS-based metric.\n* Why increase perplexity from a pruning ration of 0.6 to 0.5?\n* Shouldn't time efficiency (figure 3 right) be also compared with selfish RNN?\n* (This might also just be a misunderstanding on my side.) It seems that the optimization of e_i and e_0 was only done for the LS-based metric and not for the loss-based metric. I was wondering what would happen if e_i and e_0 were optimized for the loss-based metric instead and then tested on the LS-based metric. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}]

Author rebuttal:

Decision: Reject