Network Pruning Optimization by Simulated Annealing Algorithm
Authors: Chun Lin Kuo
Ercan Engin Kuruoglu
Wai Kin Victor Chan
Venue: ICLR 2022 Submitted (2022)
Keywords: optimization
network pruning

Abstract: One critical problem of large neural networks is over-parameterization with a large number of weight parameters. This becomes an obstacle to implement networks in edge devices as well as limiting the development of industrial applications by engineers for machine learning problems. Plenty of papers have shown that the redundant branches can be erased strategically in a fully connected network. In this work, we reduce network complexity by pruning and structure optimization. We propose to do network optimization by Simulated Annealing, a heuristic based non-convex optimization method which can potentially solve this NP-hard problem and find the global minimum for a given percentage of branch pruning given sufficient amount of time. Our results have shown that Simulated Annealing can significantly reduce the complexity of a fully connected neural network with only limited loss of performance.

Under review as a conference paper at ICLR 2022
NETWORK PRUNING OPTIMIZATION BY SIMULATED ANNEALING ALGORITHM
Anonymous authors Paper under double-blind review
ABSTRACT
One critical problem of large neural networks is over--aramann par---ammmmtrtzaiionnn with a large number of weight parameters. This becomes an obstacle to implement networks in edge devices as well as limiting the development of industrial applications by engineers for machine learning problems. Plenty of papers have shown that the redundant branches can be erased strategically in fully connected network. In this work, we reduce network complexity by pruning and structure optimization. We propose to do network optimization by Simulated Annealing, heuristic based non-convex optimization method which can potentially solve this NP-hard prob- lem and find the global minimum for given percentage of branch pruning given sufficient amount of time. Our results have shown that Simulated Annealing can significantly reduce the complexity of fully connected neural network with only limited loss of performance.
INTRODUCTION
The successful development of the artificial intelligence methods fueled by deep-learning has made important impact in multiple industries. Elaborate AI algorithm empower electronic devices to make smart decisions or foresighted predictions. However, the outstanding outcomes come with their own cost, which is the computational complexity in the case of deep-learning methods. In order to obtain a model that can accurately predict the results corresponding to the input data, a neural network should go through initialization, back propagation, gradients update, and inference process step by step. When the deep learning structure is more complicated, there are more pa- rameters and operations involving in these steps. Network compression technique is a relatively new area necessitated by the booming development of deep neural network and increasing compu- tational complexity of machine learning methods. Attracted by the extraordinary performance of recently proposed neural network architectures, there are increasingly more demands edge de- vices which possess only limited computational resources. Therefore, various network compression techniques have been tried and studied during these years. The recent mainstream strategies include quantization (Wang et al., 2019), decomposition (Zhang et al. 2015), knowledge distillation (Hin-) [on et al., 2014), and the pruning methods, which is considered compress neural network (Liu et al., 2019b) (Lin et al. 2020a}. Among those 4 computational steps of deep learners mentioned above, back propagation and gradients update make up the major part that consumes the most of computing resources. One approach by pruning those gradients that smaller than threshold (Ye al. 2020), significantly reduces the usage of hardware resources and accelerates the whole training procedure. However, the trained model would still have the same parameter size after undergoing the pruned training procedure since the actual weight parameters are not eliminated from the network from computational perspective. Thus, this approach is only helpful for training network on a powerful computer. In order to actually reduce the total size of model while inferencing in mobile electronic device, various research found in the survey (Deng ct al., 2020) focus on pruning the weight parameters directly instead of the gradients. These pruning techniques can be element-wise vector-wise, block-wise, which can correspond to unstructured or structured pruming. Behind each type of pruning technique, there are multiple criteria being setup differently. One innovative criteria is the combination of generative adversarial network (GAN) and parameters selection (Madaan et al. 2020). By evaluating the similarity of the feature maps before and after pruning process, the importance of centain parameters can be defined.
the most effective approach to

---

Under review as a conference paper at ICLR 2022
Except for evaluating the feature map with designed criteria setup in GAN, the other group of pruning strategies follow heuristic optimization algorithmss, where the network is considered as an architecture fine-tuning process. Since neural network is one type of network composed of both nodes and edges, it would be intuitive to set up the optimization problem in terms of the edge weights and the structure of the network and the pruning strategy. ABC optimization technique is an example that demonstrates the application of a heuristic algorithm on channel-wise network pruning (Lin et al. 2020b). Based on the hierarchical characteristic of the neural network structure, pruning techniques can further be split into both structured and unstructured pruning categories. When a neural network is simulated on computer, the major part of computations would be carried out in matrix operations including addition, multiplication, and dot product. Therefore, in the case of unstructured pruning, the edges or nodes are not truly being eliminated. Instead, an extra mask matrix is designed to simulate the pruning process. Even though the mask matrix can bring up high flexibility in adjusting the network configuration, the model size cannot be reduced significantly since the pruned parameters should still be conserved in the matrix zero entries. On the contrary, the structured pruning strategies sacrifice some optimality to ensure dimension reduction and avoid sparse but high dimensional matrices. The most representative cases for both unstructured and structured pruning are edge-level and node-level pruning strategies. if one edge of a network is considered as the minimum unit to reduce configuration, it would not affect the existance of all other elements in the network. However, if one node is the minimum unit to be reduce network, all other edges connected to the selected node will also need to be erased.
In choosing pruning strategies, the main factor is to determine which part of the minimum unit (edges, nodes, ... etc) is more important than the others. The existing pruning approaches do not consider the optimality of the resulting network structure in any means. The optimality under ques- tion has two aspects to it: the branch weights and the branch connections. In this work, we aim to optimize the branch connections with the constraint of pruning percentage, That is, we question the optimality of the structure obtained by pruning. In this paper, we augment the network pruning op- eration with structure optimization algorithm. For this task we propose simulated annealing (SA) algorithm. By considering the permutation of the pruned configurations as a finite Markov chain, the best remaining configuration can be cleverly searched by Simulated Annealing. Different from the other pruning approaches which defined additional fancy objective function and constraints, our approach directly takes the loss function of neural network as an objective function. The optimized network is guaranteed to be the best permutated configuration under the fixed parameters. In addi- tion, since the loss function is used both configuration optimization and parameters update with back propagation, the optimal configuration and the remaining parameters can further be trained to fine-tune the parameters seamlessly.
2 RELATED WORKS
With the increasing applications of deep learning, there are more and more demands for the porta- bility of the DNNs and two lines of research evolved over the years reduce the computational and storage requirements of DNNs. The first line of research focused on reducing the storage require- ments of DNNs; this is achieved by formulating the problem as source coding or data compression problem and making use of existing quantization methods. Examples of this approach include: low bit quantization (Hubara et al. (2018), multi-level quantization (Xu al., 2018)) and clustering based quantization (Choi et al.J 2017). More elaborate source coding schemes are also employed such as Huffman encoding in (Han et al. 2016) and trellis coded quantization (Haase et al. 2020) and entropy based coding (Wiedemann et al. 2020).
The other line of approach aims at reducing the number of parameters in the neural network by making it more sparse which is achieved by pruning, The pruning approach was motivated by the very early (pre-deeplearning) works of LeCun et al (LeCun et al., 1989) and Hassibi and Stork (Hassibi & Stork 1992), which showed that neural networks can still work well when many edges or nodes are pruned.
2.1 NETWORK PRUNING TECHNIQUES
As exemplified in deep compression (Han et al. 2016), one simple yet effective pruning criteria is to select those weights with absolute magnitude smaller than threshold. The process is executed
2

---

Under review as a conference paper at ICLR 2022
based pretrained network. Once the small weights are pruned, the remaining weights will be fine-tuned by training with the dataset again. Various pruning algorithmss differ in their choice of where they apply the threshold such the output of activation function or the gradient of each weight parameter or the magnitude of the weights.
It has also been reported in the deep compression approach (Han et al. 2016) that pruning the net- work gradually by picking up only a small fraction of total weights parameters at one time can help to maintain the performance of the pruned network. However, the sparse network still occupies lot of space in a computer because the pruned weights are all represented by zeros. In order to truely eliminate the parameters from the network, (Chen et al., 2020) proposed tight compression which converts the resulting large sparse matrix after pruning to smaller but dense matrix by moving non- zero members to matrix locations with zeros. These moves are not done in deterministic manner but instead via stochastic heuristic algorithm namely simulated annealing.
2.2 NEURAL ARCHITECTURE SEARCH (NAS)
Despite the conceptual simplicity and ease of application of pruning, it is a valid question to ask whether the resulting sparse networks are optimal. That is, do the resulting networks structures (with the weights assigned after tuning) provide globally optimal results for given data sets. The straight answer to this question is that given the prumed structures, the tuned weights provide local optimals but this observation tells us nothing about the optimality of the structures. This issue has rarely been discussed in the pruning literature. There are some work, however, which consider neural architecture search mostly out of the context of pruning.
The researchers have considered the structure of DNN as composed of operational blocks. These blocks include convolution, activation functions, normalization functions, regularization functions, and so many. According to the various combinations of these blocks, different structures of DNN can be constructed. The constructing process can be designed progress while the model is being trained. By selectively searching all possible queued operations, the optimal configuration can also be determined. When there are more types of operations, or the operation queue getting longer, the total number of possible states will grow exponentially. This could be severe problem in neural architecture search especially when it follows grid-search strategy (Zoph Le 2017). More effec- tive ways to search for the optimal combination of blocks and permutation of active branches were also proposed by Zoph (Zoph et al., 2018). which stacks the basic blocks sequentially and gradually instead of directly finding the optimal whole network architecture. One probabilistic model-basee dynamic optimization method tried to implement a penalty term to the objective function so that the gradients can lead to a better convergence based on generic framework, which determines net- work structure by a vector (Saioo & Shirakawa 2019) (Shirakawa et al. 2018). The other perspective of structure search is to construct computational graph by including all potential operational blocks (Liu et al. 2019a) (Pham et al. 2018). These approaches all focused on the macro scale of structure optimization without considering the importance each parameters contained in operation. Even though the optimal combination of blocks are determined, these structures still may have redundant parameters for weights pruning.
Optimization of structure in micro scale avoiding redundant branches seems to be rare in the litera- ture. One exception is (Nowakowski al. 2018) which utilizes genetic algorithms for the structure optimization task. Our proposed methodology. is similar in spirit to this work, and is detailed in the next section.
3 NETWORK PRUNING WITH SIMULATED ANNEALINGS
In this work, we propose a heuristic non-convex optimization algorithm, namely Simulated anneal- ing (SA) for structure optimization of partially connected neural networks after pruning. The choice of Simulated Annealing has been motivated by the success of the algorithm in various problems in- volving network/graph structures with large number of configurations and complicated cost surfaces
with various local minima (Kuruoglu & Arndt 2017), (Kuruoglu & Ayanoglu 1993), (Liu et al.
2021
SA is motivated by the annealing process in solid state physics, which aims to place the electrons in solid at their lowest energy states achieving lowest possible energy configuration (Metropolis)
3

---

Under review as a conference paper at ICLR 2022
et al. ) Similar to typical discrete optimization problems, SA is given a finite set containing all possible configurations from a fully connected neural network (-)- and a loss function L(),, and looks for c² E C such that [(c*) is minimized. The loss function defined in the optimization problem of neural network is the energy function defined solid state physics. Different from the panaroma of gradient descent algorithms, which very likely be trapped in local optima, Simulated Annealing algorithm determines the global optima by doing a search in the solution space following Markov chain random walk.
Simulated Annealing is a general algorithm which can be applied to various problems, which is documented by its success in vastly different application areas; however as any general algorithm, it requires the design of its components by the user according to the application.
The success of SA depends on careful design of three crucial mechanisms: state neighbourhood structure selection, acceptance-rejectio criteria of proposals and the cooling schedule.
3.1 CHOICE OF STATE NEIGHBORHOOD STRUCTURE
The choice of neighborhood structure dictates the possible moves the random walk can make; hence it affects significantly convergence rate. A too conservative neighbourhood structure would make the exploration of the solution space very slow while a too liberal choice may make the random walk jump over important minima and will not learn enough turning it into a blind random search algorithm on all possible configurations. However, a conservative or liberal neighbourhood structure does not always provide the same level of disadvantage process, big changes between neighborhoods can be helpful for a quick (and coarse) search of the solution space. When potentially interesting part of the solution space has been reached, smaller moves in a conservatively defined neighbourhood would be more beneficial there are more space being looked around, small changes turn to be more beneficial capture the global optimum. On the other hand starting with too conservative neighhorhoods would have made the algorithm stuck in local minima nearby the starting configuration.
advantage. At the beginning of annealing
For the network structure optimization problem, we define the states in the solution space as the configuration of branches connecting nodes in one layer to the next one. Equivalently, we consider mask matrix Mj which is filled with 1 I's at locations (i.j) where there is a surviving branch between the nodes i and j after pruning and with 0's locations (i. where there are no surviving branches.
The neighbouring states are defined to be new configurations obtained by moving one (or more) branch(es) originally at location (i.j) to ki, Equivalently, can be visualised as the move of one from matrix location ji to another position on the same row (i, k). The choice of single branch replacement corresponds to a conservative neighbourhood structure. It can be made more liberal by considering more than one replacements.
3.2 ACCEPTANCE- CICEPTANCE-RJJECTIOO RATIO
Simulated Annealing algorithm can jump out of local optimum is since it occasionally accepts a new state that increases the outcome of the loss function L(-). This mechanism reflects analogy with the electron dynamics in solid state physics, when the electron has enough thermal energy can jump over barriers and end up in higher energy states and hence avoid some local energy minima. As the case of electron dynamics, SA adopts a Boltzmann distribution to decide accepting or rejecting move or step of the random walk:
Pg = min (L.
ACC) k K-T
exex exp
(1)
At every move, -AC() = L(M) - C(-M') is calculated, if is larger than 0, the new state is accepted, otherwise uniform number is generated between [0, 1) and compared with Po. If Pa greater it is accepted, otherwise it is rejected.
3.3 CONVERGENCE
An important question is whether this accept/reject scheme and the choice of neighbourhood struc- ture ensure convergence. The answer to this question is well known in the simulated annealing and Markov chain theory literature (van Laarhoven & Aarts 1987).
4

---

Under review as a conference paper at ICLR 2022
A (finite-state) Markov chain converges to a unique stationary distribution only if it has two fun- damental properties: aperiodicity and irreducibility or equivalently ergodicity. These properties are not easy to check generally and therefore stronger property is used, namely detailed balance.
For our problem, the detailed balance condition can be expressed as:
Pc(M) P(M = Pe(M)) PPMPP))))))
Pe indicates the probability that the configuration is sampled under the representation of mask matrix M, and M' indicates the first order neighbor configuration of the Markov chain after M.
It can be shown that the accept/reject mechanism using the Boltzmann function in Equation ensures that the detailed balance condition and therefore converges to the unique stationary distri- bution (van Laarhoven & Aarts| 1987). This property is shared with the Markov chain Monte Carlo (MCMC) algorithm which aims to obtain the posterior distribution of model parameters. Both al- gorithms construct Metropolis loops via random walks over a Markov chains satisfying the detailed balance condition. The difference is that Simulated Annealing algorithm employs several Metropo- lis loops with decreasing temperature parameter T in Equation [ Hence, posterior distribution approximated by every Metropolis loop gets more and more peaked at the maximum of the station- ary distribution. In this way, the parameter values that maximize the stationary distribution of the Markov chains is obtained.
3.4 COOLING SCHEME AND HYPERPARAMETERS
Another design issue affecting the convergence rate of SA is the cooling scheme. It is known that the convergence is guaranteed in the case of logaritmic cooling, however this requires infinitely slow cooling. Instead, most users prefer geometric cooling
Tx+1 = r/Tn.
The important parameters to be set are the initial temperature (Tinit), the cooling rate (1)) between Metrolopis loops, and the Metropolis loop length (MCC). The temperature decrease controls the acceptance-rejection probability for a
state that has higher loss value. If the temperature is high, the worse states are more likely to be accepted. As the temperature is reduced increasingly, the worse states are more likely to be rejected. The decreasing rate is the main factor responsible for handling the speed of the whole annealing process. However, a faster annealing process does guarantee the convergence of optimization problem. The objective function can be minimized only when the decreasing speed is slow enough. In our work, we setup the initial T as 10 and r as 0.98. The third parameter MLL. should be large enough for the Markov chain to converge to the stationary distribution and small enough not lose time unnecessarily beyond convergence.
3.5 HEURISTIC ATTENTION MECHANISM
Attention mechanism aims at finding the importance of elements contained in a network. the element is more important than the others, its weighted factor, which is represented as number, will be larger. Generally speaking, the element can be anything except for weight parameters. For example, the elements defined in SqueezeNet (Hu al. 2018) are the channels in 2D convolutional operation. By including the weighted factors to multiply by each channel of the feature maps in the forward pass, back propagation can be applied learn the importance of each element. Different from the typical learning method to achieve attention mechanism, Simulated Annealing finds the important elements by trial-and-error with Boltzmann probability. However, unlike the continuous values given from the derivatives using back propagation, the weighted factors computed by our approach would be discrete numbers {0, 1} where and indicate that the elements are ignorable and crucial respectively. These two types of values are actually the fundamental elements composing the mask matrix (M). Being a new incorporation between SA and network pruning, we call our method the heuristic attention mechanism. The details are described in Algorithm 1 where network takes a dataset T and a set of mask matrices as inputs. The marked in the bracket indicates the e-th type of configuration represented by M.
5

---

Under review as a conference paper at ICLR 2022
3.6 PERMUTATION AFTER EDGES PRUNING
According to Algorithm 1, the mechanism starts by deciding how many less important elements eliminate randomly. Even though the elements might be eliminated in one loop, they still have chance to be recovered back during the random walk with the acceptance-rjjection mechanism. Starting from a sparser network, an optimal sub-configuration can be determined iteratively by Sim- ulated Annealing algorithm. If the total number of elements is larger, the corresponding Markov chain must be longer so that a wider space is explored. The longer length indicates that the initial temperature should be higher and the decreasing rate should be closer to so that SA has more time to search for the whole space.
While the network performance will strictly go down as there are increasingly more parameters being erased, the optimal permutation can always be determined under the pruning percentage setup in advance. Different from (Rere et al. 2015) in which applied SA to update weights parameter directly, we implemented it to decide which edges should be kept connected and which should not be in each iteration. In order to change connections in more convenient way, the actual objects that we are fine-tuning would be the mask matrices. The whole optimization process starts from a network trained using back propagation approach.
min L [T(arains Kwn, by bi) Ttran W. D
After getting the optimized weight parameters, and start the permutation process. These two steps can both be fulfilled by manipulating the mask matrices. According the pruning objective, which is minimizing the same loss function L(-) applied in E3)(3) by fine-tuning network configuration, the objective function can be formulated as follows.
small fraction of total weight parameters
argmin L (Tteaaa Mrl")
(M,")
1 Fi=1) W D. 1
Tlabblee
(4)
The objective function (3) optimizes the configuration under fixed preset fraction of randomly pruned network. When the new configuration finalized, the remaining weight parameters should be fine-tuned again make sure the network is stabilized. During the pruning process, the param- eters are not actually being erased. Instead, the numbers are masked with zeros. The difference between mask matrices and dropout operation is that the gradient conducted by back propagation cannot update the parameters being masked by zeros, but can still update the parameters in dropout case.
Min L (Terain (M 11 bi
HWw, (w, D Tlabela
(5)
Therefore, the optimal w and by are specifically indicating those remaining parameters. In order to evaluate the performance and the effectiveness of Algorithm 1 in the field of network pruning, implement it to simple and shallow fully connected network illustrated in Figure which contains only 2 hidden layers.
In both cases of one-shot and gradual pruning strategies, it basically follows the same framework composed of (3, 4, 5). Once there are more percentage of elements being eliminated, both (4) and (5) will need to be executed again in order to keep getting the optimal configuration (permutation). The efficacy of (4) can be obviously found in the following one-shot pruning experiment and the importance of (5) can be observable by comparing the results between these 2 strategies.
Here, we would like to note a shortcoming of this algorithm whether be one-shot or gradual pruning. The algorithm optimizes the structure for given set of weights which were trained for the unpruned network. The optimal weights for a new configuration is not the same with weights taken from the unpruned network. Ideally, the Algorithm should train the new structure weights at each new structure proposal, however this is avoided due to computational complexity increased by the length of the Metropolis loop. Although the network is trained finally when the metropolis loop has ended, of course the decisions that were made by Accept/Reject ratio are affected by the untrained weights. Our aim is to show that even for this suboptimal case improvements can be obtained motivating future research into structure optimization.

---

stable net- Under review as a conference paper at ICLR 2022
inputs
hidden
outputs
10
16
32
Figure I: Configuration of a shallow network. This network is designed to evaluate the effectiveness of SA in the field of network pruning. In order keep things simple, only the edges in hidden layers will be pruned.
3.7 ONE-SHOT PRUNING
One-shot pruning is direct and time-efficient approach to recompose network. Once the optimal parameters w² and d" are obtained E3.(3) with back propagation, the pruning and permutation process can be executed. These operations can be considered as some interference work. The more changes to the configuration, the higher loss value would be computed from new configuration. This trend can be clearly observed in different percentage of one-shot pruning. Since the pruning process is only executed once, the above equations, (4) and (5), will only be exccuted once, where the details are listed in Algorithm 2. In order to make sure that the per- mutation process done Algorithm can work properly, the pruning process is necessary because both connected and disconnected edges will exist in network.
However, the efficiency of one-shot pruning comes at cost, which could have been avoided by the gradual adaptation process. Its performance would significantly drop especially when there increasingly more parameters being pruned. This phenomenon can be observed from the cases by taking both MNIST and FASHION datasets as Tirain for our network in figure 2.
3.8 GRADUAL PRUNING
Gradual pruning is widely known to be better solution to trim network rather than pruning once. Instead of executing the above equations only once, gradual pruning executes them multiple times gradually decrease the total number of network connections. By doing this, the change in new configuration will get more time to fine-tune the remaining weight parameters so that the new loss value will keep close the original loss value. In the case of gradual pruning, a small fraction should be setup in advance. In this paper, we applied 2%, 5%, and 10% in total.
In the beginning of the Algorithm the fully connected network should also be well trained. For the next steps being done iteratively, a small preset fraction of weight parameters would be masked randomly so that the network will contain both connected and disconnected types of edges. After getting an optimal permutation, the network will further be trained for few more epochs to certify weight parameters still remain stable state. By repeatedly mask out more and more weight parameters, a final sparse network can be obtained.
In order to observe the effectiveness of Algorithm we also designed two baseline strategies. The first one is to randomly select weight parameters and the second one is to select the minimum I. amount absolute weight parameters (minK.) inspired by deep compression (Han et al. 2016). When there are gradually less remaining parameters, the statistics of total weight parameters are changed slightly because of the fine-tuning process in Eq.(S). This step plays an important role to prevent the performance of the network from dropping significantly when high percentage of weight parameters are pruned. addition, optimization with back propagation has strong adaptability regardless of the pruning strategy. In the case of minK., more weights updated into larger

---

Under review as a conference paper at ICLR 2022
0.72
0.923
Acouracy
0.850
tresh
mrats-gradeil--eee
Sastion-niggnnl
fastion onishot-train Gestenegraebaalieeen
0.002
50 Pruning % to tully connected network
Figure 2: The comparison between three cases from 2 different datasets: original network, one-shot pruned and trained network, and gradual pruned and trained network. Each result is obtained by computing the mean of independent and identical experiments.
absolute values and the distribution were transformed into a slight sharpened shape. As for the case of SA, the final distribution basically remained similar shape and the weight parameters at different scales were pruned relatively evenly.
4 EXPERIMENTS
According to the procedures of various pruning strategies introduced above, the remaining weight parameters and the corresponding configuration can be really different. order to evaluate the performance of the pruned network at different sparsity levels, two datasets were applied to the experiments individually in this paper. The neural network applied in our experiment is a simple structure, containing only 2 hidden layers with 32 and 16 nodes respectively. The illustration has been shown Figure 1. Simulated annealing algorithm has shown its capability to find out the optimal configuration under the situation that all parameters are fixed. To solve more complex optimization problems, more random walk is guarenteed to increase the performance of the final result. The experiments under 0, 5, 10, 20, 50, and 100 MLL have all been included in this paper. However, better optimization results come at the cost of time complexity of Algorithm 1. According to our experiments, 50 M.C has been seen to good compromise to get both good performance and tolerable time consumption.
4.1 ONE-SHOT PRUNING
The disadvantage of one-shot pruning strategy is that the accuracy will significantly drop when there are too many edges being pruned one time. The trend has been illustrated in figure 2. Before achieving the threshold of 80% sparsity level, the remaining network can still be recovered by applying back propagation to (5). Both figure 5 and figure 6 have shown the changes of network performance from the time when the network is randomly pruned to the time when the configuration is permuted and fine-tuned again. As there are more edges being pruned, the performance gap between the fine-tuned network and the original fully connected network is getting larger.
With the help of SA, our experiments have shown that the weight parameters fine-tuning process in (5) be omitted when the pruning percentage is not high. It would become much more convenient without training the network a GPU and the pruning process could even be executed edge devices. In the case of shallow network used in this paper, SA with 100 MLL almost
8

---

Under review as a conference paper at ICLR 2022
recover the performance of the network being pruned 80%. In addition, when the Metropolis loop length is big enough, the optimized configuration is guarenteed to outperform the other configuration obtained from mink strategy, which is suggested in deep compression (Han et al. 2016). The same trend can be observed in both datasets.
4.2 GRADUAL PRUNING UNDER DIFFERENT MLL
In the case of gradual pruning, there is one more hyperparameter included in the experiment, which is the fixed percentage being used to crase edges iteratively. Accordingly, we get one more di- mension to compare the experimental results. Similar to the success obtained in one-shot pruning strategy, Simulated Annealing algorithm also defeats the other two approaches in gradual pruning when the Metropolis loop length is large enough. The accuracy of the network at different time steps has been shown in figure 7 and figure 8. The network strictly undergoes the optimization process of the three equations Eq.(3, 4, 5). The trends network performance were the results after being permutated and trained to fine-tune weight parameters again. The trends indicate that the network being pruned more percentage at one time during each iteration will decrease the performance faster. In order to maintain the performance of sparser network, gradual pruning has been experimentally proved to be an valid strategy.
In addition, the fine-tuning step is capable of pushing the performance of pruned network to higher level, where the accuracy would be increased up to 1%. This trend is observable before the hidden layer is masked less than half. According the figure 7 and figure at different masked percentage for the hidden layer, SA with large metropolis loop length (50+ .MLC) always outperforms the others. The small amount of accuracy increment and large MLC are proved that they can slow down the decrease in network accuracy especially when majority of network edges are pruned. Unlike the attention mechanism applied in SqueezeNet (Hu et al. 2018) quantifying the importance factor directly to the training process optimization with back propagation, SA needs huge amount of time to locate the global optima by the Boltzmann accept reject mechanism. This is the main disadvantage of SA.
5 CONCLUSION
The most important observation of our work is that direct pruning does not necessarily lead to opti- mal network structures and changing the network structure can give us better results even a weight setting which was not optimized for the new configurations. It has also been experimentally shown that the weight parameters with small magnitude are not certain to be the less important parame- ters. With the help of SA and back propagation to update weight parameters, more lightweight configuration can be obtained without sacrificing its performance. The main shortcoming of the pre- sented algorithm is that for each structure proposal during SA metropolis loop, weight training using backpropagation was not performed. Obviously, the resulting search is suboptimal. Unfor- tunately doing weight training hand in hand with structure training turned out to be computational prohibitive and impossible for portable devices. However, the results obtained even with this sub- optimal scheme show the potential of structure optimization and motivate for further research into network optimization. Speeding up SA so that full optimization can be carried is the objective of our current research.
The other advantage of our work is that the pruning and permuting processes with Simulated An- nealing algorithm involves only forward pass. indicates that RAM space contained in a computer can be hugely saved, and the pruning process can therefore be executed on less powerful computer. In addition, our work has strong extensibility to other types of network pruning. In this paper, we took edges as the basic element to fine-tune the configuration, which generated very long finite Markov chain. However, being an intelligent state space exploration method, the Simulated Anneal- ing algorithm can be coded also for other types of basic elements, such as node and Convolution filter. By asking SA which filter to crase instead of which edge, the time complexity can be reduced to acceptable levels in the pruning process of complex network with the help of GPU.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Xizi Chen, Jingyang Zhu, Jingho Jiang, and Chi-Ying Tsui. Tight compression: Compressing enn through fine-grained pruning and weight permutation for efficient implementation. 57th ACM/IEEE Design Automation Conference (DAC), 2020.
Yoojin Choi, Mostafa El-Khamy, and Jungwon. Towards the limit of network quantization. Inter- national Conference on Learning Representations (ICLR), 2017.
Lei Deng, Guoqi Li, Song Han, Luping Shi, and Yuan Xie. Model compression and hardware acceleration for neural networks: A comprehensive survey. Proceedings of the IEEE, 108:1-48,
2020. DRL: 11011111910 TTR.//////........ 2020. 2976475
Haase, Schwarz, H. Kirchhoffer, S. Wiedemann, T. Marine, A. Marban, K. Muller, W. Samek, D. Marpe, and T. Wiegand. Dependent scalar quantization for neural network compression. Pro- ceedings- International Conference on Image Processing, ICIP, 10000-0ttober36--44 40, 2020.
Song Han, Huizi Mao, and William J Dally, Deep compression: Compressing deep neural networks with pruning. trained quantization and huffman coding. International Conference on Learning Representations (ICLR), 2016.
Babak Hassibi and David G. Stork. Second order derivatives for network pruning: Optimal brain surgeon. Neural Information Processing Systems (NIPS), 1992.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. Neural Information Processing Systems (NIPS), 2014.
Jie Hu, Li Shen, Samuel Albanie, Gang Sun, and Enhua Wu. Squeeze-and-excitation networks. Computer Vision and Pattern Recognition (CVPR), 2018.
I. Hubara, M. Courbariaux, D. Soudry, R. El- Yaniv, and Y. Bengio. Quantized neural networks: Training neural networks with low precision weights and activations. Journal of Machine Learn- ing Research, 18:1-30 2018.
E.E. Kuruoglu and P.F. Arndt. The information capacity of the genetic code: Is the natural code optimal, Journal of Theoretical Biology, 419:227-237, 2017.
Erean Engin Kuruoglu and Ender Ayanoglu. Design of finite-state machines for quantization using simulated annealing. Proceedings of the 1993 IEEE International Symposium on Information Theory, pp. 443, 1993.
Yann LeCun, John S. Denker, and Sara A. Solla. Optimal brain damage. Advances in Neural Information Processing Systems, 2:589-605, 1989.
Mingbao Lin, Rongrong Ji, Yan Wang, Yichen Zhang, Baochang Zhang, Yonghong Tian, and Ling Shao. Hrank: Filter pruning using high-rank feature map. Computer Vision and Pattern Recogni- tion (CVPR), 2020a.
Mingbao Lin, Rongrong Ji, Yuxin Zhang, Baochang Zhang, Yongjian Wu, and Yonghong Tian. Channel pruning via automatic structure search. International Joint Conferences on Artificial Intelligence (IJCAI), 2020b.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. Inter- national Conference on Learning Representations (ICLR), 2019a.
Xianggen Liu, Pengyong Li, Fandong Meng, Hao Zhou, Huasong Zhong, Jie Zhou, Lili Mou, and Sen Song. Simulated annealing for optimization of graphs and sequences. Neurocomputingg 465: 310-324, 2021.
Zechun Liu, Haoyuan Mu, Xiangyu Zhang, Zichao Guo, Xin Yang, Kwang-Ting Tim Cheng, and Jian Sun. Metapruning: Meta learning for automatic neural network channel pruning. Computer Vision and Pattern Recognition (ICCV), 2019b.
Divyam Madaan, Jinwoo Shin, and Sung-Ju Hwang. Adversarial neural pruning with latent vulner- ability suppression. International Conference on Machine Learning (ICMI), 2020.
10

---

Under review as a conference paper at ICLR 2022
Nicholas Metropolis, Arianna W. Rosenbluth, Marshall N. Rosenbluth, Augusta H. Teller, and Ed- ward Teller. Equation of state calculations by fast computing machines. The Journal of Chemical Physics, 21, 1953. URL 10.
Grzegorz Nowakowski, Yaroslaw Dorogyy, and Olena Doroga-Ivaniuk. Neural network structure optimization algorithm. Journal of Automation, Mobile Robotics Intelligent Systems, 2018. URL
JAMRII 1-2018/1
Hieu Pham, Melody Y. Guan, Barret Zoph. Quoc V. Le, and Jeff Dean. Efficient neural architecture search via parameter sharing. Proceedings of the 35th International Conference Machine Learning, pp. 4095-4104, 2018.
L.M. Rasdi Rere, Mohamad Ivan Fanany, and Aniati Murni Arymurthy. Simulated annealing algorithm for deep learning. Procedia Computer Science, 72:137-144, 2015. URL DOI:
10. 10.101675.procs wwa.ross.2115.1..
Shota Saito and Shinichi Shirakawa. Controlling model complexity in probabilistic model-based dynamic optimization of neural network structures. International Conference on Artificial Neural Networks (ICANN), 2019.
Shinichi Shirakawa, Yasushi Iwata, and Youhei Akimoto. Dynamic optimization of neural network structures using probabilistic modeling. Association for the Advancement of Artificial Intelligence (AAAT), 2018.
Peter J. M. van Laarhoven and Emile H. L., Aarts. Simulated Annealing: Theory and Applications, volume 37 of Mathematics and Its Applications. Springer, Dordrecht, 1987.
Kuan Wang, Zhijian Liu, Yujun Lin, Ji Lin, and Song Han. Haq: Hardware- aware automated quantization with mixed precision. Computer Vision and Pattern Recognition (CVPR), 2019.
Wiedemann, K.-R. Muller, and W. Samek. Compact and computationally efficient representation of deep neural networks. IEEE Transactions Neural Networks and Learning Systems, 31(3): 772-785, 2020.
Chen Xu, Jianqiang Yao, Zhouchen Lin, Wenwu Ou, Zhirong Cao, Yuanbin annd Wang, and Hong- bin Zha. Alternating multi-bit quantization for recurrent neural networks. International Confer- ence on Learning Representations (ICLR), 2018.
Xucheng Ye, Pengcheng Dai, Junyu Luo, Xin Guo, Yingjie Qi, Jianlei Yang, and Yiran Chen. Ac- celerating enn training by pruning activation gradients. European Conference on Computer Vision (ECCV), 2020.
Xiangyu Zhang, Jianhua Zou, Xiang Ming, Kaiming He, and Jian Sun. Efficient and accurate approximations of nonlinear convolutional networks. Computer Vision and Pattern Recognition (CVPR), 2015.
Barret Zoph and Quoc V. Le. Neural architecture search with reinforcement learning. International Conference on Learning Representations (ICLR), 2017.
Jonathon Barret Zoph, Vijay Jonathon Shlens, and Quoe Le. Learning transferable architectures for scalable image recognition. Computer Vision and Pattern Recognition (CVPR), 2018.
A APPENDIX
11

---

Under review as a conference paper at ICLR 2022
Algorithm Heuristic Attention Mechanism using SA
Input: (Ttrranss {M 1i=1), ).M)))) {0, 1] innxou xout ,C E R
Parameter: Tinits minn k, MLL
Notes: MLL stands for "Metropolis-tooyy Length"
Output: N(Trrains {M,
(M(c") " Tam1)
1: Let Tinit-
2: while > Tmin do
3: for in 1 MML do
4:
Compute Loss = L(N, Tilobele
5:
Randomly pick connected link in hidden layer.
6:
Randomly pick disconnected link in hidden layer.
Change connection status M (c')
Jiideen M
7:
hidden
8:
Compute Loss' C(N'(M hidden);
hhidde)) Telbbele)
(c')
9:
Loss' < Loss then
Accept new configuration N < N'
11:
else
12:
Random r(0,
exp -(Losd' Lose) then K-
13:
if >
Accept new configuration N N N'
15:
else
Remain the original network configuration
17:
end if end
18:
19:
end for
20: <- 17. f
21:
end while
return N (Trrains {M (e)) 1 )k2))
22:
Algorithm One-shot Pruning with SA (OSPSA)
Input: N (Tddaaa {Mr() }
1==1 M {0, 1] pixxtut
Parameter: r E (0, 1]1*k
Notes: indicates the ratio remained weights parameter.
Output: NTTdate (M)c")) 2 d/r
)==1)
1: Let 50%
2: wi, bū f- Minimize L(N, Tlabels) in Eq.(3)
3: r M+T,-11
4: N'([M((')
fir! ( Allorrthh 1(N)
5: we,br b + Minimize L(N', Tlabels in (5)
6: return (T (M (e) }r=))
12

---

Under review as a conference paper at ICLR 2022
Algorithm 3 Gradual Pruning with SA (GPSA)
Input: (T {M irr; - 11 M {0, 1 jinxou
Parameter: (0, 1]
Notes: r indicates the ratio remained weights parameter.
Output: (TTaat {M
(c") 1 )(=))
1: Let 10%
2: wT. by -- Minimize L(N, T.abels) in E3.(3)
3: while rı p do
4:
ri (/rr-1
5:
(MM Vir }k-)) (Ng)
6:
w., bī & Minimize L(N, Telebole) in E5)
7:
r A+M
8:
N N N
9:
end while
return N(Tdata (M(c") (rr )==1)
% useful weights mean: -0.028; P 0.0
70 % useful weights -0.027; mean: 0.002
40
pruned
35
pruned
30
25
25
Count
20
15
15
10
-0.5
-0.4
-0.3
-0.2
-0.1
0.0
0.1
0.3
-0.5
-0.4
-0.3
-0.2
-0.1
0.0
0.1
0.2
0.3
weights
weights
40 % of useful weights [ mean: -0.029; P mean: 0.0
10 % of useful weights W mean: -0.031; P mean:-On -0.006
total pruned
total
pruned
Count
-0.5
-0.3
-0.8
-0.4
0.0
weights range
weights range
Figure 3: The histogram of total weight parameters under the process of gradual pruning using
min K. strategy as = 10%. It will always determine those weights which has lowest absolute value.
13

---

Under review as a conference paper at ICLR 2022
90 % of useful weights W mean: -0.024; P mean: 0.004
70 % of useful weights W mean: -0.025; P mean: -0.018
total pruned
total pruned
Count
16
Count
20
15
10
5
-0.4
-0.2
-0.1
0.2
0.3
-0.5
-0.4
-0.3
-0.1
0.0
0.3
weights
weights
40 of useful weights| W mean: -0.025; mean: 0.014
10 % of useful weights W mean: -0.025; P mean: -0.027
35
total
total
30
pruned
25
Count
20
10
5
-0.4
-0.2
weights range
weights range
Figure 4: The histogram total weight parameters under the process of gradual pruning followed by Algorithm 10%. Those less effective weights will be determined by simulated annealing algorithm iteratively.
Reduce MNIST 99.8% of links
Reduce MNIST 99% of links
1.0
1.0
0.8
0.0
Accuracy (96)
Accuracy (96))
0.6
0.5
0.4
0.6
02
02-
0.0
0.0
Stralegies
Stralegies
Reduce MNIST 90% of links
Reduce MNIST 80% of links
1.0
1.0
0.0
0.0
Accuracy (%6)
Accuracy (96)
0.6
0.6
0.4
0.4
02
0.0
0.5
Strategies
Strategies
Figure 5: Performance changes under various pruning percentage and different permutation strate-
gies. The network is well trained by MNIST dataset.
14

---

Under review as a conference paper at ICLR 2022
Reduce FASHION 99.8% of links
Reduce FASHION 99% of links
1.0
Accuraty (96
-8 8F pneino AF prurieg
pruiing pruning
branel ugain
brincd agan
r9ndom
minK
INL
middom
min
10 Strategies
Strategies
Reduce FASHION 90% of links
Reduce FASHION 80% of links
Accurary (%6
-8 8F pnnino
puning pruking
AF pruseng
braved agon
Inaeced agar
midee
Strategies
Strategies
Figure 6: Performance changes under various pruning percentage and different permutation strate- gies. The network is well trained by FASHION dataset.
15

---

Under review as a conference paper at ICLR 2022
Eliminate 2% of edges trained by MNIST
0.96
Accuracy (%)
0.96
0.95
min K
metropolis
metropolis
metropolis 20
metropolis
metropolis
0.93
0
288228 88 949
Masked % for a hidden layer
Comparison between different pruning fractions trained by MNIST
0.98
0.97
Accuracy (%)
0.96
rondom reduce 2%
random reduce
rendom- reduce 10%
min K- reduce 2%
0.94
min reduce 10%
metropolis 50 reduce 2%
metropolis 50 reduce 10%
0.93
10
15
20
25
30
90
95
Masked % for hidden laver
Figure 7: The trends of network performance being pruned at various scales gradually. The network well trained and fine-tuned by MNIST dataset.
16

---

Under review as a conference paper at ICLR 2022
Eliminate 2% of edges trained by FASHION
0.89
0.88
Accuracy (%)
0.87
0.86
rand
min K
metropolis
metropolis 10
metropolis 20
metropolis
metropalis 100
0.84
1012144818202222266888222
0t0.://2220002221111111444428222 04033300804448802245556882222220 74 7880 868886888888 88 9698
Masked % for a hidden layer
Comparison between different pruning fractions trained by FASHION
0.89
0.88
Accuracy (%)
0.87
0.86
random reduce
random reduce 5%
random reduce 10%
reduce
min K- reduce 5%
0.85
min reduce 10%
metropolis reduce
metropolis 50 reduce 5%
metropolis
10%
0.84
10
15
20
45
50
55
60
65
70
90
95
Masked % for hidden layer
Figure 8: The trends of network performance being pruned at various scales gradually. The network well trained and fine-tuned by FASHION dataset.
17


Reviews:
[{"review_id": "hksxp8KokLW", "reviewer": "Reviewer_inN3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The topic treated in this paper is interesting and important. However, the weaknesses of this paper and the reviewer's concerns are as follows:\n\n- Many pruning algorithms have been developed so far, including pruning methods based on black-box optimization like simulated annealing. The novelty of this paper is not apparent compared to the existing pruning methods. \n\n- The effectiveness of simulated annealing for network pruning is not apparent. For example, a comparison to other black-box optimization methods, such as genetic algorithms, should be conducted to claim the effectiveness of the simulated annealing. Also, the state-of-the-art pruning methods should be included in the baseline. \n\n- Random pruning should be included as the simpler baseline.\n\n- In the experiment, the size of the neural network is too small compared to the current trend. The convolutional neural networks, such as VGG and ResNet, should be considered as the baseline networks to be pruned. The following paper is useful for experimental design.\n\nDavis Blalock, Jose Javier Gonzalez Ortiz, Jonathan Frankle, John Guttag, \"What is the state of neural network pruning?,\" arXiv:2003.03033 (2020).", "summary_of_the_paper": "This paper develops a method for neural network pruning using simulated annealing. In the proposed method, the mask matrix indicating the pruning location is optimized by simulated annealing. The proposed method is evaluated on pruning tasks of fully connected neural networks in image classification.", "summary_of_the_review": "This paper proposes the network pruning method using simulated annealing. The basic framework of the proposed method is the same as the usual pruning methods. The experimental comparison with other pruning methods is not performed. Therefore, the reviewer can recognize neither the novelty and effectiveness of the proposed method.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "dk2aIUSgr4S", "reviewer": "Reviewer_EBdG", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The idea of using simulated annealing to pruning neural networks seems interesting, but the paper suffers from the issues of 1) insufficient presentation, and 2) limited experiments. My concerns include:\n1.\tThe related work is very sparse. There are many recently published works considering neural architecture search (NAS) and network pruning (NP). But only a few of them is introduced in this study.\n2.\tThe main weakness of this study is the experiment design. The proposed method is only tested on a very simple neural network. Thus, it is difficult to judge whether this method is effective for widely-used convolution neural networks and others.\n3.\tThe performance comparison in this study is also very weak. There are many heuristic algorithms, e.g., evolutionary algorithms, that have been applied on NAS and NP. The proposed method should be compared with those SOTA ones.", "summary_of_the_paper": "  This paper presents a novel simulated annealing method for pruning neural networks. Both one-shot pruning and gradual pruning strategies are discussed in this study. Experiments are conducted based on a simple neural network structure with only two hidden layers. The results show the effectiveness of the proposed method.", "summary_of_the_review": " Although the idea seems interesting, this paper is not technically sound. Experimental results cannot fully support the main conclusion of this paper. Without extensive performance comparison, it is also hard to judge whether this new method can make a real contribution to the related community.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "bQchaZkK6C", "reviewer": "Reviewer_1vEQ", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths \n* Interesting application of simulated annealing to solve the network compression problem.\n* Detailed overview and motivation for pruning algorithms and simulated annealing.\n\nWeaknesses\n* The limited scale and scope of the experiments puts the significance of the results into question. Experiments are conducted on MNIST and FashionMNIST and appear to be single seed, with no clear comparisons to baselines. This is a much smaller scale than similar research published e.g. lottery ticket hypothesis papers. Conducting more detailed comparison to baselines would greatly strengthen the paper.\n* The organization of the paper could be improved. There is too much space spent on discussing related work and details of the simulated annealing algorithm, which can be deferred to the appendix. This leaves less room for detailing the actual method and experiments, which come off less clearly as a result. It took me some time to understand that one-shot and gradual pruning \n* There are minor grammatical papers throughout the paper. ", "summary_of_the_paper": "This paper explores the application of simulated annealing to network pruning. In the proposed algorithm, a fully trained network has some percentage of weights erased and its connections are perturbed to new configurations, with acceptance happening according to a Boltzmann distribution. Two variations of the algorithm are proposed, one where the target percentage of connections is removed in one-shot, the other where the process is gradual.", "summary_of_the_review": "I recommend rejection of the paper. Simulated annealing is not a new technique, and the limited scale of the experiments and lack of comparisons to baselines limit the utility of this work in its current form. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "1pAzpZFpPOC", "reviewer": "Reviewer_t58P", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The SA presented in the paper is easy to understand and well studied in traditional optimization community. It has proven convergence to optimality. Applying it for network pruning is certainly an interesting idea. The experiment results demonstrate the algorithm can prune up to 80% of the links without too much accuracy deterioration. The algorithm should be not difficult to implement in practice. I believe it helps small neural networks. \n\nThe paper has following weaknesses:\n1. The algorithm drops or adds a link a time, it can take very long to prune a large network. Why not drop/add a set of links so that the state neighborhood can be larger for large networks? \n2. Some content in Appendix should be included in main paper, such as Algorithm 1. Appendix is not guaranteed to be reviewed. \n3. Need more baseline comparison. How good is it when comparing with the deep compression (Han et a, 2016, Chen et al, 2020)?\n4. There are other network prune studies, for instance ASAP: Architecture Search, Anneal and Prune, by Asaf Noy, etc. What is the difference with them?\n\n\n\n\n\n", "summary_of_the_paper": "The paper proposes a simulation annealing (SA) algorithm to prune neural network by randomly dropping and adding links according to SA's acceptance-rejection criteria. Experiment results shows the algorithm is useful in reducing number of weights while maintain model accuracy. ", "summary_of_the_review": "The proposed SA method is interesting, however, I am not convinced it can scale well, and the experiments are using small networks. Need more baseline comparison to show its advantages.  ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents the use of Simulated Annealing (SA) for pruning and optimizing the architecture of a neural network. After reviewing the paper and taking into consideration of the reviewing process, here are my comments:
- The contribution of the paper and the novelty is limited and not well presented
- The related work is very sparse. It requires a major improvement.
- The main concern is about the simplistic experiments and the lack of comparison between the results of the proposal and the SOTA methods.
- Conclusions are not well supported by the results.
From the above, the paper does not fulfill the standards of the ICLR.
Decision: Reject