Network Pruning Optimization by Simulated Annealing Algorithm

One critical problem of large neural networks is over-parameterization with a large number of weight parameters . This becomes an obstacle to implement networks in edge devices as well as limiting the development of industrial applications by engineers for machine learning problems . Plenty of papers have shown that the redundant branches can be erased strategically in a fully connected network . In this work , we reduce network complexity by pruning and structure optimization . We propose to do network optimization by Simulated Annealing , a heuristic based non-convex optimization method which can potentially solve this NP-hard problem and find the global minimum for a given percentage of branch pruning given sufficient amount of time . Our results have shown that Simulated Annealing can significantly reduce the complexity of a fully connected neural network with only limited loss of performance . 1 INTRODUCTION . The successful development of the artificial intelligence methods fueled by deep-learning has made important impact in multiple industries . Elaborate AI algorithm can empower electronic devices to make smart decisions or foresighted predictions . However , the outstanding outcomes come with their own cost , which is the computational complexity in the case of deep-learning methods . In order to obtain a model that can accurately predict the results corresponding to the input data , a neural network should go through initialization , back propagation , gradients update , and inference process step by step . When the deep learning structure is more complicated , there are more parameters and operations involving in these 4 steps . Network compression technique is a relatively new area necessitated by the booming development of deep neural network and increasing computational complexity of machine learning methods . Attracted by the extraordinary performance of recently proposed neural network architectures , there are increasingly more demands on edge devices which possess only limited computational resources . Therefore , various network compression techniques have been tried and studied during these years . The recent mainstream strategies include quantization ( Wang et al. , 2019 ) , decomposition ( Zhang et al. , 2015 ) , knowledge distillation ( Hinton et al. , 2014 ) , and the pruning methods , which is considered as the most effective approach to compress a neural network ( Liu et al. , 2019b ) ( Lin et al. , 2020a ) . Among those 4 computational steps of deep learners mentioned above , back propagation and gradients update make up the major part that consumes the most of computing resources . One approach by pruning those gradients that are smaller than a threshold ( Ye et al. , 2020 ) , significantly reduces the usage of hardware resources and accelerates the whole training procedure . However , the trained model would still have the same parameter size after undergoing the pruned training procedure since the actual weight parameters are not eliminated from the network from computational perspective . Thus , this approach is only helpful for training a network on a powerful computer . In order to actually reduce the total size of a model while inferencing in a mobile electronic device , various research found in the survey ( Deng et al. , 2020 ) focus on pruning the weight parameters directly instead of the gradients . These pruning techniques can be element-wise , vector-wise , or block-wise , which can correspond to unstructured or structured pruning . Behind each type of pruning technique , there are multiple criteria being setup differently . One innovative criteria is the combination of generative adversarial network ( GAN ) and parameters selection ( Madaan et al. , 2020 ) . By evaluating the similarity of the feature maps before and after pruning process , the importance of certain parameters can be defined . Except for evaluating the feature map with a designed criteria setup in GAN , the other group of pruning strategies follow heuristic optimization algorithms , where the network is considered as an architecture fine-tuning process . Since a neural network is one type of network composed of both nodes and edges , it would be intuitive to set up the optimization problem in terms of the edge weights and the structure of the network and the pruning strategy . ABC optimization technique is an example that demonstrates the application of a heuristic algorithm on channel-wise network pruning ( Lin et al. , 2020b ) . Based on the hierarchical characteristic of the neural network structure , pruning techniques can further be split into both structured and unstructured pruning categories . When a neural network is simulated on computer , the major part of computations would be carried out in matrix operations including addition , multiplication , and dot product . Therefore , in the case of unstructured pruning , the edges or nodes are not truly being eliminated . Instead , an extra mask matrix is designed to simulate the pruning process . Even though the mask matrix can bring up high flexibility in adjusting the network configuration , the model size can not be reduced significantly since the pruned parameters should still be conserved in the matrix as zero entries . On the contrary , the structured pruning strategies sacrifice some optimality to ensure dimension reduction and avoid sparse but high dimensional matrices . The most representative cases for both unstructured and structured pruning are edge-level and node-level pruning strategies . if one edge of a network is considered as the minimum unit to reduce configuration , it would not affect the existance of all other elements in the network . However , if one node is the minimum unit to be reduce a network , all other edges connected to the selected node will also need to be erased . In choosing pruning strategies , the main factor is to determine which part of the minimum unit ( edges , nodes , ... etc ) is more important than the others . The existing pruning approaches do not consider the optimality of the resulting network structure in any means . The optimality under question has two aspects to it : the branch weights and the branch connections . In this work , we aim to optimize the branch connections with the constraint of pruning percentage , That is , we question the optimality of the structure obtained by pruning . In this paper , we augment the network pruning operation with a structure optimization algorithm . For this task we propose simulated annealing ( SA ) algorithm . By considering the permutation of the pruned configurations as a finite Markov chain , the best remaining configuration can be cleverly searched by Simulated Annealing . Different from the other pruning approaches which defined additional fancy objective function and constraints , our approach directly takes the loss function of a neural network as an objective function . The optimized network is guaranteed to be the best permutated configuration under the fixed parameters . In addition , since the loss function is used to both configuration optimization and parameters update with back propagation , the optimal configuration and the remaining parameters can further be trained to fine-tune the parameters seamlessly . 2 RELATED WORKS . With the increasing applications of deep learning , there are more and more demands for the portability of the DNNs and two lines of research evolved over the years to reduce the computational and storage requirements of DNNs . The first line of research focused on reducing the storage requirements of DNNs ; this is achieved by formulating the problem as source coding or data compression problem and making use of existing quantization methods . Examples of this approach include : low bit quantization ( Hubara et al. , 2018 ) , multi-level quantization ( Xu et al. , 2018 ) and clustering based quantization ( Choi et al. , 2017 ) . More elaborate source coding schemes are also employed such as Huffman encoding in ( Han et al. , 2016 ) and trellis coded quantization ( Haase et al. , 2020 ) and entropy based coding ( Wiedemann et al. , 2020 ) . The other line of approach aims at reducing the number of parameters in the neural network by making it more sparse which is achieved by pruning . The pruning approach was motivated by the very early ( pre-deeplearning ) works of LeCun et al ( LeCun et al. , 1989 ) and Hassibi and Stork ( Hassibi & Stork , 1992 ) , which showed that neural networks can still work well when many edges or nodes are pruned . 2.1 NETWORK PRUNING TECHNIQUES . As exemplified in deep compression ( Han et al. , 2016 ) , one simple yet effective pruning criteria is to select those weights with absolute magnitude smaller than a threshold . The process is executed based on a pretrained network . Once the small weights are pruned , the remaining weights will be fine-tuned by training with the dataset again . Various pruning algorithms differ in their choice of where they apply the threshold such as the output of activation function or the gradient of each weight parameter or the magnitude of the weights . It has also been reported in the deep compression approach ( Han et al. , 2016 ) that pruning the network gradually by picking up only a small fraction of total weights parameters at one time can help to maintain the performance of the pruned network . However , the sparse network still occupies a lot of space in a computer because the pruned weights are all represented by zeros . In order to truely eliminate the parameters from the network , ( Chen et al. , 2020 ) proposed tight compression which converts the resulting large sparse matrix after pruning to a smaller but dense matrix by moving nonzero members to matrix locations with zeros . These moves are not done in a deterministic manner but instead via stochastic heuristic algorithm namely simulated annealing . 2.2 NEURAL ARCHITECTURE SEARCH ( NAS ) . Despite the conceptual simplicity and ease of application of pruning , it is a valid question to ask whether the resulting sparse networks are optimal . That is , do the resulting networks structures ( with the weights assigned after tuning ) provide globally optimal results for given data sets . The straight answer to this question is that given the pruned structures , the tuned weights provide local optimals but this observation tells us nothing about the optimality of the structures . This issue has rarely been discussed in the pruning literature . There are some work , however , which consider neural architecture search mostly out of the context of pruning . The researchers have considered the structure of a DNN as composed of operational blocks . These blocks include convolution , activation functions , normalization functions , regularization functions , and so many . According to the various combinations of these blocks , different structures of DNN can be constructed . The constructing process can be designed to progress while the model is being trained . By selectively searching all possible queued operations , the optimal configuration can also be determined . When there are more types of operations , or the operation queue is getting longer , the total number of possible states will grow exponentially . This could be a severe problem in neural architecture search especially when it follows grid-search strategy ( Zoph & Le , 2017 ) . More effective ways to search for the optimal combination of blocks and permutation of active branches were also proposed by Zoph ( Zoph et al. , 2018 ) , which stacks the basic blocks sequentially and gradually instead of directly finding the optimal whole network architecture . One probabilistic model-based dynamic optimization method tried to implement a penalty term to the objective function so that the gradients can lead to a better convergence based on a generic framework , which determines a network structure by a vector ( Saito & Shirakawa , 2019 ) ( Shirakawa et al. , 2018 ) . The other perspective of structure search is to construct a computational graph by including all potential operational blocks ( Liu et al. , 2019a ) ( Pham et al. , 2018 ) . These approaches all focused on the macro scale of structure optimization without considering the importance of each parameters contained in an operation . Even though the optimal combination of blocks are determined , these structures still may have redundant parameters for weights pruning . Optimization of structure in micro scale avoiding redundant branches seems to be rare in the literature . One exception is ( Nowakowski et al. , 2018 ) which utilizes genetic algorithms for the structure optimization task . Our proposed methodology , is similar in spirit to this work , and is detailed in the next section .

Summary:   This paper presents a novel simulated annealing method for pruning neural networks. Both one-shot pruning and gradual pruning strategies are discussed in this study. Experiments are conducted based on a simple neural network structure with only two hidden layers. The results show the effectiveness of the proposed method.