A Directed-Evolution Method for Sparsification and Compression of Neural Networks with Application to Object Identification and Segmentation and considerations of optimal quantization using small number of bits

(Abstract) In seeking effective neural network models for use in power and resource constrained edge computing, model pruning (forced sparsification) and optimal parameter representation with diminished number of bits have become central strategies for successful deployment. Pruning became especially attractive when it was shown that a pruned large neural network outperforms a “dense” smaller network with the same memory footprint [1]. Pruning methods were proposed under the assumption that the magnitudes of the network parameters are good proxies for the relevance of those parameters to the neural network’s accuracy [2,3]. Hence, these methods attempt to zero low magnitude parameters. Pruning methods based on Hessian approximations (curvature information) around network’s loss minima were reported to consistently outperform those methods based solely on parameters’ magnitudes [4]. Both methods were shown to be superior to merely random methods since these zero parameters with no previous consideration of their impact in network accuracy [5]. These results combined appear to suggest that the first two approaches are using proxies for the relevance of parameters and that there is space for improving on those results further if the actual relevance of the parameters is more accurately determined before zeroing them. This work introduces DirectedEvolution (DE) method for sparsification of neural networks, where the relevance of parameters to the network’s accuracy is directly assessed and the parameters that produce the least effect on accuracy when tentatively zeroed are indeed zeroed. DE method avoids a potentially combinatorial explosion of all possible candidate sets of parameters to be zeroed in large networks by mimicking evolution in the natural world. DE starts at a pre-trained network and progressively “evolves” towards a much sparser network model using cycles of cumulative small steps of minimal loss of performance and re-training. In addition, DE uses a distillation context [5]. In this context, the original network is the “teacher” and DE evolves the “student” neural network to the sparsification goal while maintaining minimal divergence between teacher and student. The DE’s distillation context provides three benefits: (a) the student network captures relevant knowledge learned by the pre-trained “teacher” network, (b) the degree to which the teacher network is over-parameterized is made clear at every cycle of the DE method and (c) the dataset used in the retraining stage in each DE cycle can be much smaller than the dataset previously used in the training of the teacher network. After the desired sparsification level is reached in each layer of the network by DE, a variety of quantization alternatives are used on the surviving parameters to find the lowest number of bits for their representation with acceptable loss of accuracy. A procedure to find optimal distribution of quantization levels in each sparsified layer is presented. Suitable final lossless encoding of the surviving quantized parameters is used for the final parameter representation. DE was used in sample of representative neural networks using MNIST, Fashion-MNIST and COCO data sets with progressive larger networks. An 80 classes YOLOv3 with more than 60 million parameters network trained on COCO dataset reached 90% sparsification and correctly identifies and segments all objects identified by the original network with more than 80% confidence while using 4-bit parameter quantization. DE produced final memory footprint compression between 40x and 80x in the examples studied. It has not escaped the authors that techniques from different methods can be nested, and that once the best parameter set for sparsification is identified in a cycle of DE, a decision on zeroing only a sub-set of those parameters can be made using a combination of criteria like parameter magnitude and Hessian approximations. In this current paper, only results for the case of purely DE method are reported, and results for nesting of methods inside DE will be reported in a follow-up paper.