Pruning Deep Neural Networks. TL; Different approaches of pruning... | by Ranjeet Singh | Towards Data Science

Deep Learning models these days require a significant amount of computing, memory, and power which becomes a bottleneck in the conditions where we need real-time inference or to run models on edge devices and browsers with limited computational resources. Energy efficiency is a major concern for current deep learning models. One of the methods for tackling this efficiency is enabling inference efficiency.
Larger Model => More Memory References => More Energy
Pruning is one of the methods for inference to efficiently produce models smaller in size, more memory-efficient, more power-efficient and faster at inference with minimal loss in accuracy, other such techniques being weight sharing and quantization. Out of several aspects that deep learning takes as an inspiration from the area of Neuroscience. Pruning in deep learning is also a biologically inspired concept that we will talk later in this post.
With the progress in Deep Learning, the state of the art models are getting more and more accurate but this progress comes with a cost. I will discuss a few of these here in this blog.
Hard to distribute large models through an over-the-air update
Such long training time limits ML researcher’s productivity.
AlphaGo: 1920 CPUs and 280 GPUs, $3000 electric bill per game
On Mobile devices: drains battery
On data-center: increases TCO
Pruning
Weight Sharing
Quantization
Low-Rank Approximation
Binary / Ternary Net
Winograd Transformation
Pruning in artificial neural networks has been taken as an idea from Synaptic Pruning in the human brain where axon and dendrite completely decay and die off resulting in synapse elimination that occurs between early childhood and the onset of puberty in many mammals. Pruning starts near the time of birth and continues into the mid-20s.
Networks generally look like the one on the left: every neuron in the layer below has a connection to the layer above, but this means that we have to multiply a lot of floats together. Ideally, we’d only connect each neuron to a few others and save on doing some of the multiplications; this is called a “sparse” network.
Sparse models are easier to compress, and we can skip the zeroes during inference for latency improvements.
If you could rank the neurons in the network according to how much they contribute, you could then remove the low ranking neurons from the network, resulting in a smaller and faster network.
Getting faster/smaller networks is important for running these deep learning networks on mobile devices.
The ranking, for example, can be done according to the L1/L2 norm of neuron weights. After the pruning, the accuracy will drop (hopefully not too much if the ranking is clever), and the network is usually trained-pruned-trained-pruned iteratively to recover. If we prune too much at once, the network might be damaged so much it won’t be able to recover. So in practice, this is an iterative process — often called ‘Iterative Pruning’: Prune / Train / Repeat.
See this code by the Tensorflow team to understand iterative pruning.
Set individual weights in the weight matrix to zero. This corresponds to deleting connections as in the figure above.
Here, to achieve sparsity of k% we rank the individual weights in weight matrix W according to their magnitude, and then set to zero the smallest k%.
f = h5py.File("model_weights.h5",'r+')for k in [.25, .50, .60, .70, .80, .90, .95, .97, .99]: ranks = {} for l in list(f[‘model_weights’])[:-1]: data = f[‘model_weights’][l][l][‘kernel:0’] w = np.array(data) ranks[l]=(rankdata(np.abs(w),method=’dense’) — 1).astype(int).reshape(w.shape) lower_bound_rank = np.ceil(np.max(ranks[l])*k).astype(int) ranks[l][ranks[l]<=lower_bound_rank] = 0 ranks[l][ranks[l]>lower_bound_rank] = 1 w = w*ranks[l] data[...] = w
Set entire columns to zero in the weight matrix to zero, in effect deleting the corresponding output neuron.
Here to achieve sparsity of k% we rank the columns of a weight matrix according to their L2-norm and delete the smallest k%.
f = h5py.File("model_weights.h5",'r+')for k in [.25, .50, .60, .70, .80, .90, .95, .97, .99]: ranks = {} for l in list(f[‘model_weights’])[:-1]:     data = f[‘model_weights’][l][l][‘kernel:0’]     w = np.array(data)     norm = LA.norm(w,axis=0)     norm = np.tile(norm,(w.shape[0],1))     ranks[l] = (rankdata(norm,method=’dense’) —     1).astype(int).reshape(norm.shape)     lower_bound_rank = np.ceil(np.max(ranks[l])*k).astype(int)     ranks[l][ranks[l]<=lower_bound_rank] = 0     ranks[l][ranks[l]>lower_bound_rank] = 1     w = w*ranks[l]     data[...] = w
Naturally, as you increase the sparsity and delete more of the network, the task performance will progressively degrade. What do you anticipate the degradation curve of sparsity vs. performance to be?
Pruning results on image classification model on MNIST dataset using a simple neural network architecture given as -
Many researchers consider Pruning as an overlooked method that is going to get a lot more attention and use in practice. We showed how we can get nice results on a toy dataset using a very simple neural net architecture. I think many problems deep learning is used to solve in practice are similar to this one, using transfer learning on a limited dataset, so they can benefit from pruning too.
Code for this blog post
To prune, or not to prune: exploring the efficacy of pruning for model compression, Michael H. Zhu, Suyog Gupta, 2017
Learning to Prune Filters in Convolutional Neural Networks, Qiangui Huang et. al, 2018
Pruning deep neural networks to make them fast and small
Optimize machine learning models with Tensorflow Model Optimization Toolkit