Group Sparse Regularization for Deep Neural Networks | by Himaprasoon P.T | Towards Data Science

Paper summary of Group Sparse Regularization for Deep Neural Networks
With the advancement in hardware technologies and thereby reduced costs, training large neural networks has become a lot easier. Due to this, it has become common to throw in large networks for even simple tasks. Having too many layers in a network can lead to multiple problems like,
Overfitting
Need for Network Pruning before deploying to production / mobile / IOT devices
Then there is the problem of feature selection. Feature selection is the process of searching for a subset of all the features which are important for modeling. This is done usually as a separate pipeline before modeling starts.
Sparse group regularization(SGR) tackles these three problems/ issues
Optimizing weights of neural networkOptimizing the number of neurons for each hidden layerFeature Selection
Optimizing weights of neural network
Optimizing the number of neurons for each hidden layer
Feature Selection
and creates a single process/pipeline for tackling all of these. In neural networks, feature selection can be considered as pruning of nodes at input layer and that is how it is addressed in sparse group regularization.
If all the incoming and outgoing weights of a neuron are 0, then that node can be removed.
In the above ANN image, the red connections are weights which are zero. As all the connections from input feature X2 are zeros, this feature can be removed. Similarly for node H3, as all the outgoing weights are zeros, this node can also be removed. This is the idea of sparsity used in pruning of nodes and features.
L1 regularization (Lasso) is one way in which sparsity of weights can be achieved. In this technique the sum of absolute values of the weights are penalized during training.
New Cost = Cost + λ ||w||
As the weights are penalized by their absolute magnitude, sparsity can be achieved using L1 regularization.
Click here to know more about L1 and L2 regularization.
However, using L1(Lasso) alone cannot guarantee a systematic sparsity using which nodes can be removed. In order to remove a node, all the outgoing weights from the node should be zero. L1 can result to a different type of sparsity where it is unstructured. See the image below
Here 2 weights are zero but none of the nodes can be removed. Therefore L1 alone cannot help in pruning. This is where sparse group regularization comes to the picture.
To get a structured sparsity for pruning, we need to group outgoing connections from each neuron and force them to be zero. In order to do so, 3 different types of groups of weights are considered,
Input Groups Gin : an element gi ∈ Gin, i = 1, . . . , d is the vector of all outgoing connections from the ith input neuron to the network. In the image above (Figure 3), there are two input groups(Green). The input group of x1 will contain all the weights from x1 to the hidden layer. Input groups are used for feature selection.Hidden Groups : Gh: in this case, a single element g ∈ Gh corresponds to the vector of all outgoing connections from one of the neurons in the hidden layers of the network. In figure 3 there are two output groups ( same as number of hidden nodes).Bias groups: Gb: these are one-dimensional groups (scalars) corresponding to the biases on the network. Instead of considering each bias as a separate group, a single group containing all the bias variables can also be used.
Input Groups Gin : an element gi ∈ Gin, i = 1, . . . , d is the vector of all outgoing connections from the ith input neuron to the network. In the image above (Figure 3), there are two input groups(Green). The input group of x1 will contain all the weights from x1 to the hidden layer. Input groups are used for feature selection.
Hidden Groups : Gh: in this case, a single element g ∈ Gh corresponds to the vector of all outgoing connections from one of the neurons in the hidden layers of the network. In figure 3 there are two output groups ( same as number of hidden nodes).
Bias groups: Gb: these are one-dimensional groups (scalars) corresponding to the biases on the network. Instead of considering each bias as a separate group, a single group containing all the bias variables can also be used.
Total set of Groups G is
Sparse Group Regularization can be written as,
where |g| denotes the dimensionality of the vector g, and it ensures that each group gets weighted uniformly. Here the L2 norm of each group is used. As L2 norm cannot create sparsity unlike L1, a thresholding step is done to convert low weights to zeros.
This formulation is still suboptimal (even after thresholding) to get high level of sparsity. To enforce this, a combination of lasso and sparse group penalty is used. Sparse Group Lasso penalty is defined as,
where the second term is the L1 norm of the weights.
Figure 4 shows a comparison between the three types of regularization penalties discussed in this blog. The 2x5 matrix represent the weight matrix between a 2 dimensional input layer and a 5 dimensional output layer.
The lasso penalty (l1) removes connections without taking node level considerations. In the image, it has resulted in 40% sparsity of connections. The second neuron in the output layer (5 dim) may be removed as all the incoming connections from both the input nodes are grayed out (See column 2 in Lasso in Figure 4).
In Group Lasso (After thresholding in Sparse Group Regularization) , all the connections from the second neuron input layer are zero and therefore can be removed.
The sparse group lasso penalty combines the advantages from both the previous formulations. Two nodes (2nd and 5th) from the output layer can be removed and also the second input neuron can be removed as well. The number of connections are reduced to 70% and leads to a very compact network.
[1] Simone Scardapane , Amir Hussain and Aurelio Uncini , Group Sparse Regularization for Deep Neural Networks 2017