Finding Deep Local Optima Using Network Pruning

Artificial neural networks ( ANNs ) are very popular nowadays and offer reliable solutions to many classification problems . However , training deep neural networks ( DNN ) is time-consuming due to the large number of parameters . Recent research indicates that these DNNs might be overparameterized and different solutions have been proposed to reduce the complexity both in the number of parameters and in the training time of the neural networks . Furthermore , some researchers argue that after reducing the neural network complexity via connection pruning , the remaining weights are irrelevant and retraining the sub-network would obtain a comparable accuracy with the original one . This may hold true in most vision problems where we always enjoy a large number of training samples and research indicates that most local optima of the convolutional neural networks may be equivalent . However , in non-vision sparse datasets , especially with many irrelevant features where a standard neural network would overfit , this might not be the case and there might be many non-equivalent local optima . This paper presents empirical evidence for these statements and an empirical study of the learnability of neural networks ( NNs ) on some challenging non-linear real and simulated data with irrelevant variables . Our simulation experiments indicate that the cross-entropy loss function on XOR-like data has many local optima , and the number of local optima grows exponentially with the number of irrelevant variables . We also introduce a connection pruning method to improve the capability of NNs to find a deep local minimum even when there are irrelevant variables . Furthermore , the performance of the discovered sparse sub-network degrades considerably either by retraining from scratch or the corresponding original initialization , due to the existence of many bad optima around . Finally , we will show that the performance of neural networks for real-world experiments on sparse datasets can be recovered or even improved by discovering a good sub-network architecture via connection pruning . 1 INTRODUCTION . Artificial neural networks are very popular nowadays and they can offer good solutions to many vision problems ( Dahl et al. , 2013 ; Krizhevsky et al. , 2012 ; Le , 2013 ; He et al. , 2016 ) . However , the neural networks usually require massive amounts of training data for good generalization , which limit their applications to non-vision problems where the data is not so abundant . In this paper , we conduct an empirical investigation of the relationship between the amount of training data and the landscape of the loss function . We will see that when the training data is large , the landscape has fewer local optima , which makes it is easier to find a deep local optimum , corresponding to a model with good generalization . If the training data is limited , the energy landscape could have in some cases so many local minima that it is computationally prohibitive to find a deep local minimum with good generalization . For this experimental investigation , we will use it as a proxy of a difficult non-linear dataset generalized XOR data with extra irrelevant variables . By adjusting the dimensionality of the XOR relationship and the number of irrelevant variables we can control the degree of difficulty of the problem and see the relationship between the amount of training data and the loss landscape . For our experimental investigation , we will focus our attention to simple neural networks with one hidden layer and ReLU activation . On the XOR-based classification problems , we will show that the cross-entropy energy landscape has many shallow local minima and some deep minima that are very hard to find . We will also observe that the number of shallow local minima grows prohibitively large as the number of irrelevant variables increase , and the chances of finding a deep minimum decrease quickly with the number of irrelevant variables . This decrease is much faster for harder problems ( e.g . 5D XOR ) than for easier problems ( 3D XOR ) . We also observe that for a fixed problem , the difficulty of finding deep local minimum decreases as more training examples are used . Based on these findings , we propose a graduate node and connection pruning technique that starts with a dense network with many hidden nodes and gradually prunes connections or nodes to arrive at a sparse network with few hidden nodes . A large number of initially hidden nodes ensure high connectivity between the local minima making them easier to find . The gradual pruning ensures that the local optimum is not lost during the optimization process . We will see in experiments that this technique finds deeper local optima than when starting from a small network with random initialization or retrain the found sparse sub-network from scratch or using corresponding original initial parameters . This is a purely experimental paper that does not intend to make any theoretical contributions . Through some welldesigned experiments on data of different levels of difficulty it tries to obtain insights about the relationship between data difficulty , the number of training examples and the computational difficulty of finding a deep local optimum with good generalization . This intuition suggests that in some cases a neural network could be trained with fewer training examples at the expense of increased computational complexity in the optimization process of finding a deep local optimum , where a simple SGD or Adam optimization might not be enough . 1.1 RELATED WORK . Local minima . Recent studies Draxler et al . ( 2018 ) ; Garipov et al . ( 2018 ) have shown that the local minima of some convolutional neural networks are equivalent in the sense that they have the same loss ( energy ) value and a path can be found between the local minima along which the energy stays the same . For this reason , we will focus our attention to fully connected neural networks and find examples where the local minima have different loss values . Moreover , Soudry & Carmon ( 2016 ) proves that all differentiable local minima are global minima for the one hidden layer NNs with piecewise linear activation and square loss . However , nothing is proved for non-differentiable local minima . Network pruning . There has been quite a lot of work recently about neural network pruning , either for the purpose of improving speed and reducing complexity or giving insights about explaining the essential capability of the pruning technique . Han et al . ( 2015b ) and Han et al . ( 2015a ) propose the ” Deep Compression ” , a three-stage technique , which significantly reduces the storage requirement for training deep neural networks without affecting their accuracy . Liu et al . ( 2018 ) shows that for structured pruning methods , directly training the small target sub-network or pruned model with random initialization can achieve a comparable or even better performance than retraining using the remaining parameters after pruning . They also obtain similar results towards to a unstructured pruning method Han et al . ( 2015b ) after fine-tuning the pruned sub-network on small-scale datasets . Frankle & Carbin ( 2019 ) introduces the Lottery Tickets Hypothesis which claims that a random-initialized dense neural network contains a sub-network that can be trained in isolation with the corresponding original initialized parameters to obtain the same test accuracy of the original network after training for the same number of iterations . 2 AN EMPIRICAL STUDY OF THE NEURAL NETWORK LOCAL MINIMA . To study the local minima of neural networks , we will look at an extreme case , the XOR problem . The k-dimensional XOR is a binary classification problem that can be formulated as y ( x ) = I ( k∏ i=1 xi > 0 ) , ∀x ∈ Rp Observe that in this formulation the XOR data is p dimensional but the degree of interaction is k-dimensional , with k ≤ p. We call this data the k-D XOR in p dimensions . In this paper we will work with k ∈ { 3 , 4 , 5 } , as k = 2 is very simple . We assume that x ∈ Rp is sampled uniformly from [ −1 , 1 ] p. The XOR problem an example of data that can only be modeled by using higher order feature interactions , and for which lower order marginal models have no discrimination power . This makes it very difficult to detect what features are relevant for predicting the response y . The neural networks ( NN ) that we will study are two layer neural networks with ReLU activation for the hidden layer . These NNs can model the XOR data in case p = k = { 3 , 4 , 5 } very well given sufficiently many hidden nodes . 2.1 THE ENERGY LANDSCAPE OF THE NN ON THE XOR DATA . To see the difference between the values of the various local minima , we ran the NN starting with 100 random initializations and plotted in Figures 1 and 2 the values of the local minima sorted in increasing order for k = 4 and k = 5 . We repeated this process 10 times and also plotted the average of the sorted local minima . From Figures 1 and 2 we observe that the local minima are not equivalent , with the lowest being much smaller than the highest . We also see that for small p , the 500 hidden nodes NN has the lowest loss values , but for p = 100 the 20 hidden nodes NN obtains the lowest loss . The loss values are directly related to the training AUC ( area under the ROC curve ) . In Figures 1 and 2 , bottom , are plotted the mean train and test AUC of the models corresponding to the sorted local minima . Observe that for p = 100 the NN can not find the global optimum and the test AUCs are all close to 0.5 . These experiments indicate that when the number of irrelevant features is small , it is quite easy to find a deep local optimum with good generalization , so the NN is easily trainable . As the number of irrelevant features increases , it becomes harder and harder to find such a deep optimum , and when there are too many irrelevant features it is close to impossible to find a deep optimum with good generalization . 2.2 THE TRAINABILITY OF NEURAL NETWORKS . To see the change from an easily trainable NN ( such as shown in Figures 1 and 2 , left , to a poorly trained NN such as in Figures 1 and 2 , right , for each dimension p we train a NN with 500 hidden nodes with 10 random initializations and keep the solution with smallest loss . Then we compute the test AUC of the obtained NN . For each p we repeat this process 10 times and display in Figure 3 the average test AUC vs p. We observe that the test AUC quickly drops from close to 1 to 0.5 . The number of variables p where the test AUC gets below a threshold ( e.g . 0.8 ) depends on the number n of training examples . This drop does resemble a phase transition , from an “ easy to train ” regime where the local minima are easy to find , to the “ hard to train ” regime . Our conclusions from this empirical study are the following : • If the training data is difficult ( such as the XOR data ) , not all local minima are equivalent , since in Figures 1 and 2 there was a large difference between the test AUC of the best local minimum and the worst one . • For a fixed training size n , the number of shallow local minima quickly blows up as the number of irrelevant variables increases and finding the deep local minima becomes extremely hard . • If the number of irrelevant variables is not too large ( e.g . the middle plot from Figure 2 ) , an NN with a sufficiently large number of hidden nodes will find a deep optimum more often than one with a small number of hidden nodes . These conclusions form the basis for the proposed connection pruning methodology presented in the next section .

Summary: This paper presents a few sets of experiments related to deep local optima. The first set of experiments are to study the local minima of two layer neural networks with ReLU activation for the hidden layer and specialized for XOR problem. The paper claims that it is quite easy to find a deep local minimum with good generalization when the number of irrelevant features is small, and it becomes harder to find a deep minimum with good generation as the number of irrelevant features increases. It also claims there is a large difference between the test AUC of the best local minimum and the worse one if the training data is difficult.