Complexity of Training ReLU Neural Networks

1 INTRODUCTION . Deep neural networks ( DNNs ) are functions computed on a graph parameterized by its edge weights . More formally , the graph corresponding to a DNN is defined by input and output dimensions w0 , wk+1 ∈ Z+ , number of hidden layers k ∈ Z+ , and by specifying a sequence of k natural numbers w1 , w2 , . . . , wk representing the number of nodes in each of the hidden k-layers . The function computed on the DNN graphs is : f : = aK+1 ◦ τ ◦ aK ◦ · · · ◦ a2 ◦ τ ◦ a1 , where ◦ is function composition , τ is a nonlinear function ( applied componentwise ) called as the activation function , and ai : Rwi−1 → Rwi are affine functions . Given input and corresponding output data , the problem of training a DNN can be thought of as determining edge weights of directed layered graph for which output of the neural network matches the output data as closely as possible . Formally , given a set of input and output data { ( xi , yi ) } Ni=1 where ( xi , yi ) ∈ Rw0 × Rwk+1 , and a loss function l : Rwk+1 × Rwk+1 → R+ ( for example , l can be a norm ) the task is to determine the weights that define the affine function ai ’ s such that N∑ i=1 l ( f ( xi ) , yi ) ( 1 ) is minimized . Some commonly studied activation functions are : threshold function , sigmoid function and ReLU function . ReLU is one of the important activation functions used widely in applications and literature . However , the problem of complexity of training multi-layer fully-connected ReLU neural network remained open . This is where we add our contributions . Before formally stating the results , we take a look at current state-of-the-art in the literature . Complexity of training DNNs with threshold activation function The threshold ( sign ) function is given by sgn ( x ) : = { 1 if x > 0 −1 if x < 0 . Neural network with threshold activation function is also called as binary neural network in modern machine learning literature . It was shown by Blum et al . Blum & Rivest ( 1988 ) that problem of training a simple two layer neural network with two nodes in the first layer and one node in the second layer while using threshold activation function at all the nodes is NP-complete . The problem turns out to be equivalent to separation by two hyperplanes which was shown to be NP-complete by Megiddo ( 1988 ) . There are other hardness results such as crypto hardness for intersection of k-hyperplanes which apply to binary neural networks Shalev-Shwartz & Ben-David ( 2014 ) ; Klivans & Sherstov ( 2009 ) . In Shalev-Shwartz & Ben-David ( 2014 ) it is shown that even the problem of training a binary neural network with 3 or more nodes in first hidden layer and 1 node in second hidden layer is NP-hard . DNNs with rectified linear unit ( ReLU ) activation function Theoretical worst case results presented above , along with limited empirical successes lead to DNN ’ s going out of favor by late 1990s . However , in recent time , DNNs became popular again by advent of first-order gradient based heuristics for training . This success started with the work of Hinton et al . ( 2006 ) which gave empirical evidence that if DNNs are initialized properly then we can find good solutions in reasonable runtime . This work was soon followed by series of early successes of deep learning in natural language processing Collobert & Weston ( 2008 ) , speech recognition Mohamed et al . ( 2012 ) and visual object classification Krizhevsky et al . ( 2012 ) . It was empirically shown by Zhang et al . ( 2016 ) that a sufficiently over-parameterized neural network can be trained to global optimality . These gradient-based heuristics are not useful for binary neural networks as there is no gradient information . Even networks with sigmoid activation function fell out of favor because gradient information is not valuable when input values are large . The popular neural network architecture uses ReLU activations on which gradient based heuristics are useful . Fomally , the ReLU function is given by : [ x ] + : = max ( x , 0 ) . Related literature As discussed before , most hardness results so far are for binary neural networks Blum & Rivest ( 1988 ) ; Klivans & Sherstov ( 2009 ) ; Shalev-Shwartz & Ben-David ( 2014 ) . There are also limited results for ReLU that we discuss next : Recently , Livni et al . ( 2014 ) examined ReLU activations from the point of view that , a shifted ReLU subtracted from another ReLU yields an approximation to threshold function , so such a class of ReLU network should be as hard as binary neural network . Similar results are shown by DasGupta et al . ( 1994 ) . In both these papers , in order to model “ a shifted ReLU subtracted from another ReLU ” , the Neural network studied is not a fully connected network . More specifically , in the underlying graph of such a neural network , each node in the second hidden layer is connected to exactly one distinct node in the first hidden layer , weight of the connecting edge is set to −1 with the addition of some positive bias term . Figure 1 shows the difference between ReLU network studied by Livni et al . ( 2014 ) ; DasGupta et al . ( 1994 ) and fully connected ReLU network . Clearly , the architecture described in Livni et al . ( 2014 ) ; DasGupta et al . ( 1994 ) artificially restrict the form of the affine functions in order to prove NP-hardness . In particular , it requires connecting hidden layer matrix to be a square diagonal matrix . Due to this restriction , it was unclear whether allowing full matrix to be non-zero would make problem easy ( more parameters hence higher power to neural network function ) or hard ( more parameters so more things to decide ) . Another interesting line of research in understanding the hardness of training ReLU neural networks assumes that data is coming from some distribution . More recent work in this direction include Shamir ( 2016 ) which shows a smooth family of functions for which gradient of squared error function is not informative while training neural network over Gaussian input distribution . Song et al . ( 2017 ) consider Statistical Query ( SQ ) framework ( which contains SGD algorithms ) and show that there exists a class of special functions generated by single hidden layer neural network for which learning will require exponential number of queries ( i.e . sample gradient evaluations ) for data coming from product measure of real valued log-concave distribution . These are interesting studies in their own right and generally consider hardness with respect to the algorithms that use stochastic gradient queries to the population objective functions . In comparison , we consider the framework of NP-hardness which takes into account complete class of polynomial time algorithms , generally assumes that data is given and looks at corresponding empirical objective . Recently , Arora et al . ( 2016 ) showed that a single hidden layer ReLU network can be trained in polynomial time when dimension of input , w0 , is constant . Based on the above discussion , we see that the complexity status of training the multi-layer fullyconnected ReLU neural network remains open . Given the importance of the ReLU NN , this is an important question . In this paper , we take the first steps in resolving this question . Main Contributions . • NP-hardness : We show that the training problem for a simple two hidden layer fully connected NN which has two nodes in the first layer , one node in the second layer and ReLU activation function at all nodes is NP-hard . Underlying graph of this network is exactly the same as that in Blum et al . Blum & Rivest ( 1988 ) but all activation functions are ReLU instead of threshold function . Techniques used in the proof are different from earlier work in literature because there is no combinatorial interpretation to ReLU as opposed to threshold function . • Polynomial-time solvable cases : We present two cases where the training problem with ReLU activation function can be solved in polynomial-time . The first case is when the dimension of the input is fixed . The second case is when we have a network where the number of nodes in the first layer is equal to the number of input data points ( highly overparameterized neural network ) . This second result leads to interesting open questions that we discuss later . 2 NOTATION AND DEFINITIONS . We use the following standard set notation [ n ] : = { 1 , . . . , n } . The letter d generally denotes the dimension of input data , the output data is 1 dimensional and N denotes the number of data-points . The main training problem of interest for the paper corresponds to a neural network with 3 nodes . The underlying graph is a layered directed graph with two layers . The first layer contains two nodes and the second layer contains one node . The network is fully connected feedforward network . One can write the function corresponding to this neural network as follows : F ( x ) = θ [ w0 + w1 [ a1 ( x ) ] + + w2 [ a2 ( x ) ] + ] + , ( 2 ) where ai : Rd → R for i ∈ { 1 , 2 } are real valued affine functions , and w0 , w1 , w2 ∈ R. The output of the two affine maps a1 , a2 are the inputs to the two ReLU nodes in first hidden layer of network . The weights { w0 , w1 , w2 } denote affine map for ReLU node in second layer . The coefficient θ ∈ R is the linear map of the output layer . Henceforth , we refer to the network defined in ( 2 ) as ( 2,1 ) - ReLU Neural Network ( NN ) . As its name suggests , it has 2 ReLU nodes in first layer and 1 ReLU node in second layer . Figure 2 shows ( 2 , 1 ) -ReLU NN . Note that w [ ax+ b ] + ≡ sgn ( w ) [ |w| ( ax+ b ) ] + = sgn ( w ) [ ãx+ b̃ ] , so without loss of generality we will assume w1 , w2 ∈ { −1 , 1 } . We will refer to ( k , j ) -ReLU NN as generalization of ( 2 , 1 ) -ReLU NN where there are k ReLU nodes in first layer and j ReLU nodes in second layer . Moreover , output of ( k , j ) -ReLU NN lies in Rj . If there is only one node in the second layer , we will often drop the ′1′ and refer it as a 2-ReLU NN or k-ReLU NN depending on whether there are 2 or k nodes in the first layer . Definition 2.1 ( Decision-version of training problem ) Given a set of training data ( xi , yi ) ∈ Rd × { 1 , 0 } for i ∈ S , does there exist edge weights so that the resulting function F satisfies F ( xi ) = yi for i ∈ S. The decision version of the training problem in Definition 2.1 is asking if it is possible to find edge weights to obtain zero loss function value in the expression ( 1 ) , assuming l is a norm i.e . l ( a , b ) = 0 iff a = b .

Summary: The main result of this work is to prove that a two-layer neural network with 2 hidden neurons in the first layer and 1 hidden neuron in the second layer is NP-hard to train when all activation functions are ReLU. Similar results with the hard-thresholding activation function were proved by Blum & Rivest in 1988. The main proof consists of a reduction from the 2-hyperplane separability problem which was known to be equivalent to NNs with hard-thresholding activation functions.