Principled Deep Neural Network Training through Linear Programming

Deep Learning has received significant attention due to its impressive performance in many state-of-the-art learning tasks . Unfortunately , while very powerful , Deep Learning is not well understood theoretically and in particular only recently results for the complexity of training deep neural networks have been obtained . In this work we show that large classes of deep neural networks with various architectures ( e.g. , DNNs , CNNs , Binary Neural Networks , and ResNets ) , activation functions ( e.g. , ReLUs and leaky ReLUs ) , and loss functions ( e.g. , Hinge loss , Euclidean loss , etc ) can be trained to near optimality with desired target accuracy using linear programming in time that is exponential in the input data and parameter space dimension and polynomial in the size of the data set ; improvements of the dependence in the input dimension are known to be unlikely assuming P , NP , and improving the dependence on the parameter space dimension remains open . In particular , we obtain polynomial time algorithms for training for a given fixed network architecture . Our work applies more broadly to empirical risk minimization problems which allows us to generalize various previous results and obtain new complexity results for previously unstudied architectures in the proper learning setting . 1 Introduction . Deep Learning is a powerful tool for modeling complex learning tasks . Its versatility allows for nuanced architectures that capture various setups of interest and has demonstrated a nearly unrivaled performance on state-of-the-art learning tasks across many domains . At the same time , the fundamental behavior of Deep Learning methods is not well understood . One particular aspect that recently gained significant interest is the computational complexity of training such networks . The basic training problem is usually formulated as an empirical risk minimization problem ( ERM ) that can be phrased as min φ∈Φ 1 D D∑ i=1 ` ( f ( x̂i , φ ) , ŷi ) , ( 1 ) where ` is some loss function , ( x̂i , ŷi ) D i=1 is an i.i.d . sample from some data distribution D , and f is a neural network architecture parameterized by φ ∈ Φ with Φ being the parameter space of the considered architecture ( e.g. , network weights ) . The empirical risk minimization problem is solved in lieu of the general risk minimization problem ( GRM ) minφ∈Φ E ( x , y ) ∈D [ ` ( f ( x , φ ) , y ) ] which is usually impossible to solve due to the inaccessibility of D. Several works have studied the training problem for specific architectures , both in the proper and improper learning setup . In the former , the resulting “ predictor ” obtained from ( 1 ) is always of the form f ( · , φ̂ ) for some φ̂ ∈ Φ , whereas in the latter , the predictor is allowed to be outside the class of functions { f ( · , φ ) : φ ∈ Φ } as long as it satisfies certain approximation guarantee to the solution of ( 1 ) 1 . In both cases , all results basically establish trainability in time that is exponential in the network parameters but polynomial in the amount of data . In this work we complement and significantly extend previous work by providing a principled method to convert the empirical risk minimization problem in ( 1 ) associated with the learning problem for various architectures into a linear programming problem ( LP ) in the proper learning setting . The obtained linear programming formulations are of size roughly exponential in the input dimension and in the parameter space dimension and linear in the size of the data-set . This result provides new bounds on the computational complexity of the training problem . For an overview on Complexity Theory we refer the reader to Arora and Barak ( 2009 ) . Related Work Our work is most closely related to Goel et al . ( 2017 ) , Zhang et al . ( 2016 ) , and Arora et al . ( 2018 ) . In Zhang et al . ( 2016 ) the authors show that ` 1-regularized networks can be learned improperly in polynomial time in the size of the data ( with a possibly exponential architecture dependent constant ) for networks with ReLU-like activations ( but not actual ReLUs ) and an arbitrary number of layers k. These results were then generalized in Goel et al . ( 2017 ) to actual ReLU activations . In both cases the improper learning setup is considered , i.e. , the learned predictor is not a neural network itself and the learning problem is solved approximately for a given target accuracy . In contrast to these works , Arora et al . ( 2018 ) considered proper and exact learning however only for k = 2 ( i.e. , one hidden layer ) . In relation to these works , we consider the proper learning setup for an arbitrary number of layers k and a wide range of activations , loss functions , and architectures . As previous works , except for Arora et al . ( 2018 ) , we consider the approximate learning setup as we are solving the empirical risk minimization problem and we also establish generalization of our so-trained models . Our approach makes use of Bienstock and Muñoz ( 2018 ) that allows for reformulating non-convex optimization problems with small treewidth and discrete as well as continuous decision variables as an approximate linear programming formulations . To the best of our knowledge , there are no previous papers that propose LP-based approaches for training neural networks . There are , however , proposed uses of Mixed-Integer and Linear Programming technology in other aspects of Deep Learning . Some examples of this include feature visualization ( Fischetti and Jo , 2018 ) , generating adversarial examples ( Cheng et al. , 2017 ; Anonymous , 2019a ; Fischetti and Jo , 2018 ) , counting linear regions of a Deep Neural Network ( Serra et al. , 2017 ) , performing inference ( Amos et al. , 2017 ) and providing strong convex relaxations for trained neural networks ( Anderson et al. , 2018 ) . Contribution We first establish a general framework that allows us to reformulate ( regularized ) ERM problems arising in Deep Learning ( among others ! ) into approximate linear programs with explicit bounds on their complexity . The resulting methodology allows for providing complexity upper bounds for specific setups simply by plugging-in complexity measures for the constituting elements such as layer architecture , activation functions , and loss functions . In particular our approach overcomes limitations of previous approaches in terms of handling the accuracy of approximations of non-linearities used in the approximation functions to achieve the overall target accuracy . Principled Training through LPs . If > 0 is arbitrary , then for any sample size D there exists a dataindependent linear program , i.e. , the LP can be written down before seeing the data , with the following properties : 1In the case of Neural Networks , for example , improper learning could lead to a predictor that does not correspond to a Neural Network , but that might behave closely to one . Solving the ERM problem to -optimality . The linear program describes a polyhedron P such that for every realized data set ( X̂ , Ŷ ) = ( x̂i , ŷi ) D i=1 there is a face FX̂ , Ŷ ⊆ P such that optimizing certain linear function over FX̂ , Ŷ solves ( 1 ) to -optimality returning a feasible parametrization φ̃ ∈ Φ which is part of our hypothesis class ( i.e. , we consider proper learning ) . The face FX̂ , Ŷ ⊆ P is simply obtained from P by fixing certain variables of the linear program using the values of the actual sample ; equivalently , by Farkas ’ lemma , this can be achieved by modifying the objective function to ensure optimization over the face belonging to the data . As such , the linear program has a build-once-solve-many feature . We will also show that a possible data-dependent LP formulation is meaningless ( see Appendix B ) . Size of the linear program . The size , measured as bit complexity , of the linear program is roughly O ( ( 2L/ ) N+n+m D ) where L is a constant depending on ` , f , and Φ that we will introduce later , n , m are the dimensions of the data points , i.e. , x̂i ∈ Rn and ŷi ∈ Rm for all i ∈ [ D ] , and N is the dimension of the parameter space Φ . The overall learning algorithm is obtained then by formulating and solving the linear program , e.g. , with the ellipsoid method whose running time is polynomial in the size of the input Grötschel et al . ( 2012 ) . Even sharper size bounds can be obtained for specific architectures assuming network structure ( see Appendix F ) and our approach immediately extends to regularized ERMs ( see Appendix E ) . It is important to mention that the constant L measures a certain Lipschitzness of the ERM training problem . While not exactly requiring Lipschitz continuity in the same way , Lipschitz constants have been used before for measuring complexity in the improper learning framework ( see Goel et al . ( 2017 ) ) and more recently have been shown to be linked to generalization in Gouk et al . ( 2018 ) . Generalization . Additionally , we establish that the solutions obtained for the ERM problem via our linear programming approach generalize , utilizing techniques from stochastic optimization . We also show that using our approach one can obtain a significant improvement on the results of Goel et al . ( 2017 ) when approximating the general risk minimization problem . Due to space limitations , however , we relegate this discussion to Appendix I . Throughout this work we assume both data and parameters to be well-scaled , which is a common assumption and mainly serves to simplify the representation of our results ; the main assumption is the reasonable boundedness , which can be assumed without significant loss of generality as actual computations assume boundedness in any case ( see also Liao et al . ( 2018 ) for arguments advocating the use of normalized coefficients in neural networks ) . More specifically , we assume Φ ⊆ [ −1 , 1 ] N as well as ( x , y ) ∼ D satisfies ( x , y ) ∈ [ −1 , 1 ] n × [ −1 , 1 ] m. We point out three important features of our results . First , we provide a solution method that has provable optimality guarantees for the ERM problem , ensures generalization , and linear dependency on the data ( in terms of the complexity of the LP ) without assuming convexity of the optimization problem . To the best of our knowledge , the only result presenting optimality guarantees in a proper learning , non-convex setting is that of Arora et al . ( 2018 ) . Second , the linear program that we construct for a given sample size D is data-independent in the sense that it can be written down before seeing the actual data realization and as such it encodes reasonable approximations of all possible data sets that can be given as an input to the ERM problem . This in particular shows that our linear programs are not simply discretizing space : if one considers a discretization of data contained in [ −1 , 1 ] n × [ −1 , 1 ] m , the total number of possible data sets of size D is exponential in D , which makes the linear dependence on D of the size of our LPs a remarkable feature . Finally , our approach can be directly extended to handle commonly used regularizers as we show in Appendix E ; for ease of presentation though we omit regularizers throughout our main discussions . Complexity results for various network architectures . We apply our methodology to various well-known neural network architectures and either generalize previous results or provide completely new results . We provide an overview of our results in Table 1 , where k is the number of layers , w is width of the network , n/m are the input/output dimensions and N is the total number of parameters . We use G to denote the directed graph defining the neural network and ∆ the maximum vertex in-degree in G. In all results the node computations are linear with bias term and normalized coefficients , and activation functions with Lipschitz constant at most 1 and with 0 as a fixed point ; these include ReLU , Leaky ReLU , eLU , Tanh , among others . We would like to point out that certain improvements in the results in Table 1 can be obtained by further specifying if the ERM problem corresponds to regression or classification . For example , the choice of loss functions and the nature of the output data y ( discrete or continuous ) typically rely on this . We can exploit such features in the construction of the LPs ( see the proof of Theorem 3.1 ) and provide a sharper bound on the LP size . Nonetheless , these improvements are not especially significant and in the interest of clarity and brevity we prefer to provide a unified discussion on ERM . Missing proofs have been relegated to the appendix due to space limitations . The complexity of the training problem for the Fully Connected DNN case is arguably the most studied and , to the best of our knowledge , all training algorithms with approximation or optimality guarantees have a polynomial dependency on D only after fixing the architecture ( depth , width , input dimension , etc. ) . In our setting , once the architecture is fixed , we obtain a polynomial dependence in both D and 1/ . Moreover , our results show that in the bounded case one can obtain a training algorithm with polynomial dependence on D across all architectures , assuming very little on the specific details of the network ( loss , activation , etc ) . This answers an open question left by Arora et al . ( 2018 ) regarding the possibility of a training algorithm with polynomial dependence on D. In addition , we show that a uniform LP can be obtained without compromising that dependence on D. The reader might wonder if the exponential dependence on the other parameters of our LP sizes can be improved , namely the input dimension n + m and the parameter space dimension N ( we are ignoring for the moment the exponent involving the depth k , as it will be typically dominated by N ) . The dependence on the input dimension is unlikely to be improved due to the NP-hardness result in Blum and Rivest ( 1992 ) , whereas obtaining a polynomial dependence on the parameter space dimension remains open ( see Arora et al . ( 2018 ) ) . A recent paper Anonymous ( 2019b ) provides an NP-hard DNN training problem that becomes polynomially solvable when the input dimension is fixed . However , this result considers a fixed architecture , thus the parameter space dimension is a constant and the running time is measured with respect to D .

Summary: This work reformulates the neural network training as an LP with size that is exponential in the size of the architecture and data dimension, and polynomial in the size of the data set. They further analyze generalization properties. It extends previous works on 1-hidden-layer neural-nets (say, In Arora et al. (2018)).