Vector-output ReLU Neural Network Problems are Copositive Programs: Convex Analysis of Two Layer Networks and Polynomial-time Algorithms

1 INTRODUCTION . In this paper , we analyze vector-output two-layer ReLU neural networks from an optimization perspective . These networks , while simple , are the building blocks of deep networks which have been found to perform tremendously well for a variety of tasks . We find that vector-output networks regularized with standard weight-decay have a convex semi-infinite strong dual–a convex program with infinitely many constraints . However , this strong dual has a finite parameterization , though expressing this parameterization is non-trivial . In particular , we find that expressing a vector-output neural network as a convex program requires taking the convex hull of completely positive matrices . Thus , we find an intimate , novel connection between neural network training and copositive programs , i.e . programs over the set of completely positive matrices ( Anjos & Lasserre , 2011 ) . We describe algorithms which can be used to find the global minimum of the neural network training problem in polynomial time for data matrices of fixed rank , which holds for convolutional architectures . We also demonstrate under certain conditions that we can provably find the optimal solution to the neural network training problem using soft-thresholded Singular Value Decomposition ( SVD ) . In the general case , we introduce a relaxation to parameterize the neural network training problem , which in practice we find to be tight in many circumstances . 1.1 RELATED WORK . Our analysis focuses on the optima of finite-width neural networks . This approach contrasts with certain approaches which have attempted to analyze infinite-width neural networks , such as the Neural Tangent Kernel ( Jacot et al. , 2018 ) . Despite advancements in this direction , infinite-width neural networks do not exactly correspond to their finite-width counterparts , and thus this method of analysis is insufficient for fully explaining their success ( Arora et al. , 2019 ) . Other works may attempt to optimize neural networks with assumptions on the data distribution . Of particular interest is ( Ge et al. , 2018 ) , which demonstrates that a polynomial number of samples generated from a planted neural network model is sufficient for extracting its parameters using tensor methods , assuming the inputs are drawn from a symmetric distribution . If the input distribution to a simple convolutional neural network with one filter is Gaussian , it has also been shown that gradient descent can find the global optimum in polynomial time ( Brutzkus & Globerson , 2017 ) . In contrast to these works , we seek to find general principles for learning two-layer ReLU networks , regardless of the data distribution and without planted model assumptions . Another line of work aims to understand the success of neural networks via implicit regularization , which analyzes how models trained with Stochastic Gradient Descent ( SGD ) find solutions which generalize well , even without explicit control of the optimization objective ( Gunasekar et al. , 2017 ; Neyshabur et al. , 2014 ) . In contrast , we consider the setting of explicit regularization , which is often used in practice in the form of weight-decay , which regularizes the sum of squared norms of the network weights with a single regularization parameter β , which can be critical for neural network performance ( Golatkar et al. , 2019 ) . Our approach of analyzing finite-width neural networks with a fixed training dataset has been explored for networks with a scalar output ( Pilanci & Ergen , 2020 ; Ergen & Pilanci , 2020a ; d ) . In fact , our work here can be considered a generalization of these results . We consider a ReLU-activation two-layer network f : Rd → Rc with m neurons : f ( x ) = m∑ j=1 ( x > uj ) +v > j ( 1 ) where the function ( · ) + = max ( 0 , · ) denotes the ReLU activation , { uj ∈ Rd } mj=1 are the first-layer weights of the network , and { vj ∈ Rc } mj=1 are the second-layer weights . In the scalar-output case , the weights vj are scalars , i.e . c = 1 . Pilanci & Ergen ( 2020 ) find that the neural network training problem in this setting corresponds to a finite-dimensional convex program . However , the setting of scalar-output networks is limited . In particular , this setting can not account for tasks such as multi-class classification or multi-dimensional regression , which are some of the most common uses of neural networks . In contrast , the vector-output setting is quite general , and even greedily training and stacking such shallow vector-output networks can match or even exceed the performance of deeper networks on large datasets for classification tasks ( Belilovsky et al. , 2019 ) . We find that this important task of extending the scalar case to the vector-output case is an exceedingly non-trivial task , which generates novel insights . Thus , generalizing the results of Pilanci & Ergen ( 2020 ) is an important task for a more complete knowledge of the behavior of neural networks in practice . Certain works have also considered technical problems which arise in our analysis , though in application they are entirely different . Among these is analysis into cone-constrained PCA , as explored by Deshpande et al . ( 2014 ) and Asteris et al . ( 2014 ) . They consider the following optimization problem max u u > Ru s.t Xu ≥ 0 ; ‖u‖2 = 1 ( 2 ) This problem is in general considered NP-hard . Asteris et al . ( 2014 ) provide an exponential algorithm which runs in O ( nd ) time to find the exact solution to ( 2 ) , where X ∈ Rn×d and R ∈ Sd is a symmetric matrix . We leverage this result to show that the optimal value of the vector-output neural network training problem can be found in the worst case in exponential time with respect to r : = rank ( X ) , while in the case of a fixed-rank data matrix our algorithm is polynomial-time . In particular , convolutional networks with fixed filter sizes ( e.g. , 3 × 3 ×m convolutional kernels ) correspond to the fixed-rank data case ( e.g. , r = 9 ) . In search of a polynomial-time approximation to ( 2 ) , Deshpande et al . ( 2014 ) evaluate a relaxation of the above problem , given as max U 〈R , U〉 s.t XUX > ≥ 0 ; tr ( U ) = 1 ; U 0 ( 3 ) While the relaxation not tight in all cases , the authors find that in practice it works quite well for approximating the solution to the original optimization problem . This relaxation , in particular , corresponds to what we call a copositive relaxation , because it consists of a relaxation of the set CPCA = { uu > : ‖u‖2 = 1 , Xu ≥ 0 } . When X = I and the norm constraint is removed , CPCA is the set of completely positive matrices ( Dür , 2010 ) . Optimizing over the set of completely positive matrices is NP-hard , as is optimizing over its convex hull : C : = conv { uu > : Xu ≥ 0 } Thus , optimizing over C is a convex optimization problem which is nevertheless NP-hard . Various relaxations to C have been proposed , such as the copositive relaxation used by ( Deshpande et al. , 2014 ) above : C̃ : = { U : U 0 ; XUX > ≥ 0 } In fact , this relaxation is tight , given that u ∈ Rd and d ≤ 4 ( Burer , 2015 ; Kogan & Berman , 1993 ) . However , C ⊂ C̃ for d ≥ 5 , so the copositive relaxation provides a lower bound in the general case . These theoretical results prove insightful for understanding the neural network training objective . 1.2 CONTRIBUTIONS . • We find the semi-infinite convex strong dual for the vector-output two-layer ReLU neural network training problem , and prove that it has a finite-dimensional exact convex optimization representation . • We establish a new connection between vector-output neural networks , copositive programs and cone-constrained PCA problems , yielding new insights into the nature of vector-output neural network training , which extend upon the results of the scalar-output case . • We provide methods that globally solve the vector-output neural network training problem in polynomial time for data matrices of a fixed rank , but for the full-rank case , the complexity is necessarily exponential in d assuming P 6= NP . • We provide conditions on the training data and labels with which we can find a closedform expression for the optimal weights of a vector-output ReLU neural network using soft-thresholded SVD . • We propose a copositive relaxation to establish a heuristic for solving the neural network training problem . This copositive relaxation is often tight in practice . 2 PRELIMINARIES . In this work , we consider fitting labels Y ∈ Rn×c from inputs X ∈ Rn×d with a two layer neural network with ReLU activation and m neurons in the hidden layer . This network is trained with weight decay regularization on all of its weights , with associated parameter β > 0 . For some general loss function ` ( f ( X ) , Y ) , this gives us the non-convex primal optimization problem p∗ = min uj∈Rd vj∈Rc 1 2 ` ( f ( X ) , Y ) + β 2 m∑ j=1 ( ‖uj‖22 + ‖vj‖22 ) ( 4 ) In the simplest case , with a fully-connected network trained with squared loss1 , this becomes : p∗ = min uj∈Rd vj∈Rc 1 2 ‖ m∑ j=1 ( Xuj ) +v > j − Y ‖2F + β 2 m∑ j=1 ( ‖uj‖22 + ‖vj‖22 ) ( 5 ) 1Appendix A.6 contains extensions to general convex loss functions . However , alternative models can be considered . In particular , for example , Ergen & Pilanci ( 2020d ) consider two-layer CNNs with global average pooling , for which we can define the patch matrices { Xk } Kk=1 , which define the patches which individual convolutions operate upon . Then , the vectoroutput neural network training problem with global average pooling becomes p∗conv = min uj∈Rd vj∈Rc 1 2 ‖ K∑ k=1 m∑ j=1 ( Xkuj ) +v > j − Y ‖2F + β 2 m∑ j=1 ( ‖uj‖22 + ‖vj‖22 ) ( 6 ) We will show that in this convolutional setting , because the rank of the set of patches M : = [ X1 , X2 , · · ·XK ] > can not exceed the filter size of the convolutions , there exists an algorithm which is polynomial in all problem dimensions to find the global optimum of this problem . We note that such matrices typically exhibit rapid singular value decay due to spatial correlations , which may also motivate replacing it with an approximation of much smaller rank . In the following section , we will demonstrate how the vector-output neural network problem has a convex semi-infinite strong dual . To understand how to parameterize this semi-infinite dual in a finite fashion , we must introduce the concept of hyper-plane arrangements . We consider the set of diagonal matrices D : = { diag ( 1Xu≥0 ) : ‖u‖2 ≤ 1 } This is a finite set of diagonal matrices , dependent on the data matrix X , which indicate the set of possible arrangement activation patterns for the ReLU non-linearity , where a value of 1 indicates that the neuron is active , while 0 indicates that the neuron is inactive . In particular , we can enumerate the set of sign patterns as D = { Di } Pi=1 , where P depends on X but is in general bounded by P ≤ 2r ( e ( n− 1 ) r ) r for r : = rank ( X ) ( Pilanci & Ergen , 2020 ; Stanley et al. , 2004 ) . Note that for a fixed rank r , such as in the convolutional case above , P is polynomial in n. Using these sign patterns , we can completely characterize the range space of the first layer after the ReLU : { ( Xu ) + : ‖u‖2 ≤ 1 } = { DiXu : ‖u‖2 ≤ 1 , ( 2Di − I ) Xu ≥ 0 , i ∈ [ P ] } We also introduce a class of data matrices X for which the analysis of scalar-output neural networks simplifies greatly , as shown in ( Ergen & Pilanci , 2020b ) . These matrices are called spike-free matrices . In particular , a matrix X is called spike-free if it holds that { ( Xu ) + : ‖u‖2 ≤ 1 } = { Xu : ‖u‖2 ≤ 1 } ∩ Rn+ ( 7 ) When X is spike-free , then , the set of sign patterns D reduces to a single sign pattern , D = { I } , because of the identity in ( 7 ) . The set of spike-free matrices includes ( but is not limited to ) diagonal matrices and whitened matrices for which n ≤ d , such as the output of Zero-phase Component Analysis ( ZCA ) whitening . The setting of whitening the data matrix has been shown to improve the performance of neural networks , even in deeper settings where the whitening transformation is applied to batches of data at each layer ( Huang et al. , 2018 ) . We will see that spike-free matrices provide polynomial-time algorithms for finding the global optimum of the neural network training problem in both n and d ( Ergen & Pilanci , 2020b ) , though the same does not hold for vector-output networks .

Summary: The draft is a vector extension of [1] on studying how to approximately solve the global optima of a two-layered Relu network. The key of the analysis is to enumerate all possible sign patterns of the ReLU unit generating from specific data. Once we have the enumeration, we can also enumerate the linear area separated by ReLU, and the whole optimization problem will become a non-convex quadratic optimization problem. The non-convex quadratic can be approximately solved with its convex dual (or exactly under some conditions), or we can relax it to a copositive program (which might still be NP-hard to solve). With some assumption on the data, the sign pattern of the ReLU is a singleton, then we will have efficient algorithms to exactly recover the global optima of the two-layered network.