Vector-output ReLU Neural Network Problems are Copositive Programs: Convex Analysis of Two Layer Networks and Polynomial-time Algorithms

1 INTRODUCTION . In this paper , we analyze vector-output two-layer ReLU neural networks from an optimization perspective . These networks , while simple , are the building blocks of deep networks which have been found to perform tremendously well for a variety of tasks . We find that vector-output networks regularized with standard weight-decay have a convex semi-infinite strong dual–a convex program with infinitely many constraints . However , this strong dual has a finite parameterization , though expressing this parameterization is non-trivial . In particular , we find that expressing a vector-output neural network as a convex program requires taking the convex hull of completely positive matrices . Thus , we find an intimate , novel connection between neural network training and copositive programs , i.e . programs over the set of completely positive matrices ( Anjos & Lasserre , 2011 ) . We describe algorithms which can be used to find the global minimum of the neural network training problem in polynomial time for data matrices of fixed rank , which holds for convolutional architectures . We also demonstrate under certain conditions that we can provably find the optimal solution to the neural network training problem using soft-thresholded Singular Value Decomposition ( SVD ) . In the general case , we introduce a relaxation to parameterize the neural network training problem , which in practice we find to be tight in many circumstances . 1.1 RELATED WORK . Our analysis focuses on the optima of finite-width neural networks . This approach contrasts with certain approaches which have attempted to analyze infinite-width neural networks , such as the Neural Tangent Kernel ( Jacot et al. , 2018 ) . Despite advancements in this direction , infinite-width neural networks do not exactly correspond to their finite-width counterparts , and thus this method of analysis is insufficient for fully explaining their success ( Arora et al. , 2019 ) . Other works may attempt to optimize neural networks with assumptions on the data distribution . Of particular interest is ( Ge et al. , 2018 ) , which demonstrates that a polynomial number of samples generated from a planted neural network model is sufficient for extracting its parameters using tensor methods , assuming the inputs are drawn from a symmetric distribution . If the input distribution to a simple convolutional neural network with one filter is Gaussian , it has also been shown that gradient descent can find the global optimum in polynomial time ( Brutzkus & Globerson , 2017 ) . In contrast to these works , we seek to find general principles for learning two-layer ReLU networks , regardless of the data distribution and without planted model assumptions . Another line of work aims to understand the success of neural networks via implicit regularization , which analyzes how models trained with Stochastic Gradient Descent ( SGD ) find solutions which generalize well , even without explicit control of the optimization objective ( Gunasekar et al. , 2017 ; Neyshabur et al. , 2014 ) . In contrast , we consider the setting of explicit regularization , which is often used in practice in the form of weight-decay , which regularizes the sum of squared norms of the network weights with a single regularization parameter β , which can be critical for neural network performance ( Golatkar et al. , 2019 ) . Our approach of analyzing finite-width neural networks with a fixed training dataset has been explored for networks with a scalar output ( Pilanci & Ergen , 2020 ; Ergen & Pilanci , 2020a ; d ) . In fact , our work here can be considered a generalization of these results . We consider a ReLU-activation two-layer network f : Rd → Rc with m neurons : f ( x ) = m∑ j=1 ( x > uj ) +v > j ( 1 ) where the function ( · ) + = max ( 0 , · ) denotes the ReLU activation , { uj ∈ Rd } mj=1 are the first-layer weights of the network , and { vj ∈ Rc } mj=1 are the second-layer weights . In the scalar-output case , the weights vj are scalars , i.e . c = 1 . Pilanci & Ergen ( 2020 ) find that the neural network training problem in this setting corresponds to a finite-dimensional convex program . However , the setting of scalar-output networks is limited . In particular , this setting can not account for tasks such as multi-class classification or multi-dimensional regression , which are some of the most common uses of neural networks . In contrast , the vector-output setting is quite general , and even greedily training and stacking such shallow vector-output networks can match or even exceed the performance of deeper networks on large datasets for classification tasks ( Belilovsky et al. , 2019 ) . We find that this important task of extending the scalar case to the vector-output case is an exceedingly non-trivial task , which generates novel insights . Thus , generalizing the results of Pilanci & Ergen ( 2020 ) is an important task for a more complete knowledge of the behavior of neural networks in practice . Certain works have also considered technical problems which arise in our analysis , though in application they are entirely different . Among these is analysis into cone-constrained PCA , as explored by Deshpande et al . ( 2014 ) and Asteris et al . ( 2014 ) . They consider the following optimization problem max u u > Ru s.t Xu ≥ 0 ; ‖u‖2 = 1 ( 2 ) This problem is in general considered NP-hard . Asteris et al . ( 2014 ) provide an exponential algorithm which runs in O ( nd ) time to find the exact solution to ( 2 ) , where X ∈ Rn×d and R ∈ Sd is a symmetric matrix . We leverage this result to show that the optimal value of the vector-output neural network training problem can be found in the worst case in exponential time with respect to r : = rank ( X ) , while in the case of a fixed-rank data matrix our algorithm is polynomial-time . In particular , convolutional networks with fixed filter sizes ( e.g. , 3 × 3 ×m convolutional kernels ) correspond to the fixed-rank data case ( e.g. , r = 9 ) . In search of a polynomial-time approximation to ( 2 ) , Deshpande et al . ( 2014 ) evaluate a relaxation of the above problem , given as max U 〈R , U〉 s.t XUX > ≥ 0 ; tr ( U ) = 1 ; U 0 ( 3 ) While the relaxation not tight in all cases , the authors find that in practice it works quite well for approximating the solution to the original optimization problem . This relaxation , in particular , corresponds to what we call a copositive relaxation , because it consists of a relaxation of the set CPCA = { uu > : ‖u‖2 = 1 , Xu ≥ 0 } . When X = I and the norm constraint is removed , CPCA is the set of completely positive matrices ( Dür , 2010 ) . Optimizing over the set of completely positive matrices is NP-hard , as is optimizing over its convex hull : C : = conv { uu > : Xu ≥ 0 } Thus , optimizing over C is a convex optimization problem which is nevertheless NP-hard . Various relaxations to C have been proposed , such as the copositive relaxation used by ( Deshpande et al. , 2014 ) above : C̃ : = { U : U 0 ; XUX > ≥ 0 } In fact , this relaxation is tight , given that u ∈ Rd and d ≤ 4 ( Burer , 2015 ; Kogan & Berman , 1993 ) . However , C ⊂ C̃ for d ≥ 5 , so the copositive relaxation provides a lower bound in the general case . These theoretical results prove insightful for understanding the neural network training objective . 1.2 CONTRIBUTIONS . • We find the semi-infinite convex strong dual for the vector-output two-layer ReLU neural network training problem , and prove that it has a finite-dimensional exact convex optimization representation . • We establish a new connection between vector-output neural networks , copositive programs and cone-constrained PCA problems , yielding new insights into the nature of vector-output neural network training , which extend upon the results of the scalar-output case . • We provide methods that globally solve the vector-output neural network training problem in polynomial time for data matrices of a fixed rank , but for the full-rank case , the complexity is necessarily exponential in d assuming P 6= NP . • We provide conditions on the training data and labels with which we can find a closedform expression for the optimal weights of a vector-output ReLU neural network using soft-thresholded SVD . • We propose a copositive relaxation to establish a heuristic for solving the neural network training problem . This copositive relaxation is often tight in practice . 2 PRELIMINARIES . In this work , we consider fitting labels Y ∈ Rn×c from inputs X ∈ Rn×d with a two layer neural network with ReLU activation and m neurons in the hidden layer . This network is trained with weight decay regularization on all of its weights , with associated parameter β > 0 . For some general loss function ` ( f ( X ) , Y ) , this gives us the non-convex primal optimization problem p∗ = min uj∈Rd vj∈Rc 1 2 ` ( f ( X ) , Y ) + β 2 m∑ j=1 ( ‖uj‖22 + ‖vj‖22 ) ( 4 ) In the simplest case , with a fully-connected network trained with squared loss1 , this becomes : p∗ = min uj∈Rd vj∈Rc 1 2 ‖ m∑ j=1 ( Xuj ) +v > j − Y ‖2F + β 2 m∑ j=1 ( ‖uj‖22 + ‖vj‖22 ) ( 5 ) 1Appendix A.6 contains extensions to general convex loss functions . However , alternative models can be considered . In particular , for example , Ergen & Pilanci ( 2020d ) consider two-layer CNNs with global average pooling , for which we can define the patch matrices { Xk } Kk=1 , which define the patches which individual convolutions operate upon . Then , the vectoroutput neural network training problem with global average pooling becomes p∗conv = min uj∈Rd vj∈Rc 1 2 ‖ K∑ k=1 m∑ j=1 ( Xkuj ) +v > j − Y ‖2F + β 2 m∑ j=1 ( ‖uj‖22 + ‖vj‖22 ) ( 6 ) We will show that in this convolutional setting , because the rank of the set of patches M : = [ X1 , X2 , · · ·XK ] > can not exceed the filter size of the convolutions , there exists an algorithm which is polynomial in all problem dimensions to find the global optimum of this problem . We note that such matrices typically exhibit rapid singular value decay due to spatial correlations , which may also motivate replacing it with an approximation of much smaller rank . In the following section , we will demonstrate how the vector-output neural network problem has a convex semi-infinite strong dual . To understand how to parameterize this semi-infinite dual in a finite fashion , we must introduce the concept of hyper-plane arrangements . We consider the set of diagonal matrices D : = { diag ( 1Xu≥0 ) : ‖u‖2 ≤ 1 } This is a finite set of diagonal matrices , dependent on the data matrix X , which indicate the set of possible arrangement activation patterns for the ReLU non-linearity , where a value of 1 indicates that the neuron is active , while 0 indicates that the neuron is inactive . In particular , we can enumerate the set of sign patterns as D = { Di } Pi=1 , where P depends on X but is in general bounded by P ≤ 2r ( e ( n− 1 ) r ) r for r : = rank ( X ) ( Pilanci & Ergen , 2020 ; Stanley et al. , 2004 ) . Note that for a fixed rank r , such as in the convolutional case above , P is polynomial in n. Using these sign patterns , we can completely characterize the range space of the first layer after the ReLU : { ( Xu ) + : ‖u‖2 ≤ 1 } = { DiXu : ‖u‖2 ≤ 1 , ( 2Di − I ) Xu ≥ 0 , i ∈ [ P ] } We also introduce a class of data matrices X for which the analysis of scalar-output neural networks simplifies greatly , as shown in ( Ergen & Pilanci , 2020b ) . These matrices are called spike-free matrices . In particular , a matrix X is called spike-free if it holds that { ( Xu ) + : ‖u‖2 ≤ 1 } = { Xu : ‖u‖2 ≤ 1 } ∩ Rn+ ( 7 ) When X is spike-free , then , the set of sign patterns D reduces to a single sign pattern , D = { I } , because of the identity in ( 7 ) . The set of spike-free matrices includes ( but is not limited to ) diagonal matrices and whitened matrices for which n ≤ d , such as the output of Zero-phase Component Analysis ( ZCA ) whitening . The setting of whitening the data matrix has been shown to improve the performance of neural networks , even in deeper settings where the whitening transformation is applied to batches of data at each layer ( Huang et al. , 2018 ) . We will see that spike-free matrices provide polynomial-time algorithms for finding the global optimum of the neural network training problem in both n and d ( Ergen & Pilanci , 2020b ) , though the same does not hold for vector-output networks .

Summary: The paper proposes a convex formulation for shallow neural networks with one hidden layer and vectorial outputs. This is an extension on a line of previous works (Ergen & Pilanci, 2020a) and (Ergen & Pilanci, 2020b) where similar results have been established for the case of scalar outputs. A Frank-Wolfe algorithm for finding the global optimum of the resulting convex program is proposed and evaluated on smaller datasets.