Revealing the Structure of Deep Neural Networks via Convex Duality

1 INTRODUCTION . Deep neural networks ( DNNs ) have become extremely popular due to their success in machine learning applications . Even though DNNs are highly over-parameterized and non-convex , simple first-order algorithms , e.g. , Stochastic Gradient Descent ( SGD ) , can be used to successfully train them . Moreover , recent work has shown that highly over-parameterized networks trained with SGD obtain simple solutions that generalize well ( Savarese et al. , 2019 ; Parhi & Nowak , 2019 ; Ergen & Pilanci , 2020a ; b ) , where two-layer ReLU networks with the minimum Euclidean norm solution and zero training error are proven to fit a linear spline model in 1D regression . Therefore , regularizing the solution towards smaller norm weights might be the key to understand the generalization properties of DNNs . However , analyzing DNNs is still theoretically elusive even in the absence of nonlinear activations . Therefore , we study norm regularized DNNs and develop a framework based on convex duality such that a set of optimal solutions to the training problem can be analytically characterized . Deep linear networks have been the subject of extensive theoretical analysis due to their tractability . A line of research ( Saxe et al. , 2013 ; Arora et al. , 2018a ; Laurent & Brecht , 2018 ; Du & Hu , 2019 ; Shamir , 2018 ) focused on GD training dynamics , however , they lack the analysis of generalization properties of deep networks . Another line of research ( Gunasekar et al. , 2017 ; Arora et al. , 2019 ; Bhojanapalli et al. , 2016 ) studied the generalization properties via matrix factorization and showed that linear networks trained with GD converge to minimum nuclear norm solutions . Later on , Arora et al . ( 2018b ) ; Du et al . ( 2018 ) showed that gradient flow enforces the layer weights to align . Ji & Telgarsky ( 2019 ) further proved that each layer weight matrix is asymptotically rank-one . These results provide insights to characterize the structure of the optimal layer weights , however , they require multiple strong assumptions , e.g. , linearly separable training data and strictly decreasing loss function , which makes the results impractical . Furthermore , Zhang et al . ( 2019 ) provided some characterizations for nonstandard networks , which are valid for hinge loss and specific regularizations where the data matrix is included . Unlike these studies , we introduce a complete characterization for the regularized deep network training problem without requiring such assumptions . Our contributions : 1 ) We introduce a convex analytic framework that characterizes a set of optimal solutions to regularized training problems as the extreme points of a convex set , which is valid for vector outputs and popular loss functions including squared , cross entropy and hinge loss1 ; 2 ) For deep linear networks with K outputs , we prove that each optimal layer weight matrix aligns 1Extensions to other loss functions , e.g. , cross entropy and hinge loss , are presented Appendix A.1 with the previous layers and becomes rank-K via convex duality ; 3 ) For deep ReLU networks , we obtain the same weight alignment result for whitened or rank-one data matrices . As a corollary , we achieve closed-form solutions for the optimal hidden layer weights when data is whitened or rank-one ( see Theorem 4.1 and 4.3 ) . As another corollary , we prove that the optimal networks are linear spline interpolators for one-dimensional , i.e. , rank-one , data which generalizes the two-layer results for one-dimensional data in Savarese et al . ( 2019 ) ; Parhi & Nowak ( 2019 ) ; Ergen & Pilanci ( 2020a ; b ) to arbitrary depth . We note that the analysis of ReLU networks for the one dimensional data considered in these works is non-trivial , which is a special case of our rank-one/whitened data assumption . Notation : We denote matrices/vectors as uppercase/lowercase bold letters . We use 0k ( or 1k ) and Ik to denote a vector of zeros ( or ones ) and the identity matrix of size k , respectively . We denote the set of integers from 1 to n as [ n ] . To denote Frobenius , operator , and nuclear norms , we use ‖ · ‖F , ‖ · ‖2 , and ‖ · ‖∗ , respectively . Furthermore , σmax ( · ) and σmin ( · ) represent the maximum and minimum singular values , respectively and B2 is defined as B2 : = { u ∈ Rd | ‖u‖2 ≤ 1 } . 1.1 OVERVIEW OF OUR RESULTS . We consider an L-layer network with layer weights Wl ∈ Rml−1×ml , ∀l ∈ [ L ] , where m0 = d and mL = 1 , respectively . Then , given a data matrix X ∈ Rn×d , the output is fθ , L ( X ) = AL−1wL , Al = g ( Al−1Wl ) ∀l ∈ [ L − 1 ] , where A0 = X and g ( · ) is the activation function . Given a label vector y ∈ Rn , training problem can be formulated as follows min { θl } Ll=1 L ( fθ , L ( X ) , y ) + βR ( θ ) , ( 1 ) where L ( · , · ) is an arbitrary loss function , R ( θ ) is regularization for the layer weights , β > 0 is a regularization parameter , θl = { Wl , ml } , and θ : = { θl } Ll=1 . In the paper , for the sake of presentation simplicity , we illustrate the conventional training setup with squared loss and ` 2-norm regularization , i.e. , L ( fθ , L ( X ) , y ) = ‖fθ , L ( X ) − y‖22 and R ( θ ) = ∑L l=1 ‖Wl‖2F . However , our analysis is valid for arbitrary loss functions and different regularization terms as proven in Appendix . Thus , we consider the following optimization problem P ∗ = min { θl } Ll=1 L ( fθ , L ( X ) , y ) + β L∑ l=1 ‖Wl‖2F . ( 2 ) Next , we show that the minimum ` 22 norm is equivalent to minimum ` 1 norm after a rescaling . Lemma 1.1 . The following problems are equivalent : min { θl } Ll=1 L ( fθ , L ( X ) , y ) + β L∑ l=1 ‖Wl‖2F = min { θl } Ll=1 , t L ( fθ , L ( X ) , y ) + 2β‖wL‖1 + β ( L− 2 ) t2 s.t . wL−1 , j ∈ B2 , ‖Wl‖F ≤ t , ∀l ∈ [ L− 2 ] , where wL−1 , j denotes the jth column of WL−1 . Using Lemma 1.12 , we first take the dual with respect to the output layer weights wL and then change the order of min-max to achieve the following dual deep network training problem , which provides a lower bound 3 P ∗ ≥D∗ = min t max λ min wL−1 , j∈B2 , ∀j ‖Wl‖F≤t , ∀l∈ [ L−2 ] −L∗ ( λ ) + β ( L− 2 ) t2 s.t . ‖ATL−1λ‖∞ ≤ 2β . 2The proof is presented in Appendix A.3 . 3For the definitions and details see Appendix A.1 . To the best of our knowledge , the above dual deep network characterization is novel . Using this result , we first characterize a set of weights that minimize the objective via the optimality conditions and active constraints in the dual objective . We then prove the optimality of these weights by proving strong duality , i.e. , P ∗ = D∗ , for deep networks . We then show that , for deep linear networks with K outputs , optimal weight matrices are rank-K and align with the previous layers . More importantly , the same analysis and conclusions also apply to deep ReLU networks with K outputs when the input is whitened and/or rank-one . To the best of our knowledge , this is the first work providing a complete characterization for deep ReLU networks via convex duality . Based on this analysis , we even obtain closed-form solutions for the optimal layer weights . As a corollary , we show that deep ReLU networks fit a linear spline interpolation when the input is a one-dimensional dataset . We also provide an experiment in Figure 1 to verify this claim . We emphasize that this result was previously known only for two-layer networks ( Savarese et al. , 2019 ; Parhi & Nowak , 2019 ; Ergen & Pilanci , 2020a ; b ) and here we extend it to arbitrary depth L ( see Table 1 for details ) . 2 WARMUP : TWO-LAYER LINEAR NETWORKS . To illustrate an application of the convex dual D∗ , we consider the simple case of two-layer linear networks with the output fθ,2 ( X ) = XW1w2 and define the parameter space as θ ∈ Θ = { ( W1 , w2 , m ) |W1 ∈ Rd×m , w2 ∈ Rm , m ∈ Z+ } . Motivated by recent results ( Neyshabur et al. , 2014 ; Chizat & Bach , 2018 ; Savarese et al. , 2019 ; Parhi & Nowak , 2019 ; Ergen & Pilanci , 2020a ; b ) , we first focus on a minimum norm4 variant of equation 1 when L ( fθ , L ( X ) , y ) = ‖fθ , L ( X ) − y‖22 and then extend it to equation 1 . The minimum norm primal training problem can be written as min θ∈Θ ‖W1‖2F + ‖w2‖22 s.t . fθ,2 ( X ) = y . ( 3 ) Using Lemma A.15 , we equivalently have P ∗ = min θ∈Θ ‖w2‖1 s.t . fθ,2 ( X ) = y , w1 , j ∈ B2 , ∀j , ( 4 ) which has the following dual form . Theorem 2.1 . The dual of the problem in equation 4 is given by P ∗ ≥ D∗ = max λ∈Rn λTy s.t . max w1∈B2 ∣∣λTXw1∣∣ ≤ 1 . ( 5 ) For finite width networks , there exists a finitem such that strong duality holds , i.e. , P ∗ = D∗ , and an optimal W1 for equation 4 satisfies ‖ ( XW∗1 ) Tλ∗‖∞ = 1 , where λ∗ is the dual optimal parameter . Using Theorem 2.1 , we now characterize the optimal neurons as the extreme points of a convex set . Corollary 2.1 . Theorem 2.1 implies that the optimal neurons are extreme points which solve the following problem arg maxw1∈B2 |λ ∗TXw1 | . Definition 1 . We call the maximizers of the constraint in Corollary 2.1 extreme points . From Theorem 2.1 , we have the following dual problem max λ λTy s.t . max w1∈B2 |λTXw1| ≤ 1 . ( 6 ) Let X = UxΣxVTx be the singular value decomposition ( SVD ) of X 6 . If we assume that there exists w∗ such that Xw∗ = y due to Proposition 2.1 , then equation 6 is equivalent to max λ̃ λ̃TΣxw̃ ∗ s.t . ‖ΣTx λ̃‖2 ≤ 1 , ( 7 ) where λ̃ = UTxλ and w̃ ∗ = VTxw ∗ . Notice that in equation 7 , we use an alternative formulation for the constraint , i.e. , ‖XTλ‖2 ≤ 1 instead of |λTXw1| ≤ 1 , ∀w1 ∈ B2 since the extreme point is achieved when w1 = XTλ/‖XTλ‖2 . Given rank ( X ) = r ≤ min { n , d } , we have λ̃TΣxw̃ ∗ = λ̃TΣx [ Ir 0r×d−r 0d−r×r 0d−r×d−r ] w̃∗︸ ︷︷ ︸ w∗r ≤ ‖ΣTx λ̃‖2‖w̃∗r‖2 ≤ ‖w̃∗r‖2 , ( 8 ) 4This corresponds to weak regularization , i.e. , β → 0 in equation 1 ( see e.g . Wei et al . ( 2018 ) . ) . 5All the equivalence lemmas and proofs are presented in Appendix A.3 . 6In this paper , we use full SVD unless otherwise stated . which shows that the maximum objective value is achieved when ΣTx λ̃ = c1w̃ ∗ r . Thus , we have w∗1 = VxΣ T x λ̃ ‖VxΣTx λ̃‖2 = Vxw̃ ∗ r ‖w̃∗r‖2 = PXT ( w∗ ) ‖PXT ( w∗ ) ‖2 , where PXT ( · ) projects its input onto the range of XT . In the following results , we show that one can consider a planted model without loss of generality and prove strong duality for equation 4 . Proposition 2.1 . [ Du & Hu ( 2019 ) ] Given w∗ = arg minw ‖Xw − y‖2 , we have arg min W1 , w2 ‖XW1w2 −Xw∗‖22 = arg min W1 , w2 ‖XW1w2 − y‖22 . Theorem 2.2 . Let { X , y } be feasible for equation 4 , then strong duality holds for finite width networks .

Summary: The authors consider training neural networks with a variety of losses and regularization (such as weight decay).  The authors introduce a novel convex-dual formulation which allows them to characterize optimal solutions as being extreme points of particular convex sets.    For multi-layer linear networks, the authors prove that the optimal weight matrices have rank equal to the number of outputs of the network, and whose singular vectors align with those of neighboring layers.  For ReLU nets in one-dimension, the authors prove that optimal solutions act as linear spline interpolators (the kinks between linear pieces occur at data points), and the authors prove closed form expressions for optimal weights at intermediate layers when input data is whitened.