Parallel Deep Neural Networks Have Zero Duality Gap

1 INTRODUCTION . Deep neural networks demonstrate outstanding representation and generalization abilities in popular learning problems ranging from computer vision , natural language processing to recommendation system . Although the training problem of deep neural networks is a highly non-convex optimization problem , simple algorithms , such as stochastic gradient descent , can find a solution with good generalization properties . The non-convex and non-linear nature of neural networks render the theoretical understanding of neural networks extremely challenging . The Lagrangian dual problem ( Boyd et al. , 2004 ) plays an important role in the theory of convex and non-convex optimization . For convex optimization problems , the convex duality is an important tool to determine its optimal value and to characterize the optimal solutions . Even for a non-convex primal problem , the dual problem is a convex optimization problem the can be solved efficiently . As a result of weak duality , the optimal value of the dual problem serves as a non-trivial lower bound for the optimal primal objective value . Although the duality gap is non-zero for non-convex problems , the dual problem provides a convex relaxation of the non-convex primal problem . For example , the semi-definite programming relaxation of the two-way partitioning problem can be derived from its dual problem Boyd et al . ( 2004 ) . The convex duality also has important applications in machine learning . In ( Paternain et al. , 2019 ) , the design problem of an all-encompassing reward can be formulated as a constrained reinforcement learning problem , which is shown to have zero duality . This property gives a theoretical convergence guarantee of the primal-dual algorithm for solving this problem . Meanwhile , the minimax generative adversarial net ( GAN ) training problem can be tackled using duality ( Farnia & Tse , 2018 ) . In lines of recent works , the convex duality can also be applied for analyzing the optimal layer weights of two-layer neural networks with linear or ReLU activations ( Pilanci & Ergen , 2020 ; Ergen & Pilanci , 2020a ) . Based on the convex duality framework , the training problem of two-layer neural networks with ReLU activation can be represented in terms of a single convex program in ( Pilanci & Ergen , 2020 ) . Such convex optimization formulations are extended to two-layer and three-layer convolutional neural network training problems in ( Ergen & Pilanci , 2020c ) . Strong duality also holds for deep linear neural networks with scalar output ( Ergen & Pilanci , 2020b ) . The convex optimization formulation essentially gives a detailed characterization of the global optimum of the training problem . This enables us to examine in numerical experiments whether popular optimizers for neural networks , such as gradient descent or stochastic gradient descent , converge to the global optimum of the training loss . Admittedly , the zero duality gap is hard to achieve for deep neural networks , especially for those with vector outputs . This imposes more difficulty to train deep neural networks . Fortunately , neural networks with parallel structures ( also known as multi-branch architecture ) appear to be easier to train . Practically , the usage of parallel neural networks dates back to AlexNet ( Krizhevsky et al. , 2012 ) . Modern neural network architecture including Inception ( Szegedy et al. , 2017 ) , Xception ( Chollet , 2017 ) and SqueezeNet ( Iandola et al. , 2016 ) utilize the parallel structure . As the “ parallel ” version of ResNet ( He et al. , 2016a ; b ) , ResNeXt ( Xie et al. , 2017 ) and Wide ResNet ( Zagoruyko & Komodakis , 2016 ) exhibit improved performance on many applications . Recently , it was shown that neural networks with the parallel architecture have smaller duality gaps ( Zhang et al. , 2019 ) compared to standard neural networks . On the other hand , it is known that overparameterized parallel neural networks have benign training landscapes ( Haeffele & Vidal , 2017 ) . For training ` 2 loss with deep linear networks using Schatten norm regularization , Zhang et al . ( 2019 ) show that there is no duality gap . From another perspective , the standard two-layer network is equivalent to the parallel two-layer network . This may also explain why there is no duality gap for two-layer neural networks . 1.1 CONTRIBUTIONS . Following the convex duality framework introduced in ( Ergen & Pilanci , 2020b ; a ) , which showed the duality gap is zero for two-layer networks , we go beyond two-layer and study the convex duality for vector-output deep neural networks with linear activation and ReLU activation . Surprisingly , we prove that three-layer networks may have duality gaps depending on their architecture , unlike two-layer neural networks which always have zero duality gap . We summarize our contributions as follows . • For training standard vector-output deep linear networks using ` 2 regularization , we precisely calculate the optimal value of the primal and dual problems and show that the duality gap is non-zero , i.e. , Lagrangian relaxation is inexact . We also demonstrate that the ` 2- regularization on the parameter explicitly forces a tendency toward a low-rank solution , which is boosted with the depth . However , we show that the optimal solution is available in closed-form . • For parallel deep linear networks , with certain convex regularization , we show that the duality gap is zero , i.e , Lagrangian relaxation is exact . • For training vector-output three-layer ReLU networks with standard architecture using ` 2 regularization , even when the data is whitened , we show that the duality gap is non-zero . The gap can be closed by replacing the neural network to one with parallel architecture . • For parallel deep ReLU networks of arbitrary depth , with certain convex regularization , we prove strong duality , i.e. , the duality gap is zero . Remarkably , this guarantees that there is a convex program equivalent to the original deep ReLU neural network problem . We summarize the duality gaps for parallel/standard neural network in Table 1 . 1.2 NOTATIONS . We use bold capital letters to represent matrices and bold lowercase letters to represent vectors . Denote [ n ] = { 1 , . . . , n } . For a matrix Wl ∈ Rm1×m2 , for i ∈ [ m1 ] and j ∈ [ m2 ] , we denote wcoll , i as its i-th column and wrowl , j as its j-th row . Throughout the paper , X ∈ RN×d is the data matrix consisting of d dimensional N samples and Y ∈ RN×K is the label matrix for a regression/classification task with K outputs . 1.3 MOTIVATIONS AND BACKGROUND . Recently a series of papers ( Pilanci & Ergen , 2020 ; Ergen & Pilanci , 2020b ; a ) studied two-layer neural networks via convex duality and proved that strong duality holds for these architectures . Particularly , these prior works consider the following weight decay regularized training framework for classification/regression tasks . Given a data matrix X ∈ RN×d consisting of d dimensional N samples and the corresponding label matrix y ∈ RN , the weight-decay regularized training problem for a scalar-output neural network with m hidden neurons can be written as follows P : = min W1 , w2 1 2 ‖φ ( XW1 ) w2 − y‖22 + β 2 ( ‖W1‖2F + ‖w2‖22 ) , ( 1 ) where W1 ∈ Rd×m and w2 ∈ Rm are the layer weights , β > 0 is a regularization parameter , and φ is the activation function , which can be linear φ ( z ) = z or ReLU φ ( z ) = max { z , 0 } . Then , one can take the dual of ( 1 ) with respect to W1 and w2 obtain the following dual optimization problem D : = max λ −1 2 ‖λ− y‖22 + 1 2 ‖y‖22 , s.t . max w1 : ‖w1‖2≤1 |λTφ ( Xw1 ) | ≤ β . ( 2 ) We first note that since the training problem ( 1 ) is non-convex , strong duality may not hold , i.e. , P ≥ D. Surprisingly , as shown in Pilanci & Ergen ( 2020 ) ; Ergen & Pilanci ( 2020b ; a ) , strong duality in fact holds , i.e. , P = D , for two-layer networks and therefore one can derive exact convex representations for the non-convex training problem in ( 1 ) . However , extensions of this approach to deeper and state-of-the-art architectures are not available in the literature . Based on this observation , the central question we address in this paper is : Does strong duality hold for deep neural networks ? Depending on the answer to the question above , an immediate next questions we address is Can we characterize the duality gap ( P-D ) ? Is there an architecture for which strong duality holds independent of the depth ? Consequently , throughout the paper , we provide a full characterization of convex duality for deeper neural networks . Then , based on this characterization , we propose a modified architecture for which strong duality holds regardless of depth . 1.4 CONVEX DUALITY FOR TWO-LAYER NEURAL NETWORKS . We briefly review the convex duality for two-layer neural networks introduced in ( Ergen & Pilanci , 2020b ; a ) . Consider the following weight-decay regularized training problem for a vector-output neural network architecture with m hidden neurons min W1 , W2 1 2 ‖φ ( XW1 ) W2 −Y‖2F + β 2 ( ‖W1‖2F + ‖W2‖2F ) , ( 3 ) where W1 ∈ Rd×m and W2 ∈ Rm×K are the variables , and β > 0 is a regularization parameter . Here φ is the activation function , which can be linear φ ( z ) = z or ReLU φ ( z ) = max { z , 0 } . As long as the network is sufficiently overparameterized , there exists a feasible solution for such that φ ( XW1 ) W2 = Y . Then , a minimal norm variant1 of the training problem in ( 3 ) is given by min W1 , W2 1 2 ( ‖W1‖2F + ‖W2‖2F ) s.t . φ ( XW1 ) W2 = Y . ( 4 ) As shown in ( Pilanci & Ergen , 2020 ) , after a suitable rescaling , this problem can be reformulated as min W1 , W2 m∑ j=1 ‖wrow2 , j ‖2 s.t . φ ( XW1 ) W2 = Y , ‖wcol1 , j‖2 ≤ 1 , j ∈ [ m ] . ( 5 ) where [ m ] = { 1 , . . . , m } . Here wrow2 , j represents the j-th row of W2 and wcol1 , j denotes the j-th column of W1 . The rescaling does not change the solution to ( 4 ) . By taking the dual with respect to W1 and W2 , the dual problem of ( 5 ) with respect to variables is a convex optimization problem given by max Λ tr ( ΛTY ) , s.t . max u : ‖u‖2≤1 ‖ΛTφ ( Xu ) ‖2 ≤ 1 , ( 6 ) where Λ ∈ RN×K is the dual variable . Provided that m ≥ m∗ , where m∗ ≤ N + 1 , the strong duality holds , i.e. , the optimal value of the primal problem ( 5 ) equals to the optimal value of the dual problem ( 6 ) . 1.5 ORGANIZATION . This paper is organized as follows . In Section 2 , we review standard neural networks and introduce parallel architectures . For deep linear networks , we derive primal and dual problems for both standard and parallel architectures and provide calculations of optimal values of these problems in Section 3 . We derive primal and dual problems for three-layer ReLU networks with standard architecture and precisely calculate the optimal values for whitened data in Section 4 . We also show that deep ReLU networks with parallel structures have no duality gap .

Summary: Using duality theory, this paper studies the duality gap between prime and dual optimization problems for deep linear (or relu) networks that have sequential or parallel structure. The major claim is that, assuming the networks is sufficiently over-parameterized, linear networks of depths at least 3 have nonzero duality gap, while strong duality holds for deep parallel networks. Similar conclusions are given for relu networks.