Parallel Deep Neural Networks Have Zero Duality Gap

1 INTRODUCTION . Deep neural networks demonstrate outstanding representation and generalization abilities in popular learning problems ranging from computer vision , natural language processing to recommendation system . Although the training problem of deep neural networks is a highly non-convex optimization problem , simple algorithms , such as stochastic gradient descent , can find a solution with good generalization properties . The non-convex and non-linear nature of neural networks render the theoretical understanding of neural networks extremely challenging . The Lagrangian dual problem ( Boyd et al. , 2004 ) plays an important role in the theory of convex and non-convex optimization . For convex optimization problems , the convex duality is an important tool to determine its optimal value and to characterize the optimal solutions . Even for a non-convex primal problem , the dual problem is a convex optimization problem the can be solved efficiently . As a result of weak duality , the optimal value of the dual problem serves as a non-trivial lower bound for the optimal primal objective value . Although the duality gap is non-zero for non-convex problems , the dual problem provides a convex relaxation of the non-convex primal problem . For example , the semi-definite programming relaxation of the two-way partitioning problem can be derived from its dual problem Boyd et al . ( 2004 ) . The convex duality also has important applications in machine learning . In ( Paternain et al. , 2019 ) , the design problem of an all-encompassing reward can be formulated as a constrained reinforcement learning problem , which is shown to have zero duality . This property gives a theoretical convergence guarantee of the primal-dual algorithm for solving this problem . Meanwhile , the minimax generative adversarial net ( GAN ) training problem can be tackled using duality ( Farnia & Tse , 2018 ) . In lines of recent works , the convex duality can also be applied for analyzing the optimal layer weights of two-layer neural networks with linear or ReLU activations ( Pilanci & Ergen , 2020 ; Ergen & Pilanci , 2020a ) . Based on the convex duality framework , the training problem of two-layer neural networks with ReLU activation can be represented in terms of a single convex program in ( Pilanci & Ergen , 2020 ) . Such convex optimization formulations are extended to two-layer and three-layer convolutional neural network training problems in ( Ergen & Pilanci , 2020c ) . Strong duality also holds for deep linear neural networks with scalar output ( Ergen & Pilanci , 2020b ) . The convex optimization formulation essentially gives a detailed characterization of the global optimum of the training problem . This enables us to examine in numerical experiments whether popular optimizers for neural networks , such as gradient descent or stochastic gradient descent , converge to the global optimum of the training loss . Admittedly , the zero duality gap is hard to achieve for deep neural networks , especially for those with vector outputs . This imposes more difficulty to train deep neural networks . Fortunately , neural networks with parallel structures ( also known as multi-branch architecture ) appear to be easier to train . Practically , the usage of parallel neural networks dates back to AlexNet ( Krizhevsky et al. , 2012 ) . Modern neural network architecture including Inception ( Szegedy et al. , 2017 ) , Xception ( Chollet , 2017 ) and SqueezeNet ( Iandola et al. , 2016 ) utilize the parallel structure . As the “ parallel ” version of ResNet ( He et al. , 2016a ; b ) , ResNeXt ( Xie et al. , 2017 ) and Wide ResNet ( Zagoruyko & Komodakis , 2016 ) exhibit improved performance on many applications . Recently , it was shown that neural networks with the parallel architecture have smaller duality gaps ( Zhang et al. , 2019 ) compared to standard neural networks . On the other hand , it is known that overparameterized parallel neural networks have benign training landscapes ( Haeffele & Vidal , 2017 ) . For training ` 2 loss with deep linear networks using Schatten norm regularization , Zhang et al . ( 2019 ) show that there is no duality gap . From another perspective , the standard two-layer network is equivalent to the parallel two-layer network . This may also explain why there is no duality gap for two-layer neural networks . 1.1 CONTRIBUTIONS . Following the convex duality framework introduced in ( Ergen & Pilanci , 2020b ; a ) , which showed the duality gap is zero for two-layer networks , we go beyond two-layer and study the convex duality for vector-output deep neural networks with linear activation and ReLU activation . Surprisingly , we prove that three-layer networks may have duality gaps depending on their architecture , unlike two-layer neural networks which always have zero duality gap . We summarize our contributions as follows . • For training standard vector-output deep linear networks using ` 2 regularization , we precisely calculate the optimal value of the primal and dual problems and show that the duality gap is non-zero , i.e. , Lagrangian relaxation is inexact . We also demonstrate that the ` 2- regularization on the parameter explicitly forces a tendency toward a low-rank solution , which is boosted with the depth . However , we show that the optimal solution is available in closed-form . • For parallel deep linear networks , with certain convex regularization , we show that the duality gap is zero , i.e , Lagrangian relaxation is exact . • For training vector-output three-layer ReLU networks with standard architecture using ` 2 regularization , even when the data is whitened , we show that the duality gap is non-zero . The gap can be closed by replacing the neural network to one with parallel architecture . • For parallel deep ReLU networks of arbitrary depth , with certain convex regularization , we prove strong duality , i.e. , the duality gap is zero . Remarkably , this guarantees that there is a convex program equivalent to the original deep ReLU neural network problem . We summarize the duality gaps for parallel/standard neural network in Table 1 . 1.2 NOTATIONS . We use bold capital letters to represent matrices and bold lowercase letters to represent vectors . Denote [ n ] = { 1 , . . . , n } . For a matrix Wl ∈ Rm1×m2 , for i ∈ [ m1 ] and j ∈ [ m2 ] , we denote wcoll , i as its i-th column and wrowl , j as its j-th row . Throughout the paper , X ∈ RN×d is the data matrix consisting of d dimensional N samples and Y ∈ RN×K is the label matrix for a regression/classification task with K outputs . 1.3 MOTIVATIONS AND BACKGROUND . Recently a series of papers ( Pilanci & Ergen , 2020 ; Ergen & Pilanci , 2020b ; a ) studied two-layer neural networks via convex duality and proved that strong duality holds for these architectures . Particularly , these prior works consider the following weight decay regularized training framework for classification/regression tasks . Given a data matrix X ∈ RN×d consisting of d dimensional N samples and the corresponding label matrix y ∈ RN , the weight-decay regularized training problem for a scalar-output neural network with m hidden neurons can be written as follows P : = min W1 , w2 1 2 ‖φ ( XW1 ) w2 − y‖22 + β 2 ( ‖W1‖2F + ‖w2‖22 ) , ( 1 ) where W1 ∈ Rd×m and w2 ∈ Rm are the layer weights , β > 0 is a regularization parameter , and φ is the activation function , which can be linear φ ( z ) = z or ReLU φ ( z ) = max { z , 0 } . Then , one can take the dual of ( 1 ) with respect to W1 and w2 obtain the following dual optimization problem D : = max λ −1 2 ‖λ− y‖22 + 1 2 ‖y‖22 , s.t . max w1 : ‖w1‖2≤1 |λTφ ( Xw1 ) | ≤ β . ( 2 ) We first note that since the training problem ( 1 ) is non-convex , strong duality may not hold , i.e. , P ≥ D. Surprisingly , as shown in Pilanci & Ergen ( 2020 ) ; Ergen & Pilanci ( 2020b ; a ) , strong duality in fact holds , i.e. , P = D , for two-layer networks and therefore one can derive exact convex representations for the non-convex training problem in ( 1 ) . However , extensions of this approach to deeper and state-of-the-art architectures are not available in the literature . Based on this observation , the central question we address in this paper is : Does strong duality hold for deep neural networks ? Depending on the answer to the question above , an immediate next questions we address is Can we characterize the duality gap ( P-D ) ? Is there an architecture for which strong duality holds independent of the depth ? Consequently , throughout the paper , we provide a full characterization of convex duality for deeper neural networks . Then , based on this characterization , we propose a modified architecture for which strong duality holds regardless of depth . 1.4 CONVEX DUALITY FOR TWO-LAYER NEURAL NETWORKS . We briefly review the convex duality for two-layer neural networks introduced in ( Ergen & Pilanci , 2020b ; a ) . Consider the following weight-decay regularized training problem for a vector-output neural network architecture with m hidden neurons min W1 , W2 1 2 ‖φ ( XW1 ) W2 −Y‖2F + β 2 ( ‖W1‖2F + ‖W2‖2F ) , ( 3 ) where W1 ∈ Rd×m and W2 ∈ Rm×K are the variables , and β > 0 is a regularization parameter . Here φ is the activation function , which can be linear φ ( z ) = z or ReLU φ ( z ) = max { z , 0 } . As long as the network is sufficiently overparameterized , there exists a feasible solution for such that φ ( XW1 ) W2 = Y . Then , a minimal norm variant1 of the training problem in ( 3 ) is given by min W1 , W2 1 2 ( ‖W1‖2F + ‖W2‖2F ) s.t . φ ( XW1 ) W2 = Y . ( 4 ) As shown in ( Pilanci & Ergen , 2020 ) , after a suitable rescaling , this problem can be reformulated as min W1 , W2 m∑ j=1 ‖wrow2 , j ‖2 s.t . φ ( XW1 ) W2 = Y , ‖wcol1 , j‖2 ≤ 1 , j ∈ [ m ] . ( 5 ) where [ m ] = { 1 , . . . , m } . Here wrow2 , j represents the j-th row of W2 and wcol1 , j denotes the j-th column of W1 . The rescaling does not change the solution to ( 4 ) . By taking the dual with respect to W1 and W2 , the dual problem of ( 5 ) with respect to variables is a convex optimization problem given by max Λ tr ( ΛTY ) , s.t . max u : ‖u‖2≤1 ‖ΛTφ ( Xu ) ‖2 ≤ 1 , ( 6 ) where Λ ∈ RN×K is the dual variable . Provided that m ≥ m∗ , where m∗ ≤ N + 1 , the strong duality holds , i.e. , the optimal value of the primal problem ( 5 ) equals to the optimal value of the dual problem ( 6 ) . 1.5 ORGANIZATION . This paper is organized as follows . In Section 2 , we review standard neural networks and introduce parallel architectures . For deep linear networks , we derive primal and dual problems for both standard and parallel architectures and provide calculations of optimal values of these problems in Section 3 . We derive primal and dual problems for three-layer ReLU networks with standard architecture and precisely calculate the optimal values for whitened data in Section 4 . We also show that deep ReLU networks with parallel structures have no duality gap .

Summary: The authors theoretically analyzed the dual of minimum norm optimization of over-parameterized deep neural networks. Specifically, they focused on deep neural networks with linear and ReLU activation functions, as well as parallel architectures where the output of each branch is a scalar. First, the authors considered the linear NNs and showed that the duality gap of general deep NN is non-zero, but for parallel NN with enough number of parallel branches, that would be zero. Next, they considered neural networks with ReLU activation functions and derived the duality gap for standard 3 layers neural networks. However, for an arbitrary deep parallel NN with ReLU activation, the duality gap would be zero if the L-th power of Frobenius norms of the weights is used as the regularizer.