Implicit Convex Regularizers of CNN Architectures: Convex Optimization of Two- and Three-Layer Networks in Polynomial Time

1 INTRODUCTION . Convolutional Neural Networks ( CNNs ) have shown a remarkable success across various machine learning problems ( LeCun et al. , 2015 ) . However , our theoretical understanding of CNNs still remains restricted , where the main challenge arises from the highly non-convex and nonlinear structure of CNNs with nonlinear activations such as ReLU . Hence , we study the training problem for various CNN architectures with ReLU activations and introduce equivalent finite dimensional convex formulations that can be used to globally optimize these architectures . Our results characterize the role of network architecture in terms of equivalent convex regularizers . Remarkably , we prove that the proposed methods are polynomial time with respect to all problem parameters . Convex neural network training was previously considered in Bengio et al . ( 2006 ) ; Bach ( 2017 ) . However , these studies are restricted to two-layer fully connected networks with infinite width , thus , the optimization problem involves infinite dimensional variables . Moreover , it has been shown that even adding a single neuron to a neural network leads to a non-convex optimization problem which can not be solved efficiently ( Bach , 2017 ) . Another line of research in Parhi & Nowak ( 2019 ) ; Ergen & Pilanci ( 2019 ; 2020a ; b ; c ; d ) ; Pilanci & Ergen ( 2020 ) ; Savarese et al . ( 2019 ) ; Gunasekar et al . ( 2018 ) ; Maennel et al . ( 2018 ) ; Blanc et al . ( 2019 ) ; Zhang et al . ( 2016 ) focuses on the effect of implicit and explicit regularization in neural network training and aims to explain why the resulting network generalizes well . Among these studies , Parhi & Nowak ( 2019 ) ; Ergen & Pilanci ( 2020b ; c ; d ) ; Savarese et al . ( 2019 ) proved that the minimum ` 2 norm two-layer network that perfectly fits a one dimensional dataset outputs the linear spline interpolation . Moreover , Gunasekar et al . ( 2018 ) studied certain linear convolutional networks and revealed an implicit non-convex quasi-norm regularization . However , as the number of layers increases , the regularization approaches to ` 0 quasi-norm , which is not computationally tractable . Recently , Pilanci & Ergen ( 2020 ) showed that two-layer CNNs with linear activations can be equivalently optimized as nuclear and ` 1 norm regularized convex problems . Although all the norm characterizations provided by these studies are insightful for future research , existing results are quite restricted due to linear activations , simple settings or intractable problems . Shallow CNNs and their representational power : As opposed to their relatively simple and shallow architecture , CNNs with two/three layers are very powerful and efficient models . Belilovsky et al . ( 2019 ) show that greedy training of two/three layer CNNs can achieve comparable performance to deeper models , e.g. , VGG-11 ( Simonyan & Zisserman , 2014 ) . However , a full theoretical understanding and interpretable description of CNNs even with a single hidden layer is lacking in the literature . Our contributions : Our contributions can be summarized as follows : • We develop convex programs that are polynomial time with respect to all input parameters : the number of samples , data dimension , and the number of neurons to globally train CNNs . To the best of our knowledge , this is the first work characterizing polynomial time trainability of nonconvex CNN models . More importantly , we achieve this complexity with explicit and interpretable convex optimization problems . Consequently , training CNNs , especially in practice , can be further accelerated by leveraging extensive tools available from convex optimization theory . • Our work reveals a hidden regularization mechanism behind CNNs and characterizes how the architecture and pooling strategies , e.g. , max-pooling , average pooling , and flattening , dramatically alter the regularizer . As we show , ranging from ` 1 and ` 2 norm to nuclear norm ( see Table 1 for details ) , ReLU CNNs exhibit an extremely rich and elegant regularization structure which is implicitly enforced by architectural choices . In convex optimization and signal processing , ` 1 , ` 2 and nuclear norm regularizations are well studied , where these structures have been applied in compressed sensing , inverse problems , and matrix completion . Our results bring light to unexplored and promising connections of ReLU CNNs with these established disciplines . Notation and preliminaries : We denote matrices/vectors as uppercase/lowercase bold letters , for which a subscript indicates a certain element/column . We use Ik for the identity matrix of size k. We denote the set of integers from 1 to n as [ n ] . Moreover , ‖ · ‖F and ‖ · ‖∗ are Frobenius and nuclear norms and Bp : = { u ∈ Cd : ‖u‖p ≤ 1 } is the unit ` p ball . We also use 1 [ x ≥ 0 ] as an indicator . To keep the presentation simple , we will use a regression framework with scalar outputs and squared loss . However , we also note that all of our results can be extended to vector outputs and arbitrary convex regression and classification loss functions . We present these extensions in Appendix . In our regression framework , we denote the input data matrix and the corresponding label vector as X ∈ Rn×d and y ∈ Rn , respectively . Moreover , we represent the patch matrices , i.e. , subsets of columns , extracted from X as Xk ∈ Rn×h , k ∈ [ K ] , where h denotes the filter size . With this notation , { Xku } Kk=1 describes a convolution operation between the filter u ∈ Rh and the data matrix X . Throughout the paper , we will use the ReLU activation function defined as ( x ) + = max { 0 , x } . However , since CNN training problems with ReLUs are not convex in their conventional form , below we introduce an alternative formulation for this activation , which will be crucial for our derivations . Prior Work ( Pilanci & Ergen , 2020 ) : Recently , Pilanci & Ergen ( 2020 ) introduced an exact convex formulation for training two-layer fully connected ReLU networks in polynomial time for training data X ∈ Rn×d of constant rank , where the model is a standard two-layer scalar output network fθ ( X ) : = ∑m j=1 ( Xuj ) + αj . However , this model has three main limitations . First , as noted by the authors , even though the algorithm is polynomial time , i.e. , O ( nr ) , provided that r : = rank ( X ) , the complexity is exponential in r = d , i.e. , O ( nd ) , if X is full rank . Additionally , as a direct consequence of their model , the analysis is limited to fully connected architectures . Although they briefly analyzed some CNN architectures in Section 4 , as emphasized by the authors , these are either fully linear ( without ReLU ) or separable over the patch index k as fully connected models , which do not correspond to weight sharing in classical CNN architectures in practice . Finally , their analysis does not extend to three-layer architectures with two ReLU layers since the analysis of two ReLU layers is significantly more challenging . On the contrary , we prove that classical CNN architectures can be globally optimized by standard convex solvers in polynomial time independent of the rank 1The results on two-layer CNNs are presented in Appendix A.4 . 2This refers to an L-layer network with only one ReLU layer and circular convolutions . 1.1 HYPERPLANE ARRANGEMENTS . LetH be the set of all hyperplane arrangement patterns of X , defined as the following set H : = ⋃ { { sign ( Xw ) } : w ∈ Rd } , which has finitely many elements , i.e. , |H| ≤ NH < ∞ , NH ∈ N. We now define a collection of sets that correspond to positive signs for each element in H , by S : = { { ∪hi=1 { i } } : h ∈ H } . We first note that ReLU is an elementwise function that masks the negative entries of a vector or matrix . Hence , given a set S ∈ S , we define a diagonal mask matrix D ( S ) ∈ Rn×n defined as D ( S ) ii : = 1 [ i ∈ S ] . Then , we have an alternative representation for ReLU as ( Xw ) + = D ( S ) Xw given D ( S ) Xw ≥ 0 and ( In −D ( S ) ) Xw ≤ 0 . Note that these constraints can be compactly defined as ( 2D ( S ) − In ) Xw ≥ 0 . If we denote the cardinality of S as P , i.e. , the number of regions in a partition of Rd by hyperplanes passing through the origin and are perpendicular to the rows of the data matrix X with r : = rank ( X ) ≤ min ( n , d ) , then P can be upper-bounded as follow P ≤ 2 r−1∑ k=0 ( n− 1 k ) ≤ 2r ( e ( n− 1 ) r ) r ( Ojha , 2000 ; Stanley et al. , 2004 ; Winder , 1966 ; Cover , 1965 ) ( see Appendix A.2 for details ) . 1.2 CONVOLUTIONAL HYPERPLANE ARRANGEMENTS . We now define a notion of hyperplane arrangements for CNNs , where we introduce the patch matrices { Xk } Kk=1 instead of directly operating on X . We first construct a new data matrix as M = [ X1 ; X2 ; . . . XK ] ∈ RnK×h . We then define convolutional hyperplane arrangements as the hyperplane arrangements for M and denote the cardinality of this set as Pconv . Then , we have Pconv ≤ 2 rc−1∑ k=0 ( nK − 1 k ) ≤ 2rc ( e ( nK − 1 ) rc ) rc where rc : = rank ( M ) ≤ h and K = ⌊ d−h stride ⌋ + 1 . Note that when the filter size h is fixed , Pconv is polynomial in n and d. Similarly , we consider hyperplane arrangements for circular CNNs followed by a linear pooling layer , i.e. , XUw , where U ∈ Rd×d is a circulant matrix generated by the elements u ∈ Rh . Then , we define circular convolutional hyperplane arrangements and denote the cardinality of this set as Pcconv , which is exponential in the rank of the circular patch matrices , i.e. , rcc . Remark 1.1 . There exist P hyperplane arrangements of X where P is exponential in r. Thus , if X is full rank , r = d , then P can be exponentially large in the dimension d. As we will show , this makes the training problem for fully connected networks challenging . On the other hand , for CNNs , the number of relevant hyperplane arrangements Pconv is exponential in rc . If M is full rank , then rc = h d and accordingly Pconv P . This shows that the parameter sharing structure in CNNs enables a significant reduction in the number of possible hyperplane arrangements . Consequently , as shown in the sequel and Table 2 , our results imply that the complexity of training problem is significantly lower compared to fully connected networks .

Summary: The paper studies the non-convex optimization problem of training CNNs with ReLU activations under different choices for the CNN architecture, and shows how these can be framed as convex problems with a poly time complexity w.r.t. relevant variables. The derived convex problems provide valuable insights on how the CNN architecture induces different weight regularizers by giving them in explicit form -- these show a rich connection between the architecture and regularizer.