Computational Separation Between Convolutional and Fully-Connected Networks

1 INTRODUCTION . Convolutional neural networks ( LeCun et al. , 1998 ; Krizhevsky et al. , 2012 ) achieve state-of-the-art performance on every possible task in computer vision . However , while the empirical success of convolutional networks is indisputable , the advantage of using them is not well understood from a theoretical perspective . Specifically , we consider the following fundamental question : Why do convolutional networks ( CNNs ) perform better than fully-connected networks ( FCNs ) ? Clearly , when considering expressive power , FCNs have a big advantage . Since convolution is a linear operation , any CNN can be expressed using a FCN , whereas FCNs can express a strictly larger family of functions . So , any advantage of CNNs due to expressivity can be leveraged by FCNs as well . Therefore , expressive power does not explain the superiority of CNNs over FCNs . There are several possible explanations to the superiority of CNNs over FCNs : parameter efficiency ( and hence lower sample complexity ) , weight sharing , and locality prior . The main result of this paper is arguing that locality is a key factor by proving a computational separation between CNNs and FCNs based on locality . But , before that , let ’ s discuss the other possible explanations . First , we observe that CNNs seem to be much more efficient in utilizing their parameters . A FCN needs to use a greater number of parameters compared to an equivalent CNN : each neuron of a CNN is limited to a small receptive field , and moreover , many of the parameters of the CNN are shared . From classical results in learning theory , using a large number of param- eters may result in inferior generalization . So , can the advantage of CNNs be explained simply by counting parameters ? To answer this question , we observe the performance of CNN and FCN based architecture of various widths and depths trained on the CIFAR-10 dataset . For each architecture , we observe the final test accuracy versus the number of trainable parameters . The results are shown in Figure 1 . As can be seen , CNNs have a clear advantage over FCNs , regardless of the number of parameters used . As is often observed , a large number of parameters does not hurt the performance of neural networks , and so parameter efficiency can not explain the advantage of CNNs . This is in line with various theoretical works on optimization of neural networks , which show that over-parameterization is beneficial for convergence of gradient-descent ( e.g. , Du et al . ( 2018 ) ; Soltanolkotabi et al . ( 2018 ) ; Li & Liang ( 2018 ) ) . The superiority of CNNs can be also attributed to the extensive weight sharing between the different convolutional filters . Indeed , it has been previously shown that weight sharing is important for the optimization of neural networks ( Shalev-Shwartz et al. , 2017b ) . Moreover , the translation-invariant nature of CNNs , which relies on weight sharing , is often observed to be beneficial in various signal processing tasks ( Kauderer-Abrams , 2017 ; Kayhan & Gemert , 2020 ) . So , how much does the weight sharing contribute to the superiority of CNNs over FCNs ? To understand the effect of weight sharing on the behavior of CNNs , it is useful to study locallyconnected network ( LCN ) architectures , which are similar to CNNs , but have no weight sharing between the kernels of the network . While CNNs are far more popular in practice ( also due to the fact that they are much more efficient in terms of model size ) , LCNs have also been used in different contexts ( e.g. , Bruna et al . ( 2013 ) ; Chen et al . ( 2015 ) ; Liu et al . ( 2020 ) ) . It has been recently observed that in some cases , the performance of LCNs is on par with CNNs ( Neyshabur , 2020 ) . So , even if weight sharing explains some of the advantage of CNNs , it clearly doesn ’ t tell the whole story . Finally , a key property of CNN architectures is their strong utilization of locality in the data . Each neuron in a CNN is limited to a local receptive field of the input , hence encoding a strong locality bias . In this work we demonstrate how CNNs can leverage the local structure of the input , giving them a clear advantage in terms of computational complexity . Our results hint that locality is the principal property that explains the advantage of using CNNs . Our main result is a computational separation result between CNNs and FCNs . To show this result , we introduce a family of functions that have a very strong local structure , which we call k-patterns . A k-pattern is a function that is determined by k consecutive bits of the input . We show that for inputs of n bits , when the target function is a ( log n ) -pattern , training a CNN of polynomial size with gradient-descent achieves small error in polynomial time . However , gradient-descent will fail to learn ( log n ) -patterns , when training a FCN of polynomial-size . 1.1 RELATED WORK . It has been empirically observed that CNN architectures perform much better than FCNs on computer vision tasks , such as digit recognition and image classification ( e.g. , Urban et al . ( 2017 ) ; Driss et al . ( 2017 ) ) . While some works have applied various techniques to improve the performance of FCNs ( Lin et al . ( 2015 ) ; Fernando et al . ( 2016 ) ; Neyshabur ( 2020 ) ) , there is still a gap between performance of CNNs and FCNs , where the former give very good performance “ out-of-the-box ” . The focus of this work is to understand , from a theoretical perspective , why CNNs give superior performance when trained on input with strong local structure . Various theoretical works show the advantage of architectures that leverage local and hierarchical structure . The work of Poggio et al . ( 2015 ) shows the advantage of using deep hierarchical models over wide and shallow functions . These results are extended in Poggio et al . ( 2017 ) , showing an exponential gap between deep and shallow networks , when approximating locally compositional functions . The works of Mossel ( 2016 ) ; Malach & Shalev-Shwartz ( 2018 ) study learnability of deep hierarchical models . The work of Cohen et al . ( 2017 ) analyzes the expressive efficiency of convolutional networks via hierarchical tensor decomposition . While all these works show that indeed CNNs powerful due to their hierarchical nature and the efficiency of utilizing local structure , they do not explain why these models are superior to fully-connected models . There are a few works that provide a theoretical analysis of CNN optimization . The works of Brutzkus & Globerson ( 2017 ) ; Du et al . ( 2018 ) show that gradient-descent can learn a shallow CNN with a single filter , under various distributional assumptions . The work of Zhang et al . ( 2017 ) shows learnability of a convex relaxation of convolutional networks . While these works focus on computational properties of learning CNNs , as we do in this work , they do not compare CNNs to FCNs , but focus only on the behavior of CNNs . The works of Cohen & Shashua ( 2016 ) ; Novak et al . ( 2018 ) study the implicit bias of simplified CNN models . However , these result are focused on generalization properties of CNNs , and not on computational efficiency of the optimization . 2 DEFINITIONS AND NOTATIONS . Let X = { ±1 } n be our instance space , and let Y = { ±1 } be the label space . Throughout the paper , we focus on learning a binary classification problem using the hinge-loss : ` ( ŷ , y ) = max { 1 yŷ , 0 } . Given some distribution D over X , some target function f : X ! Y and some hypothesis h : X ! Y , we define the loss of h with respect to f on the distribution D by : Lf , D ( h ) = E x⇠D [ ` ( h ( x ) , f ( x ) ) ] The goal of a supervised learning algorithm is , given access to examples sampled from D and labeled by f , to find a hypothesis h that minimizes Lf , D ( h ) . We focus on the gradient-descent ( GD ) algorithm : given some parametric hypothesis class H = { hw : w 2 Rq } , gradient-descent starts with some ( randomly initialized ) hypothesis hw ( 0 ) and , for some learning rate ⌘ > 0 , updates : w ( t ) = w ( t 1 ) ⌘rwLf , D ( hw ( t 1 ) ) We compare the behavior of gradient-descent , when learning two possible neural network architectures : a convolutional network ( CNN ) and a fully-connected network ( FCN ) . Definition 1 . A convolutional network hu , W , b is defined as follows : hu , W , b ( x ) = n kX j=1 D u ( j ) , ( Wxj ... j+k 1 + b ) E for activation function , with kernel W 2 Rq⇥k , bias b 2 Rq and readout layer u ( 1 ) , . . . , u ( n ) 2 Rq . Note that this is a standard depth-2 CNN with kernel k , stride 1 and q filters . Definition 2 . A fully-connected network hu , w , b is defined as follows : hu , w , b ( x ) = qX i=1 ui ⇣D w ( i ) , x E + bi ⌘ for activation function , first layer w ( 1 ) , . . . , w ( q ) 2 Rn , bias b 2 Rq and second layer u 2 Rq . We demonstrate the advantage of CNNs over FCNs by observing a problem that can be learned using CNNs , but is hard to learn using FCNs . We call this problem the k-pattern problem : Definition 3 . A function f : X ! Y is a k-pattern , if for some g : { ±1 } k ! Y and index j⇤ : f ( x ) = g ( xj⇤ ... j⇤+k 1 ) Namely , a k-pattern is a function that depends only on a small pattern of consecutive bits of the input . The k-pattern problem is the problem of learning k-patterns : for some k-pattern f and some distribution D over X , given access to D labeled by f , find a hypothesis h with Lf , D ( h ) ✏ . We note that a similar problem has been studied in Golovnev et al . ( 2017 ) , providing results on PAC learnability of a related target class . 3 CNNS EFFICIENTLY LEARN ( log n ) -PATTERNS The main result in this section shows that gradient-descent can learn k-patterns when training convolutional networks for poly ( 2k , n ) iterations , and when the network has poly ( 2k , n ) neurons : Theorem 4 . Assume we uniformly initialize W ( 0 ) ⇠ { ±1/k } q⇥k , bi = 1/k 1 and u ( 0 , j ) = 0 for every j . Assume the activation satisfies | | c , | 0 | 1 , for some constant c. Fix some > 0 , some k-pattern f and some distribution D over X . Then , if q > 2k+3 log ( 2k/ ) , with probability at least 1 over the initialization , when training a convolutional network hu , W , b using gradient descent with ⌘ = p np qT we have : 1 T TX t=1 Lf , D ( hu ( t ) , W ( t ) , b ) 2cn2k22k q + 2 ( 2kk ) 2 p qn + c 2 n 1.5p q T Before we prove the theorem , observe that the above immediately implies that when k = O ( log n ) , gradient-descent can efficiently learn to solve the k-pattern problem , when training a CNN : Corollary 5 . Let k = O ( log n ) . Then , running GD on a CNN with q = O ( ✏ 2n3 log2 n ) neurons for T = O ( ✏ 2n3 log n ) iterations , using a sample S ⇠ D of size O ( ✏ 2nkq log ( nkq/ ) ) , learns the k-pattern problem up to accuracy ✏ w.p . 1 . Proof . Sample S ⇠ D , and let bD be the uniform distribution over S. Then , from Theorem 4 and the choice of q and T there exists t 2 [ T ] with L f , bD ( hu ( t ) , W ( t ) , b ) ✏/2 , i.e . GD finds a hypothesis with train loss at most ✏/2 . Now , using the fact the VC dimension of depth-2 ReLU networks with W weights is O ( W logW ) ( see Bartlett et al . ( 2019 ) ) , we can bound the generalization gap by ✏/2 . To prove Theorem 4 , we show that , for a large enough CNN , the k-pattern problem becomes linearly separable , after applying the first layer of the randomly initialized CNN : Lemma 6 . Assume we uniformly initialize W ⇠ { ±1/k } q⇥k and bi = 1/k 1 . Fix some > 0 . Then if q > 2k+3 log ( 2k/ ) , w.p . 1 over the choice of W , for every k-pattern f there exist u⇤ ( 1 ) , . . . , u⇤ ( n k ) 2 Rq with u⇤ ( j⇤ ) 2 k+1 kp q and u⇤ ( j ) = 0 for j 6= j⇤ , s.t . hu⇤ , W , b = f ( x ) . Proof . Fix some z 2 { ±1 } k , then for every w ( i ) ⇠ { ±1/k } k , we have : P ⇥ sign ( w ( i ) ) = z ⇤ = 2 k. Denote by Jz ✓ [ q ] the subset of indexes satisfying signw ( i ) = z , for every i 2 Jz , and note that EW |Jz| = q2 k. From Chernoff bound : P ⇥ |Jz| q2 k /2 ⇤ e q2 k/8 2 k by choosing q > 2k+3 log ( 2k/ ) . So , using the union bound , w.p . at least 1 , for every z 2 { ±1 } k we have |Jz| q2 k 1 . By the choice of bi we have ( ⌦ w ( i ) , z ↵ + bi ) = ( 1/k ) 1 { signw ( i ) = z } . Now , fix some k-pattern f , where f ( x ) = g ( xj⇤ , ... , j⇤+k 1 ) . For every i 2 Jz we choose u ⇤ ( j⇤ ) i = k |Jz|g ( z ) and u ⇤ ( j ) = 0 for every j 6= j⇤ . Therefore , we get : hu⇤ , W , b ( x ) = n kX j=1 D u⇤ ( j ) , ( Wxj ... j+k 1 + b ) E = X z2 { ±1 } k i2Jz u⇤ ( j ⇤ ) i ⇣D w ( i ) , xj⇤ ... j⇤+k 1 E + bi ⌘ = X z2 { ±1 } k 1 { z = xj⇤ ... j⇤+k 1 } g ( z ) = g ( xj⇤ ... j⇤+k 1 ) = f ( x ) Note that by definition of u⇤ ( j ⇤ ) we have u⇤ ( j⇤ ) 2 = P z2 { ±1 } k P i2Jz k 2 |Jz|2 4 ( 2 k k ) 2 q . Comment 7 . Admittedly , the initialization assumed above is non-standard , but is favorable for the analysis . A similar result can be shown for more natural initialization ( e.g. , normal distribution ) , using known results from random features analysis ( for example , Bresler & Nagaraj ( 2020 ) ) . From Lemma 6 and known results on learning linear classifiers with gradient-descent , solving the k-pattern problem can be achieved by optimizing the second layer of a randomly initialized CNN . However , since in gradient-descent we optimize both layers of the network , we need a more refined analysis to show that full gradient-descent learns to solve the problem . We follow the scheme introduced in Daniely ( 2017 ) , adapting it our setting . We start by showing that the first layer of the network does not deviate from the initialization during the training : Lemma 8 . We have u ( T , j ) ⌘Tpq for all j 2 [ n k ] , and W ( 0 ) W ( T ) c⌘2T 2n p qk We can now bound the difference in the loss when the weights of the first layer change during the training process : Lemma 9 . For every u⇤ we have : Lf , D ( hu⇤ , W ( T ) , b ) Lf , D ( hu⇤ , W ( 0 ) , b ) c⌘2T 2nkpq n kX j=1 u⇤ ( j ) The proofs of Lemma 8 and Lemma 9 are shown in the appendix . Finally , we use the following result on the convergence of online gradient-descent to show that gradient-descent converges to a good solution . The proof of the Theorem is given in Shalev-Shwartz et al . ( 2011 ) , with an adaptation to a similar setting in Daniely & Malach ( 2020 ) . Theorem 10 . ( Online Gradient Descent ) Fix some ⌘ , and let f1 , . . . , fT be some sequence of convex functions . Fix some ✓1 , and update ✓t+1 = ✓t ⌘rft ( ✓t ) . Then for every ✓⇤ the following holds : 1 T TX t=1 ft ( ✓t ) 1 T TX t=1 ft ( ✓ ⇤ ) + 1 2⌘T k✓ ⇤ k 2 + k✓1k 1 T TX t=1 krft ( ✓t ) k+ ⌘ 1 T TX t=1 krft ( ✓t ) k 2 Proof of Theorem 4 . From Lemma 6 , with probability at least 1 over the initialization , there exist u⇤ ( 1 ) , . . . , u⇤ ( n k ) 2 Rq with u⇤ ( 1 ) 2 k+1 kp q and u⇤ ( j ) = 0 for j > 1 such that hu⇤ , W ( 0 ) , b ( x ) = f ( x ) , and so Lf , D ( hu⇤ , W ( 0 ) , b ) = 0 . Using Theorem 10 , since Lf , D ( hu , W , b ) is convex with respect to u , we have : 1 T TX t=1 Lf , D ( hu ( t ) , W ( t ) , b ) 1 T TX t=1 Lf , D ( hu⇤ , W ( t ) , b ) + 1 2⌘T n kX j=1 u⇤ ( j ) 2 + ⌘ 1 T TX t=1 @ @ u Lf , D ( fu ( t ) , W ( t ) , b ) 2 1 T TX t=1 Lf , D ( hu⇤ , W ( t ) , b ) + 2 ( 2kk ) 2 q⌘T + c2⌘nq = ( ⇤ ) Using Lemma 9 we have : ( ⇤ ) 1 T TX t=1 Lf , D ( hu⇤ , W ( 0 ) , b ) + c⌘ 2 T 2 nk p q n kX j=1 u⇤ ( j ) + 2 ( 2kk ) 2 q⌘T + c2⌘nq 2c⌘2T 2nk22k + 2 ( 2kk ) 2 q⌘T + c2⌘nq Now , choosing ⌘ = p np qT we get the required .

Summary: It is well-known that neural networks (NN) perform very well in various areas and in particular if one looks at computer vision convolutional neural networks perform very well. Although convolutional neural networks (CNN) are limited in their architecture (since they only allow nearest-neighbour connections) compared to fully-connected NNs (FCNN), their superiority in performance is unclear. In this paper they answer the following fundamental question: can one formally show that CNNs are better than FCNNs for a specific learning task? In this direction they answer in the affirmative.  In particular, more than just giving an example, they show that an interesting property called locality, instead of other parameters like parameter and efficiency weight sharing is the reason for its superior performance.