Why Are Convolutional Nets More Sample-Efficient than Fully-Connected Nets?

1 INTRODUCTION . Deep convolutional nets ( “ ConvNets ” ) are at the center of the deep learning revolution ( Krizhevsky et al. , 2012 ; He et al. , 2016 ; Huang et al. , 2017 ) . For many tasks , especially in vision , convolutional architectures perform significantly better their fully-connected ( “ FC ” ) counterparts , at least given the same amount of training data . Practitioners explain this phenomenon at an intuitive level by pointing out that convolutional architectures have better “ inductive bias ” , which intuitively means the following : ( i ) ConvNet is a better match to the underlying structure of image data , and thus are able to achieve low training loss with far fewer parameters ( ii ) models with fewer total number of parameters generalize better . Surprisingly , the above intuition about the better inductive bias of ConvNets over FC nets has never been made mathematically rigorous . The natural way to make it rigorous would be to show explicit learning tasks that require far more training samples on FC nets than for ConvNets . ( Here “ task ” means , as usual in learning theory , a distribution on data points , and binary labels for them generated given using a fixed labeling function . ) Surprisingly , the standard repertoire of lower bound techniques in ML theory does not seem capable of demonstrating such a separation . The reason is that any ConvNet can be simulated by an FC net of sufficient width , since a training algorithm can just zero out unneeded connections and do weight sharing as needed . Thus the key issue is not an expressiveness per se , but the combination of architecture plus the training algorithm . But if the training algorithm must be accounted for , the usual hurdle arises that we lack good mathematical understanding of the dynamics of deep net training ( whether FC or ConvNet ) . How then can one establish the limitations of “ FC nets + current training algorithms ” ? ( Indeed , many lower bound techniques in PAC learning theory are information theoretic and ignore the training algorithm . ) The current paper makes significant progress on the above problem by exhibiting simple tasks that require Ω ( d2 ) factor more training samples for FC nets than for ConvNets , where d is the data dimension . ( In fact this is shown even for 1-dimensional ConvNets ; the lowerbound easily extends to 2-D ConvNets . ) The lower bound holds for FC nets trained with any of the popular algorithms listed in Table 1 . ( The reader can concretely think of vanilla SGD with Gaussian initialization of network weights , though the proof allows use of momentum , ` 2 regularization , and various learning rate schedules . ) Our proof relies on the fact that these popular algorithms lead to an orthogonalequivariance property on the trained FC nets , which says that at the end of training the FC net —no matter how deep or how wide — will make the same predictions even if we apply orthogonal transformation on all datapoints ( i.e. , both training and test ) . This notion is inspired by Ng ( 2004 ) ( where it is named “ orthogonal invariant ” ) , which showed the power of logistic regression with ` 1 regularization versus other learners . For a variety of learners ( including kernels and FC nets ) that paper described explicit tasks where the learner has Ω ( d ) higher sample complexity than logistic regression with ` 1 regularization . The lower bound example and technique can also be extended to show a ( weak ) separation between FC nets and ConvNets . ( See Section 4.2 ) Our separation is quantitatively stronger than the result one gets using Ng ( 2004 ) because the sample complexity gap is Ω ( d2 ) vs O ( 1 ) , and not Ω ( d ) vs O ( 1 ) . But in a more subtle way our result is conceptually far stronger : the technique of Ng ( 2004 ) seems incapable of exhibiting a sample gap of more than O ( 1 ) between Convnets and FC nets in our framework . The reason is that the technique of Ng ( 2004 ) can exhibit a hard task for FC nets only after fixing the training algorithm . But there are infinitely many training algorithms once we account for hyperparameters associated in various epochs with LR schedules , ` 2 regularizer and momentum . Thus Ng ( 2004 ) ’ s technique can not exclude the possibility that the hard task for “ FC net + Algorithm 1 ” is easy for “ FC net + Algorithm 2 ” . Note that we do not claim any issues with the results claimed in Ng ( 2004 ) ; merely that the technique can not lead to a proper separation between ConvNets and FC nets , when the FC nets are allowed to be trained with any of the infinitely many training algorithms . ( Section 4.2 spells out in more detail the technical difference between our technique and Ng ’ s idea . ) The reader may now be wondering what is the single task that is easy for ConvNets but hard for FC nets trained with any standard algorithm ? A simple example is the following : data distribution in Rd is standard Gaussian , and target labeling function is the sign of ∑d/2 i=1 x 2 i − ∑d i=d/2+1 x 2 i . Figure 1 shows that this task is indeed much more difficult for FC nets . Furthermore , the task is also hard in practice for data distributions other than Gaussian ; the figure shows that a sizeable performance gap exists even on CIFAR images with such a target label . Extension to broader class of algorithms . The orthogonal-equivariance property holds for many types of practical training algorithms , but not all . Notable exceptions are adaptive gradient methods ( e.g . Adam and AdaGrad ) , ` 1 regularizer , and initialization methods that are not spherically symmetric . To prove a lower bound against FC nets with these algorithms , we identify a property , permutationinvariance , which is satisfied by nets trained using such algorithms . We then demonstrate a single and natural task on Rd × { ±1 } that resembles real-life image texture classification , on which we prove any permutation-invariant learning algorithm requires Ω ( d ) training examples to generalize , while Empirical Risk Minimization with O ( 1 ) examples can learn a convolutional net . Paper structure . In Section 2 we discuss about related works . In section 3 , we define the notation and terminologies . In Section 4 , we give two warmup examples and an overview for the proof technique for the main theorem . In Section 5 , we present our main results on the lower bound of orthogonal and permutation equivariant algorithms . 2 RELATED WORKS . Du et al . ( 2018 ) attempted to investigate the reason why convolutional nets are more sample efficient . Specifically they prove O ( 1 ) samples suffice for learning a convolutional filter and also proved a Ω ( d ) min-max lower bound for learning the class of linear classifiers . Their lower bound is against learning a class of distributions , and their work fails to serve as a sample complexity separation , because their upper and lower bounds are proved on different classes of tasks . Arjevani & Shamir ( 2016 ) also considered the notion of distribution-specific hardness of learning neural nets . They focused on proving running time complexity lower bounds against so-called `` orthogonally invariant '' and `` linearly invariant '' algorithms . However , here we focus on sample complexity . Recently , there has been progress in showing lower bounds against learning with kernels . Wei et al . ( 2019 ) constructed a single task on which they proved a sample complexity separation between learning with neural networks vs. with neural tangent kernels . Notably the lower bound is specific to neural tangent kernels ( Jacot et al. , 2018 ) . Relatedly , Allen-Zhu & Li ( 2019 ) showed a sample complexity lower bound against all kernels for a family of tasks , i.e. , learning k-XOR on the hypercube . 3 NOTATION AND PRELIMINARIES . We will use X = Rd , Y = { −1 , 1 } to denote the domain of the data and label and H = { h | h : X → Y } to denote the hypothesis class . Formally , given a joint distribution P , the error of a hypothesis h ∈ H is defined as errP ( h ) : = Px , y∼P [ h ( x ) 6= y ] . If h is a random hypothesis , we define errP ( h ) : = Px , y∼P , h [ h ( x ) 6= y ] for convenience . A class of joint distributions supported on X × Y is referred as a problem , P . We use ‖·‖2 to denote the spectrum norm and ‖·‖F to denote the Frobenius norm of a matrix . We use A ≤ B to denote that B − A is a semi-definite positive matrix . We also use O ( d ) and GL ( d ) to denote the d-dimensional orthogonal group and general linear group respectively . We use Bd 2 p to denote the unit Schatten-p norm ball in Rd×d . We useN ( µ , Σ ) to denote Gaussian distribution with mean µ and covariance Σ . For random variables X and Y , we denote X is equal to Y in distribution by X d= Y . In this work , we also always use PX to denote the distributions on X and P to denote the distributions supported jointly on X × Y . Given an input distribution PX and a hypothesis h , we define PX h as the joint distribution on X × Y , such that ( PX h ) ( S ) = P ( { x| ( x , h ( x ) ) ∈ S } ) , ∀S ⊂ X × Y . In other words , to sample ( X , Y ) ∼ PX h means to first sample X ∼ PX , and then set Y = h ( X ) . For a family of input distributions PX and a hypothesis class H , we define PX H = { PX h | PX ∈ PX , h ∈ H } . In this work all joint distribution P can be written as PX h for some h , i.e . PY|X is deterministic . For set S ⊂ X and 1-1 map g : X → X , we define g ( S ) = { g ( x ) |x ∈ S } . We use ◦ to denote function composition . ( f ◦ g ) ( x ) is defined as f ( g ( x ) ) , and for function classes F , G , F ◦ G = { f ◦ g | f ∈ F , g ∈ G } . For any distribution PX supported on X , we define PX ◦ g as the distribution such that ( PX ◦ g ) ( S ) = PX ( g ( S ) ) . In other words , if X ∼ PX ⇐⇒ g−1 ( X ) ∼ PX ◦ g , because ∀S ⊆ X , P X∼PX [ g−1 ( X ) ∈ S ] = P X∼PX [ X ∈ g ( S ) ] = [ PX ◦ g ] ( S ) . Algorithm 1 Iterative algorithm A Require : Initial parameter distribution Pinit supported in W = Rm , total iterations T , training dataset { xi , yi } ni=1 , parametric modelM : W → H , iterative update rule F ( W , M , { xi , yi } n i=1 ) Ensure : Hypothesis h : X → Y . Sample W ( 0 ) ∼ Pinit . for t = 0 to T − 1 do W ( t+1 ) = F ( W ( t ) , M , { xi , yi } ni=1 ) . return h = sign [ M [ W ( T ) ] ] . For any joint distribution P of form P = PX h , we define P ◦ g = ( PX ◦ g ) ( h ◦ g ) . In other words , ( X , Y ) ∼ P ⇐⇒ ( g−1 ( X ) , Y ) ∼ P ◦ g. For any distribution class P and group G acting on X , we define P ◦ G as { P ◦ g | P ∈ P , g ∈ G } . Definition 3.1 . A deterministic supervised Learning Algorithm A is a mapping from a sequence of training data , { ( xi , yi ) } ni=1 ∈ ( X × Y ) n , to a hypothesis A ( { ( xi , yi ) } ni=1 ) ∈ H ⊆ YX . The algorithmA could also be randomized , in which case the outputA ( { ( xi , yi ) } ni=1 ) is a distribution on hypotheses . Two randomized algorithms A and A′ are the same if for any input , their outputs have the same distribution in function space , which is denoted by A ( { xi , yi } ni=1 ) d = A′ ( { xi , yi } ni=1 ) . Definition 3.2 ( Equivariant Algorithms ) . A learning algorithm is equivariant under group GX ( or GX -equivariant ) if and only if for any dataset { xi , yi } ni=1 ∈ ( X × Y ) n and ∀g ∈ GX , x ∈ X , A ( { g ( xi ) , yi } ni=1 ) ◦ g = A ( { xi , yi } n i=1 ) , or A ( { g ( xi ) , yi } n i=1 ) ( g ( x ) ) = [ A ( { xi , yi } n i=1 ) ] ( x ) . 1 Definition 3.3 ( Sample Complexity ) . Given a problem P and a randomized learning algorithm A , δ , ε ∈ [ 0 , 1 ] , we define the ( ε , δ ) -sample complexity , denoted N ( A , P , ε , δ ) , as the smallest number n ∈ N such that ∀P ∈ P , w.p . 1− δ over the randomness of { xi , yi } ni=1 , errP ( A ( { xi , yi } n i=1 ) ) ≤ ε . We also define the ε-expected sample complexity for a problem P , denoted N ∗ ( A , P , ε ) , as the smallest number n ∈ N such that ∀P ∈ P , E ( xi , yi ) ∼P [ errP ( A ( { xi , yi } ni=1 ) ) ] ≤ ε . By definition , we have N ∗ ( A , P , ε+ δ ) ≤ N ( A , P , ε , δ ) ≤ N ∗ ( A , P , εδ ) , ∀ε , δ ∈ [ 0 , 1 ] .

Summary: The paper presents an interesting analysis of MLP and convnets, where they show a gap between the number of required training examples to generalize well. They show that due to orthogonality invariance in MLP training, then more examples are required compare to convnet, where one example is needed. This approach, which relies on an older result, provides an intuition as to the success of resnet.