On the Inductive Bias of a CNN for Distributions with Orthogonal Patterns

1 INTRODUCTION . Convolutional neural networks ( CNNs ) have achieved remarkable performance in various computer vision tasks ( Krizhevsky et al. , 2012 ; Xu et al. , 2015 ; Taigman et al. , 2014 ) . In practice , these networks typically have more parameters than needed to achieve zero train error ( i.e. , are overparameterized ) . Despite non-convexity and the potential problem of overfitting , training these models with gradient based methods leads to solutions with low test error . It is still largely unknown why such simple optimization algorithms have outstanding test performance for learning overparameterized convolutional networks . Recently , there have been major efforts to provide generalization guarantees for overparameterized CNNs . However , current generalization guarantees either depend on the number of channels of the network ( Long & Sedghi , 2020 ) or hold under specific constraints on the weights ( Li et al. , 2018 ) . Clearly , the generalization of overparameterized CNNs depends on both the learning algorithm ( gradient-based methods ) and unique properties of the data . Providing generalization guarantees while incorporating these factors is a major challenge . Indeed , this requires analyzing non-convex optimization methods and mathematically defining properties of the data , which is extremely difficult for real-world problems . Therefore , it is necessary to first understand simple settings which are amenable to theoretical and empirical analysis and share salient features with real-world problems . Towards this goal , we analyze a simplified pattern recognition task where all patterns in the images are orthogonal and the classification is binary . The architecture is a 3-layer overparameterized convolutional neural network and it is learned using stochastic gradient descent ( SGD ) . We take a unique approach that combines novel empirical observations with theoretical guarantees to provide a novel generalization bound which is independent of the number of channels and is a low-degree polynomial of the filter dimension , which is usually low in practice . Empirically , we identify a novel property of the solutions found by SGD . We observe that the statistics of patterns in the training data govern the magnitude of the dot-product between learned pattern detectors and their detected patterns . Specifically , patterns that appear almost exclusively in one of the classes will have a large dot-product with the channels that detect them . On the other hand , patterns that appear roughly equally in both classes , will have a low dot-product with their detecting channels . We formally define this as the “ Pattern Statistics Inductive Bias ” condition ( PSI ) and provide empirical evidence that PSI holds across a large number of instances . We also prove that SGD indeed satisfies PSI in a simple setup of two points in the training set . Under the assumption that PSI holds , we analyze the sample complexity and prove that it is at most O ( d2 log d ) , where d is the filter dimension . In contrast , we show that the VC dimension of the class of functions we consider is exponential in d , and thus there exist other learning algorithms ( not SGD ) that will have exponential sample complexity . Together , these results provide firm evidence that even though SGD can in principle overfit , it is nonetheless biased towards solutions which are determined by the statistics of the patterns in the training set and consequently it has good generalization performance . We perform experiments with overparamterized CNNs on a variant of MNIST that has nonorthogonal patterns . We use our analysis to better understand why SGD has low sample complexity in this setting . We empirically show that the inductive bias of SGD is similar to PSI . This suggests that the idea of PSI is not unique to the orthogonal case and can be useful for understanding overparameterized CNNs in other challenging settings . 2 RELATED WORK . Several recent works have studied the generalization properties of overparameterized CNNs . Some of these propose generalization bounds that depend on the number of channels ( Long & Sedghi , 2020 ; Jiang et al. , 2019 ) . Others provide guarantees for CNNs with constraints on the weights ( Zhou & Feng , 2018 ; Li et al. , 2018 ) . Convergence of gradient descent to KKT points of the max-margin problem is shown in Lyu & Li ( 2020 ) and Nacson et al . ( 2019 ) for homogeneous models . However , their results do not provide generalization guarantees in our setting . Gunasekar et al . ( 2018 ) study the inductive bias of linear CNNs . Yu et al . ( 2019 ) study a pattern classification problem similar to ours . However , their analysis holds for an unbounded hinge loss which is not used in practice . Furthermore , their sample complexity depends on the network size , and thus does not explain why large networks do not overfit . Other works have studied learning under certain ground truth distributions . For example , Brutzkus & Globerson ( 2019 ) study a simple extension of the XOR problem , showing that overparameterized CNNs generalize better than smaller CNNs . Single-channel CNNs are analyzed in ( Du et al. , 2018b ; a ; Brutzkus & Globerson , 2017 ; Du et al. , 2018c ) . Other works study the inductive bias of gradient descent on fully connected linear or non-linear networks ( Ji & Telgarsky , 2019 ; Arora et al. , 2019a ; Wei et al. , 2019 ; Brutzkus et al. , 2018 ; Dziugaite & Roy , 2017 ; Allen-Zhu et al. , 2019 ; Chizat & Bach , 2020 ) . Fully connected networks were also analyzed via the NTK approximation ( Du et al. , 2019 ; 2018d ; Arora et al. , 2019b ; Fiat et al. , 2019 ) . Kushilevitz & Roth ( 1996 ) ; Shvaytser ( 1990 ) study the learnability of visual patterns distribution . However , our focus is on learnability using a specific algorithm and architecture : SGD trained on overparameterized CNNs . 3 THE ORTHOGONAL PATTERNS PROBLEM . Data Generating Distribution : We consider a learning problem that captures a key property of visual classification . Many visual classes are characterized by the existence of certain patterns . For example an 8 will typically contain an x like pattern somewhere in the image . Here we consider an abstraction of this behavior where images consist of a set of patterns . Furthermore , each class is characterized by a pattern that appears exclusively in it . We define this formally below . Let P be a set of orthogonal vectors in Rd , where ∣P ∣ ≤ d. For simplicity , we assume that ∥p∥2 = 1 for all p ∈ P . We consider input vectors x with n patterns of dimension d. Formally , x = ( x [ 1 ] , ... , x [ n ] ) ∈ Rnd where x [ i ] ∈ P is the ith pattern of x and n < d. We denote p ∈ x if x contains the pattern p ∈ P .1 Let P ( x ) = { p ∈ x ∣ p ∈ P } denote the set of all patterns in x . 1We say that x contains p if there exists j such that x [ j ] = p. Next , we define how labeled points are generated . Consider three non-overlapping sets of patterns : P− , P+ , Ps ⊂ P whose disjoint union is P . P+ is the set of positive patterns , P− the set of negative patterns andPs is the set of spurious patterns . For simplicity , in this work we consider the case where ∣P+∣ = ∣P−∣ = 1 . We denote , P = { p1 , p2 , ... , p∣P ∣ } , P+ = { p1 } and P− = { p2 } . For convenience , we also refer to a set of patterns A as a set of the indices of the patterns , .e.g. , we denote i ∈ A if pi ∈ A . We consider distributions D over ( x , y ) ∈ Rnd × { ±1 } with the following properties : ( 1 ) P ( y = 1 ) = P ( y = −1 ) = 1 2 . ( 2 ) Given y = 1 , a vector x is sampled as follows . Choose the positive pattern p1 and randomly choose a set of n− 1 patterns from Ps . Denote this set of n chosen patterns by A . Let x be some x′ such that P ( x′ ) = A , i.e. , the location of each pattern in x is chosen arbitrarily.2 For example , if n = 3 and the n−1 patterns are p3 , p7 this can result in samples such as ( [ p1 , p7 , p3 ] ,1 ) or ( [ p3 , p1 , p7 ] ,1 ) . ( 3 ) Similarly for y = −1 , only choose p2 ∈ P− instead of p1 . We will consider several distributions that satisfy the above and have different sampling schemes for the spurious patterns ( see Sec . 7 ) . Fig . 4 in the supplementary shows an example of samples generated using the above procedure . Note that any distribution which satisfies the above is linearly separable and each vector x can be classified solely based on whether p1 ∈ x or p2 ∈ x. Neural Architecture : For learning the above pattern detection problems , a natural model in this context is a 3-layer network with a convolutional layer , followed by ReLU , max pooling and a fullyconnected layer . Each channel in the first layer can be thought of as a detector for a given pattern . We say that a detector detects pattern p ∈ P , if p has the largest dot product with the detector among all patterns in P+ ∪Ps or P− ∪Ps and this dot product is positive.3 For simplicity we fix the weights on the last linear layer to values ±1.4 Let 2k denote the number of channels . We partition the channels into two sets : w ( 1 ) , . . . , w ( k ) and u ( 1 ) , . . . , u ( k ) . These will have weights of +1 and −1 in the output respectively . Finally , let W ∈ R2k×2 be the weight matrix whose rows are w ( i ) followed by u ( i ) . For an input x = ( x [ 1 ] , ... , x [ n ] ) ∈ Rnd where x [ i ] ∈ Rd , the output of the network is : NW ( x ) = k ∑ i=1 [ max j { σ ( w ( i ) ⋅x [ j ] ) } −max j { σ ( u ( i ) ⋅x [ j ] ) } ] ( 1 ) where σ ( x ) = max { 0 , x } is the ReLU activation . Let H denote the class of all networks NW in Eq . 1 , with k > 0 . Finally , we note thatH can perfectly fit the distribution D above , by setting k = 1 , w ( 1 ) = p1 and u ( 1 ) = p2 . Therefore , for k > 1 the network is overparameterized . Training Algorithm : Let S be a training set withm IID samples fromD . We consider minimizing the hinge loss : ` ( W ) = 1 m ∑ ( xi , yi ) ∈Smax { 1 − yiNW ( xi ) ,0 } . For optimization , we use SGD with constant learning rate η . The parameters W are initialized as IID Gaussians with zero mean and standard deviation σg . Let Wt be the weight matrix at iteration t of SGD . Similarly let w ( i ) t , u ( i ) t be the corresponding vectors at iteration t. Detection Ratios : We now define the notion of detection ratios . The detection ratios are a property of the model and will be key to our analysis . We first define the set of neurons that are maximally activated by pattern pi among all patterns in Ps ∪P+ ( i.e. , all detectors of pattern pi ) :5 W+ ( i ) = { j ∣ argmax l∈Ps∪P+ w ( j ) ⋅ pl = i , w ( j ) ⋅ pi > 0 } U+ ( i ) = { j ∣ argmax l∈Ps∪P+ u ( j ) ⋅ pl = i , u ( j ) t ⋅ pi > 0 } ( 2 ) Next we defineM+w ( i ) = ∑j∈W+ ( i ) w ( j ) ⋅pi andM+u ( i ) = ∑j∈U+ ( i ) u ( j ) ⋅pi . The quantityM+w ( i ) is the sum of the dot products between a pattern pi and its detectors w ( j ) . M+w ( i ) can be interpreted 2The order of the patterns will not matter , because the convolutional network is invariant to it . 3The reason we consider these two sets of patterns will be clear when we discuss detection ratios . 4Note that this does not affect the expressive power of the network . 5Given a tie between sets , we assume the weight is assigned arbitrarily to one of them . as the overall response of w ( j ) detectors of pattern pi . Similarly , we define M + u ( i ) , which is like M+w ( i ) , only with detectors u ( j ) . For all pi ∈ P+ ∪ Ps we refer to M+u ( i ) M+w ( 1 ) as positive detection ratios . The detection ratio can be interpreted as the ratio between the undesired response of pattern detectors of pi and the desired response of discriminative pattern detectors ( detectors of p1 ) . Therefore , we would like this ratio to be small . Indeed , for any positive point x+ we have that : NW ( x+ ) ≥M+w ( 1 ) − ∑ pi∈Ps∪P+ M+u ( i ) =M+w ( 1 ) ⎛ ⎝ 1 − ∑ pi∈Ps∪P+ M+u ( i ) M+w ( 1 ) ⎞ ⎠ ( 3 ) where the inequality follows since positive points have only patterns in P+ ∪ Ps , by Eq . 1 and the definitions of M+w ( i ) and M+u ( i ) . Notice that if all positive detection ratios are small , then the positive point is classified correctly . We will empirically show that for SGD , the magnitude of the detection ratios are governed by the statistics of the patterns in the training set , which will imply our generalization result . Similarly , we define W− ( i ) , U− ( i ) and M−w ( i ) , M−u ( i ) , where the only difference is using P− instead ofP+ . Furthermore , for all pi ∈ P−∪Ps we say that M−w ( i ) M−u ( 2 ) are negative detection ratios . Then , as in Eq . 3 we have for all negative points x− that : −NW ( x− ) ≥ M−u ( 2 ) ( 1 −∑pi∈Ps∪P− M−w ( i ) M−u ( 2 ) ) . This shows that if negative detection ratios are small , then all negative points are classified correctly . In the rest of the paper , we refer to both positive and negative detection ratios as detection ratios . Empirical Pattern Bias : In a given training set , patterns will appear in both positive and negative examples . The following measure captures how well-balanced are the patterns between the labels . For any pattern pi ∈ P , define the following statistic of the training set : si = 1 m m ∑ j=1 yj1 { pi ∈ xj } ( 4 ) The detection ratios define quantities of the learned model . On the other hand , Eq . 4 is a quantity of the sampled training set . In the next section , we define the PSI property , which specifies how these two measures should be related to guarantee good generalization for the learned model .

Summary: In this manuscript the authors derive theoretical analysis for the generalization guarantees of a naïve CNN (3-layers) where the task is a simplified binary classification task, under the assumption that the images contain orthogonal patches (a naïve assumption). They define a statistical phenomenon that holds in SGD in the proposed setting and call it Pattern Statistics Inductive Bias (PSI). Informally, this means that the magnitude of the dot-product between the learned pattern detectors and their detected patterns is correlated with the distribution of the patterns in the data.   They prove that if a learning algorithm  satisfies PSI then its sample complexity is O(d^2 log (d)), where d is the dimension of the filter. According to their empirical derivation SGD satisfies this property. In contrast there exist learning algorithms that have exponential sample complexity.