What Breaks The Curse of Dimensionality in Deep Learning?

Although learning in high dimensions is commonly believed to suffer from the1 curse of dimensionality , modern machine learning methods often exhibit an as-2 tonishing power to tackle a wide range of challenging real-world learning prob-3 lems without using abundant amounts of data . How exactly these methods break4 this curse remains a fundamental open question in the theory of deep learning.5 While previous efforts have investigated this question by studying the data ( D ) ,6 model ( M ) , and inference algorithm ( I ) as independent modules , in this paper7 we analyzes the triple ( D , M , I ) as an integrated system . We examine the basic8 symmetries of such systems , focusing on four of the main architectures in deep9 learning : fully-connected networks ( FCN ) , locally-connected networks ( LCN ) , and10 convolutional networks with and without pooling ( GAP/VEC ) . By computing an11 eigen-decomposition of the infinite-width limits ( aka Neural Kernels ) of these12 architectures , we characterize how inductive biases ( locality , weight-sharing , pool-13 ing , etc ) and the breaking of spurious symmetries can affect the performance of14 these learning systems . Our theoretical analysis shows that for many real-world15 tasks it is locality rather than symmetry that provides the first-order remedy to the16 curse of dimensionality . Empirical results on state-of-the-art models on ImageNet17 corroborate our results.18 1 Introduction19 Statistical problems with high-dimensional data are frequently plagued by the curse of dimensionality,20 in which the number of samples required to solve the problem grows rapidly with the dimensionality21 of the input . Classical theory explains this phenomenon as the consequence of basic geometric and22 algebraic properties of high-dimensional spaces ; for example , the number of -cubes inside a unit23 cube in Rd grows exponentially like −d , and the number of degree r polynomials in Rd grows like a24 power-law dr . Since for real-world problems d is typically in the hundreds or thousands , classical25 wisdom suggests that learning is likely to be infeasible . However , starting from the groundbreaking26 work AlexNet [ 1 ] , practitioners in deep learning have tackled a wide range of difficult real-world27 learning problems ( [ 2–6 ] ) in high dimensions , once believed by many to be out-of-scope of current28 techniques . The astonishing success of modern machine learning methods clearly contradicts the29 curse of dimensinonality and therefore poses the fundamental question : mathematically , how do30 modern machine learning methods break the curse of dimensionality ? 31 To answer this question , we must trace back to the most fundamental ingredients of machine learning32 methods . They are the data ( D ) , the model ( M ) , and the inference algorithm ( I ) .33 Data ( D ) is of course central in machine learning . In the classical learning theory setting , the learning34 objective usually has a power-law decay m−β as the function of the number of training samples35 m. The theoretical bound on β is usually tiny , owing to the curse of dimensionality , and is of36 Submitted to 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Do not distribute . limited practical utility for high-dimensional data . On the other hand , empirical measurements of37 β in state-of-the-art deep learning models typically reveal values of β that are not at all small ( e.g.38 β = 0.43 for ResNet in Fig.S2 ) even though d is quite large ( e.g . d ∼ 105 for ImageNet ) . This39 example suggests that the learning curve must have important functional dependence onM and I.40 Indeed , as we will observe later , many of the best performing methods exhibit learning curves for41 which β = β ( m ) actually increases as m becomes larger , i.e . data makes the usage of data more42 efficient . We call this phenomenon DIDE , for data improves data efficiency.43 Designing machine learning models ( M ) that maximize data-efficiency is critical to the success44 of solving real-world tasks . Indeed , breakthroughs in machine learning are often driven by novel45 architectures LeNet [ 7 ] , AlexNet [ 1 ] , Transformer [ 2 ] , etc . While some of the inductive biases of these46 methods are clear ( e.g . translation symmetries of CNNs ) , others tend to build off of prior empirical47 success and are less well-understood ( e.g . the implicit bias of SGD ) . To build our understanding of48 these biases and how they affect learning , we conduct a theoretical analysis of them in the infinite-49 width setting [ 8–12 ] , which preserves most salient aspects of the architecture while enabling tractable50 calculations . We classify all phenomena that could be explained by infinite networks alone as the51 consequences of inductive biases.52 The inference procedure ( I ) is what enables learning in machine learning methods . It is widely53 believed that modern inference methods , specifically gradient descent and variants , ‘ implicitly ‘ bias54 the solutions of the networks towards those that generalize well and away from those that generalize55 poorly [ 13–15 ] . The effects of the inference algorithm are intimately tied to the specifics of the model56 ( e.g . weight-sharing ) and the data ( e.g . augmentation ) , and might not be fully understood with a57 fixed-data , fixed-model analysis . Indeed , good performance may derive from interactions between58 ( M , I ) , or ( D , I ) , or even ( D , M , I ) . In Sec . 3.1 , we demonstrate the DIDE effect for a particular59 choice of ( D , M , I ) and show that this effect disappears if any one of D , M , or I is altered.60 The above discussion highlights the insufficiency of treating D , M , and I as separate non-interacting61 modules . They must be considered as an integrated system . Throughout this paper , we will refer to62 the triplet ( D , M , I ) as a ( machine ) learning system and the tuple ( M , I ) as the learning algorithm63 of the system that operates on D. We summarize our contributions below.64 1 . We surface the basic symmetries of various ( D , M , I ) associated to four of the main ar-65 chitectures in deep learning FCNn ( fully-connected networks ) , LCNn ( locally-connected66 networks ) , VECn/GAPn ( convolution networks with a flattening /a global average pooling67 readout layer ) , their infinite width counterparts FCN∞/LCN∞/VEC∞/GAP∞ . Treating68 FCNn/∞ as the baseline model , we show that the locality from LCNn and the weight-sharing69 from VECn/GAPn break spurious symmetries and lead to better systems . Empirically , we70 examine the relation between the symmetries and the performance of the systems in the71 infinite width setting and finite width setting with various of interventions . Surprisingly,72 we observe that state-of-the-art learning system ( EfficientNet [ 16 ] ) on ImageNet can learn73 almost equally well even the coordinate of the data are transformed by the symmetry group74 defined by LCNn.75 2 . We show that although the weight-sharing from VECn provides coordinate information of76 the data to the system , as the width gets larger , it becomes harder for the learning algorithm77 to explore such information and at infinite width , the system restores the symmetry group78 that is identical to LCNn , and is completely unaware of the coordinate information . As a79 consequence , the performance of the network , as a function of width , monotonically decays80 [ 12 ] . This is in stark contrast to recent finding that the performance of network is positively81 correlated to its width . We show that this phenomenon continues to hold even with various82 interventions ( larger learning rate and l2 regularization ) to the training procedures . However,83 with more data ( e.g . data augmentation ) VECn can be on par with GAPn.84 3 . The function space defined by LCNn is a super set of that defined by VECn . We prove the85 opposite is true . Therefore , VECn is able to express functions in the space with a stronger86 inductive bias GAPn ( translation invariance ) and functions in a seemingly much larger87 class LCNn . We hypothesize that as the dataset grows , the learned functions using VECn is88 transitioned away from those learned using LCNn and become closer to those learned using89 GAPn . This suggests , even though the prior ( provided by human ) is not 100 % correct , with90 the help of more data , gradient descent might be able to correct it , a possible explanation of91 DIDE.92 4 . When the input space is the product of hyperspheres , we eigendecompose the kernels93 associated to one-hidden layer infinite width network , FCN∞ , VEC∞ = LCN∞ and GAP∞.94 We treat FCN∞ as the baseline , whose order r eigenspace has dimension of order dr95 and eigenvalues of order d−r for r ≥ 0 [ 17 ] . We show that locality alone ( i.e . VEC∞ ) 96 dramatically reduces the dimension of the r-eigenspace for r ≥ 2 and the spectral gap97 between all r-eigenspaces but r = 0 and r = 1 , making learning of higher order eigenspaces98 feasible with dramatically fewer samples and gradient steps . In addition , pooling ( i.e.99 GAP∞ ) reduces the dimension of r-eigenspace for r ≥ 1 by a factor equal to the size of the100 pooling window , but it does not change the spectra in an essential way.101 Our empirical and theoretical results surface the importance of locality which , we believe , provides102 the first-order remedy to the curse of dimensionality for many real-world tasks and which has been103 largely overlooked.104 2 Preliminary and Notation105 2.1 Neural Networks106 We focus our presentation on the supervised learning setting and more concretely , on image107 recognition . Let D ⊆ ( Rd ) 3 × Rk ≡ R3d × Rk denote the data set ( training and test ) and108 X = { x : ( x , y ) ∈ D } and Y = { y : ( x , y ) ∈ D } denote the input space ( images ) and label space,109 respectively . Here d is the spatial dimension ( e.g . d = 32× 32 for CIFAR-10 ) of the images and 3 is110 the total number of channels ( i.e . RGB ) . We use FCNn to denote a L-hidden layer fully-connected111 network with identical hidden widths nl = n ∈ N for l = 1 , ... , L and with readout width nL+1 = k112 ( the number of logits ) . For each x ∈ R3d = ( Rd ) 3 , we use hl ( x ) , xl ( x ) ∈ Rnl to represent the pre-113 and post-activation functions at layer l with input x . The recurrence relation FCN is given by114 { hl+1 = xlW l+1 xl+1 = φ ( hl+1 ) and W li , j = 1√nlωlij , ωlij ∼ N ( 0 , 1 ) ( 1 ) where φ is a point-wise activation function , W l+1 ∈ Rnl×nl+1 are the weights and ωlij are the115 trainable parameters , drawn i.i.d . from a standard Gaussian∼ N ( 0 , 1 ) at initialization . For simplicity116 of the presentation , the bias terms and the hyperparameters ( the variances of the weights ) are omitted.117 Adding them back won ’ t affect the conclusion of the paper.118 For convolutional networks or locally-connected networks , the inputs are treated as tensors in ( Rd ) 3.119 The recurrent relation of convolutional networks can be written as120 xl+1α , j = φ ( h l+1 α , j ) and h l+1 α , j ≡ 1√ ( 2k + 1 ) nl nl∑ j=1 k∑ β=−k xlα+β , iω l ij , β ( 2 ) Here α ∈ [ d ] denote the spatial location , i/j ∈ [ n ] denotes the fanin/fanout channel indices . For121 notational convenience , we assume circular padding and stride equal to 1 for all layers . The features122 of the penultimate layer are 2D tensors and there are two commonly used approaches to map them123 to the logit layer : stack a dense layer after either vectorizing the 2D tensor to a 1D vector or124 applying a global average pooling layer to each channel . We use VECn/GAPn to denote the network125 obtain from the former/latter , which are known to be equipped with the inductive biases translation126 equivariant/invariant . The readout layer of VECn/GAPn could be written as127 xL+1j = 1√ dn ∑ α∈ [ d ] xLα , iw L+1 α , ij , x L+1 j = 1√ n ∑ i∈ [ n ] 1 d ∑ α∈ [ d ] xLα , i wL+1ij ( 3 ) We briefly remark the the key difference between the two . In VECn , each pixel in the penultimate128 layer has its own ( independent random ) variable while pixels within the same channel shared the129 same ( random ) variable in GAPn . It is clear that the function space of VECn contains that of GAPn.130 Locally Connected Networks LCNn [ 18 , 19 ] are convolutional network without weight sharing131 between spatial locations . LCNn preserve the connectivity pattern , and thus topology , of a convnet.132 Mathematically , the current formula is defind as in Equation 2 with all the shared parameters ωlij , β133 replaced by unshared ωlij , α , β ∼ N ( 0 , 1 ) 134 In this note , we assume that the LCNn are always associated with a vectorization readout layer and it135 is clear , as a function space , LCNn is a super set of VECn . Interestingly , the opposite is also true.136 Theorem 2.1 ( Sec . B ) . Let VECn/LCNn/GAPn denote the set of functions that can be represented137 by L-hidden layer VECn/LCNn/GAPn networks with hidden width n. Then138 GAPn ⊆ VECn ⊆ LCNn ⊆ VECdn ( 4 ) The significance of this theorem is that if we consider the function space VECn as a soft prior,139 gradient descent could move it closer to a better prior GAPn ( translation invariance ) if the average140 pooling is ( approximately ) learned in the readout layer or it might remain close to LCNn.141 2.2 Gradient Descent Training142 We use f to denote any functions defined by the architectures above and θ to denote the collection143 of all parameters . Denote by θt the time-dependence of the parameters and by θ0 their initial144 values . We use ft ( x ) ≡ f ( x , θt ) ∈ Rk to denote the output ( or logits ) of the neural network at145 time t. Let ` ( ŷ , y ) : Rk × Rk → R denote the loss function where the first/second argument is146 the prediction/true label . By applying continuous time gradient descent to minimize the objective147 L = ∑ ( x , y ) ∈D ` ( ft ( x , θ ) , y ) , the evolution of the parameters θ and the logits f can be written as148 θ̇t = −∇θft ( XT ) T∇ft ( XT ) L , ḟt ( XT ) = ∇θft ( XT ) θ̇t = − Θ̂t ( XT , XT ) ∇ft ( XT ) L ( 5 ) where ft ( XT ) = vec ( [ ft ( x ) ] x∈XT ) , the k|D|× 1 vector of concatenated logits for all examples , and149 ∇ft ( XT ) L is the gradient of the loss with respect to the model ’ s output , ft ( XT ) . Θ̂t ≡ Θ̂t ( XT , XT ) 150 is the tangent kernel at time t , which is a k|D| × k|D| kernel matrix151 Θ̂t = ∇θft ( XT ) ∇θft ( XT ) T ( 6 ) One can define the tangent kernel for general arguments , e.g . Θ̂t ( x , XT ) where x is test input . At152 finite-width , Θ̂ will depend on the specific random draw of the parameters and evolve with time . As153 such , for a test point x the prediction ft ( x ) depends on the random initalization and is also stochastic.154 Note that the parameters are initialized randomly and the randomness will be carried out through the155 training procedure . As a consequence , the prediction functions are stochastic.156 2.3 Infinite Network : Gaussian Processes and the Neural Tangent Kernels157 Neural Networks as Gaussian Processes ( NNGP ) . As the width n → ∞ , at initialization the158 output f0 ( X ) forms a Gaussian Process f0 ( X ) ∼ GP ( 0 , K ( X , X ) ) , known as the NNGP [ 8 , 20 , 21 ] .159 HereK is the GP kernel and can be computed in closed form for a variety of architectures . By treating160 this infinite width network as a Bayesian model ( aka Bayesian Neural Networks ) and applying161 Bayesian inference , the posterior is also a GP162 N ( K ( X∗ , XT ) K−1 ( XT , XT ) YT , K ( X∗ , X∗ ) −K ( X∗ , X ) K ( X , X ) −1K ( X∗ , X ) T ) ( 7 ) Neural Tangent Kernelss ( NTK ) . Recent advance in global convergence theory of over-163 parameterized networks [ 22–25 , 12 ] has shown that under certain assumptions , the tangent kernels is164 almost stationary over the course of training and is concentrated on its infinite width limit Θ in the165 sense there is a constant C independent of t and the network ’ s width n such that166 sup t≥0 ‖Θ̂t ( XT , XT ) −Θ ( XT , XT ) ‖F + ‖Θ̂t ( XT , X∗ ) −Θ ( XT , X∗ ) ‖F ≤ C√ n . ( 8 ) where is the infinite width limit of Θ at initialization , whose existence has been proved in [ 22 , 26 ] .167 As such , when the loss is the mean squared error ( MSE ) , the mean prediction ( marginarized over168 random initialization ) has the following closed form169 f ( X∗ ) = Θ ( X∗ , XT ) Θ−1 ( XT , XT ) ( I − e−ηΘ ( XT , XT ) t ) Y , ( 9 ) Letting t → ∞ , the above solution is the same as that of the kernel ridgeless regression using the170 infinite width tangent kernel Θ . We use FCN∞ ( x ) , LCN∞ ( x ) , VEC∞ ( x ) and GAP∞ ( x ) to denote171 the infinite width solutions ( either the GP inference or the NTK regression ) for the corresponding172 architectures , where we have suppressed the dependence on the training data ( XT , YT ) .173 3 Symmetries of Machine Learning Systems174 Symmetry is fundamental in physical systems . So is it in machine learning systems . We explore175 symmetries of various machine learning systems in this section . Given D = ( X , Y ) and a transforma-176 tion on the input space τ : R3d → R3d , we set τ ( D ) = ( τ ( X ) , Y ) . Let O ( 3d ) denote the orthogonal177 group on the flatten input space R3d . The subgroup O ( 3 ) d ≤ O ( 3d ) operates on the un-flattened178 input ( Rd ) 3 , whose element rotates each pixel xα ∈ R3 by an independent element τα ∈ O ( 3 ) . The179 smaller subgroup O ( 3 ) ⊗ Id ≤ O ( 3 ) d applies the shared rotation ( i.e . τα = τ to all xα for α ∈ [ d ] ) .180 We use P ( 3d ) to denote the permutation group on R3d and P ( 3 ) d and P ( 3 ) ⊗ Id are defined similarly.181 Note that rotating X by τ is equivalent to transfer the original coordinate system by the adjoint182 tranformation τ∗ = τ−1.183 For a deterministic ( stochastic ) learning algorithmA = ( M , I ) , we useA ( DT ) to denote the learned184 function ( distribution of the learned functions ) using training set DT . We use Aτ ( DT ) to denote185 the learned function ( s ) using τ ( DT ) and makes prediction on the transformed test point τ ( X∗ ) . In186 another word , the learning algorithm is conducted in the input space whose coordinate system is187 transformed by τ−1.188 Definition 1 . Let G be a group of transformations R3d → R3d . We say a deterministic ( stochastic ) 189 learning algorithm A = ( M , I ) is g-invariant if A = Ag ( A =d Ag ) . In this case , we say the190 system ( D , M , I ) is g-invariant and use the notation ( D , M , I ) = ( gD , M , I ) . If this holds for all191 g ∈ G , then we say the algorithm and the system are G-invariant.192 If ( M , I ) is the algorithm of minimum norm linear regressor , then ( D , M , I ) is O ( 3 ) d-invariant ; 193 see Sec.G for more details . Note that the symmetry ( invariance ) in our definition is a property of a194 system and is different from the notion of symmetry that are commonly used in the machine learning195 community , which is a property of a function ( e.g . translation invariance ) .196 Theorem 3.1 ( Sec.C ) . If the parameters of the networks are initialized with iid N ( 0 , 1 ) , then197 • FCNn/∞ are O ( 3d ) -invariant.198 • LCNn/∞ are O ( 3 ) d-invariant.199 • VECn is O ( 3 ) ⊗Id-invariant and VEC∞200 is O ( 3 ) d-invariant.201 • GAPn/∞ are O ( 3 ) ⊗ Id-invariant.202 The O ( 3d ) -invariant of FCN∞ is because the NTK/NNGP kernel is an inner product kernel , namely,203 there is a function k such that the kernels have the form k ( 〈x , x′〉 ) . The O ( 3d ) -invariant of finite204 width FCNn is due to the Gaussian initialization of the first layer which was first observed and205 proved in [ 27 ] . Rotating the input by τ ∈ O ( 3d ) is equivalent to rotating the weight matrix ω of206 the first layer by τ∗ . Since for ω ∈ N ( 0 , 1 ) 3d τ∗ω =d ω , at random initialization , the distribution207 of the output functions ( the prior ) are unchanged if all inputs are rotated by the same element in208 O ( 3d ) . This property continues to hold throughout the course of ( continue/discrete ) gradient descent209 training with/without L2-regularization and Bayesian posterior inference . For the same reason , LCNn210 is O ( 3 ) d-invariant because each patch of the image uses independent Gaussian random variables.211 However , weight-sharing in VECn and GAPn breaks the O ( 3 ) d symmetry , reducing it to O ( 3 ) ⊗ Id.212 For infinite networks , LCN∞ = VEC∞ [ 28–31 ] . The kernels of VEC∞ and GAP∞ are of the forms213 ΘVEC ( x , x ′ ) = k ( { 〈xα , x′α〉 } α∈ [ d ] ) and ΘGAP ( x , x′ ) = k ( { 〈xα , x′α′〉 } α , α′∈ [ d ] ) , ( 10 ) resp . The former depends only on the inner product between pixels in the same spatial location,214 breaking the O ( 3d ) symmetry and reducing it to O ( 3 ) d. In addition , the latter depends also on the215 inner products of pixels across different spatial locations due to pooling , which breaks the O ( 3 ) d216 symmetry and reduces it to O ( 3 ) ⊗ Id.217 Note that dim ( O ( 3d ) ) = 3d ( 3d− 1 ) /2 , dim ( O ( 3 ) d ) = 3d and dim ( O ( 3 ) ⊗ Id ) = 3 . LCNn/VEC∞218 dramatically reduces the dimension of the symmetry group . It is worth mentioning that while219 dim ( O ( 3d ) ) many pairs of rotated and unrotated images are needed to recover the exact rotation in220 O ( 3d ) , only 3 pairs are sufficient for O ( 3 ) d , same as that of O ( 3 ) ⊗ Id . The results of the paper221 are presented in the most vanilla setting . Our methods can easily extend to more complicated222 architectures like ResNet [ 32 ] , MLP-Mixer [ 33 ] and etc . The symmetry groups of such systems223 need to be computed in a case-by-case manner by identifying the invariant group of the random224 initialization and training procedures.225 3.1 Empirical Supports and Observations226 Performance under Rotations . We examinate the performance of : FCN , VEC , LCN , GAP and227 LAP4/8 , when the coordinates of the data are transformed by six different groups ( x-axis in Fig.1 ) 228 using the standard dataset CIFAR-10 . , Here LAP4/8 is the same as GAP except the readout layer is229 replaced by the Local Average Pooling with window size 4×4/8×8 . We consider 4 types of training230 methods : ( 1 ) NTK , i.e . infinite networks ( 2 ) NN , our baseline for finite width neural network which231 is trained with momemtum using a small learning rate and without L2 regularizer and the network232 is centered ( +C ) to reduce the variance from random initialization ( 3 ) NN+= NN+LR+L2−C , i.e.233 using a larger learning rate ( +LR ) , adding L2 regularization ( +L2 ) ) and removing the centering ( −C ) 234 ( 4 ) NN++=NN++DA , adding MixUp [ 34 ] data augmentation ( +DA ) to NN+ . Overall , we observe235 that , for most of the cases in NTK/NN/NN+ , adding spurious symmetry to a system ( D , M , I ) 236 degrades the performance towards that of the system invariant to that symmetry . Surprisingly , in the237 baseline NN , performance of VECn+ O ( 3 ) ⊗ Id rotation is slightly worse than that of VECn+ O ( 3 ) d238 and than that of LCNn , indicating that the system with M = VECn is likely operating closely239 on the O ( 3 ) d symmetry . The interventions −C+L2+LR in NN+ distinguishes the performance of240 VECn + O ( 3 ) ⊗ Id from VECn + O ( 3 ) d and +DA eventually closes the performance gap between241 VECn + O ( 3 ) ⊗ Id and GAPn + O ( 3 ) ⊗ Id , helping the system to be aware of the smaller symmetry242 O ( 3 ) ⊗ Id , escaping from the O ( 3 ) d symmetry.243 Symmetry Breaking of VECn . Assuming Equation 8 , namely , the network is in the NTK regime,244 lim n→∞ |EVECn ( x ) − VEC∞ ( x ) |+ lim n→∞ |EVECn ( x ) − EVECτn ( x ) | ≤ Cn− 1 2 ( 11 ) where the expectation E is over random initialization and VECn ( x ) is the prediction of the test point245 x when t = ∞ , i.e . training loss is 0 . VECτn is the prediction of the τ -rotated system , τ ∈ O ( 3 ) d.246 The O ( 3 ) d symmetry is restored as n→∞ . As such , for large n , the system is approximately O ( 3 ) d-247 invariant . We randomly sample τ ∈ O ( 3 ) d and use the exponential map to construct a continuous248 interpolation τt ∈ O ( 3 ) d with τ0 = Id and τ1 = τ . We train the network as in NN++ ( +LR+L2−C ) 249 using different n and τt and average the predictions over 10 random initialization as an approximation250 of EVECτtn ( x ) . Not surprisingly , as n increases and/or t increases , ( 1 ) test performance decays251 monotonically ( left panel in Fig.2 ) , ( 2 ) the distance to EGAPn increases monotonically ( middle252 panel ) and ( 3 ) distance to VEC∞ decrease monotonically ( right panel ) . Clearly , the coordinate253 information from the data is utilized by smaller width VECn.254 26 29 211 215 Training Set Size 2-6 2-5 2-4 M SE +FlipUnaugmented GAPn GAP VECn VEC LCNn Figure 3 : Data Bends Learning Curve of VECn . We study the effect of training set size to the network ’ s performance for various models . In the small dataset regime , the slope of the learning curve ( in the log-log plot ) of VECn is similar to that of VEC∞ and FCNn . However , as the dataset gets larger , the slope increases significantly . This is hinted by Theorem 2.1 . 0.70 0.72 0.74 0.76 0.78 0.80 0.82 0.84 ImageNet Top-1 Accuracy ( % ) 0.60 0.64 0.68 0.72 0.76 0.80 0.84 O ( 3 ) d - Im ag eN et T op -1 A cc ur ac y ( % ) B0 B1 B3 B5 B7 ResNet18 ResNet34 ResNet50 ResNet101 ResNet200 O ( 3 ) d Images Clean Images f ( x ) = x 105 106 Training Set Size 1 2 3 4 5 6 Cr os s E nt ro py = 0.43 = 0.32 = 0.49 ResNet50 + Clean Images ResNet50 + O ( 3 ) d-Images 105 106 Training Set Size 1 2 3 4 5 6 Cr os s E nt ro py = 0.45 = 0.36 = 0.17 = 0.08 ResNet101 + Clean Images ResNet101 + O ( 3 ) d-Images MixerSmall + Clean Images MixerSmall + O ( 3 ) d-Images Figure 4 : With coordinate of the input data rotated by O ( 3 ) d , state of the art models learn as good as without rotation . middle/right : slopes of the learning curves increases due to more data . DIDE DIDE for VECn . To understand the role of data , we vary the training set size of Cifar10 from about255 26 to 50k ( the whole un-augmented training set ) and to 100k ( adding left-right flip augmentation ) and256 plot the learning curves in Fig.3 . We observe dramatic speedup of learning for VECn in the larger257 data set regime , which isn ’ t the case for VEC∞ ( kernel ) , LCNn , GAP∞ and even for GAPn after258 m = 212 . We argue that this is due to the prior ( the function space defined by the model ) is too large259 ( and not optimal ) for the task and the coupled effect of more data together with inference procedures260 corrects the prior , as it is suggested by Theorem 2.1.261 DIDE for SOTA models . In the middle and right panels of Fig.S2 , we provide additional evidence262 in a larger scale setting . We generate learning curves of ImageNet using ResNet50 and MLP-Mixer,263 a very recent architecture that contains no convolution layers except the first layer , which is a264 convolution with filter size and stride equal to ( 16 , 16 ) ( patches are disjoint ) . The symmetry group265 associated to ResNet is similar to that of GAPn which is relatively small . However , the symmetry266 group induced by the first layer of the Mixer is O ( 3× 162 ) ⊗I142 , where 3×162 is number of entries267 in the ( 16 , 16 , 3 ) patch ( RGB channels ) and 142 = 2242/162 is the number of patches . Although268 the dimension of O ( 3× 162 ) ⊗ I142 is quite large ( about ( 3 × 162 ) 2/2 ) , it is still dramatically269 smaller than that of applying a fully-connected layer to the flatten images , which O ( 3× 2242 ) ( about270 ( 3× 2242 ) 2/2 ) . In the middle panel of Fig.S2 , we observe an almost perfect power-law scaling for271 the learning curve for the ResNet50 system with unrotated images . When the images are rotated by272 O ( 3 ) d ( d = 2242 ) , the learning curve is relatively flat in the smaller data regime ( green dashed line ) .273 However , the data set grows , it eventually catches up ( purple dashed line ) as that of the unrotated274 setting ; see Sec.E for ResNet34/101 . In the third panel , we see the learning curves are much flatter275 ( red ) for the Mixer and even more so for the rotated images ( green ) . Again , these curves are bent276 towards that of ResNet50 with unrorated images as data increases , indicating the prior was being277 corrected.278 Finally , in the left panel of Fig.S2 , we compare the accuracy of state-of-the-art models trained on both279 unrotated and O ( 3 ) d rotated images . Surprisingly , the gap between the two are not large and becomes280 smaller for better performant models . For EfficientNet B7 1 , the top-1 accuracy of the rotated system281 is only 1.2 % off from the unroated one.282 4 Eigenecomposition of Neural Kernels283 To get insights into the inductive biases , we eigendecompose the kernels using spherical harmon-284 ics . We assume the input space X = { ξ = ( ξ0 , . . . , ξp−1 ) ∈ ( √ d0S ( d0−1 ) ) p } ⊆ Rd0p , i.e.285 1Still under training the p-product of ( d0 − 1 ) -sphere with radius √ d0 . We call ξi ∈ √ d0S ( d0−1 ) a mini-patch and286 ( ξi , ξi+1 , . . . , ξi+s−1 ) ∈ ( √ d0S ( d0−1 ) ) s } a patch for i ∈ [ p ] , where circular boundary condition is287 assumed . We consider the asymptotic limit when d0 = dα , p = d1−α and d = pd0 →∞ and treat288 0 < α < 1 and s as fixed constant . The input space X is associated with the product measure289 µ ≡ σpd0 , where σd0 is the normalized uniform measure on √ d0S ( d0−1 ) . The kernels associated to290 the one-hidden layer infinite networks ( either NNGP or NTK ) has the following general forms291 k 1 p ∑ i∈ [ p ] ξTi ηi/d0 1 p ∑ i∈ [ p ] k 1 s ∑ b∈ [ s ] ξTi+bηi+b/d0 1 p2 ∑ i , j∈ [ p ] k 1 s ∑ b∈ [ s ] ξTi+bηj+b/d0 , ( 12 ) although that exact form of the ( positive definite ) kernel function k : R→ R depends on the kernel292 types ( NNGP vs NTK ) , activations , hyperparameters and etc . We assume the kernel is sufficiently293 smooth in ( −1 , 1 ) and the Tayor expansion of k ( r ) converges uniformly in [ −1 , 1 ] for sufficiently294 many r ∈ N. We use the notation that A ∼ B if there are positive constants c and C such that295 cA ≤ B ≤ CA for d sufficiently large . We use K to represent any kernels above and consider it as a296 Hilbert–Schmidt operator on L2 ( X , µ ) 297 Kf ( ξ ) = ∫ X K ( ξ , η ) f ( η ) dµ , f ∈ L2 ( X , µ ) , ( 13 ) which is well-defined since µ is a probability measure and k is bounded . Let ~r = ( r0 , . . . , rp−1 ) ∈ Np,298 τ the shifting operator τ~r = ( rp−1 , r0 , . . . , rp−2 ) . The s-banded subset of Np is defined to be299 B ( Np , s ) = { ~r ∈ Np : dist ( argmaxjrj 6= 0 , argminjrj 6= 0 ) ≤ s− 1 } ( 14 ) which is a quantifier used to restrict the support of a function on a patch . Here dist ( i , j ) = min { |i−300 j| , p− |i− j| } , a distance defined on the cyclic group [ p ] = Z/pZ . The quotient space B ( Np , s ) /τ301 denotes a subset of B ( Np , s ) by identifying ~v = ~v′ as the same element if ~v = τa~r′ for some a ∈ [ p ] .302 Finally , Yrj , lj ( ξj ) is used to denote the lj-th spherical harmonic of degree rj in the unit sphere303 S ( d0−1 ) and has unit norm under the normalized measure on S ( d0−1 ) . As such Yrj , lj ( ξj/ √ d0 ) ∈304 L2 ( √ d0S ( d0−1 ) , σd0 ) has unit norm . Recall that the total number of spherical harmonic of degree305 rj in S ( d0−1 ) is N ( d0 , rj ) = ( 2rj + d0 − 2 ) ( d0+rj−3 rj−1 ) ∼ drj0 /rj ! as d0 →∞ . We use N ( d0 , ~r ) =306 ∏ j∈ [ p ] N ( d0 , rj ) and [ N ( d0 , ~r ) ] = ∏ j∈ [ p ] [ N ( d0 , rj ) ] , resp . Let307 ~Y~r , ~l ( ξ ) = ∏ j∈ [ p ] Yrj , lj ( ξj ) ( 15 ) The following theorem shows that locality ( VEC∞ ) dramatically reduces both the dimensions of308 Eigendecomposition r ≥ 1 eigenspaces and the spectral gap between them . In addition , pooling309 ( i.e . translation symmetry of GAPn ) reduces their dimensions by a factor of p. See Sec.E for the310 implication of this theorem to learning.311 Theorem 4.1 . [ Sec.D ] We have the following eigendecomposition for the integral operator K312 H = ⋃ r∈N H ( r ) = ⋃ r∈N ⋃ ~r∈Q ( K , r ) H ( ~r ) , ( 16 ) where Q ( K , r ) is a quantifier defined below . If r = 0 , H ( 0 ) is the space of constant functions and the313 eigenvalue is ∼ k ( 0 ) . For r ≥ 1,314 1. if K = KFCN , then Q ( K , r ) = { ~r ∈ Np : |~r| = r } and the unit eigenfunctions are315 H ( ~r ) = span { Y~r , ~l } ~l∈ [ B ( d0 , ~r ) ] dim ( H ( r ) ) ∼ dr and λ ( H ( ~r ) ) ∼ d−rδ ( k ( r ) ( 0 ) ) ( 17 ) 2. if K = KVEC , Q ( K , r ) = { ~r ∈ B ( Np , s ) : |~r| = r } the unit eigenfunctions are316 H ( ~r ) VEC = span { Y~r , ~l } ~l∈ [ B ( d0 , ~r ) ] dim ( H ( r ) VEC ) ∼ p ( sd0 ) r = srd1−α+rα and λ ( H ( ~r ) VEC ) ∼ p−1 ( sd0 ) −rδ ( k ( r ) ( 0 ) ) ( 18 ) 3. and finally , if K = KGAP , then Q ( K , r ) = { ~r ∈ B ( Np , s ) /τ : |~r| = r } , the unit eigenfunc-317 tions are318 H ( ~r ) GAP = span { 1√ p ∑ τ∈ [ p ] Y~r , ~l ( τξ ) } ~l∈ [ B ( d0 , ~r ) ] dim ( H ( r ) GAP ) ∼ ( sd0 ) r = srdrα and λ ( H ( ~r ) GAP ) ∼ p−1 ( sd0 ) −rδ ( k ( r ) ( 0 ) ) ( 19 ) 5 Related Work319 The study of infinite networks dates back to seminal work by Neal [ 8 ] who showed the convergence of320 single hidden-layer networks to Gaussian Processes ( GPs ) . Recently , there has been renewed interest321 in studying random , infinite , networks starting with concurrent work on “ conjugate kernels ” [ 10 , 35 ] 322 and “ mean-field theory ” [ 9 , 36 ] , taking a statistical learning and statistical physics view of points,323 resp . Since then this analysis has been extended to include a wide range for architectures [ 20 , 21 , 37,324 29 , 26 , 38 ] . The inducing kernel is often referred to as the Neural Network Gaussian Process ( NNGP ) 325 kernel . The neural tangent kernel ( NTK ) , first introduced in Jacot et al . [ 22 ] , along with followup326 work [ 12 , 39 ] showed that the distribution of functions induced by gradient descent for infinite-width327 networks is a Gaussian Process with NTK as the kernel.328 The study of implicit bias ( regularization ) of gradient descent has received considerable interests.329 The work [ 15 , 40–43 ] demonstrate the convergence of SGD to the maximal margin solution for330 logistic-type of losses during late time training . [ 44–50 ] study the early-time SGD dynamics , spectral331 biases of neural networks . These results aim to explain the order of learning of neural networks:332 functions of less complexity are usually learned before more complex functions.333 [ 27 ] is the first to show that the prediction functions obtained from training FCN depend , in addition334 on the labels , only on the covariance of the input data . This implies our result regarding the O ( 3d ) 335 invariance of FCN . By utilizing this symmetry , recent work [ 51 ] constructs a particular task where336 the label function is a second order polynomial of the inputs and show that orthogonal invariance337 algorithm requires sample size of order d2 while there is a convnet requires only O ( 1 ) samples . Their338 convnet essentially corresponds to the d0 = s = 1 and r = 2 case of Theorem 4.1 , in which the339 dimension of this eigenspace ( and indeed of all r-eigenspace by treating r as a finite constant as340 d→∞ ) of GAP∞ is O ( 1 ) while the dimension of the 2-eigenspace of FCN∞ is of order d2.341 6 Conclusion342 In this paper , we consider machine learning methods as an integrated system of data , models and343 inference algorithms and study the basic symmetries of various machine learning systems . We surface344 the importance of locality in modern machine learning systems through large scale empirical study345 and through an eigendecomposition of one-layer infinite networks . However , we haven ’ t addressed346 the two import questions ( 1 ) theoretical characterization of the effect of composing locality and ( 2 ) 347 the mathematical understanding of DIDE and how the prior is corrected by the coupled effect of data348 and gradient descent . We leave them to future work.349 Checklist350 The checklist follows the references . Please read the checklist guidelines carefully for information on351 how to answer these questions . For each question , change the default [ TODO ] to [ Yes ] , [ No ] , or352 [ N/A ] . You are strongly encouraged to include a justification to your answer , either by referencing353 the appropriate section of your paper or providing a brief inline description . For example:354 • Did you include the license to the code and datasets ? [ Yes ] See Section ? ? .355 • Did you include the license to the code and datasets ? [ No ] The code and the data are356 proprietary.357 • Did you include the license to the code and datasets ? [ N/A ] 358 Please do not modify the questions and only use the provided macros for your answers . Note that the359 Checklist section does not count towards the page limit . In your paper , please delete this instructions360 block and only keep the Checklist section heading above along with the questions/answers below.361 1 . For all authors ... 362 ( a ) Do the main claims made in the abstract and introduction accurately reflect the paper ’ s363 contributions and scope ? [ Yes ] 364 ( b ) Did you describe the limitations of your work ? [ Yes ] 365 ( c ) Did you discuss any potential negative societal impacts of your work ? [ N/A ] 366 ( d ) Have you read the ethics review guidelines and ensured that your paper conforms to367 them ? [ Yes ] 368 2 . If you are including theoretical results ... 369 ( a ) Did you state the full set of assumptions of all theoretical results ? [ Yes ] 370 ( b ) Did you include complete proofs of all theoretical results ? [ Yes ] 371 3 . If you ran experiments ... 372 ( a ) Did you include the code , data , and instructions needed to reproduce the main experi-373 mental results ( either in the supplemental material or as a URL ) ? [ No ] 374 ( b ) Did you specify all the training details ( e.g. , data splits , hyperparameters , how they375 were chosen ) ? [ No ] 376 ( c ) Did you report error bars ( e.g. , with respect to the random seed after running experi-377 ments multiple times ) ? [ Yes ] 378 ( d ) Did you include the total amount of compute and the type of resources used ( e.g. , type379 of GPUs , internal cluster , or cloud provider ) ? [ No ] 380 4 . If you are using existing assets ( e.g. , code , data , models ) or curating/releasing new assets ... 381 ( a ) If your work uses existing assets , did you cite the creators ? [ TODO ] 382 ( b ) Did you mention the license of the assets ? [ TODO ] 383 ( c ) Did you include any new assets either in the supplemental material or as a URL ? 384 [ TODO ] 385 ( d ) Did you discuss whether and how consent was obtained from people whose data you ’ re386 using/curating ? [ TODO ] 387 ( e ) Did you discuss whether the data you are using/curating contains personally identifiable388 information or offensive content ? [ TODO ] 389 5 . If you used crowdsourcing or conducted research with human subjects ... 390 ( a ) Did you include the full text of instructions given to participants and screenshots , if391 applicable ? [ TODO ] 392 ( b ) Did you describe any potential participant risks , with links to Institutional Review393 Board ( IRB ) approvals , if applicable ? [ TODO ] 394 ( c ) Did you include the estimated hourly wage paid to participants and the total amount395 spent on participant compensation ? [ TODO ] 396 References397 [ 1 ] Alex Krizhevsky , Ilya Sutskever , and Geoffrey E Hinton . Imagenet classification with deep convolutional398 neural networks . In Advances in neural information processing systems , pages 1097–1105 , 2012.399 [ 2 ] Ashish Vaswani , Noam Shazeer , Niki Parmar , Jakob Uszkoreit , Llion Jones , Aidan N Gomez , Lukasz400 Kaiser , and Illia Polosukhin . Attention is all you need . arXiv preprint arXiv:1706.03762 , 2017.401 [ 3 ] Jacob Devlin , Ming-Wei Chang , Kenton Lee , and Kristina Toutanova . Bert : Pre-training of deep bidirec-402 tional transformers for language understanding . arXiv preprint arXiv:1810.04805 , 2018.403 [ 4 ] David Silver , Aja Huang , Chris J Maddison , Arthur Guez , Laurent Sifre , George Van Den Driessche , Julian404 Schrittwieser , Ioannis Antonoglou , Veda Panneershelvam , Marc Lanctot , et al . Mastering the game of go405 with deep neural networks and tree search . Nature , 529 ( 7587 ) :484–489 , 2016.406 [ 5 ] Andrew W Senior , Richard Evans , John Jumper , James Kirkpatrick , Laurent Sifre , Tim Green , Chongli407 Qin , Augustin Žídek , Alexander WR Nelson , Alex Bridgland , et al . Improved protein structure prediction408 using potentials from deep learning . Nature , 577 ( 7792 ) :706–710 , 2020.409 [ 6 ] Jared Kaplan , Sam McCandlish , Tom Henighan , Tom B Brown , Benjamin Chess , Rewon Child , Scott Gray,410 Alec Radford , Jeffrey Wu , and Dario Amodei . Scaling laws for neural language models . arXiv preprint411 arXiv:2001.08361 , 2020.412 [ 7 ] Yann LeCun , Léon Bottou , Yoshua Bengio , and Patrick Haffner . Gradient-based learning applied to413 document recognition . Proceedings of the IEEE , 86 ( 11 ) :2278–2324 , 1998.414 [ 8 ] Radford M. Neal . Priors for infinite networks ( tech . rep. no . crg-tr-94-1 ) . University of Toronto , 1994.415 [ 9 ] Ben Poole , Subhaneil Lahiri , Maithra Raghu , Jascha Sohl-Dickstein , and Surya Ganguli . Exponential416 expressivity in deep neural networks through transient chaos . In Advances In Neural Information Processing417 Systems , 2016.418 [ 10 ] Amit Daniely , Roy Frostig , and Yoram Singer . Toward deeper understanding of neural networks : The419 power of initialization and a dual view on expressivity . In Advances In Neural Information Processing420 Systems , 2016.421 [ 11 ] Arthur Jacot , Franck Gabriel , and Clément Hongler . Neural tangent kernel : Convergence and generalization422 in neural networks . arXiv preprint arXiv:1806.07572 , 2018.423 [ 12 ] Jaehoon Lee , Lechao Xiao , Samuel S. Schoenholz , Yasaman Bahri , Roman Novak , Jascha Sohl-Dickstein,424 and Jeffrey Pennington . Wide neural networks of any depth evolve as linear models under gradient descent.425 In Advances in Neural Information Processing Systems , 2019.426 [ 13 ] Behnam Neyshabur . Implicit regularization in deep learning . arXiv preprint arXiv:1709.01953 , 2017.427 [ 14 ] Suriya Gunasekar , Jason Lee , Daniel Soudry , and Nathan Srebro . Implicit bias of gradient descent on428 linear convolutional networks . arXiv preprint arXiv:1806.00468 , 2018.429 [ 15 ] Daniel Soudry , Elad Hoffer , Mor Shpigel Nacson , Suriya Gunasekar , and Nathan Srebro . The implicit bias430 of gradient descent on separable data , 2018.431 [ 16 ] Mingxing Tan and Quoc Le . Efficientnet : Rethinking model scaling for convolutional neural networks . In432 International Conference on Machine Learning , pages 6105–6114 . PMLR , 2019.433 [ 17 ] Alex J Smola , Zoltan L Ovari , Robert C Williamson , et al . Regularization with dot-product kernels.434 Advances in neural information processing systems , pages 308–314 , 2001.435 [ 18 ] Kunihiko Fukushima . Cognitron : A self-organizing multilayered neural network . Biological cybernetics,436 20 ( 3-4 ) :121–136 , 1975.437 [ 19 ] Yann Lecun . Generalization and network design strategies . In Connectionism in perspective . Elsevier,438 1989.439 [ 20 ] Jaehoon Lee , Yasaman Bahri , Roman Novak , Sam Schoenholz , Jeffrey Pennington , and Jascha Sohl-440 dickstein . Deep neural networks as gaussian processes . In International Conference on Learning Repre-441 sentations , 2018.442 [ 21 ] Alexander G. de G. Matthews , Jiri Hron , Mark Rowland , Richard E. Turner , and Zoubin Ghahramani.443 Gaussian process behaviour in wide deep neural networks . In International Conference on Learning444 Representations , 2018.445 [ 22 ] Arthur Jacot , Franck Gabriel , and Clement Hongler . Neural tangent kernel : Convergence and generalization446 in neural networks . In Advances in Neural Information Processing Systems , 2018.447 [ 23 ] Simon S Du , Jason D Lee , Haochuan Li , Liwei Wang , and Xiyu Zhai . Gradient descent finds global448 minima of deep neural networks . arXiv preprint arXiv:1811.03804 , 2018.449 [ 24 ] Zeyuan Allen-Zhu , Yuanzhi Li , and Zhao Song . A convergence theory for deep learning via over-450 parameterization . In International Conference on Machine Learning , 2018.451 [ 25 ] Difan Zou , Yuan Cao , Dongruo Zhou , and Quanquan Gu . Gradient descent optimizes over-parameterized452 deep relu networks . Machine Learning , 109 ( 3 ) :467–492 , 2020.453 [ 26 ] Greg Yang . Scaling limits of wide neural networks with weight sharing : Gaussian process behavior,454 gradient independence , and neural tangent kernel derivation . arXiv preprint arXiv:1902.04760 , 2019.455 [ 27 ] Neha S. Wadia , Daniel Duckworth , Samuel S. Schoenholz , Ethan Dyer , and Jascha Sohl-Dickstein.456 Whitening and second order optimization both destroy information about the dataset , and can make457 generalization impossible . arxiv preprint arXiv:2008.07545 , 2020.458 [ 28 ] Lechao Xiao , Yasaman Bahri , Jascha Sohl-Dickstein , Samuel Schoenholz , and Jeffrey Pennington . Dy-459 namical isometry and a mean field theory of cnns : How to train 10,000-layer vanilla convolutional neural460 networks . In International Conference on Machine Learning , pages 5393–5402 , 2018.461 [ 29 ] Roman Novak , Lechao Xiao , Jaehoon Lee , Yasaman Bahri , Greg Yang , Jiri Hron , Daniel A. Abolafia,462 Jeffrey Pennington , and Jascha Sohl-Dickstein . Bayesian deep convolutional networks with many channels463 are gaussian processes . In International Conference on Learning Representations , 2019.464 [ 30 ] Adrià Garriga-Alonso , Laurence Aitchison , and Carl Edward Rasmussen . Deep convolutional networks as465 shallow gaussian processes . In International Conference on Learning Representations , 2019.466 [ 31 ] Roman Novak , Lechao Xiao , Jiri Hron , Jaehoon Lee , Alexander A Alemi , Jascha Sohl-Dickstein , and467 Samuel S Schoenholz . Neural tangents : Fast and easy infinite neural networks in python . arXiv preprint468 arXiv:1912.02803 , 2019.469 [ 32 ] Kaiming He , Xiangyu Zhang , Shaoqing Ren , and Jian Sun . Deep residual learning for image recognition.470 In Conference on Computer Vision and Pattern Recognition , pages 770–778 , 2016.471 [ 33 ] Ilya Tolstikhin , Neil Houlsby , Alexander Kolesnikov , Lucas Beyer , Xiaohua Zhai , Thomas Unterthiner,472 Jessica Yung , Daniel Keysers , Jakob Uszkoreit , Mario Lucic , et al . Mlp-mixer : An all-mlp architecture for473 vision . arXiv preprint arXiv:2105.01601 , 2021.474 [ 34 ] Hongyi Zhang , Moustapha Cisse , Yann N. Dauphin , and David Lopez-Paz . mixup : Beyond empirical475 risk minimization . In International Conference on Learning Representations , 2018 . URL https : //476 openreview.net/forum ? id=r1Ddp1-Rb.477 [ 35 ] Amit Daniely . SGD learns the conjugate kernel class of the network . In Advances in Neural Information478 Processing Systems 30 . 2017.479 [ 36 ] Samuel S Schoenholz , Justin Gilmer , Surya Ganguli , and Jascha Sohl-Dickstein . Deep information480 propagation . International Conference on Learning Representations , 2017.481 [ 37 ] Lechao Xiao , Yasaman Bahri , Jascha Sohl-Dickstein , Samuel Schoenholz , and Jeffrey Pennington . Dy-482 namical isometry and a mean field theory of CNNs : How to train 10,000-layer vanilla convolutional neural483 networks . In International Conference on Machine Learning , 2018.484 [ 38 ] Jiri Hron , Yasaman Bahri , Jascha Sohl-Dickstein , and Roman Novak . Infinite attention : Nngp and ntk for485 deep attention networks , 2020.486 [ 39 ] Lenaic Chizat , Edouard Oyallon , and Francis Bach . On lazy training in differentiable programming . In487 Advances in Neural Information Processing Systems , pages 2937–2947 , 2019.488 [ 40 ] Kaifeng Lyu and Jian Li . Gradient descent maximizes the margin of homogeneous neural networks . In489 International Conference on Learning Representations , 2020 . URL https : //openreview.net/forum ? 490 id=SJeLIgBKPS.491 [ 41 ] Ziwei Ji and Matus Telgarsky . The implicit bias of gradient descent on nonseparable data . In Conference492 on Learning Theory , pages 1772–1798 , 2019.493 [ 42 ] Ziwei Ji and Matus Jan Telgarsky . Gradient descent aligns the layers of deep linear networks . In 7th494 International Conference on Learning Representations , ICLR 2019 , 2019.495 [ 43 ] Lénaïc Chizat and Francis Bach . Implicit bias of gradient descent for wide two-layer neural networks496 trained with the logistic loss . In Jacob Abernethy and Shivani Agarwal , editors , Proceedings of Thirty497 Third Conference on Learning Theory , volume 125 of Proceedings of Machine Learning Research , pages498 1305–1338 . PMLR , 09–12 Jul 2020 . URL http : //proceedings.mlr.press/v125/chizat20a.html.499 [ 44 ] Preetum Nakkiran , Gal Kaplun , Dimitris Kalimeris , Tristan Yang , Benjamin L Edelman , Fred Zhang,500 and Boaz Barak . Sgd on neural networks learns functions of increasing complexity . arXiv preprint501 arXiv:1905.11604 , 2019.502 [ 45 ] Wei Hu , Lechao Xiao , Ben Adlam , and Jeffrey Pennington . The surprising simplicity of the early-time503 learning dynamics of neural networks . arXiv preprint arXiv:2006.14599 , 2020.504 [ 46 ] Nasim Rahaman , Aristide Baratin , Devansh Arpit , Felix Draxler , Min Lin , Fred Hamprecht , Yoshua505 Bengio , and Aaron Courville . On the spectral bias of neural networks . In International Conference on506 Machine Learning , pages 5301–5310 . PMLR , 2019.507 [ 47 ] Zhiqin John Xu . Understanding training and generalization in deep learning by fourier analysis . arXiv508 preprint arXiv:1808.04295 , 2018.509 [ 48 ] Zhi-Qin John Xu , Yaoyu Zhang , Tao Luo , Yanyang Xiao , and Zheng Ma . Frequency principle : Fourier510 analysis sheds light on deep neural networks . arXiv preprint arXiv:1901.06523 , 2019.511 [ 49 ] Lili Su and Pengkun Yang . On learning over-parameterized neural networks : A functional approximation512 perspective . arXiv preprint arXiv:1905.10826 , 2019.513 [ 50 ] Greg Yang and Hadi Salman . A fine-grained spectral perspective on neural networks . arXiv preprint514 arXiv:1907.10599 , 2019.515 [ 51 ] Zhiyuan Li , Yi Zhang , and Sanjeev Arora . Why are convolutional nets more sample-efficient than516 fully-connected nets ? arXiv preprint arXiv:2010.08515 , 2020.517

Summary: The paper proposed an viewpoint to analyze deep learning methods as an integrated system of data (D), models (M), and inference algorithms (I) by studying the symmetric of the triplet (D, M, I) (in terms of transformations) on specific network models for vision including fully/locally connected networks (F/LCN), convolutional network w/o pooling(VEC/GAP). The key result in symmetry is that when networks are initialized with i.i.d. standard Gaussian distribution, then the considered neural networks are invariant against orthogonal group on the flatten input space when networks are over-parameterized.