On Symmetry and Initialization for Neural Networks

1 INTRODUCTION . Building a theory that can help to understand neural networks and guide their construction is one of the current challenges of machine learning . Here we wish to shed some light on the role symmetry plays in the construction of neural networks . It is well-known that symmetry can be used to enhance the performance of neural networks . For example , convolutional neural networks ( CNNs ) ( see Lecun et al . ( 1998 ) ) use the translational symmetry of images to classify images better than fully connected neural networks . Our focus is on the role of symmetry in the initialization stage . We show that symmetry-based initialization can be the difference between failure and success . On a high-level , the study of neural networks can be partitioned to three different aspects . Expressiveness Given an architecture , what are the functions it can approximate well ? Training Given a network with a “ proper ” architecture , can the network fit the training data and in a reasonable time ? Generalization Given that the training seemed successful , will the true error be small as well ? We study these aspects for the first “ non trivial ” case of neural networks , networks with one hidden layer . We are mostly interested in the initialization phase . If we take a network with the appropriate architecture , we can always initialize it to the desired function . A standard method ( that induces a non trivial learning problem ) is using random weights to initialize the network . A different reasonable choice is to require the initialization to be useful for an entire class of functions . We follow the latter option . Our focus is on the role of symmetry . We consider the following class of symmetric functions S= Sn = { n ∑ i=0 ai ·1|x|=i : a1 , . . . , an ∈ { ±1 } } , where x ∈ { 0,1 } n and |x| = ∑i xi . The functions in this class are invariant under arbitrary permutations of the input ’ s coordinates . The parity function π ( x ) = ( −1 ) |x| and the majority function are well-known examples of symmetric functions . Expressiveness for this class was explored by Minsky and Papert ( 1988 ) . They showed that the parity function can not be represented using a network with limited “ connectivity ” . Contrastingly , if we use a fully connected network with one hidden layer and a common activation function ( like sign , sigmoid , or ReLU ) only O ( n ) neurons are needed . We provide such explicit representations for all functions in S ; see Lemmas 1 and 2 . We also provide useful information on both the training phase and generalization capabilities of the neural network . We show that , with proper initialization , the training process ( using standard SGD ) efficiently converges to zero empirical error , and that consequently the network has small true error as well . Theorem 1 . There exists a constant c > 1 so that the following holds . There exists a network with one hidden layer , cn neurons with sigmoid or ReLU activations , and an initialization such that for all distributions D over X = { 0,1 } n and all functions f ∈ S with sample size m≥ c ( n+ log ( 1/δ ) ) /ε , after performing poly ( n ) SGD updates with a fixed step size h = 1/poly ( n ) it holds that P xm∼Dm ( { S : Pr x∼D ( NS ( x ) 6= f ( x ) ) > ε } ) < δ where S = { ( x1 , f ( x1 ) ) , ... , ( xm , f ( xm ) ) } and NS ( x ) is the network after training over S. The number of parameters in the network described in Theorem 1 is Ω ( n2 ) . So in general one could expect overfitting when the sample size is as small as O ( n ) . Nevertheless , the theorem provides generalization guarantees , even for such a small sample size . The initialization phase plays an important role in proving Theorem 1 . To emphasize this , we report an empirical phenomenon ( this is “ folklore ” ) . We show that a network can not learn parity from a random initialization ( see Section 5.3 ) . On one hand , if the network size is big , we can bring the empirical error to zero ( as suggested in Soudry and Carmon ( 2016 ) ) , but the true error is close to 1/2 . On the other hand , if its size is too small , the network is not even able to achieve small empirical error ( see Figure 5 ) . We observe a similar phenomenon also for a random symmetric function . An open question remains : why is it true that a sample of size polynomial in n does not suffice to learn parity ( with random initialization ) ? A similar phenomenon was theoretically explained by Shamir ( 2016 ) and Song et al . ( 2017 ) . The parity function belongs to the class of all parities P= Pn = { πs ( x ) = ( −1 ) s·x : s ∈ X } where · is the standard inner product . This class is efficiently PAC-learnable with O ( n ) samples using Gaussian elimination . A continuous version of P was studied by Shamir ( 2016 ) and Song et al . ( 2017 ) . To study the training phase , they used a generalized notion of statistical queries ( SQ ) ; see Kearns ( 1998 ) . In this framework , they show that most functions in the class P can not be efficiently learned ( roughly stated , learning the class requires an exponential amount of resources ) . This framework , however , does not seem to capture actual training of neural networks using SGD . For example , it is not clear if one SGD update corresponds to a single query in this model . In addition , typically one receives a dataset and performs the training by going over it many times , whereas the query model estimates the gradient using a fresh batch of samples in each iteration . The query model also assumes the noise to be adversarial , an assumption that does not necessarily hold in reality . Finally , the SQ-based lower bound holds for every initialization ( in particular , for the initialization we use here ) , so it does not capture the efficient training process Theorem 1 describes . Theorem 1 shows , however , that with symmetry-based initialization , parity can be efficiently learned . So , in a nutshell , parity can not be learned as part of P , but it can be learned as part of S. One could wonder why the hardness proof for P can not be applied for S as both classes consist of many input sensitive functions . The answer lies in the fact that P has a far bigger statistical dimension than S ( all functions in P are orthogonal to each other , unlike S ) . The proof of the theorem utilizes the different behavior of the two layers in the network . SGD is performed using a step size h that is polynomially small in n. The analysis shows that in a polynomial number of steps that is independent of the choice of h the following two properties hold : ( i ) the output neuron reaches a “ good ” state and ( ii ) the hidden layer does not change in a “ meaningful ” way . These two properties hold when h is small enough . In Section 5.2 , we experiment with large values of h. We see that , although the training error is zero , the true error becomes large . Here is a high level description of the proof . The ` neurons in the hidden layer define an “ embedding ” of the inputs space X = { 0,1 } n into R ` ( a.k.a . the feature map ) . This embedding changes in time according to the training examples and process . The proof shows that if at any point in time this embedding has good enough margin , then training with standard SGD quickly converges . This is explained in more detail in Section 3 . It remains an interesting open problem to understand this phenomenon in greater generality , using a cleaner and more abstract language . 1.1 BACKGROUND . To better understand the context of our research , we survey previous related works . The expressiveness and limitations of neural networks were studied in several works such as Rahimi and Recht ( 2008 ) ; Telgarsky ( 2016 ) ; Eldan and Shamir ( 2016 ) and Arora et al . ( 2016 ) . Constructions of small ReLU networks for the parity function appeared in several previous works , such as Wilamowski et al . ( 2003 ) , Arslanov et al . ( 2016 ) , Arslanov et al . ( 2002 ) and Masato Iyoda et al . ( 2003 ) . Constant depth circuits for the parity function were also studied in the context of computational complexity theory , see for example Furst et al . ( 1981 ) , Ajtai ( 1983 ) and Håstad ( 1987 ) . The training phase of neural networks was also studied in many works . Here we list several works that seem most related to ours . Daniely ( 2017 ) analyzed SGD for general neural network architecture and showed that the training error can be nullified , e.g. , for the class of bounded degree polynomials ( see also Andoni et al . ( 2014 ) ) . Jacot et al . ( 2018 ) studied neural tangent kernels ( NTK ) , an infinite width analogue of neural networks . Du et al . ( 2018 ) showed that randomly initialized shallow ReLU networks nullify the training error , as long as the number of samples is smaller than the number of neurons in the hidden layer . Their analysis only deals with optimization over the first layer ( so that the weights of the output neuron are fixed ) . Chizat and Bach ( 2018 ) provided another analysis of the latter two works . Allen-Zhu et al . ( 2018b ) showed that over-parametrized neural networks can achieve zero training error , as as long as the data points are not too close to one another and the weights of the output neuron are fixed . Zou et al . ( 2018 ) provided guarantees for zero training error , assuming the two classes are separated by a positive margin . Convergence and generalization guarantees for neural networks were studied in the following works . Brutzkus et al . ( 2017 ) studied linearly separable data . Li and Liang ( 2018 ) studied well separated distributions . Allen-Zhu et al . ( 2018a ) gave generalization guarantees in expectation for SGD . Arora et al . ( 2019 ) gave data-dependent generalization bounds for GD . All these works optimized only over the hidden layer ( the output layer is fixed after initialization ) . Margins play an important role in learning , and we also use it in our proof . Sokolic et al . ( 2016 ) , Sokolic et al . ( 2017 ) , Bartlett et al . ( 2017 ) and Sun et al . ( 2015 ) gave generalization bounds for neural networks that are based on their margin when the training ends . From a practical perspective , Elsayed et al . ( 2018 ) , Romero and Alquezar ( 2002 ) and Liu et al . ( 2016 ) suggested different training algorithms that optimize the margin . As discussed above , it seems difficult for neural networks to learn parities . Song et al . ( 2017 ) and Shamir ( 2016 ) demonstrated this using the language statistical queries ( SQ ) . This is a valuable language , but it misses some central aspects of training neural networks . SQ seems to be closely related to GD , but does not seem to capture SGD . SQ also shows that many of the parities functions ⊗i∈Sxi are difficult to learn , but it does not imply that the parity function ⊗i∈ [ n ] xi is difficult to learn . Abbe and Sandon ( 2018 ) demonstrated a similar phenomenon in a setting that is closer to the “ real life ” mechanics of neural networks . We suggest that taking the symmetries of the learning problem into account can make the difference between failure and success . Several works suggested different neural architectures that take symmetries into account ; see Zaheer et al . ( 2017 ) , Gens and Domingos ( 2014 ) , and Cohen and Welling ( 2016 ) .

Summary: The paper studies the problem of learning the class of symmetric boolean function, that is, functions that depend only on |x| = \sum_i x_i. The paper shows that with proper initialization, one-hidden layer over-parametrized networks can learn this class of functions. The main observation that the authors make is that the last layer weights are updated as in the Perceptron algorithm and as long as the first layer has learned a large-margin representation, the first-layer weights do not change much. The authors experimentally validate their theory and additionally show that random initialization fails to converge to a low test-error solution while their special initialization works.