Neural networks are a priori biased towards Boolean functions with low entropy

1 INTRODUCTION . In order to generalise beyond training data , learning algorithms need some sort of inductive bias . The particular form of the inductive bias dictates the performance of the algorithm . For one of the most important machine learning techniques , deep neural networks ( DNNs ) ( LeCun et al. , 2015 ) , sources of inductive bias can include the architecture of the networks , e.g . the number of layers , how they are connected , say as a fully connected network ( FCN ) or as a convolutional neural net ( CNN ) , and the type of optimisation algorithm used , e.g . stochastic gradient descent ( SGD ) versus full gradient descent ( GD ) . Many further methods such as dropout ( Srivastava et al. , 2014 ) , weight decay ( Krogh & Hertz , 1992 ) and early stopping ( Morgan & Bourlard , 1990 ) have been proposed as techniques to improve the inductive bias towards desired solutions that generalise well . What is particularly surprising about DNNs is that they are highly expressive and work well in the heavily overparameterised regime where traditional learning theory would predict poor generalisation due to overfitting ( Zhang et al. , 2016 ) . DNNs must therefore have a strong intrinsic bias that allows for good generalisation , in spite of being in the overparameterised regime . Here we study the intrinsic bias of the parameter-function map for neural networks , defined in ( VallePérez et al. , 2018 ) as the map between a set of parameters and the function that the neural network represents . In particular , we define the a-priori probability P ( f ) of a DNN as the probability that a particular function f is produced upon random sampling ( or initialisation ) of the weight and threshold bias parameters . The prior at initialization , P ( f ) , should inform the inductive bias of SGD-trained neural networks , as long as SGD approximates Bayesian inference with P ( f ) as prior sufficiently well Valle-Pérez et al . ( 2018 ) . We explain this connection further , and give some evidence supporting this behavior of SGD , in Appendix L. This supports the idea studying neural networks with random parameters Poole et al . ( 2016 ) ; Lee et al . ( 2018 ) ; Schoenholz et al . ( 2017 ) ; Garriga-Alonso et al . ( 2018 ) ; Novak et al . ( 2018 ) is not just relevant to find good initializations for optimization , but also to understand their generalization . A naive null-model for P ( f ) might suggest that without further information , one should expect that all functions are equally likely . However , recent very general arguments ( Dingle et al. , 2018 ) based on the coding theorem from Algorithmic Information Theory ( AIT ) ( Li et al. , 2008 ) have instead suggested that for a wide range of maps M that obey a number of conditions such as being simple ( they have a low Kolmogorov complexity K ( M ) ) and redundancy ( multiple inputs map to the same output ) then if they are sufficiently biased , they will be exponentially biased towards outputs of low Kolmogorov complexity . The parameter-function map of neural networks satisfies these conditions , and it was found empirically ( Valle-Pérez et al. , 2018 ) that , as predicted in ( Dingle et al. , 2018 ) , the probability P ( f ) of obtaining a function f upon random sampling of parameter weights satisfies the following simplicity-bias bound P ( f ) . 2− ( bK̃ ( f ) +a ) , ( 1 ) where K̃ ( f ) is a computable approximation of the true Kolmogorov complexity K ( f ) , and a and b are constants that depend on the network , but not on the functions . It is widely expected that real world data is highly structured , and so has a relatively low Kolmogorov complexity ( Hinton & Van Camp , 1993 ; Schmidhuber , 1997 ) . The simplicity bias described above may therefore be an important source of the inductive bias that allows DNNs to generalise so well ( and not overfit ) in the highly over-parameterised regime ( Valle-Pérez et al. , 2018 ) . Nevertheless , this bound has limitations . Firstly , the only rigorously proven result is for the true Kolmogorov complexity version of the bound in the case of large enough K ( f ) . Although it has been found to work remarkably well for small systems and computable approximations to Kolmogorov complexity ( Valle-Pérez et al. , 2018 ; Dingle et al. , 2018 ) , this success is not yet fully understood theoretically . Secondly , it does not explain why models like DNNs are biased ; it only explains that , if they are biased , they should be biased towards simplicity . Also , the AIT bound is very general – it predicts a probability P ( f ) that depends mainly on the function , and only weakly on the network . It may therefore not capture some variations in the bias that are due to details of the network architecture , and which may be important for practical applications . For these reasons it is of interest to obtain a finer quantitative understanding of the simplicity bias of neural networks . Some work has been done in this direction , showing that infinitely wide neural networks are biased towards functions which are robust to changes in the input ( De Palma et al. , 2018 ) , showing that “ flatness ” is connected to function smoothness ( Wu et al. , 2016 ) , or arguing that low Fourier frequencies are learned first by a ReLU neural network ( Rahaman et al. , 2018 ; Yang & Salman , 2019 ) . All of these papers take some notion of “ smoothness ” as tractable proxy for the complexity of a function . One generally expects smoother functions to be simpler , although this is clearly a very rough measure of the Kolmogorov complexity . 2 SUMMARY OF KEY RESULTS . In this paper we study how likely different Boolean functions , defined as f : { 0 , 1 } n → { 0 , 1 } , are obtained upon randomly chosen weights of neural networks . Our key results are aimed at fleshing out with more precision and rigour what the inductive biases of ( very ) simple neural networks are , and how they arise For this , we study the prior distribution over functions P ( f ) , upon random initialization of the parameters , which reflects the inductive bias of training algorithms that approximate Bayesian inference ( see Appendix L for a detailed explanation , and data on how well SGD follows this behaviour ) . We focus our study on a notion of complexity , namely the “ entropy , ” H ( f ) , of a Boolean function f , defined as the binary entropy of the fraction of possible inputs to f that f maps to 1 . This quantity essentially measures the amount of class imbalance of the function , and is complementary to previous works studying notions of smoothness as a proxy for complexity . 1 . In Section 4 we study a simple perceptron with no threshold bias term , and with weights w sampled from a distribution which is symmetric under reflections along the coordinate axes . Let the random variable T correspond to the number of points in { 0 , 1 } n which that fall above the decision boundary of the network ( i.e . T=| { x ∈ { 0 , 1 } n : 〈w , x〉 > 0 } | ) upon i.i.d . random initialisation of the weights . We prove that T is distributed uniformly , i.e . P ( T = t ) = 2−n for 0 ≤ t < 2n . Let Ft be the set of all functions with T = t that the perceptron can produce and let |Ft| be its size ( cf . Definition 3.4 ) . We expect |Ft| for t ∼ 2n−1 ( high entropy ) to be ( much ) larger than |Ft| for extreme values of t ( low entropy ) . The average probability of obtaining a particular function f which maps t inputs to 1 is 2−n/|Ft| . The perceptron therefore shows a strong bias towards functions with low entropy , in the sense that individual functions with low entropy have , on average , higher probability than individual functions with high entropy . 2 . In Section 4.3 , we show that within the sets Ft , there is a further bias , and in some cases this is clearly towards simple functions which correlates with Lempel-Ziv complexity ( Lempel & Ziv , 1976 ; Dingle et al. , 2018 ) , as predicted in ( Valle-Pérez et al. , 2018 ) . 3 . In Section 4.4 , we show that adding a threshold bias term to a perceptron significantly increases the bias towards low entropy . 4 . In Section 5.1 , we provide a new expressivity bound for Boolean functions : DNNs with input size n , l hidden layers each with width n+2n−1−log2 l+1 and a single output neuron can express all 22 n Boolean functions over n variables . 5 . In Section 5.2 we generalise our results to neural networks with multiple layers , proving ( in the infinite-width limit ) that the bias towards low entropy increases with the number of ReLU-activated layers . In Appendix J , we also show some empirical evidence that the results derived in this paper seem to generalize beyond the assumptions of our theoretical analysis , to more complicated data distributions ( MNIST , CIFAR ) and architectures ( CNNs ) . Finally , in Appendix M , we show preliminary results on the effect of entropy-like biases in P ( f ) on learning class-imbalanced data . 3 DEFINITIONS , TERMINOLOGY , AND NOTATION . Definition 3.1 ( DNNs ) . Fully connected feed-forward neural networks with activations σ and a single output neuron form a parameterised function family f ( x ) on inputs x ∈ Rn . This can be defined recursively , for L hidden layers for 1 ≤ l ≤ L , as f ( x ) = 1 ( h ( L+1 ) ( x ) ) , h ( l+1 ) ( x ) = wlσ ( h ( l ) ) + bl , h ( 1 ) ( x ) = w0x+ b0 , where 1 ( X ) is the Heaviside step function defined as 1 if X > 0 and 0 otherwise , and σ is an activation function that acts element-wise . The wl ∈ Rnl+l×nl are the weights , and bl ∈ Rnl+1 are the threshold bias weights at layer l , where nl is the number of hidden neurons in the l-th layer . nL+1 is the number of outputs ( 1 in this paper ) , and n0 is the dimension of the inputs ( which we will also refer to as n ) . We will refer to the whole set of parameters ( wl and bl , 1 ≤ l ≤ L ) as θ . In the case of perceptrons we use fθ ( x ) = σ ( 〈w , x〉 + b ) to specify a network . We define the parameter-function map as in ( Valle-Pérez et al. , 2018 ) below . Definition 3.2 ( Parameter-function map ) . Consider a parameterised supervised model , and let the input space be X and the output space be Y . The space of functions the model can express is F ⊂ Y|X| . If the model has p real valued parameters , taking values within a set Θ ⊆ Rp , the parameter function mapM is defined M : Θ→ F θ 7→ fθ where fθ is the function corresponding to parameters θ . In this paper we are interested in the Boolean functions that neural networks express . We consider the 0-1 Boolean hypercube { 0 , 1 } n as the input domain . Definition 3.3 . The function T ( f ) is defined as the number of points in the hypercube { 0 , 1 } n that are mapped to 1 by the action of a neural network f . For example , for a perceptron this function is defined as , T ( f ) = T ( w , b ) = ∑ x∈ { 0,1 } n 1 ( 〈x , w〉+ b ) . ( 2 ) We will sometimes use T ( w , b ) if the neural network is a perceptron . Definition 3.4 ( Ft and P ( t ) ) . We define the set Ft to be the set of functions expressible by some modelM ( e.g . a perceptron , a neural network ) which all have the same value of T ( f ) , Ft = { f ∈ FM|T ( f ) = t } , where FM is the set of all functions expressible by M. Given a probability measure P on the weights θ , we define the probability measure P ( T = t ) : = P ( θ : fθ ∈ Ft ) We can also define T ( f ) and P ( T = t ) in the natural way for sets of input points other than { 0 , 1 } n , the context making clear what definition is being used . Definition 3.5 . The entropy H ( f ) of a Boolean function f : { 0 , 1 } ∗ → { 0 , 1 } is defined as H ( f ) = −p log2 p − ( 1 − p ) log2 ( 1− p ) , where p = Tf/2n . It is the binary entropy of the fraction p of possible inputs to f that f maps to 1 or equivalently , the binary entropy of the fraction of 1 ’ s in the right-hand column of the truth table of f . Definition 3.6 . We define the Boolean complexity KBool ( f ) of a function f as the number of binary connectives in the shortest Boolean formula that expresses f . Note that Boolean complexity can be defined in other ways as well . For example , KBool ( f ) is sometimes defined as the number of connectives ( rather than binary connectives ) in the shortest formula that expresses f , or as the depth of the most shallow formula that expresses f . These definitions tend to give similar values for the complexity of a given function , and so they are largely interchangeable in most contexts . We use the definition above because it makes our calculations easier .

Summary: The topic of the paper is the inductive bias of neural networks. The authors study a simple model, namely a perceptron with no bias term viewed as a mapping from {0,1}^n->{0,1}. They show that initializing weights with a distribution that is symmetric under coordinate sign flips corresponds to an initialization in function space that is biased towards low-entropy functions. They also exhibit empirical evidence that by adding a bias term, or by using multiple layers, this tendency towards low entropy appears to increase. They also prove a bound on the minimal size of a network in order for it to represent all boolean functions. Finally, they prove a result that suggests that for ReLU networks with infinite widths the bias towards low-entropy function does indeed increase with depth.