On Learning Read-once DNFs With Neural Networks

1 INTRODUCTION . The training objective of overparameterized neural networks is non-convex and contains multiple global minima with different generalization properties . Therefore , just minimizing the training objective does not guarantee good generalization performance . Nonetheless , neural networks trained in practice with gradient-based methods show good test performance across numerous tasks ( Krizhevsky et al. , 2012 ; Silver et al. , 2016 ) , suggesting an inductive bias towards desirable solutions . Understanding this inductive bias and how it depends on the algorithm , architecture and data is one of the major open problems in machine learning ( Zhang et al. , 2017 ; Neyshabur et al. , 2018 ) . In recent years , there have been major efforts to tackle this challenge . One line of works considers the Neural Tangent Kernel ( NTK ) approximation of neural networks which reduces to a convex optimization problem ( Jacot et al. , 2018 ) . However , it has been shown that the NTK approximation is limited and does not accurately model neural networks as they are used in practice ( Yehudai & Shamir , 2019 ; Daniely & Malach , 2020 ) . Other works tackle the non-convexity directly for specific cases . However , current results are either for very simplified settings ( e.g. , diagonal linear networks Woodworth et al. , 2019 ) or for specific cases such as regression with 2-layer models and Gaussian distributions ( Li et al. , 2020 ) , or for impractical settings with infinitely wide two-layer networks ( Chizat & Bach , 2020 ) . Presumably , the reason for this relatively limited progress is the lack of general mathematical tools to analyze the non-convexity directly , except for a few simplified cases . One approach to make progress on this front is to use empirical tools in addition to theory , when the theoretical analysis is not tractable . In this work , we use this approach and study the inductive bias in a challenging and novel setting which is not addressed in previous theoretical works . Concretely , we consider learning read-once DNFs under the uniform distribution with a one-hidden layer , nonhomogeneous convex network with ReLU activations and gradient descent ( GD ) .1 In computational learning theory , the problem of learning DNFs has a long history . Learning DNFs is hard ( Pitt & Valiant , 1988 ) and the best known algorithms for learning DNFs under the uniform distribution run in quasi-polynomial time ( Verbeurgt , 1990 ) . On the other hand , for learning read-once DNFs under the uniform distribution there exist efficient learning algorithms ( Mansour & Schain , 2001 ) .2 Therefore , it is interesting to understand whether neural networks can learn readonce DNFs under the uniform distribution and this motivates the study of the inductive bias in this case . To better understand the inductive bias , we focus on the population setting . We show that even in this setting , where the training set consists of all possible binary vectors , there exist global minima of the training objective with significantly different properties . For example , a global minimum which memorizes the training points in its neurons , and another minimum whose neurons align exactly with the terms of the DNF , which we call a DNF-recovery solution . Figure 1a-b shows an example of these global minima . Therefore , the key question is what is the inductive bias of GD in this case . Namely , to which global minimum does it converge ? To address this question , we provide a computer-assisted proof for the convergence of GD in low dimensional DNFs . We circumvent the difficulty of floating point errors in the computer-assisted proof by utilizing a unique feature of our setting that allows to perform calculations in integers . We prove that under a symmetric initialization , the global minimum that GD converges to is similar to a DNF-recovery solution . Figure 1c shows an example of the global minimum GD converges to , which indeed looks similar to the DNF-recovery solution in Figure 1b . We prove , using the computer assisted proof , that after a simple procedure of pruning and rounding of the weights , we can obtain the exact DNF-recovery solution from the model that GD converges to . Consequently , the terms of the DNF can be reconstructed from the network weights . We provide additional theoretical results for the population setting . We show that for a symmetric initialization , gradient descent has the following unique stability property : if at some iteration a neuron is aligned with a term of a DNF , it will continue to be aligned with the term for all subsequent iterations . This gives further evidence that GD is biased towards neurons that are aligned with terms . We also study minimum l2 norm solutions of our problem , inspired by recent works that show connections between norm minimization and GD for homogeneous models ( Lyu & Li , 2020 ; Chizat & Bach , 2020 ) . We prove that the l2 minimum norm solutions are all DNF-recovery solutions . We corroborate our findings with empirical results which show that our conclusions hold more broadly . Specifically , we perform experiments on DNFs of higher dimension , standard one-hidden layer neural networks and Gaussian initialization . Taken together , our results demonstrate that gradi- 1Non- homogeneity here is a result of a bias in the second layer . 2In a read-once DNF each literal appears at most once . See Section 3 for a formal definition . ent descent can recover simple descriptions of Boolean functions from data , a fact that has important implications on the question of interpretability . 2 RELATED WORK . Recently , several works studied the inductive bias of two-layer homogeneous networks and show connections between gradient methods and margin maximization ( Chizat & Bach , 2020 ; Lyu & Li , 2020 ; Ji & Telgarsky , 2020 ) . Preliminary results for non-homogeneous networks were provided in Nacson et al . ( 2019 ) . We note that these results do not provide convergence guarantees and hold under assumptions on the dynamics of gradient methods ( e.g. , that they reach a certain loss value ) . Other works study fully connected neural networks under certain assumptions on the data such as linearly separable data ( Brutzkus et al. , 2018 ) or Gaussian data ( Safran & Shamir , 2018 ) . Malach & Shalev-Shwartz ( 2019 ) show that certain structured Boolean circuits can be learned with a network architecture that is specialized for their data structure . Fully connected networks were also analyzed via the NTK approximation ( Du et al. , 2019 ; 2018 ; Arora et al. , 2019 ; Fiat et al. , 2019 ) . Another line of works ( Saad & Solla , 1996 ; Goldt et al. , 2019 ; Tian , 2019 ) studies neural networks in a student-teacher setting and shows a “ specialization ” phenomenon , where a subset of student neurons aligns with teacher neurons . The main difference from our setting is that we consider classification on binary data , and they consider regression tasks on non-discrete data ( e.g. , Gaussian ) . Furthermore , we perform exact DNF recovery which is unique in our classification setting . Bengio et al . ( 2006 ) ; Bach ( 2017 ) study convex networks with infinitely many hidden units and devise convex relaxation algorithms . In this work , we consider convex networks with finitely many hidden neurons and study gradient descent on a non-convex objective . Rudin ( 2019 ) argues that methods for explaining large neural networks should be avoided because networks are too complex for humans to understand . However , we show , albeit in a restricted setting , that learned networks can be rather simple , and are easily mapped to the underlying DNF . 3 PROBLEM FORMULATION . DNFs and Read-Once DNFs : In what follows , we use [ n ] to denote the set { 1,2 , ... , n } , and let X = { ±1 } D and Y = { ±1 } . DNFs ( e.g. , see O ’ Donnell , 2014 ) are usually defined on inputs with entries in { 0,1 } to an output in { 0,1 } . In this work , we consider DNFs on inputs with entries in { ±1 } and output in { ±1 } . Therefore , we will use the following notation for DNFs . Let t∗1 , ... , t∗K ∈ { 0,1 } D. We refer to each t∗n as a term and define its set of active indices by An = { j ∈ [ D ] ∣ t∗nj = 1 } , where t∗nj is the jth entry of t ∗ n. We define a DNF f ∗ ∶ X → Y with K terms t∗1 , ... , t∗K as follows : f∗ ( x ) = 1 if ∃n ∈ [ K ] s.t . x ⋅ t∗n = ∣An∣ , and otherwise f∗ ( x ) = −1 . Notice that f∗ is monotone . We refer to t∗n as term n of f ∗ and we say that a sample x ∈ X satisfies the term t∗n if x ⋅ t∗n = ∣An∣ . We refer to ∣An∣ as the size of the term t∗n . To compare our notation with the standard one , for example , the DNF ( x1∧x2 ) ∨ ( x3∧x4 ) with 4 inputs has terms t∗1 = ( 1,1,0,0 ) and t∗2 = ( 0,0,1,1 ) . We will use the standard notation when convenient , e.g. , as in Figure 1 . In this work we will focus on read-once DNFs where for all i ≠ j ∈ [ K ] , Ai ∩Aj = ∅ Learning Setup : Let D be a the uniform distribution over X and f∗ be a monotone read-once DNF ...... 3 . We consider learning f∗ given a training set S ⊆ X × Y , where for each ( x , y ) ∈ S , x is sampled IID from D and y = f∗ ( x ) . Denote Sx = { x ∣ ( x , y ) ∈ S } and the positive samples by Sp = { x ∣ ( x,1 ) ∈ S } . Neural Architecture : We consider a convex one-hidden layer neural network ( NN ) with r hidden units and parameters ( W , b ) ∈ RrD ×Rr which is defined by : N ( x ; W , b ) = ∑ i∈ [ r ] σ ( wi ⋅x + bi ) − 1 ( 1 ) 3In the case of the uniform distribution , we can assume monotone DNFs WLOG . This follows since for the uniform distribution ( that has IID Bernoulli ( 0.5 ) entries ) , by symmetry , any negated literal can be replaced with the original literal ( without negation ) and all our results still hold . where σ ( x ) = max { 0 , x } is the ReLU function , wi is the ith row of W and bi is the ith entry of b . Note that the network is not homogeneous and therefore recent results on homogeneous networks do not apply ( see Section 2 ) . The network is convex because it is a sum of convex ReLU functions . Training Loss : To learn f∗ we aim to minimize the following hinge loss : L ( W , b ) = 1∣S∣ ∑ ( x , y ) ∈S max { 0,1 − yN ( x ; W , b ) } ( 2 ) We note that the above loss function is generally non-convex ( even though the network N is convex ) . For optimization we use Gradient Descent ( GD ) with a fixed learning rate η . We denote the initialization of GD by ( W ( 0 ) , b ( 0 ) ) and the weights at iteration t by ( W ( t ) , b ( t ) ) . If the iteration index is clear from context we omit it and use ( W , b ) . See Section A for the gradient update . In this work we will mainly focus on the population case . This corresponds to optimizing the loss in Eq . ( 2 ) with Sx = X .4 Many works have studied it as a proxy to understand the performance in the empirical case ( e.g. , Brutzkus & Globerson , 2017 ; Daniely & Malach , 2020 ) . Remark 3.1 . The convex network considered here is a good test-bed for understanding the inductive bias of learning read-once DNFs with one-hidden layer NNs because : ( 1 ) It has the same expressive power as standard NNs for implementing Boolean functions ( Section 4 ) ( 2 ) It outperforms standard NNs on learning read-once DNFs in our setting ( Section 8 ) ( 3 ) Its analysis can be used to better understand the inductive bias of standard NNs ( Section 8 ) .

Summary: In this paper the aim in to understand the inductive bias of neural networks learning DNFs. The focus is in convex neural networks and gradient descent. It is shown that under a symmetric initialization, the global minimum that gradient descent converges to is similar to a DNF-recovery solution. Further, experimental evaluation demonstrates that gradient descent can recover read-once DNFs from data.