On Learning Read-once DNFs With Neural Networks

1 INTRODUCTION . The training objective of overparameterized neural networks is non-convex and contains multiple global minima with different generalization properties . Therefore , just minimizing the training objective does not guarantee good generalization performance . Nonetheless , neural networks trained in practice with gradient-based methods show good test performance across numerous tasks ( Krizhevsky et al. , 2012 ; Silver et al. , 2016 ) , suggesting an inductive bias towards desirable solutions . Understanding this inductive bias and how it depends on the algorithm , architecture and data is one of the major open problems in machine learning ( Zhang et al. , 2017 ; Neyshabur et al. , 2018 ) . In recent years , there have been major efforts to tackle this challenge . One line of works considers the Neural Tangent Kernel ( NTK ) approximation of neural networks which reduces to a convex optimization problem ( Jacot et al. , 2018 ) . However , it has been shown that the NTK approximation is limited and does not accurately model neural networks as they are used in practice ( Yehudai & Shamir , 2019 ; Daniely & Malach , 2020 ) . Other works tackle the non-convexity directly for specific cases . However , current results are either for very simplified settings ( e.g. , diagonal linear networks Woodworth et al. , 2019 ) or for specific cases such as regression with 2-layer models and Gaussian distributions ( Li et al. , 2020 ) , or for impractical settings with infinitely wide two-layer networks ( Chizat & Bach , 2020 ) . Presumably , the reason for this relatively limited progress is the lack of general mathematical tools to analyze the non-convexity directly , except for a few simplified cases . One approach to make progress on this front is to use empirical tools in addition to theory , when the theoretical analysis is not tractable . In this work , we use this approach and study the inductive bias in a challenging and novel setting which is not addressed in previous theoretical works . Concretely , we consider learning read-once DNFs under the uniform distribution with a one-hidden layer , nonhomogeneous convex network with ReLU activations and gradient descent ( GD ) .1 In computational learning theory , the problem of learning DNFs has a long history . Learning DNFs is hard ( Pitt & Valiant , 1988 ) and the best known algorithms for learning DNFs under the uniform distribution run in quasi-polynomial time ( Verbeurgt , 1990 ) . On the other hand , for learning read-once DNFs under the uniform distribution there exist efficient learning algorithms ( Mansour & Schain , 2001 ) .2 Therefore , it is interesting to understand whether neural networks can learn readonce DNFs under the uniform distribution and this motivates the study of the inductive bias in this case . To better understand the inductive bias , we focus on the population setting . We show that even in this setting , where the training set consists of all possible binary vectors , there exist global minima of the training objective with significantly different properties . For example , a global minimum which memorizes the training points in its neurons , and another minimum whose neurons align exactly with the terms of the DNF , which we call a DNF-recovery solution . Figure 1a-b shows an example of these global minima . Therefore , the key question is what is the inductive bias of GD in this case . Namely , to which global minimum does it converge ? To address this question , we provide a computer-assisted proof for the convergence of GD in low dimensional DNFs . We circumvent the difficulty of floating point errors in the computer-assisted proof by utilizing a unique feature of our setting that allows to perform calculations in integers . We prove that under a symmetric initialization , the global minimum that GD converges to is similar to a DNF-recovery solution . Figure 1c shows an example of the global minimum GD converges to , which indeed looks similar to the DNF-recovery solution in Figure 1b . We prove , using the computer assisted proof , that after a simple procedure of pruning and rounding of the weights , we can obtain the exact DNF-recovery solution from the model that GD converges to . Consequently , the terms of the DNF can be reconstructed from the network weights . We provide additional theoretical results for the population setting . We show that for a symmetric initialization , gradient descent has the following unique stability property : if at some iteration a neuron is aligned with a term of a DNF , it will continue to be aligned with the term for all subsequent iterations . This gives further evidence that GD is biased towards neurons that are aligned with terms . We also study minimum l2 norm solutions of our problem , inspired by recent works that show connections between norm minimization and GD for homogeneous models ( Lyu & Li , 2020 ; Chizat & Bach , 2020 ) . We prove that the l2 minimum norm solutions are all DNF-recovery solutions . We corroborate our findings with empirical results which show that our conclusions hold more broadly . Specifically , we perform experiments on DNFs of higher dimension , standard one-hidden layer neural networks and Gaussian initialization . Taken together , our results demonstrate that gradi- 1Non- homogeneity here is a result of a bias in the second layer . 2In a read-once DNF each literal appears at most once . See Section 3 for a formal definition . ent descent can recover simple descriptions of Boolean functions from data , a fact that has important implications on the question of interpretability . 2 RELATED WORK . Recently , several works studied the inductive bias of two-layer homogeneous networks and show connections between gradient methods and margin maximization ( Chizat & Bach , 2020 ; Lyu & Li , 2020 ; Ji & Telgarsky , 2020 ) . Preliminary results for non-homogeneous networks were provided in Nacson et al . ( 2019 ) . We note that these results do not provide convergence guarantees and hold under assumptions on the dynamics of gradient methods ( e.g. , that they reach a certain loss value ) . Other works study fully connected neural networks under certain assumptions on the data such as linearly separable data ( Brutzkus et al. , 2018 ) or Gaussian data ( Safran & Shamir , 2018 ) . Malach & Shalev-Shwartz ( 2019 ) show that certain structured Boolean circuits can be learned with a network architecture that is specialized for their data structure . Fully connected networks were also analyzed via the NTK approximation ( Du et al. , 2019 ; 2018 ; Arora et al. , 2019 ; Fiat et al. , 2019 ) . Another line of works ( Saad & Solla , 1996 ; Goldt et al. , 2019 ; Tian , 2019 ) studies neural networks in a student-teacher setting and shows a “ specialization ” phenomenon , where a subset of student neurons aligns with teacher neurons . The main difference from our setting is that we consider classification on binary data , and they consider regression tasks on non-discrete data ( e.g. , Gaussian ) . Furthermore , we perform exact DNF recovery which is unique in our classification setting . Bengio et al . ( 2006 ) ; Bach ( 2017 ) study convex networks with infinitely many hidden units and devise convex relaxation algorithms . In this work , we consider convex networks with finitely many hidden neurons and study gradient descent on a non-convex objective . Rudin ( 2019 ) argues that methods for explaining large neural networks should be avoided because networks are too complex for humans to understand . However , we show , albeit in a restricted setting , that learned networks can be rather simple , and are easily mapped to the underlying DNF . 3 PROBLEM FORMULATION . DNFs and Read-Once DNFs : In what follows , we use [ n ] to denote the set { 1,2 , ... , n } , and let X = { ±1 } D and Y = { ±1 } . DNFs ( e.g. , see O ’ Donnell , 2014 ) are usually defined on inputs with entries in { 0,1 } to an output in { 0,1 } . In this work , we consider DNFs on inputs with entries in { ±1 } and output in { ±1 } . Therefore , we will use the following notation for DNFs . Let t∗1 , ... , t∗K ∈ { 0,1 } D. We refer to each t∗n as a term and define its set of active indices by An = { j ∈ [ D ] ∣ t∗nj = 1 } , where t∗nj is the jth entry of t ∗ n. We define a DNF f ∗ ∶ X → Y with K terms t∗1 , ... , t∗K as follows : f∗ ( x ) = 1 if ∃n ∈ [ K ] s.t . x ⋅ t∗n = ∣An∣ , and otherwise f∗ ( x ) = −1 . Notice that f∗ is monotone . We refer to t∗n as term n of f ∗ and we say that a sample x ∈ X satisfies the term t∗n if x ⋅ t∗n = ∣An∣ . We refer to ∣An∣ as the size of the term t∗n . To compare our notation with the standard one , for example , the DNF ( x1∧x2 ) ∨ ( x3∧x4 ) with 4 inputs has terms t∗1 = ( 1,1,0,0 ) and t∗2 = ( 0,0,1,1 ) . We will use the standard notation when convenient , e.g. , as in Figure 1 . In this work we will focus on read-once DNFs where for all i ≠ j ∈ [ K ] , Ai ∩Aj = ∅ Learning Setup : Let D be a the uniform distribution over X and f∗ be a monotone read-once DNF ...... 3 . We consider learning f∗ given a training set S ⊆ X × Y , where for each ( x , y ) ∈ S , x is sampled IID from D and y = f∗ ( x ) . Denote Sx = { x ∣ ( x , y ) ∈ S } and the positive samples by Sp = { x ∣ ( x,1 ) ∈ S } . Neural Architecture : We consider a convex one-hidden layer neural network ( NN ) with r hidden units and parameters ( W , b ) ∈ RrD ×Rr which is defined by : N ( x ; W , b ) = ∑ i∈ [ r ] σ ( wi ⋅x + bi ) − 1 ( 1 ) 3In the case of the uniform distribution , we can assume monotone DNFs WLOG . This follows since for the uniform distribution ( that has IID Bernoulli ( 0.5 ) entries ) , by symmetry , any negated literal can be replaced with the original literal ( without negation ) and all our results still hold . where σ ( x ) = max { 0 , x } is the ReLU function , wi is the ith row of W and bi is the ith entry of b . Note that the network is not homogeneous and therefore recent results on homogeneous networks do not apply ( see Section 2 ) . The network is convex because it is a sum of convex ReLU functions . Training Loss : To learn f∗ we aim to minimize the following hinge loss : L ( W , b ) = 1∣S∣ ∑ ( x , y ) ∈S max { 0,1 − yN ( x ; W , b ) } ( 2 ) We note that the above loss function is generally non-convex ( even though the network N is convex ) . For optimization we use Gradient Descent ( GD ) with a fixed learning rate η . We denote the initialization of GD by ( W ( 0 ) , b ( 0 ) ) and the weights at iteration t by ( W ( t ) , b ( t ) ) . If the iteration index is clear from context we omit it and use ( W , b ) . See Section A for the gradient update . In this work we will mainly focus on the population case . This corresponds to optimizing the loss in Eq . ( 2 ) with Sx = X .4 Many works have studied it as a proxy to understand the performance in the empirical case ( e.g. , Brutzkus & Globerson , 2017 ; Daniely & Malach , 2020 ) . Remark 3.1 . The convex network considered here is a good test-bed for understanding the inductive bias of learning read-once DNFs with one-hidden layer NNs because : ( 1 ) It has the same expressive power as standard NNs for implementing Boolean functions ( Section 4 ) ( 2 ) It outperforms standard NNs on learning read-once DNFs in our setting ( Section 8 ) ( 3 ) Its analysis can be used to better understand the inductive bias of standard NNs ( Section 8 ) .

Summary: This paper investigates the problem of learning monotone read-once DNF formulas using convex neural networks. Specifically, the authors explore the distribution-specific PAC setting, where training samples are drawn independently according to the uniform distributions and are labeled according to a target monotone read-once DNF. The main contribution of this study is essentially empirical: convex neural nets, trained with GD for minimizing the cumulative hinge loss, converge to global minima for which neural units coincide with the monomials of the target DNF. This remarkable stability is corroborated by theoretical insights about global minima.