Learning Boolean Circuits with Neural Networks

1 INTRODUCTION AND MOTIVATION . It is well known ( e.g . Livni et al . ( 2014 ) ) that while deep neural-networks can express any function that can be run efficiently on a computer , in the general case , training them is computationally hard . Despite this theoretic pessimism , in practice , deep neural networks are successfully trained on real world datasets . Bridging this theoretical-practical gap seems to be the holy grail of theoretical machine learning nowadays . Maybe the most natural direction to bridge this gap is to find a property of data distributions that determines whether training is computationally easy or hard . The goal of this paper is to propose such a property . To motivate this , we first recall the k-parity problem : the input is n bits , there is a subset of k relevant bits ( which are unknown to the learner ) , and the output should be 1 if the number of 1 ’ s among the relevant bits is even and −1 otherwise . It is well known ( e.g . Shalev-Shwartz et al . ( 2017 ) ) that the parity problem can be expressed by a fully connected two layer network or by depth log ( n ) locally connected 1 network . We observe the behavior of a one hidden-layer neural network trained on the k-parity problem , in two different instances : first , when the underlying distribution is the uniform distribution ( i.e . the probability to see every bit is 12 ) ; and second , when the underlying distribution is a slightly biased product distribution ( the probability for every bit to be 1 is 0.6 ) . As can be clearly seen in figure 1 , adding a slight bias to the probability of each bit dramatically affects the behavior of the network : while on the uniform distribution the training process completely fails , in the biased case it converges to a perfect solution . This simple experiment shows that a small change in the underlying distribution can cause a dramatic change in the trainability of neural-networks . A key property that differentiates the uniform from the biased distribution is the correlation between input bits and the target label . While in the uniform distribution , the correlation between each bit and the label is zero , in the biased case every bit of the k bits in the parity has a non-negligible correlation to the label ( we show this formally in section 5 ) . So , local correlations between bits of the input and the target label seems to be a promising property which separates easy and hard distributions . In this paper , we analyze the problem of learning tree-structured Boolean circuits with neuralnetworks . The key property that we assume is having sufficient correlation between every gate in the circuit and the label . We show that a variant of gradient-descent can efficiently learn such 1i.e . every two adjacent neurons are only connected to one neuron in the upper layer . circuits for some families of distributions , where at the same time , without correlation , gradientdescent is likely to fail . More concretely , we discuss specific target functions and distributions that satisfy the local correlation requirement . We show that for most product distributions , gradientdescent learns the ( log n ) -parity problem ( parity on log n bits of an input with dimension n ) . We further show that for every circuit with AND/OR/NOT gates , there exists a generative distribution , such that gradient-descent recovers the Boolean circuit exactly . Admittedly , as the primary focus of this paper is on theoretical analysis , the distributions we study are synthetic in nature . However , to explain the empirical success of neural-networks , we need to verify whether the local correlation property holds for natural datasets as well . To confirm this , we perform the following simple experiment : we train a network with two hidden-layers on a single random patch from images in the ImageNet dataset . We observe that even on a complex task such as ImageNet , a network that gets only a 3×3 patch as an input , achieves 2.6 % top-5 accuracy — much better than a random guess ( 0.5 % top-5 accuracy ) . The full results of the experiment are detailed in the appendix . This experiment highlights that , to some extent , natural datasets display a local correlation property : even a few “ bits ” of the input already have some non-negligible information on the target label . 2 RELATED WORK . In recent years , the success of neural-networks has inspired an ongoing theoretical research , trying to explain empirical observations about their behavior . Some theoretical works show failure cases of neural-networks . Other works give different guarantees on various learning algorithms for neuralnetworks . In this section , we cover the main works that are relevant to our paper . Failures of gradient-based algorithms . Various works have shown different examples demonstrating failures of gradient-based algorithm . The work of Shamir ( 2018 ) shows failures of gradient descent , both in learning natural target functions and in learning natural distributions . The work of Shalev-Shwartz et al . ( 2017 ) shows that gradient-descent fails to learn parities and linear-periodic functions under the uniform distribution . In Das et al . ( 2019 ) , a hardness result for learning random deep networks is shown . Other similar failure cases are also covered in Abbe & Sandon ( 2018 ) ; Malach & Shalev-Shwartz ( 2019 ) . While the details of these works differ , they all share the same key principal - if there is no local correlation , gradient-descent fails . Our work complements these results , showing that in some cases , when there are local correlations to the target , gradient-descent succeeds to learn the target function . Learning neural-networks with gradient-descent . Recently , a large number of papers have provided positive results on learning neural-networks with gradient-descent . Generally speaking , most of these works show that over-parametrized neural-networks , deep or shallow , achieve performance that is competitive with kernel-SVM . Daniely ( 2017 ) shows that SGD learns the conjugate kernel associated with the architecture of the network , for a wide enough neural-network . The work of Brutzkus et al . ( 2017 ) shows that SGD learns a neural-network with good generalization , when the target function is linear . A growing number of works show that for a specific kernel induced by the network activation , called the Neural Tangent Kernel ( NTK ) , gradient-descent learns overparametrized networks , for target functions with small norm in the reproducing kernel Hilbert space ( see the works of Jacot et al . ( 2018 ) ; Xie et al . ( 2016 ) ; Oymak & Soltanolkotabi ( 2018 ) ; Allen-Zhu et al . ( 2018a ; b ) ; Oymak & Soltanolkotabi ( 2019 ) ; Arora et al . ( 2019 ) ; Du et al . ( 2018 ) ; Ma et al . ( 2019 ) ; Lee et al . ( 2019 ) ) . While these results show that learning neural-networks with gradientdescent is not hopeless , they are in some sense disappointing — in practice , neural-networks achieve performance that are far better than SVM , a fact that is not explained by these works . A few results do discuss success cases of gradient-descent that go beyond the kernel-based analysis ( Brutzkus & Globerson , 2017 ; 2019 ; Allen-Zhu & Li , 2019 ; Yehudai & Shamir , 2019 ) . However , these works still focus on very simple cases , such as learning a single neuron , or learning shallow neural-networks in restricted settings . In this work we deal with learning deep networks , going beyond the common reduction to linear classes of functions . Layerwise optimization algorithms . In this paper , we analyze the behavior of layerwise gradientdescent — optimizing one layer at a time , instead of the common practice to optimize the full network end-to-end . We do so since such algorithm greatly simplifies our theoretical analysis . While layerwise training is not a common practice , recent works ( Belilovsky et al. , 2018 ; 2019 ) have shown that such algorithms achieve performance that are competitive with the standard end-to-end approach , scaling up to the ImageNet dataset . We note that other theoretical works have studied iterative algorithms that learn neural-networks layer-by-layer ( Arora et al. , 2014 ; Malach & ShalevShwartz , 2018 ) . However , our work focuses specifically on layerwise gradient-descent , considering the problem of learning Boolean circuits . Learning Boolean Circuits . The problem of learning Boolean circuits has been studied in the classical literature of theoretical machine learning . The work of Kearns et al . ( 1987 ) gives various positive and negative results on the learnability of Boolean Formulas , including Boolean circuits . The work of Linial et al . ( 1989 ) introduces an algorithm that learns a constant-depth circuit in quasipolynomial time . Another work by Kalai ( 2018 ) discusses various properties of learning Boolean formulas and Boolean circuits . Our work differs from the above in various aspects . Our main focus is learning deep neural-networks with gradient descent , where the target function is implemented by a Boolean circuit , and we do not aim to study the learnability of Boolean circuits in general . Furthermore , we consider Boolean circuits where a gate can take any Boolean functions , and not only AND/OR/NOT , as is often considered in the literature of Boolean circuits . On the other hand , we restrict ourselves to the problem of learning circuits with a fixed structure of full binary trees . We are not aware of any work studying a problem similar to ours . 3 PROBLEM SETTING . We consider the problem of learning binary classification functions over the Boolean cube . So , let X = { ±1 } n be the instance space and Y = { ±1 } be the label set . Throughout the paper , we assume the target function is given by a Boolean circuit . In general , such assumption effectively does not limit the set of target functions , as any computable function can be implemented by a Boolean circuit . We define a circuit C to be a directed graph with n input nodes and a single output node , where each inner node has exactly two incoming edges , and is labeled by some arbitrary Boolean function f : { ±1 } 2 → { ±1 } , which we call a gate 2 . For each node v in C we denote by γ ( v ) ∈ { f : { ±1 } 2 → { ±1 } } its gate . We recursively define hv , C : { ±1 } n → { ±1 } to be : hv , C ( x ) = γ ( v ) ( hu1 , C ( x ) , hu2 , C ( x ) ) where u1 , u2 are the two nodes with outcoming edges to v. Finally , define hC = ho , C , where o is the output node . We study the problem of learning the target function hC , when C is a full binary tree , and n = 2d , where d is the depth of the tree . The leaves of the tree are the input bits , ordered by x1 , . . . xn . Admittedly , such assumption greatly limits the set of target functions , but still gives a rather rich family of functions . For example , such circuit can calculate the parity function on any k bits of the input ( the function calculated by f ( x ) = ∏ i∈I xi for some set of indexes I ) . We note that the total number of functions calculated by such tree grows like 6n , as shown in Farhoodi et al . ( 2019 ) . 2Note that in the literature on Boolean circuits it is often assumed that the gates are limited to being AND/OR and NOT . We allow the gates to take any Boolean function , which makes this model somewhat stronger . We introduce a few notations that are used in the sequel . Fix some tree structured binary circuit C. This circuit has d levels , and we denote vi , j the j-th node in the i-th level of the tree , and denote γi , j = γ ( vi , j ) . Fix some i ∈ [ d ] , let ni : = 2i , and denote by Γi : { ±1 } ni → { ±1 } ni/2 the function calculated by the i-th level of the circuit : Γi ( x ) = ( γi−1,1 ( x1 , x2 ) , . . . , γi−1 , ni/2 ( xni−1 , xni ) ) For i < i′ , we denote : Γi ... i′ : = Γi ◦ · · · ◦ Γi′ . So , the full circuit is given by hC ( x ) = Γ1 ... d ( x ) . As noted , our goal is to learn Boolean circuits with neural-networks . To do so , we use a network architecture that aims to imitate the Boolean circuits described above . We replace each Boolean gate with a neural-gate : a one hidden-layer ReLU network , with a hard-tanh3 activation on its output . Formally , let σ be the ReLU activation , and let φ be the hard-tanh activation , so : σ ( x ) = max ( x , 0 ) , φ ( x ) = −1 x ≤ −1 x x ∈ ( −1 , 1 ) 1 x ≥ 1 Define a neural-gate to be a neural-network with one hidden layer , input dimension 2 , with ReLU activation for the hidden-layer and hard-tanh for the output node . Namely , denote gw , v : R2 → R such that : gw , v ( x ) = φ ( k∑ l=1 viσ ( 〈wl , x〉 ) ) Notice that a neural-gate gw , v of width 4 or more can implement any Boolean gate . That is , we can replace any Boolean gate with a neural-gate , and maintain the same expressive power . To implement the full Boolean circuit defined above , we construct a deep network of depth d ( the depth of the Boolean circuit ) , with the same structure as the Boolean circuit . We define d blocks , each block has neural-gates with the same structure and connectivity as the Boolean circuit . A block BW ( i ) , V ( i ) : R2 i → R2i−1 , is defined by : BW ( i ) , V ( i ) ( x ) = [ gw ( i,1 ) , v ( i,1 ) ( x1 , x2 ) , gw ( i,2 ) , v ( i,2 ) ( x3 , x4 ) , . . . , gw ( i,2i−1 ) , v ( i,2i−1 ) ( x2i−1 , x2i ) ] We consider the process of training neural-networks of the form NW , V = BW ( 1 ) , V ( 1 ) ◦ · · · ◦ BW ( d ) , V ( d ) . Notice that indeed , a network NW , V can implement any tree-structured Boolean circuit of depth d. In practice , neural-networks are trained with gradient-based optimization algorithm , in an end-to-end fashion . That is , the weights of all the layers are optimized together , with gradient updates on a given sample . To simplify the analysis , we instead consider a layerwise optimization algorithm , that performs gradient updates layer-by-layer . While this approach is much less popular , it has been recently shown to achieve performance that are comparable to end-to-end training , scaling up to the ImageNet dataset ( Belilovsky et al. , 2018 ) . Denote by P the average-pooling operator , defined by P ( x1 , . . . , xn ) = 1n ∑n i=1 xi . Denote the hinge-loss by ` ( ŷ , y ) = max ( 1 − yŷ , 0 ) and denote the loss on the distribution by LD ( f ) = E ( x , y ) ∼D [ ` ( f ( x ) , y ) ] . For a sample S ⊆ X × Y , denote the loss on the sample by LS ( f ) = 1 |S| ∑ ( x , y ) ∈S ` ( f ( x ) , y ) . The layerwise gradient-descent algorithm for learning deep networks is described in algorithm 1 . For simplicity , we assume that the second layer of every neural-gate is fixed , such that v ∈ { ±1 } . Notice that this does not limit the expressive power of the network . Algorithm 1 iteratively optimizes the output of the network ’ s layers , starting from the bottom-most layer . For each layer , the average-pooling operator is applied to reduce the output of the layer to a single bit , and this output is optimized with respect to the target label . Note that in fact , we can equivalently optimize each neural-gate separately and achieve the same algorithm . However , we present a layerwise training process to conform with algorithms used in practice . 3We chose to use the hard-tanh activation over the more popular tanh activation since it simplifies our theoretical analysis . However , we believe the same results can be given for the tanh activation . Algorithm 1 Layerwise Gradient-Descent input : Sample S ⊆ X × Y , number of iterations T ∈ N , learning rate η ∈ R. Let Nd ← id for i = d . . . 1 do Initialize W ( i ) 0 , V ( i ) 0 . for t = 1 . . . T do Update W ( i ) t ←W ( i ) t−1 − η ∂∂W ( i ) t−1 LS ( P ( BW ( i ) t−1 , V ( i ) 0 ◦ Ni ) ) end for Update Ni−1 ← BW ( i ) T , V ( i ) 0 ◦ Ni end for Return N0

Summary: This paper aims to study the correlation between the neural network's input and output by abstracting the network as a binary tree Boolean circuit problem. The paper is well-written, motivations are clearly presented, and literature reviews are well placed. The contributions are mainly theoretical, and the experimental plots are simply used for concept illustrations, therefore the correctness of the theoretical analysis has no empirical evaluations.