The staircase property: How hierarchical structure can guide deep learning

1 Introduction . It has been observed empirically that neural networks can learn hierarchically . For example , a ‘ car ’ may be detected by first understanding simpler concepts like ‘ door ’ , ’ wheel ’ , and so forth in intermediate layers , which are then combined in deeper layers ( c.f . [ 1 , 2 ] ) . However , on the theoretical side , the mechanisms by which such hierarchical learning occurs are not yet fully understood . In this paper we are motivated by the following question : Can we identify naturally structured and interpretable classes of hierarchical functions , and show how regular1DNNs are able to learn them ? This is a refinement of the generic objective of trying to understand DNNs : We identify several key desiderata for any theoretical result in this direction . ( 1 ) Natural structure : We aim to capture naturally occurring data of interest , so the structural assumption must make conceptual sense . ( 2 ) Interpretability : If we hope to clearly interpret the inner workings of neural networks , understanding both how they classify and also how they learn , then we need a model for data that is interpretable to 1The notion of regularity is specified in Definition 2.1 ; this means network architectures and initializations that have homogeneity properties within layers , in contrast to the emulation architectures in [ 3 , 4 ] . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . begin with . Interpretation of the representations occurring within a neural network is most clearly expressed with respect to structural properties of the data . Finally , ( 3 ) Regularity of the network : The network architecture and initialization should be symmetric in a sense defined later on . This prevents using carefully-crafted architectures and initializations to emulate general learning algorithms [ 3 , 4 ] . We view this type of restriction as being partway towards considering practical neural networks that learn in a blackbox fashion . The results in this paper aim to satisfy all three high-level objectives . The relation with prior work is discussed in Section 1.1 . This paper proposes a new structurally-defined class of hierarchical functions and proves guarantees for learning by regular neural networks . In order to describe this structure , we first recall that any function f : { +1 , 1 } n ! R can be decomposed in the Fourier-Walsh basis as f ( x ) = X S✓ [ n ] f̂ ( S ) S ( x ) , where f̂ ( S ) : = hf , Si , S ( x ) : = Y i2S xi ( 1 ) and the inner product between two functions is hf , gi = Ef ( X ) g ( X ) for X ⇠ Unif ( { +1 , 1 } n ) . This decomposition expresses f ( x ) as a sum of components , each of which is a monomial S ( x ) , weighted by the Fourier coefficient f̂ ( S ) . Our definition of hierarchical structure is motivated by an observation regarding two closely related functions , “ high-degree monomials ” and “ staircase functions ” , the latter of which can be learned efficiently and the former of which can not . Monomials with no hierarchical structure The class of monomials of any degree k where k n/2 ( i.e. , the class { S } S✓ [ n ] , |S|=k ) is efficiently learnable by Statistical Query ( SQ ) algorithms if and only if k is constant [ 5 , 6 ] , and the same holds for noisy Gradient Descent ( GD ) on neural nets with polynomially-many parameters [ 5 ] , and for noisy Stochastic Gradient Descent ( SGD ) where the batch-size is sufficiently large compared to the gradients ’ precision [ 3 , 4 ] . This was also noted in [ 7 ] which shows that gradients carry little information to reconstruct S for large |S| , and hence gradient-based training is expected to fail . Thus , we can think of a component S as simple and easily learnable if the degree |S| is small and complex and harder to learn if the degree |S| is large . Staircase functions with hierarchical structure Now , instead of a single monomial , consider the following staircase function ( and its orbit class induced by permutations of the inputs ) , which is a sum of monomials of increasing degree : Sk ( x ) = x1 + x1x2 + x1x2x3 + x1x2x3x4 + · · ·+ 1 : k . ( 2 ) Here Sk ( x ) has a hierarchical structure , where x1 builds up to x1x2 , which builds up to x1x2x3 , and so on until the degree-k monomial 1 : k. Our experiments in Fig . 2 show a dramatic difference between learning a single monomial 1 : k and learning the staircase function Sk . Even with n = 30 and k = 10 , the same network with 5 ReLU ResNet layers and the same hyperparameters can easily learn Sk to a vanishing error ( Fig . 2b ) whereas , as expected , it can not learn 1 : k even up to any non-trivial error since 1 : k is a high-degree monomial ( Fig . 2a ) . An explanation for this phenomenon is that the neural network learns the staircase function Sk ( x ) by first learning a degree-1 approximation that picks up the feature x1 , and then uses this to more readily learn a degree-2 approximation that picks up the feature x1x2 , and so on , progressively incrementing the degree of the approximation and ‘ climbing the staircase ’ up to the large degrees . We refer to Fig . 1a for an illustration . This is indeed the learning mechanism , as we can see once we plot the Fourier coefficients of the network output against training iteration . Indeed , in Fig . 2c we see that the network trained to learn 1:10 can not learn any Fourier coefficient relevant to 1:10 whereas in Fig . 2d it is clear that the network trained to learn S10 learns the relevant Fourier coefficients in order of increasing complexity and eventually reaches the 1:10 coefficient . Main results We shed light on this phenomenon , proving that certain regular networks efficiently learn the staircase function Sk ( x ) , and , more generally , functions satisfying this structural property : Definition 1.1 ( Staircase property ) . For any M > 1 , a function g : { 1 , 1 } n ! R satisfies the [ 1/M , M ] -staircase property over the unbiased binary hypercube if : • for all S ⇢ [ n ] , if ĝ ( S ) 6= 0 then |ĝ ( S ) | 2 [ 1/M , M ] . • for all S ⇢ [ n ] , if ĝ ( S ) 6= 0 and |S| 2 , there is S0 ⇢ S such that |S \ S0| = 1 and ĝ ( S0 ) 6= 0 . Furthermore , g is said to be an s-sparse polynomial if | { S : ĝ ( S ) 6= 0 } | s. The parameters M and s appear naturally since a PAC-learning algorithm for s-sparse polynomials satisfying the [ 1/M , M ] -staircase property must use a number of samples that depends polynomially on M and s. Our theoretical result is informally summarized as follows , and we remark that the proof shows that the neural network progressively learns approximations of higher degree : Theorem 1.2 ( Informal statement of Theorem 2.2 ) . Let g : { 1 , 1 } n ! R be an unknown s-sparse polynomial satisfying the [ 1/M , M ] -staircase property . Given access to random samples from { ( x , g ( x ) ) } x⇠ { 1,1 } n , there is a regular neural network architecture that approximately learns g in poly ( n , s , M ) time and samples when trained with layerwise stochastic coordinate descent . Even though we only consider hierarchical functions over the Boolean hypercube { 1 , 1 } n in our theoretical result , we believe that the techniques used in this work can be extended to other function spaces of interest , exploiting the orthonormality of the corresponding Fourier basis functions . For this reason we give a fairly general definition of hierarchical functions in Section 3 that goes beyond the Boolean hypercube , as well as beyond the strict notion of increasing chains . This more general class of functions is of further interest because it includes as special cases well-studied classes such as biased sparse parities and decision trees in a smoothed complexity setting ( see Section 3.1 ) . 1.1 Related Work . Statistical query emulation results For the general class of polynomial-size neural network architectures with any choice of initialization , it is known that SGD on a sufficiently small batch-size can learn2 any function class ( including functions satisfying the staircase property ) that is efficiently learnable from samples [ 3 ] , while GD can learn any function class that is efficiently learnable from statistical queries ( SQ ) [ 4 ] . However , these results rely on highly non-regular architectures and initializations , with different parts of the nets responsible for different tasks that emulate the computations of general learning algorithms . In particular , it is not known how to obtain the emulation results of [ 4 ] for “ regular ” architectures and initializations as defined in Definition 2.1 . In contrast , our architecture in Theorem 2.2 is a regular neural network in this sense , and our analysis further illustrates how features are built greedily over depth rather than by emulating a given algorithm . Consider also the orbit class under permutations of the inputs of the “ truncated staircase function ” , Sj ! k ( x ) = P k i=j i : k ( x ) , for 1 j k n. Note that this class is efficiently SQ-learnable when k = n , since the monomial 1 : n is always present and one can recursively check which sub-monomial is present or not by checking at most n monomials at each step . However , we conjecture that Sj ! n ( x ) is not learnable by regular networks trained with gradient descent if min ( n j , j ) = ! ( 1 ) . Therefore such truncated staircases provide a candidate for separating gradient-based learning on regular networks versus general , non-regular networks that allow for emulating any SQ algorithm [ 4 ] . Hierarchical models of data Explicitly adding hierarchical structures into machine learning algorithms such as hierarchical Bayesian modeling and hierarchical linear modeling has proved successful in various machine learning tasks beyond deep learning [ 8–11 ] . For image data , [ 12 , 13 ] propose hierarchical generative models of images and use them to motivate deep convolutional architectures , although these works do not prove that deep learning learns these generative models . [ 14 ] similarly proposes a ‘ deep rendering model ’ which hierarchically models levels of abstraction present in data , but does not prove learnability . [ 15 ] gives a training algorithm for deep convolutional networks that provably learns a deep generative model of images . The paper [ 16 ] proposes a generative model of data motivated by evolutionary processes , and proves in a formal sense that “ deep ” algorithms can learn these models , whereas shallow algorithms can not . In contrast to our work , the “ deep ” algorithms considered by [ 16 ] are not descent algorithms on regular deep neural network architectures . In [ 17 ] it is shown that during training of two-layer ReLU networks with SGD , the lower frequency components of the target function are learned first . Unfortunately , their results have an exponential dependence on the degree . In our work , we leverage depth and the hierarchical Boolean function structure to ensure that higher-level Fourier coefficients are learned efficiently . Finally , [ 18 ] , studies learning Boolean circuits of depth O ( log n ) via neural networks under product distributions using layer-wise gradient descent . While [ 18 ] requires the architecture to match the Boolean circuit being learned , in contrast , our architecture is regular and independent of the function learned . Power of depth Several works have studied how representation power depends on depth . [ 19 ] shows that deep networks can represent a class of compositionally-created functions more efficiently than shallow networks . [ 20 ] shows that certain smooth radial functions can be easily represented by three-layer networks but need exponentially-many neurons to be represented by a two-layer network . Based on an analysis of learning fractal distributions related to the Cantor set , [ 21 ] conjectures that if shallow networks are poorly represent a target function , then a deep network can not be trained efficiently using gradient based methods . [ 22 ] presents a depth separation result by showing that deep networks can produce highly oscillatory functions by building on the oscillations layer by layer . [ 23 ] uses this phenomenon to show a sharp representation theorem for arbitrary-depth ReLU networks . Other theoretical works have proved depth separation theorems for training . [ 24 ] prove that a twohidden-layer neural network where the first hidden layer is kept random and only the second layer is trained , provably outperforms a just one hidden layer network . In [ 25 , 26 ] it is proved that deep networks trained end-to-end with SGD and quadratic activation functions can efficiently learn a non-trivial concept class hierarchically , whereas kernel methods and lower-depth networks provably fail to do so . The class of functions studied by [ 25 , 26 ] are those representable as the sum of neurons in a teacher network that is well-conditioned , has quadratic activations and has a depth of at most log log n , where n is the number of inputs . This function class is expressive but incomparable to the hierarchical function class studied in our work ( e.g. , we can learn polynomials up to degree n , 2These reductions are for polynomial-time algorithms and for polynomial precisions on the gradients . whereas [ 26 ] is limited to degree log ( n ) ) . Furthermore , our function class has the advantage of being naturally interpretable , with complex features ( the high-order monomials ) being built in a transparent way from simple features ( the low-order monomials ) . In [ 27–29 ] , gradient dynamics are explored for the simplified case of deep linear networks , where an ‘ incremental learning ’ phenomenon is observed in which the singular values are learned sequentially , one after the other . This phenomenon is reminiscent of the incremental learning of the Fourier coefficients of the Boolean function in our setting ( Fig . 2 ) . For real world data sets , [ 30 ] empirically shows that in many data sets of interest , simple neural networks trained with SGD first fit a linear classifier to the data and then progressively improve the approximation , similar in spirit to the theoretical results in this paper . Neural Tangent Kernel and random features A sequence of papers have studied convergence of overparametrized ( or Neural Tangent Kernel regime ) neural networks to the global minimizer of empirical loss when trained via gradient descent . In this regime , they show that neural networks behave like kernel methods and give training and/or generalization guarantees . Because of the reduction to kernels , these results are essentially non-hierarchical . [ 31 , 32 ] in fact show that deep networks in the NTK regime behave no better than shallow networks . We refer to [ 26 ] for a review of the literature related to NTK and shallow learning . Finally , we mention the related works [ 33–35 ] which consider learning low-degree ( degree q ) polynomials over Rn , without a hierarchical structure assumption . They require n⌦ ( q ) neurons to learn such functions , which is super-polynomial once q 1 . The results hold in the random features regime , known to be weaker than NTK .

Summary: In order to formalize the intuition that neural nets learn by building hierarchical features, this paper considers a model of learning Boolean functions shoe spectrum satisfies a certain  "staircase" structure. Roughly speaking, for every non-zero coefficient in the support, there is a subset of it that also has a non-zero coefficient. This paper shows that a certain family of neural networks will successfully learn such functions. In contrast, earning parity functions along is hard.