Improved Generalization Bound of Permutation Invariant Deep Neural Networks

√ n ! where n is a number of permuting coordinates of data . Moreover , we prove that an approximation power of invariant deep neural networks can achieve an optimal rate , though the networks are restricted to be invariant . To achieve the results , we develop several new proof techniques such as correspondence with a fundamental domain and a scale-sensitive metric entropy . 1 INTRODUCTION . A learning task with permutation invariant data frequently appears in various situations in data analysis . A typical example is learning on sets such as a point cloud , namely , the data are given as a set of points and permuting the points in the data does not change a result of its prediction . Another example is learning with graphs which contain a huge number of edges and nodes . Such the tasks are very common in various scientific fields ( Ntampaka et al. , 2016 ; Ravanbakhsh et al. , 2016 ; Faber et al. , 2016 ) , hence , numerous deep neural networks have been developed to handle such the data with invariance ( Zaheer et al. , 2017 ; Li et al. , 2018a ; Su et al. , 2018 ; Li et al. , 2018b ; Yang et al. , 2018 ; Xu et al. , 2018 ) . The succeeding methods show that their networks for invariance can greatly improve the accuracy with a limited size of networks and data . An important question with invariant data is to understand the reason for the empirical high accuracy from theoretical aspects . Since invariant data are high-dimensional in general , learning theory claims that the high-dimensionality reduces its generalization performance . However , the methods for invariant data achieve better accuracy , thus it contradicts the theoretical principle . Though several theoretical studies ( Maron et al . ( 2019 ) and Sannai et al . ( 2019 ) ) prove a universal approximation property of neural networks for invariant data and guarantee that invariant deep neural networks have sufficient expressive power , the generalization power of the invariant deep neural networks is left as an open question . In this paper , we prove a theoretical bound for generalization of invariant deep neural networks . To show an overview of our result , we provide a simplified version as follows . We consider a supervised-learning problem with m pairs of observations ( Xi , Yi ) where Xi are regarded as pdimensional vectors , and Xi can divided to n coordinates and each of them have D = p/n dimension . Also , let fSn denote a function by a deep neural network which satisfies an invariant property , f ( x ) = f ( σ · x ) holds for any x ∈ Rn×D where σ is an arbitrary permutation of D-dimensional coordinates in x . Also , we define Rm ( f ) = m−1 ∑m i=1 L ( Yi , f ( Xi ) ) and R ( f ) = E [ L ( Y , f ( X ) ) ] as an empirical and expected loss value L ( Y , f ( X ) ) . Then , we show that following : Theorem 1 ( Informal version of Theorem 2 ) . Let fSn be a function by a deep neural network which takes p-dimensional inputs and invariant to any permutations of n coordinates . Then , for sufficiently small ε > 0 , we obtain R ( fSn ) ≤ Rm ( fSn ) + √ C n ! mεp +O ( log ( 1/ε ) ) , with probability at least 1−O ( ε ) . Here , C > 0 is a constant independent of m and n. As a consequence of Theorem 1 , the generalization bound is improved by √ n ! by the invariant property . Since the number of coordinates n is huge in practice , e.g . there are n ≥ 1 , 000 points in the point cloud data in Zaheer et al . ( 2017 ) and hence √ n ! ≥ 101,000 holds , we show that the derived generalization bound is largely improved by invariance . Further , we also derive a rate of approximation of neural networks with invariance ( Theorem 4 ) and its optimality , thus we show that an invariance property for deep neural networks does not reduce an expressive power . From a technical aspect , we develop mainly three proof technique to obtain the improved bound in Theorem 1 . Firstly , we introduce a notion of a fundamental domain to handle invariance of functions and evaluate the complexity of the domain ( Lemma 1 ) . Secondly , we show a one-to-one correspondence between a function by invariant deep neural networks and a function on the fundamental domain ( Proposition 2 ) . Thirdly , we develop a scale-sensitive covering number to control a volume of invariant functions with neural networks ( Proposition 5 ) . Based on the techniques , we can connect a generalization analysis to the invariance of deep neural networks . We summarize the contributions of this paper as follow : • We investigate the generalization bound of deep neural networks which are invariant to permutation of n coordinates , then we show that the bound is improved by √ n ! . • We derive a rate of approximation of invariant deep neural networks . The result shows that the approximation rate is optimal . • We develop several proof techniques to achieve the bound such as a complexity analysis for a fundamental domain and a scale-sensitive metric entropy . 1.1 NOTATION . For a vector b ∈ RD , its d-th element is denoted by bd . Also , b−d : = ( b1 , ... , bd−1 , bd+1 , ... , bD ) ∈ RD−1 is a vector without bd . ‖b‖q : = ( ∑D j=d b q d ) 1/q is the q-norm for q ∈ [ 0 , ∞ ] . For a tensor A ∈ RD1×D2 , a ( d1 , d2 ) -th element of A is written as Ad1 , d2 . For a function f : Ω → R with a set Ω , ‖f‖Lq : = ( ∫ Ω |f ( x ) |qdx ) 1/q denotes the Lq-norm for q ∈ [ 0 , ∞ ] . For a subset Λ ⊂ Ω , f Λ denotes the restriction of f to Λ . For an integer z , z ! = ∏n j=1 j denotes a factorial of z . For a set Ω with a norm ‖ · ‖ , N ( ε , Ω , ‖ · ‖ ) : = inf { N : ∃ { ωj } Nj=1 s.t . ∪Nj=1 { ω : ‖ω − ωj‖ ≤ ε } ⊃ Ω } is a covering number of Ω with ε > 0 . For a set Ω , idΩ or id denotes the identity map on Ω , namely idΩ ( x ) = x for any x ∈ Ω . For a subset ∆ ⊂ Rn , int ( ∆ ) denotes the set of the inner points of ∆ . 2 PROBLEM SETTING . 2.1 INVARIANT DEEP NEURAL NETWORK . We define a set of permutation Sn in this paper . Consider x ∈ Rn×D where n be a number of coordinates in x and D be a dimension of each coordinate . Then , an action σ ∈ Sn on x is defined as ( σ · x ) i , d = xσ−1 ( i ) , d , i = 1 , ... , n , d = 1 , ... , D , here , σ is a permutation of indexes i . Also , we define an invariant property for general functions . Definition 1 ( Sn-Invariant/Equivariant Function ) . For a set X ⊂ Rn×D , we say that a map f : X → RM is • Sn-invariant ( or simply invariant ) if f ( σ · x ) = f ( x ) for any σ ∈ Sn and any x ∈ X , • Sn-equivariant ( or simply equivariant ) if there is an Sn-action on RM and f ( σ · x ) = σ · f ( x ) for any σ ∈ Sn and any x ∈ X . In this paper , we mainly treat fully connected deep neural networks with a ReLU activation function . The ReLU activation function is defined by ReLU ( x ) = max ( 0 , x ) . Deep neural networks are built by stacking blocks which consist of a linear map and a ReLU activation . More formally , it is a function Zi : Rdi → Rdi+1 defined by Zi ( x ) = ReLU ( Wix + bi ) , where Wi ∈ Rdi+1×di and bi ∈ Rdi+1 for i = 1 , ... , H . Here , H is a depth of the deep neural network and di is a width of the i-th layer . An output of deep neural networks is formulated as f ( x ) : = ZH ◦ ZH−1 . . . Z2 ◦ Z1 ( x ) . ( 1 ) Let FDNN be a set of functions by deep neural networks . We also consider an invariant deep neural network defined as follows : Definition 2 ( Invariant Deep Neural Network ) . f ∈ FDNN is a function by a Sn-invariant deep neural network , if f is a Sn-invariant function . Let FSnDNN ⊂ FDNN be a set of functions by Sn-invariant deep neural networks . The definition is a general notion and it contains several explicit invariant deep neural networks . We provide several representative examples as follow . Example 1 ( Deep Sets ) . Zaheer et al . ( 2017 ) develops an architecture for invariant deep neural networks by utilizing layer-wise equivariance . Their architecture consists of equivariant layers ` 1 , ... , ` j , an invariant linear layer h , and a fully-connected layer f ′ . For each ` i.i = 1 , .. , j , its parameter matrix is defined as Wi = λI + γ ( 11 > ) , λ , γ ∈ R,1 = [ 1 , ... , 1 ] > , which makes ` i as a layer-wise equivariant function . They show that f = f ′ ◦ h ◦ ` j ◦ · · · ` 1 is an invariant function . Its illustration is provided in Figure 1 . Example 2 ( Invariant Feature Extraction ) . Let e is a mapping for invariant feature extraction which will be explicitly constructed by deep neural networks in Proposition 2 . Then , a function f = g ◦ e where g is a function by deep neural networks with a restricted domain . Figure 2 provides its image . 2.2 LEARNING PROBLEM WITH INVARIANT NETWORK . Problem formulation : Let I = [ 0 , 1 ] n×D be an input space with dimension p = dD . Let Y be an output space . Also , let L : Y × Y → R be a loss function which satisfies supy , y′∈Y |L ( y , y′ ) | ≤ 1 and 1-Lipschitz continuous . Let P ∗ ( x , y ) be the true unknown distribution on I×Y , and for f : I → Y , R ( f ) = E ( X , Y ) ∼P∗ [ L ( f∗ ( X ) , Y ) ] be the expected risk of f . Also , suppose we observe a training datasetDm : = { ( X1 , Y1 ) , ... , ( Xm , Ym ) } of sizem . LetRm ( f ) : = m−1 ∑m i=1 L ( f ( Xi ) , Yi ) be the empirical risk of f . A goal of this study to investigate the expected lossR ( f ) with a function f from a set of functions as a hypothesis set . Learning with Invariant Network : We consider learning with a hypothesis set by invariant deep networks . Namely , we fix an architecture of deep neural networks preserves fSn ∈ FSnDNN to be an invariant function . Then , we evaluate the expected loss R ( fSn ) .

Summary: This paper presents a derivation of a generalization bound for neural networks designed specifically to deal with permutation invariant data (such as point clouds). The heart of the contribution is that the bound includes a  1/n! (i.e. 1 / (n-factorial)) factor to the major term, where n is the number of permutable elements there are in a data example (think: number of points in a point cloud). This term goes some way towards making the bound tight.