Improved Generalization Bound of Permutation Invariant Deep Neural Networks

√ n ! where n is a number of permuting coordinates of data . Moreover , we prove that an approximation power of invariant deep neural networks can achieve an optimal rate , though the networks are restricted to be invariant . To achieve the results , we develop several new proof techniques such as correspondence with a fundamental domain and a scale-sensitive metric entropy . 1 INTRODUCTION . A learning task with permutation invariant data frequently appears in various situations in data analysis . A typical example is learning on sets such as a point cloud , namely , the data are given as a set of points and permuting the points in the data does not change a result of its prediction . Another example is learning with graphs which contain a huge number of edges and nodes . Such the tasks are very common in various scientific fields ( Ntampaka et al. , 2016 ; Ravanbakhsh et al. , 2016 ; Faber et al. , 2016 ) , hence , numerous deep neural networks have been developed to handle such the data with invariance ( Zaheer et al. , 2017 ; Li et al. , 2018a ; Su et al. , 2018 ; Li et al. , 2018b ; Yang et al. , 2018 ; Xu et al. , 2018 ) . The succeeding methods show that their networks for invariance can greatly improve the accuracy with a limited size of networks and data . An important question with invariant data is to understand the reason for the empirical high accuracy from theoretical aspects . Since invariant data are high-dimensional in general , learning theory claims that the high-dimensionality reduces its generalization performance . However , the methods for invariant data achieve better accuracy , thus it contradicts the theoretical principle . Though several theoretical studies ( Maron et al . ( 2019 ) and Sannai et al . ( 2019 ) ) prove a universal approximation property of neural networks for invariant data and guarantee that invariant deep neural networks have sufficient expressive power , the generalization power of the invariant deep neural networks is left as an open question . In this paper , we prove a theoretical bound for generalization of invariant deep neural networks . To show an overview of our result , we provide a simplified version as follows . We consider a supervised-learning problem with m pairs of observations ( Xi , Yi ) where Xi are regarded as pdimensional vectors , and Xi can divided to n coordinates and each of them have D = p/n dimension . Also , let fSn denote a function by a deep neural network which satisfies an invariant property , f ( x ) = f ( σ · x ) holds for any x ∈ Rn×D where σ is an arbitrary permutation of D-dimensional coordinates in x . Also , we define Rm ( f ) = m−1 ∑m i=1 L ( Yi , f ( Xi ) ) and R ( f ) = E [ L ( Y , f ( X ) ) ] as an empirical and expected loss value L ( Y , f ( X ) ) . Then , we show that following : Theorem 1 ( Informal version of Theorem 2 ) . Let fSn be a function by a deep neural network which takes p-dimensional inputs and invariant to any permutations of n coordinates . Then , for sufficiently small ε > 0 , we obtain R ( fSn ) ≤ Rm ( fSn ) + √ C n ! mεp +O ( log ( 1/ε ) ) , with probability at least 1−O ( ε ) . Here , C > 0 is a constant independent of m and n. As a consequence of Theorem 1 , the generalization bound is improved by √ n ! by the invariant property . Since the number of coordinates n is huge in practice , e.g . there are n ≥ 1 , 000 points in the point cloud data in Zaheer et al . ( 2017 ) and hence √ n ! ≥ 101,000 holds , we show that the derived generalization bound is largely improved by invariance . Further , we also derive a rate of approximation of neural networks with invariance ( Theorem 4 ) and its optimality , thus we show that an invariance property for deep neural networks does not reduce an expressive power . From a technical aspect , we develop mainly three proof technique to obtain the improved bound in Theorem 1 . Firstly , we introduce a notion of a fundamental domain to handle invariance of functions and evaluate the complexity of the domain ( Lemma 1 ) . Secondly , we show a one-to-one correspondence between a function by invariant deep neural networks and a function on the fundamental domain ( Proposition 2 ) . Thirdly , we develop a scale-sensitive covering number to control a volume of invariant functions with neural networks ( Proposition 5 ) . Based on the techniques , we can connect a generalization analysis to the invariance of deep neural networks . We summarize the contributions of this paper as follow : • We investigate the generalization bound of deep neural networks which are invariant to permutation of n coordinates , then we show that the bound is improved by √ n ! . • We derive a rate of approximation of invariant deep neural networks . The result shows that the approximation rate is optimal . • We develop several proof techniques to achieve the bound such as a complexity analysis for a fundamental domain and a scale-sensitive metric entropy . 1.1 NOTATION . For a vector b ∈ RD , its d-th element is denoted by bd . Also , b−d : = ( b1 , ... , bd−1 , bd+1 , ... , bD ) ∈ RD−1 is a vector without bd . ‖b‖q : = ( ∑D j=d b q d ) 1/q is the q-norm for q ∈ [ 0 , ∞ ] . For a tensor A ∈ RD1×D2 , a ( d1 , d2 ) -th element of A is written as Ad1 , d2 . For a function f : Ω → R with a set Ω , ‖f‖Lq : = ( ∫ Ω |f ( x ) |qdx ) 1/q denotes the Lq-norm for q ∈ [ 0 , ∞ ] . For a subset Λ ⊂ Ω , f Λ denotes the restriction of f to Λ . For an integer z , z ! = ∏n j=1 j denotes a factorial of z . For a set Ω with a norm ‖ · ‖ , N ( ε , Ω , ‖ · ‖ ) : = inf { N : ∃ { ωj } Nj=1 s.t . ∪Nj=1 { ω : ‖ω − ωj‖ ≤ ε } ⊃ Ω } is a covering number of Ω with ε > 0 . For a set Ω , idΩ or id denotes the identity map on Ω , namely idΩ ( x ) = x for any x ∈ Ω . For a subset ∆ ⊂ Rn , int ( ∆ ) denotes the set of the inner points of ∆ . 2 PROBLEM SETTING . 2.1 INVARIANT DEEP NEURAL NETWORK . We define a set of permutation Sn in this paper . Consider x ∈ Rn×D where n be a number of coordinates in x and D be a dimension of each coordinate . Then , an action σ ∈ Sn on x is defined as ( σ · x ) i , d = xσ−1 ( i ) , d , i = 1 , ... , n , d = 1 , ... , D , here , σ is a permutation of indexes i . Also , we define an invariant property for general functions . Definition 1 ( Sn-Invariant/Equivariant Function ) . For a set X ⊂ Rn×D , we say that a map f : X → RM is • Sn-invariant ( or simply invariant ) if f ( σ · x ) = f ( x ) for any σ ∈ Sn and any x ∈ X , • Sn-equivariant ( or simply equivariant ) if there is an Sn-action on RM and f ( σ · x ) = σ · f ( x ) for any σ ∈ Sn and any x ∈ X . In this paper , we mainly treat fully connected deep neural networks with a ReLU activation function . The ReLU activation function is defined by ReLU ( x ) = max ( 0 , x ) . Deep neural networks are built by stacking blocks which consist of a linear map and a ReLU activation . More formally , it is a function Zi : Rdi → Rdi+1 defined by Zi ( x ) = ReLU ( Wix + bi ) , where Wi ∈ Rdi+1×di and bi ∈ Rdi+1 for i = 1 , ... , H . Here , H is a depth of the deep neural network and di is a width of the i-th layer . An output of deep neural networks is formulated as f ( x ) : = ZH ◦ ZH−1 . . . Z2 ◦ Z1 ( x ) . ( 1 ) Let FDNN be a set of functions by deep neural networks . We also consider an invariant deep neural network defined as follows : Definition 2 ( Invariant Deep Neural Network ) . f ∈ FDNN is a function by a Sn-invariant deep neural network , if f is a Sn-invariant function . Let FSnDNN ⊂ FDNN be a set of functions by Sn-invariant deep neural networks . The definition is a general notion and it contains several explicit invariant deep neural networks . We provide several representative examples as follow . Example 1 ( Deep Sets ) . Zaheer et al . ( 2017 ) develops an architecture for invariant deep neural networks by utilizing layer-wise equivariance . Their architecture consists of equivariant layers ` 1 , ... , ` j , an invariant linear layer h , and a fully-connected layer f ′ . For each ` i.i = 1 , .. , j , its parameter matrix is defined as Wi = λI + γ ( 11 > ) , λ , γ ∈ R,1 = [ 1 , ... , 1 ] > , which makes ` i as a layer-wise equivariant function . They show that f = f ′ ◦ h ◦ ` j ◦ · · · ` 1 is an invariant function . Its illustration is provided in Figure 1 . Example 2 ( Invariant Feature Extraction ) . Let e is a mapping for invariant feature extraction which will be explicitly constructed by deep neural networks in Proposition 2 . Then , a function f = g ◦ e where g is a function by deep neural networks with a restricted domain . Figure 2 provides its image . 2.2 LEARNING PROBLEM WITH INVARIANT NETWORK . Problem formulation : Let I = [ 0 , 1 ] n×D be an input space with dimension p = dD . Let Y be an output space . Also , let L : Y × Y → R be a loss function which satisfies supy , y′∈Y |L ( y , y′ ) | ≤ 1 and 1-Lipschitz continuous . Let P ∗ ( x , y ) be the true unknown distribution on I×Y , and for f : I → Y , R ( f ) = E ( X , Y ) ∼P∗ [ L ( f∗ ( X ) , Y ) ] be the expected risk of f . Also , suppose we observe a training datasetDm : = { ( X1 , Y1 ) , ... , ( Xm , Ym ) } of sizem . LetRm ( f ) : = m−1 ∑m i=1 L ( f ( Xi ) , Yi ) be the empirical risk of f . A goal of this study to investigate the expected lossR ( f ) with a function f from a set of functions as a hypothesis set . Learning with Invariant Network : We consider learning with a hypothesis set by invariant deep networks . Namely , we fix an architecture of deep neural networks preserves fSn ∈ FSnDNN to be an invariant function . Then , we evaluate the expected loss R ( fSn ) .

Summary: This paper derives a generalization bound for permutation invariant networks. The main idea is to prove that the bound is inversely proportional to the square-root of the number of possible permutations to the input. The key result is Theorem 3 that bounds the covering number of a neural network (defined under an approximation control bound, Thm 4) using the number of permutations. The paper proves the theorem by showing that the space of input permutations can reduced to group actions over a fundamental domain, and deriving a bound for the covering number of the fundamental domain (Lemma 1), which is then extended to derive the same for the neural network setting. For the permutation invariance setting, the fundamental domain is obtained via the sorting operator.