Universal Approximation Theorem for Equivariant Maps by Group CNNs

Group symmetry is inherent in a wide variety of data distributions . Data processing that preserves symmetry is described as an equivariant map and often effective in achieving high performance . Convolutional neural networks ( CNNs ) have been known as models with equivariance and shown to approximate equivariant maps for some specific groups . However , universal approximation theorems for CNNs have been separately derived with individual techniques according to each group and setting . This paper provides a unified method to obtain universal approximation theorems for equivariant maps by CNNs in various settings . As its significant advantage , we can handle non-linear equivariant maps between infinite-dimensional spaces for non-compact groups . 1 INTRODUCTION . Deep neural networks have been widely used as models to approximate underlying functions in various machine learning tasks . The expressive power of fully-connected deep neural networks was first mathematically guaranteed by the universal approximation theorem in Cybenko ( 1989 ) , which states that any continuous function on a compact domain can be approximated with any precision by an appropriate neural network with sufficient width and depth . Beyond the classical result stated above , several types of variants of the universal approximation theorem have also been investigated under different conditions . Among a wide variety of deep neural networks , convolutional neural networks ( CNNs ) have achieved impressive performance for real applications . In particular , almost all of state-of-the-art models for image recognition are based on CNNs . These successes are closely related to the property that performing CNNs commute with translation on pixel coordinate . That is , CNNs can conserve symmetry about translation in image data . In general , this kind of property for symmetry is known as the equivariance , which is a generalization of the invariance . When a data distribution has some symmetry and the task to be solved relates to the symmetry , data processing is desired to be equivariant on the symmetry . In recent years , different types of symmetry have been focused per each task , and it has been proven that CNNs can approximate arbitrary equivariant data processing for specific symmetry . These results are mathematically captured as the universal approximation for equivariant maps and represent the theoretical validity of the use of CNNs . In order to theoretically correctly handle symmetric structures , we have to carefully consider the structure of data space where data distributions are defined . For example , in image recognition tasks , image data are often supposed to have symmetry for translation . When each image data is acquired , there are finite pixels equipped with an image sensor , and an image data is represented by a finitedimensional vector in a Euclidean space Rd , where d is the number of pixels . However , we note that the finiteness of pixels stems from the limit of the image sensor and a raw scene behind the image data is thought to be modelled by an element in RS with continuous spatial coordinates S , where RS is a set of functions from S to R. Then , the element in RS is regarded as a functional representation of the image data in Rd . In this paper , in order to appropriately formulate data symmetry , we treat both typical data representation in finite-dimensional settings and functional representation in infinite-dimensional settings in a unified manner . 1.1 RELATED WORKS . Symmetry and functional representation . Symmetry is mathematically described in terms of groups and has become an essential concept in machine learning . Gordon et al . ( 2019 ) point out that , when data symmetry is represented by a infinite group like the translation group , equivariant maps , which are symmetry-preserving processing , can not be captured as maps between finitedimensional spaces but can be described by maps between infinite-dimensional function spaces . As a related study about symmetry-preserving processing , Finzi et al . ( 2020 ) propose group convolution of functional representations and investigate practical computational methods such as discretization and localization . Universal approximation for continuous maps . The universal approximation theorem , which is the main objective of this paper , is one of the most classical mathematical theorems of neural networks . The universal approximation theorem states that a feedforward fully-connected network ( FNN ) with a single hidden layer containing finite neurons can approximate a continuous function on a compact subset of Rd . Cybenko ( 1989 ) proved this theorem for the sigmoid activation function . After his work , some researchers showed similar results to generalize the sigmoidal function to a larger class of activation functions as Barron ( 1994 ) , Hornik et al . ( 1989 ) , Funahashi ( 1989 ) , Kůrková ( 1992 ) and Sonoda & Murata ( 2017 ) . These results were approximations to functional representations between finite-dimensional vector spaces , but recently Guss & Salakhutdinov ( 2019 ) generalized them to continuous maps between infinite-dimensional function spaces in Guss & Salakhutdinov ( 2019 ) . Equivariant neural networks . The concept of group-invariant neural networks was first introduced in Shawe-Taylor ( 1989 ) in the case of permutation groups . In addition to the invariant case , Zaheer et al . ( 2017a ) designed group-equivariant neural networks for permutation groups and obtained excellent results in many applications . Maron et al . ( 2019a ; 2020 ) consider and develop a theory of equivariant tensor networks for general finite groups . Petersen & Voigtlaender ( 2020 ) established a connection between group CNNs , which are equivariant networks , and FNNs for group finites . However , symmetry are not limited to finite groups . Convolutional neural networks ( CNNs ) was designed to be equivariant for translation groups and achieved impressive performance in a wide variety of tasks . Gens & Domingos ( 2014 ) proposed architectures that are based on CNNs and invariant to more general groups including affine groups . Motivated by CNN ’ s experimental success , many researchers have further generalized this by using group theory . Kondor & Trivedi ( 2018 ) proved that , when a group is compact and the group action is transitive , a neural network constrained by some homogeneous structure is equivariant if and only if it becomes a group CNN . Universal approximation for equivariant maps . Compared to the vast studies about universal approximation for continuous maps , there are few existing studies about universal approximation for equivariant maps . Sannai et al . ( 2019 ) ; Ravanbakhsh ( 2020 ) ; Keriven & Peyré ( 2019 ) considered the equivariant model for finite groups and proved universal approximation property of them by attributing it to the results of Maron et al . ( 2019b ) . Cohen et al . ( 2019 ) considered group convolution on a homogeneous space and proved that a linear equivariant map is always convolution-like . Yarotsky ( 2018 ) proved universal approximation theorems for nonlinear equivariant maps by CNNlike models when groups are the d-dimensional translation group T ( d ) = Rd or the 2-dimensional Euclidean group SE ( 2 ) . However , when groups are more general , universal approximation theorems for non-linear equivariant maps have not been obtained . 1.2 PAPER ORGANIZATION AND OUR CONTRIBUTIONS . The paper is organized as follows . In section 2 , we introduce the definition of group equivariant maps and provide the essential property that equivariant maps have one-to-one correspondence to theoretically tractable maps called generators . In section 3 , we define fully-connected and group convolutional neural networks between function spaces . This formulation is suitable to represent data symmetry . Then , we provide a main theorem called the conversion theorem that can convert FNNs to CNNs . In section 4 , using the conversion theorem , we derive universal approximation theorems for non-linear equivariant maps by group CNNs . In particular , this is the first universal approximation theorem for equivariant maps in infinite-dimensional settings . We note that finite and infinite groups are handled in a unified manner . In section 5 , we provide concluding remarks and mention future works . 2 GROUP EQUIVARIANCE . 2.1 PRELIMINARIES . We introduce definitions and terminology used in the later discussion . Functional representation . In this paper , sets denoted by S , T and G are assumed to be locally compact , σ-compact , Hausdorff spaces . When S is a set , we denote by RS the set of all maps from S to R and by ‖ · ‖∞ the supremum norm . We call S of RS the index set . We denote by C ( S ) the set of all continuous maps from S to R. We denote by C0 ( S ) the set of continuous functions from S to R which vanish at infinity1 . For a Borel space S with some measure µ , we denote the set of integrable functions from S to R with respect to µ as L1µ ( S ) . For a subset B ⊂ S , the restriction map RB : RS → RB is defined by RB ( x ) = x|B , where x ∈ RS and x|B is the restriction of the domain of x onto B . When S is a finite set , RS is identified with the finite-dimensional Euclidean space R|S| , where |S| is the cardinality of S. In this sense , RS for general sets S is a generalization of Euclidean spaces . However , RS itself is often intractable for an infinite set S. In such cases , we instead consider C ( S ) , C0 ( S ) or Lp ( S ) as relatively tractable subspaces of RS . Group action . We denote the identity element in a group G by 1 . We assume that the action of a group G on a set S is continuous . We denote by g · s the left action of g ∈ G to s ∈ S . Then we call Gs : = { g · s|g ∈ G } the orbit of s ∈ S . From the definition , we have S = ⋃ s∈S Gs . When a subset B ⊂ S is the set of representative elements from all orbits , it satisfies the disjoint condition S = ⊔ s∈B Gs . Then , we call B a base space2 and define the projection PB : S → B by mapping s ∈ S to the representative element in B ∩Gs . When a group G acts on sets S and T , the action of G on the product space S ×T is defined by g · ( s , t ) : = ( g · s , g · t ) . When a group G acts on a index set S , the G-translation operators Tg : RS → RS for g ∈ G are defined by Tg [ x ] ( s ) : = x ( g−1 · s ) , where x ∈ RS and s ∈ S . We often denote Tg [ x ] simply by g ·x for brevity . Then , group translation determine the action3 of G on RS . 2.2 GROUP EQUIVARIANT MAPS . In this section , we introduce group equivariant maps and show their basic properties . First , we define group equivariance . Definition 1 ( Group Equivariance ) . Suppose that a group G acts on sets S and T . Then , a map F : RS → RT is called G-equivariant when F [ g · x ] = g · F [ x ] holds for any g ∈ G and x ∈ RS . An example of an equivariant map in image processing is provided in Figure 1 . To clarify the degree of freedom of equivariant maps , we define the generator of equivariant maps . Definition 2 ( Generator ) . Let B ⊂ T be a base space with respect to the action of G on T . For a G-equivariant map F : RS → RT , we call FB : = RB ◦ F the generator of F . The following theorem shows that equivariant maps can be represented by their generators . Theorem 3 ( Degree of Freedom of Equivariant Maps ) . Let a group G act on sets S and T , and B ⊂ T a base space . Then , a G-equivariant map F : RS → RT has one-to-one correspondence to its generator FB . A detailed version of Theorem 3 is proved in Section A.1 . 1A function f on a locally compact space is said to vanish at infinity if , for any ϵ , there exists a compact subset K ⊂ S such that sups∈S\K |f ( s ) | < ϵ . 2The choice of the base space is not unique in general . However , the topological structure of a base space can be induced by the quotient space S/G . 3We note that Tg ◦ Tg′ = Tg′g and the group translation operator is the action of G on RS from the right .

Summary: This paper considers a certain generalization of convolutional neural networks and equivariant linear networks to the infinite dimensional case, while covering also the discrete case, and offers a universality result. In more detail, the paper first characterizes  equivariant maps as the unique  extensions of "generator", namely regular maps that provide target functions (or vector) defined over a basic domain. In other words, any map that takes functions (or vectors) into functions (or vectors) defined over the representatives from the symmetry's equivalent classes can be extended uniquely to an equivariant map by enlarging its target domain according to the equivariance rule. Second, infinite dimensional fully connected networks (FNNs) and general (equivariant) convolution neural networks (CNNs) are described. The main result of the paper is the Conversion Theorem (Theorem 11), and its consequences. The theorem specify the conditions under which an FNN can be approximated by a CNN. Since FNNs are known to be universal this implies universality of CNNs.