MANIFOLDNET: A DEEP NEURAL NETWORK FOR MANIFOLD-VALUED DATA

1 INTRODUCTION . Convolutional neural networks ( CNNs ) have attracted enormous attention in the past decade due to their significant success in Computer Vision , Speech Analysis and other fields . CNNs were pioneered by LeCun et al . ( 1998 ) and gained much popularity ever since their significant success on Imagenet data reported in Krizhevsky et al . ( 2012 ) . CNNs have traditionally been restricted to dealing with data residing in vector spaces . However , in the past few years , there is growing interest in generalizing the CNNs and deep networks in general to data that reside on smooth non-Euclidean spaces . In this context , at the outset , it would be useful to categorize problems into 1 ) those that involve data as samples of real-valued functions defined on a manifold and 2 ) those that are simply manifold-valued and hence are sample points on a manifold . In the context of input data being samples of functions defined on smooth manifolds , recently there has been a flurry of activity in developing methods that can cope specifically with samples of functions on a sphere that are encountered in many applications such as , omnidirectional cameras on drones , robots etc. , meteorological data and many others . The key property that allows learned weight sharing in CNNs is the equivariance to translations . The simplest technique to achieve equivariance is via data augmentation ( Krizhevsky et al. , 2012 ; Dieleman et al. , 2015 ) . Cascade of wavelet transforms to achieve equivariance was shown in Bruna & Mallat ( 2013 ) ; Oyallon & Mallat ( 2015 ) . In Gens ( 2014 ) , authors describe ‘ Symnet ’ , which achieves invariance to symmetry group actions . Equivariance to discrete group actions was achieved through parameter sharing in Ravanbakhsh et al . ( 2017 ) . For the case of data on a spherical domain , one considers exploiting equivariance to the rotation group . Several research groups recently reported spherical-CNNs ( SCNNs ) to accommodate such an equivariance in defining the convolution of functions ( Worrall et al. , 2017 ; Cohen & Welling , 2016 ; Cohen et al. , 2018 ) . In another recent work Esteves et al . ( 2017 ) , authors describe a polar transformer network , which is equivariant to rotations and scaling transformations . By combining this with a spatial transformer ( Jaderberg et al. , 2015 ) , they achieve equivariance to translations as well . More generally , equivariance of convolution operations to group actions admitted by the underlying manifold is what is needed to this end and most recent work reported in Chakraborty et al . ( 2018 ) ; Kondor & Trivedi ( 2018 ) achieves this for Riemannian homogeneous spaces . In this paper we will consider the second problem , namely , when the input data are sample points on known Riemannian manifolds for example , the manifold of symmetric positive definite matrices , SPD ( n ) , the special orthogonal group , SO ( n ) , the n-sphere , Sn , the Grassmannian , Gr ( p , n ) , and others . To be precise , the domain of interest is an n-dimensional field of points sampled from a Riemannian manifold . There is very little prior work that we are aware of on DNNs that can cope with input data samples residing on these manifolds with the exception of Huang et al . ( 2016 ) ; Huang & Van Gool ( 2017 ) . In Huang et al . ( 2016 ) , authors presented a deep network architecture for classification of hand-crafted features residing on a Grassmann manifold that form the input to the network . In Huang & Van Gool ( 2017 ) , authors presented a deep network architecture for data on SPD ( n ) . In both of these works , the architecture does not involve the use of any convolution or equivalent operations on Gr ( p , n ) or SPD ( n ) . Further , it does not use the natural invariant metric or intrinsic operations on the Grassmannian or the SPD ( n ) in the network blocks . Using intrinsic operations within the layers guarantees that the result remains on the manifold and hence does not require any projection ( extrinsic ) operations to ensure the result lies in the same space . Further , using extrinsic operations can yield results that are susceptible to significant inaccuracies when the data variance is large ( Salehian et al. , 2015 ) . Moreover , since there are no convolution type operations defined for data on these manifolds in their network , it can not be considered a generalization to the CNN and as a consequence does not consider equivariance property to the action of the group of isometries denoted by I ( M ) , admitted by the manifoldM . There are several deep networks reported in literature to deal with cases when data reside on 2- manifolds encountered in Computer Vision and Graphics for modeling shapes of objects . Some of these are based on graph-based representations of points on the surfaces in 3D and a generalization of CNNs to graphs ( Henaff et al. , 2015 ; Defferrard et al. , 2016 ) . There is also recent work in Masci et al . ( 2015 ) where the authors presented a deep network called geodesic CNN ( GCNN ) , where convolutions are performed in local geodesic polar charts constructed on the manifold . For more literature on deep networks for data on 2-manifolds , we refer the interested reader to a recent survey paper Bronstein et al . ( 2017 ) and references therein . In this paper , we present a novel DNN framework called the ManifoldNet . This is a potential analog of a CNN and can cope with input data sampled from a Riemannian manifold . The intuition in defining the analog relies on the equivariance property . Note that convolution of functions in vector spaces are equivariant to translations in the spatial domain and in the pixel domain . I.e . if the input pixels are all translated by a fixed amount , then the output pixels are equally translated . Further , it is easy to show that traditional convolutions are equivalent to computing the weighted mean ( Goh et al. , 2011 ) . Hence , for the case of manifold-valued data , we can define the analogous operation of a weighted Fréchet mean ( wFM ) and prove that it is equivariant to the action of I ( M ) . This will be achieved in a subsequent section . Our key contributions in this work are : [ presented in section 2 ] ( i ) we define the analog of convolution operations for manifold-valued data to be one of estimating the wFM for which we present a provably convergent , efficient and recursive estimator . ( ii ) A proof of equivariance of wFM to the action of I ( M ) . This equivariance allows the network to share weights within the layers . ( iii ) A novel deep architecture involving the Riemannian counterparts to the conventional CNN units . [ presented in section 3 ] ( iv ) Several real data experiments on classification and reconstruction demonstrating the performance of the ManifoldNet . 2 GROUP ACTION EQUIVARIANT NETWORK FOR MANIFOLD-VALUED DATA . In this section , we will define the equivalent of a convolution operation on Riemannian manifolds . As mentioned in the introduction , the domain of interest is an n-dimensional field of manifold valued points . Before formally defining such an operation and building the DNN for the manifold-valued data , dubbed a ManifoldNet , we first present some relevant concepts from differential geometry that will be used in the rest of the paper . Preliminaries . Let ( M , gM ) be a orientable complete Riemannian manifold with a Riemannian metric gM , i.e. , ( ∀x ∈ M ) gMx : TxM× TxM → R is a bi-linear symmetric positive definite map , where TxM is the tangent space ofM at x ∈ M. Let d : M×M → [ 0 , ∞ ) be the metric ( distance ) induced by the Riemannian metric gM . With a slight abuse of notation we will denote a Riemannian manifold ( M , gM ) byM unless specified otherwise . Let ∆ be the supremum of the sectional curvatures ofM . Definition 1 . Let p ∈ M , r > 0 . Define Br ( p ) = { q ∈M|d ( p , q ) < r } to be a open ball at p of radius r. Definition 2 . ( Groisser , 2004 ) The local injectivity radius at p ∈M , rinj ( p ) , is defined as rinj ( p ) = sup { r|Expp : ( Br ( 0 ) ⊂ TpM ) →M is defined and is a diffeomorphism onto its image } . The injectivity radius Manton ( 2004 ) ofM is defined as rinj ( M ) = infp∈M { rinj ( p ) } . Within Br ( p ) , where r ≤ rinj ( M ) , the mapping Exp−1p : Br ( p ) → U ⊂ TpM , is called the inverse Exponential/ Log map . Definition 3 . ( Kendall , 1990 ) An open ball Br ( p ) is a regular geodesic ball if r < rinj ( p ) and r < π/ ( 2∆1/2 ) . In Definition 3 and below , we interpret 1/∆1/2 as∞ if ∆ ≤ 0 . It is well known that , if p and q are two points in a regular geodesic ball Br ( p ) , then they are joined by a unique geodesic within Br ( p ) ( Kendall , 1990 ) . Definition 4 . ( Chavel , 2006 ) U ⊂ M is strongly convex if for all p , q ∈ U , there exists a unique length minimizing geodesic segment between p and q and the geodesic segment lies entirely in U . Definition 5 . ( Groisser , 2004 ) Let p ∈ M. The local convexity radius at p , rcvx ( p ) , is defined as rcvx ( p ) = sup { r ≤ rinj ( p ) |Br ( p ) is strongly convex } . The convexity radius ofM is defined as rcvx ( M ) = infp∈M { rcvx ( p ) } . For the rest of the paper , we will assume that the samples onM lie inside an open ball U = Br ( p ) where r = min { rcvx ( M ) , rinj ( M ) } , for some p ∈ M , unless mentioned otherwise . Now , we are ready to define the operations necessary to develop the ManifoldNet . 2.1 WFM ONM AS A GENERALIZATION OF CONVOLUTION We will now define a convolution type operation on points sampled from M. This convolution operation will perform an averaging operation over a moving window , but will replace weighted sums with weighted intrinsic averages . Let { Xi } Ni=1 be the manifold-valued samples onM . We define the convolution type operation onM as the weighted Fréchet mean ( wFM ) ( Maurice Fréchet , 1948 ) of the samples { Xi } Ni=1 . Also , by the aforementioned condition on the samples , the existence and uniqueness of FM is guaranteed ( Afsari , 2011 ) . As mentioned earlier , it is easy to show ( see Goh et al . ( 2011 ) ) . that convolution ψ∗ = b ? a of two functions a : X ⊂ Rn → R and b : X ⊂ Rn → R can be formulated as computation of the weighted mean ψ∗ = argminψ ∫ a ( u ) ( ψ − b̃u ) 2du , where , ∀x ∈ X , b̃u ( x ) = b ( u + x ) and ∫ a ( x ) dx = 1 . Here , f2 for any function f is defined pointwise . Further , the defining property of convolutions in vector spaces is the linear translation equivariance . Since weighted mean in vector spaces can be generalized to wFM on manifolds and further , wFM can be shown ( see below ) to be equivariant to group actions admitted by the manifold , we claim that wFM is a generalization of convolution operations to manifold-valued data . Let { wi } Ni=1 be the weights such that they satisfy the convexity constraint , i.e. , ∀i , wi > 0 and∑ i wi = 1 , then wFM , wFM ( { Xi } , { wi } ) is defined as : wFM ( { Xi } , { wi } ) = argmin M∈M N∑ i=1 wid 2 ( Xi , M ) ( 1 ) Analogous to the equivariance property of convolution translations in vector spaces , we will now proceed to show that the wFM is equivariant under the action of the group of isometries ofM . We will first formally define the group of isometries ofM ( let us denote it by G ) and then define the equivariance property and show that wFM is G-equivariant . Definition 6 ( Group of isometries ofM ( I ( M ) ) ) . A diffeomorphism φ : M→M is an isometry if it preserves distance , i.e. , d ( φ ( x ) , φ ( y ) ) = d ( x , y ) . The set I ( M ) of all isometries ofM forms a group with respect to function composition . Rather than write an isometry as a function φ , we will write it as a group action . Henceforth , let G denote the group I ( M ) , and for g ∈ G , and x ∈M , let g.x denote the result of applying the isometry g to point x. ClearlyM is a G set ( see Dummit & Foote ( 2004 ) for the definition of a G set ) . We will now define equivariance and show that wFM , is G-equivariant . Definition 7 ( Equivariance ) . LetX and Y beG sets . Then , F : X → Y is said to beG-equivariant if ∀g ∈ G , ∀x ∈ X , F ( g.x ) = g.F ( x ) . Let U ⊂ M be an open ball inside which FM exists and is unique , let P be the set consists of all possible finite subsets of U . Theorem 1 . Given { wi } satisfying the convex constraint , let F : P → U be a function defined by { Xi } 7→ wFM ( { Xi } , { wi } ) . Then , F is G-equivariant . Proof . Let g ∈ G and { Xi } Ni=1 ∈ P , now , let M∗ = wFM ( { Xi } , { wi } ) , as g.F ( { Xi } ) = g.M∗ , it suffices to show g.M∗ is wFM ( { g.Xi } , { wi } ) ( assuming the existence and uniqueness of wFM ( { g.Xi } , { wi } ) which is stated in the following claim ) . Claim : Let U = Br ( p ) for some r > 0 and p ∈ M. Then , { g.Xi } ⊂ Br ( g.p ) and hence wFM ( { g.Xi } , { wi } ) exists and is unique . Let M̃ be wFM ( { g.Xi } , { wi } ) . Then , ∑N i=1 wid 2 ( g.Xi , M̃ ) = ∑N i=1 wid 2 ( Xi , g −1.M̃ ) . Since , M∗ = wFM ( { Xi } , { wi } ) , hence , M∗ = g−1.M̃ , i.e. , M̃ = g.M∗ . Thus , g.M∗ = wFM ( { g.Xi } , { wi } ) , which implies F is G-equivariant . Now we give some examples ofM with the corresponding group of isometries G. LetM = SPD ( n ) ( the space of n×n symmetric positive-definite matrices ) . Let d be the Stein metric on SPD ( n ) . Then , the group of isometries G is O ( n ) ( the space of n× n orthogonal matrices ) . A class of Riemannian manifolds on which G acts transitively are called Riemannian homogeneous spaces . We can see that on a Riemannian homogeneous spaceM , wFM is G-equivariant . Equipped with a G-equivariant operator onM , we can claim that the wFM ( defined above ) is a valid convolution operator since group equivariance is a unique defining property of a convolution operator . The rest of this subsection will be devoted to developing an efficient way to compute wFM . Let ωM > 0 be the Riemannian volume form . Let pX be the probability density of a U -valued random variable X with respect to ωM on U ⊂ M , so that Pr ( X ∈ A ) = ∫ A pX ( Y ) ω M ( Y ) for any Borel-measurable subset A of U . Let Y ∈ U , we can define the expectation of the real valued random variable d2 ( , Y ) : U → R by E [ d2 ( , Y ) ] = ∫ U d2 ( X , Y ) pX ( X ) ω M ( X ) . Now , let w : U → ( 0 , ∞ ) be an integrable function and ∫ U w ( X ) ωM ( X ) = 1 . Then , observe that , Ew [ d2 ( , Y ) ] : = ∫ U w ( X ) d2 ( X , Y ) pX ( X ) ω M ( X ) = C ∫ U d2 ( X , Y ) p̃X ( X ) ωM ( X ) = C Ẽ [ d2 ( , Y ) ] . Here , p̃X is the probability density corresponding to the probability measure P̃r defined by , P̃r ( X ∈ X ) = ∫ X p̃X ( Y ) ω M ( Y ) : = ∫ X 1 C pX ( Y ) w ( Y ) ω M ( Y ) , where , X lies in the Borel σ-algebra over U and C = ∫ U pX ( Y ) w ( Y ) ω M ( Y ) . Note that the constant C > 0 , since pX is a probability density , w > 0 andM is orientable . Thus , Ew [ d2 ( , Y ) ] with respect to pX is proportional to Ẽ [ d2 ( , Y ) ] with respect to p̃X . Now , we will state the following proposition ( the proof is in the appendix section ) . Proposition 2 . ( i ) supp ( pX ) = supp ( p̃X ) . ( ii ) wFE ( X , w ) = FE ( X̃ ) . Let { Xi } Ni=1 be samples drawn from pX and { X̃i } N i=1 be samples drawn from p̃X . In order to compute wFM , we will now present an online algorithm ( inductive FM Estimator – dubbed iFME ) . Given , { Xi } Ni=1 ⊂ U and { wi : = w ( Xi ) } N i=1 such that ∀i , wi > 0 , the nth estimate , Mn of wFM ( { Xi } , { wi } ) is given by the following recursion : M1 = X1 Mn = Γ Xn Mn−1 ( wn∑n j=1 wj ) . ( 2 ) In the above equation , ΓYX : [ 0 , 1 ] → U is the shortest geodesic curve from X to Y . Observe that , in general wFM is defined with ∑N i=1 wi = 1 , but in above definition , ∑N i=1 wi 6= 1 . We can normalize { wi } to get { w̃i } by w̃i = wi/ ( ∑ i wi ) , but then Eq . 2 will not change as w̃n/ ( ∑n j=1 w̃j ) = wn/ ( ∑n j=1 wj ) . This gives us an efficient inductive/recursive way to define convolution operation onM . Now , we state that the proposed wFM estimator is consistent ( the proof is in the appendix ) . Proposition 3 . Using the above notations and assumptions , let { Xi } Ni=1 be i.i.d . samples drawn from pX onM . Let the wFE be finite . Then , MN converges a.s. to wFE as N →∞ . 2.2 NONLINEAR OPERATION BETWEEN WFM-LAYERS FORM-VALUED DATA In the traditional CNN model , we need a nonlinear function between two convolutional layers similar to ReLU and softmax . As argued in Mallat ( 2016 ) , any nonlinear function used in CNN is basically a contraction mapping . Formally , let F be a nonlinear mapping from U to V . Let assume , U and V are metric spaces equipped with metric dU and dV respectively . Then , F is a contraction mapping iff ∃c < 1 such that , dV ( F ( x ) , F ( y ) ) ≤ c dU ( x , y ) . F is a non-expansive mapping ( Mallat , 2016 ) iff dV ( F ( x ) , F ( y ) ) ≤ dU ( x , y ) . One can easily see that the popular choices for nonlinear operations like ReLU , sigmoid are in fact non-expansive mappings . Now , we will show that the function wFM as defined in 1 , is a contraction mapping for non-trivial choices of weights . Let { Xi } Ni=1 and { Yj } M j=1 be the two set of samples onM . Without any loss of generality , assume N ≤M . We consider the set UM = U × · · · × U︸ ︷︷ ︸ M times . Clearly { Yj } Mj=1 ∈ U M and we embed { Xi } Ni=1 in UM as follows : we construct { X̃i } M i=1 from { Xi } Ni=1 by defining X̃i = X ( i−1 ) modN+1 . Let us denote the embedding by ι . Now , define the distance on UM as d ( { X̃i } M i=1 , { Yj } Mj=1 ) = maxi , j d ( Xi , Yj ) . The choice of weights for wFM is said to be trivial if one of the weights is 1 and hence the rest are 0 . Proposition 4 . For all nontrivial choices of { αi } Ni=1 and { βj } M j=1 satisfying the convexity constraint , ∃c < 1 such that , d ( wFM ( { Xi } Ni=1 , { αi } N i=1 ) , wFM ( { Yj } Mi=1 , { βj } M i=1 ) ) ≤ c d ( ι ( { Xi } Ni=1 ) , { Yj } Mj=1 ) ( 3 )

Summary: This paper introduces a generalization of convolution operations to manifold-valued data using  the computation of the weighted Frechet mean (wFM). Without applying any non-linear units and pooling layers, the paper proposes to merely stack suching generalized convolutional layers to construct a deep network for the data residing on Riemannian manifolds. The evaluations on video classification and reconstruction tasks show the advantages of the introduced network over some baselines.