MANIFOLDNET: A DEEP NEURAL NETWORK FOR MANIFOLD-VALUED DATA
Year: 2018

Abstract:
Developing deep neural networks (DNNs) for manifold-valued data sets has gained much interest of late in the deep learning research community. Examples of manifold-valued data include data from omnidirectional cameras on automobiles, drones etc., diffusion magnetic resonance imaging, elastography and others. In this paper, we present a novel theoretical framework for DNNs to cope with manifold-valued data inputs. In doing this generalization, we draw parallels to the widely popular convolutional neural networks (CNNs). We call our network the ManifoldNet. As in vector spaces where convolutions are equivalent to computing the weighted mean of functions, an analogous definition for manifold-valued data can be constructed involving the computation of the weighted Fréchet Mean (wFM). To this end, we present a provably convergent recursive computation of the wFM of the given data, where the weights makeup the convolution mask, to be learned. Further, we prove that the proposed wFM layer achieves a contraction mapping and hence the ManifoldNet does not need the additional non-linear ReLU unit used in standard CNNs. Operations such as pooling in traditional CNN are no longer necessary in this setting since wFM is already a pooling type operation. Analogous to the equivariance of convolution in Euclidean space to translations, we prove that the wFM is equivariant to the action of the group of isometries admitted by the Riemannian manifold on which the data reside. This equivariance property facilitates weight sharing within the network. We present experiments, using the ManifoldNet framework, to achieve video classification and image reconstruction using an auto-encoder+decoder setting. Experimental results demonstrate the efficacy of ManifoldNet in the context of classification and reconstruction accuracy.

heading: 1 INTRODUCTION
text: Convolutional neural networks (CNNs) have attracted enormous attention in the past decade due to their significant success in Computer Vision, Speech Analysis and other fields. CNNs were pioneered by LeCun et al. (1998) and gained much popularity ever since their significant success on Imagenet data reported in Krizhevsky et al. (2012). CNNs have traditionally been restricted to dealing with data residing in vector spaces. However, in the past few years, there is growing interest in generalizing the CNNs and deep networks in general to data that reside on smooth non-Euclidean spaces. In this context, at the outset, it would be useful to categorize problems into 1) those that involve data as samples of real-valued functions defined on a manifold and 2) those that are simply manifold-valued and hence are sample points on a manifold.
In the context of input data being samples of functions defined on smooth manifolds, recently there has been a flurry of activity in developing methods that can cope specifically with samples of functions on a sphere that are encountered in many applications such as, omnidirectional cameras on drones, robots etc., meteorological data and many others. The key property that allows learned weight sharing in CNNs is the equivariance to translations. The simplest technique to achieve equivariance is via data augmentation (Krizhevsky et al., 2012; Dieleman et al., 2015). Cascade of wavelet transforms to achieve equivariance was shown in Bruna & Mallat (2013); Oyallon & Mallat (2015). In Gens (2014), authors describe ‘Symnet’, which achieves invariance to symmetry group actions. Equivariance to discrete group actions was achieved through parameter sharing in Ravanbakhsh et al. (2017). For the case of data on a spherical domain, one considers exploiting equivariance to the rotation
group. Several research groups recently reported spherical-CNNs (SCNNs) to accommodate such an equivariance in defining the convolution of functions (Worrall et al., 2017; Cohen & Welling, 2016; Cohen et al., 2018). In another recent work Esteves et al. (2017), authors describe a polar transformer network, which is equivariant to rotations and scaling transformations. By combining this with a spatial transformer (Jaderberg et al., 2015), they achieve equivariance to translations as well. More generally, equivariance of convolution operations to group actions admitted by the underlying manifold is what is needed to this end and most recent work reported in Chakraborty et al. (2018); Kondor & Trivedi (2018) achieves this for Riemannian homogeneous spaces.
In this paper we will consider the second problem, namely, when the input data are sample points on known Riemannian manifolds for example, the manifold of symmetric positive definite matrices, SPD(n), the special orthogonal group, SO(n), the n-sphere, Sn, the Grassmannian, Gr(p, n), and others. To be precise, the domain of interest is an n-dimensional field of points sampled from a Riemannian manifold. There is very little prior work that we are aware of on DNNs that can cope with input data samples residing on these manifolds with the exception of Huang et al. (2016); Huang & Van Gool (2017). In Huang et al. (2016), authors presented a deep network architecture for classification of hand-crafted features residing on a Grassmann manifold that form the input to the network. In Huang & Van Gool (2017), authors presented a deep network architecture for data on SPD(n). In both of these works, the architecture does not involve the use of any convolution or equivalent operations on Gr(p, n) or SPD(n). Further, it does not use the natural invariant metric or intrinsic operations on the Grassmannian or the SPD(n) in the network blocks. Using intrinsic operations within the layers guarantees that the result remains on the manifold and hence does not require any projection (extrinsic) operations to ensure the result lies in the same space. Further, using extrinsic operations can yield results that are susceptible to significant inaccuracies when the data variance is large (Salehian et al., 2015). Moreover, since there are no convolution type operations defined for data on these manifolds in their network, it can not be considered a generalization to the CNN and as a consequence does not consider equivariance property to the action of the group of isometries denoted by I (M), admitted by the manifoldM. There are several deep networks reported in literature to deal with cases when data reside on 2- manifolds encountered in Computer Vision and Graphics for modeling shapes of objects. Some of these are based on graph-based representations of points on the surfaces in 3D and a generalization of CNNs to graphs (Henaff et al., 2015; Defferrard et al., 2016). There is also recent work in Masci et al. (2015) where the authors presented a deep network called geodesic CNN (GCNN), where convolutions are performed in local geodesic polar charts constructed on the manifold. For more literature on deep networks for data on 2-manifolds, we refer the interested reader to a recent survey paper Bronstein et al. (2017) and references therein.
In this paper, we present a novel DNN framework called the ManifoldNet. This is a potential analog of a CNN and can cope with input data sampled from a Riemannian manifold. The intuition in defining the analog relies on the equivariance property. Note that convolution of functions in vector spaces are equivariant to translations in the spatial domain and in the pixel domain. I.e. if the input pixels are all translated by a fixed amount, then the output pixels are equally translated. Further, it is easy to show that traditional convolutions are equivalent to computing the weighted mean (Goh et al., 2011). Hence, for the case of manifold-valued data, we can define the analogous operation of a weighted Fréchet mean (wFM) and prove that it is equivariant to the action of I (M). This will be achieved in a subsequent section. Our key contributions in this work are: [presented in section 2](i) we define the analog of convolution operations for manifold-valued data to be one of estimating the wFM for which we present a provably convergent, efficient and recursive estimator. (ii) A proof of equivariance of wFM to the action of I (M). This equivariance allows the network to share weights within the layers. (iii) A novel deep architecture involving the Riemannian counterparts to the conventional CNN units. [presented in section 3] (iv) Several real data experiments on classification and reconstruction demonstrating the performance of the ManifoldNet.

heading: 2 GROUP ACTION EQUIVARIANT NETWORK FOR MANIFOLD-VALUED DATA
text: In this section, we will define the equivalent of a convolution operation on Riemannian manifolds. As mentioned in the introduction, the domain of interest is an n-dimensional field of manifold valued points. Before formally defining such an operation and building the DNN for the manifold-valued
data, dubbed a ManifoldNet, we first present some relevant concepts from differential geometry that will be used in the rest of the paper.
Preliminaries. Let (M, gM) be a orientable complete Riemannian manifold with a Riemannian metric gM, i.e., (∀x ∈ M) gMx : TxM× TxM → R is a bi-linear symmetric positive definite map, where TxM is the tangent space ofM at x ∈ M. Let d :M×M → [0,∞) be the metric (distance) induced by the Riemannian metric gM. With a slight abuse of notation we will denote a Riemannian manifold (M, gM) byM unless specified otherwise. Let ∆ be the supremum of the sectional curvatures ofM. Definition 1. Let p ∈ M, r > 0. Define Br(p) = {q ∈M|d(p, q) < r} to be a open ball at p of radius r. Definition 2. (Groisser, 2004) The local injectivity radius at p ∈M, rinj(p), is defined as rinj(p) = sup { r|Expp : (Br(0) ⊂ TpM)→M is defined and is a diffeomorphism onto its image}. The injectivity radius Manton (2004) ofM is defined as rinj(M) = infp∈M {rinj(p)}.
Within Br(p), where r ≤ rinj(M), the mapping Exp−1p : Br(p)→ U ⊂ TpM, is called the inverse Exponential/ Log map. Definition 3. (Kendall, 1990) An open ball Br(p) is a regular geodesic ball if r < rinj(p) and r < π/ ( 2∆1/2 ) .
In Definition 3 and below, we interpret 1/∆1/2 as∞ if ∆ ≤ 0. It is well known that, if p and q are two points in a regular geodesic ball Br(p), then they are joined by a unique geodesic within Br(p) (Kendall, 1990). Definition 4. (Chavel, 2006) U ⊂ M is strongly convex if for all p, q ∈ U , there exists a unique length minimizing geodesic segment between p and q and the geodesic segment lies entirely in U . Definition 5. (Groisser, 2004) Let p ∈ M. The local convexity radius at p, rcvx(p), is defined as rcvx(p) = sup {r ≤ rinj(p)|Br(p) is strongly convex}. The convexity radius ofM is defined as rcvx(M) = infp∈M {rcvx(p)}.
For the rest of the paper, we will assume that the samples onM lie inside an open ball U = Br(p) where r = min {rcvx(M), rinj(M)}, for some p ∈ M, unless mentioned otherwise. Now, we are ready to define the operations necessary to develop the ManifoldNet.
2.1 WFM ONM AS A GENERALIZATION OF CONVOLUTION
We will now define a convolution type operation on points sampled from M. This convolution operation will perform an averaging operation over a moving window, but will replace weighted sums with weighted intrinsic averages. Let {Xi}Ni=1 be the manifold-valued samples onM. We define the convolution type operation onM as the weighted Fréchet mean (wFM) (Maurice Fréchet, 1948) of the samples {Xi}Ni=1. Also, by the aforementioned condition on the samples, the existence and uniqueness of FM is guaranteed (Afsari, 2011). As mentioned earlier, it is easy to show (see Goh et al. (2011)). that convolution ψ∗ = b ? a of two functions a : X ⊂ Rn → R and b : X ⊂ Rn → R can be formulated as computation of the weighted mean ψ∗ = argminψ ∫ a(u)(ψ − b̃u)2du, where,
∀x ∈ X, b̃u(x) = b (u + x) and ∫ a(x)dx = 1. Here, f2 for any function f is defined pointwise. Further, the defining property of convolutions in vector spaces is the linear translation equivariance. Since weighted mean in vector spaces can be generalized to wFM on manifolds and further, wFM can be shown (see below) to be equivariant to group actions admitted by the manifold, we claim that wFM is a generalization of convolution operations to manifold-valued data. Let {wi}Ni=1 be the weights such that they satisfy the convexity constraint, i.e., ∀i, wi > 0 and∑ i wi = 1, then wFM, wFM ({Xi} , {wi}) is defined as:
wFM ({Xi} , {wi}) = argmin M∈M N∑ i=1 wid 2 (Xi,M) (1)
Analogous to the equivariance property of convolution translations in vector spaces, we will now proceed to show that the wFM is equivariant under the action of the group of isometries ofM. We
will first formally define the group of isometries ofM (let us denote it by G) and then define the equivariance property and show that wFM is G-equivariant. Definition 6 (Group of isometries ofM (I (M))). A diffeomorphism φ :M→M is an isometry if it preserves distance, i.e., d (φ (x) , φ (y)) = d (x, y). The set I(M) of all isometries ofM forms a group with respect to function composition. Rather than write an isometry as a function φ, we will write it as a group action. Henceforth, let G denote the group I(M), and for g ∈ G, and x ∈M, let g.x denote the result of applying the isometry g to point x.
ClearlyM is a G set (see Dummit & Foote (2004) for the definition of a G set). We will now define equivariance and show that wFM, is G-equivariant. Definition 7 (Equivariance). LetX and Y beG sets. Then, F : X → Y is said to beG-equivariant if ∀g ∈ G, ∀x ∈ X , F (g.x) = g.F (x).
Let U ⊂ M be an open ball inside which FM exists and is unique, let P be the set consists of all possible finite subsets of U .
Theorem 1. Given {wi} satisfying the convex constraint, let F : P → U be a function defined by {Xi} 7→ wFM ({Xi} , {wi}). Then, F is G-equivariant.
Proof. Let g ∈ G and {Xi}Ni=1 ∈ P , now, let M∗ = wFM ({Xi} , {wi}), as g.F ({Xi}) = g.M∗, it suffices to show g.M∗ is wFM ({g.Xi} , {wi}) (assuming the existence and uniqueness of wFM ({g.Xi} , {wi}) which is stated in the following claim). Claim: Let U = Br (p) for some r > 0 and p ∈ M. Then, {g.Xi} ⊂ Br (g.p) and hence wFM ({g.Xi} , {wi}) exists and is unique.
Let M̃ be wFM ({g.Xi} , {wi}). Then, ∑N i=1 wid 2 ( g.Xi, M̃ ) = ∑N i=1 wid 2 ( Xi, g −1.M̃ ) .
Since, M∗ = wFM ({Xi} , {wi}), hence, M∗ = g−1.M̃ , i.e., M̃ = g.M∗. Thus, g.M∗ = wFM ({g.Xi} , {wi}), which implies F is G-equivariant.
Now we give some examples ofM with the corresponding group of isometries G. LetM = SPD(n) (the space of n×n symmetric positive-definite matrices). Let d be the Stein metric on SPD(n). Then, the group of isometries G is O(n) (the space of n× n orthogonal matrices). A class of Riemannian manifolds on which G acts transitively are called Riemannian homogeneous spaces. We can see that on a Riemannian homogeneous spaceM, wFM is G-equivariant. Equipped with a G-equivariant operator onM, we can claim that the wFM (defined above) is a valid convolution operator since group equivariance is a unique defining property of a convolution operator. The rest of this subsection will be devoted to developing an efficient way to compute wFM. Let ωM > 0 be the Riemannian volume form. Let pX be the probability density of a U -valued random variable X with respect to ωM on U ⊂ M, so that Pr (X ∈ A) = ∫ A pX(Y )ω
M (Y ) for any Borel-measurable subset A of U. Let Y ∈ U , we can define the expectation of the real valued random variable d2(, Y ) : U → R by E [ d2(, Y ) ] = ∫ U d2(X,Y )pX(X)ω
M(X). Now, let w : U → (0,∞) be an integrable function and ∫ U w (X)ωM (X) = 1.
Then, observe that, Ew [ d2(, Y ) ] := ∫ U w(X)d2(X,Y )pX(X)ω M(X) = C ∫ U d2(X,Y )p̃X(X)
ωM(X) = C Ẽ [ d2(, Y ) ] . Here, p̃X is the probability density corresponding to the probability
measure P̃r defined by, P̃r (X ∈ X) = ∫ X p̃X(Y )ω M(Y ) := ∫ X 1 C pX(Y )w(Y )ω
M(Y ), where, X lies in the Borel σ-algebra over U and C = ∫ U pX(Y )w(Y )ω
M(Y ). Note that the constant C > 0, since pX is a probability density, w > 0 andM is orientable. Thus, Ew [ d2(, Y ) ] with respect to pX
is proportional to Ẽ [ d2(, Y ) ] with respect to p̃X .
Now, we will state the following proposition (the proof is in the appendix section). Proposition 2. (i) supp (pX) = supp (p̃X). (ii) wFE (X, w) = FE ( X̃ ) . Let {Xi}Ni=1 be samples drawn from pX and { X̃i }N i=1 be samples drawn from p̃X . In order to compute wFM, we will now present an online algorithm (inductive FM Estimator – dubbed iFME).
Given, {Xi}Ni=1 ⊂ U and {wi := w (Xi)} N i=1 such that ∀i, wi > 0, the nth estimate, Mn of wFM ({Xi} , {wi}) is given by the following recursion:
M1 = X1 Mn = Γ Xn Mn−1 ( wn∑n j=1 wj ) . (2)
In the above equation, ΓYX : [0, 1]→ U is the shortest geodesic curve from X to Y . Observe that, in general wFM is defined with ∑N i=1 wi = 1, but in above definition, ∑N i=1 wi 6= 1. We can normalize
{wi} to get {w̃i} by w̃i = wi/ ( ∑ i wi), but then Eq. 2 will not change as w̃n/ (∑n j=1 w̃j ) =
wn/ (∑n j=1 wj ) . This gives us an efficient inductive/recursive way to define convolution operation onM. Now, we state that the proposed wFM estimator is consistent (the proof is in the appendix).
Proposition 3. Using the above notations and assumptions, let {Xi}Ni=1 be i.i.d. samples drawn from pX onM. Let the wFE be finite. Then, MN converges a.s. to wFE as N →∞.
2.2 NONLINEAR OPERATION BETWEEN WFM-LAYERS FORM-VALUED DATA
In the traditional CNN model, we need a nonlinear function between two convolutional layers similar to ReLU and softmax. As argued in Mallat (2016), any nonlinear function used in CNN is basically a contraction mapping. Formally, let F be a nonlinear mapping from U to V . Let assume, U and V are metric spaces equipped with metric dU and dV respectively. Then, F is a contraction mapping iff ∃c < 1 such that, dV (F (x), F (y)) ≤ c dU (x, y). F is a non-expansive mapping (Mallat, 2016) iff dV (F (x), F (y)) ≤ dU (x, y).
One can easily see that the popular choices for nonlinear operations like ReLU, sigmoid are in fact non-expansive mappings. Now, we will show that the function wFM as defined in 1, is a contraction mapping for non-trivial choices of weights. Let {Xi}Ni=1 and {Yj} M j=1 be the two set of samples onM. Without any loss of generality, assume N ≤M . We consider the set UM = U × · · · × U︸ ︷︷ ︸
M times
. Clearly {Yj}Mj=1 ∈ U M
and we embed {Xi}Ni=1 in UM as follows: we construct { X̃i }M i=1
from {Xi}Ni=1 by defining X̃i = X(i−1)modN+1. Let us denote the embedding
by ι. Now, define the distance on UM as d ({
X̃i }M i=1 , {Yj}Mj=1 ) = maxi,j d (Xi, Yj). The choice
of weights for wFM is said to be trivial if one of the weights is 1 and hence the rest are 0.
Proposition 4. For all nontrivial choices of {αi}Ni=1 and {βj} M j=1 satisfying the convexity constraint , ∃c < 1 such that,
d ( wFM ( {Xi}Ni=1 , {αi} N i=1 ) ,wFM ( {Yj}Mi=1 , {βj} M i=1 )) ≤ c d ( ι ( {Xi}Ni=1 ) , {Yj}Mj=1 ) (3)

heading: 2.3 THE INVARIANT (LAST) LAYER
text: We will form a deep network by cascading multiple wFM blocks each of which acts as a convolutiontype layer. Each convolutional-type layer is equivariant to the group action, and hence at the end of the cascaded convolutional layers, the output is equivariant to the group action applied to the input of the network. Let d be the number of output channels each of which outputs a wFM, hence each of the channels is equivariant to the group action. However, in order to build a network that yields an output which is invariant to the group action, we now seek the last layer (i.e., the analogue to a linear classifier) to be invariant to the group action. The last layer is thus constructed as follows: Let {Z1, · · · , Zd} ⊂ M be the output of d channels and Mu = FM ( {Zi}di=1 ) =
wFM ( {Zi}di=1 , {1/d} d 1 ) be the unweighted FM of the outputs {Zi}di=1. Then, we construct a layer
with d outputs whose ith output oi = d (Mu, Zi). Let c be the number of classes for the classification task, then, a fully connected (FC) layer with inputs {oi} and c output nodes is build. A softmax operation is then used at the c output nodes to obtain the outputs {yi}ci=1. In the following proposition we claim that this last layer with {Zi}di=1 inputs and {yi} c i=1 outputs is group invariant.
Proposition 5. The last layer with {Zi}di=1 inputs and {yi} c i=1 outputs is group invariant.
In Fig. 1 we present a schematic of ManifoldNet depicting the different layers of processing the manifold-valued data as described above in Sections 2.1-2.3.

heading: 3 EXPERIMENTS
text: In this section we present performance of the ManifoldNet framework on several standard computer vision problems. The breadth of application coverage here includes classification and reconstruction problems. We begin with a video classification problem and then present a reconstruction problem using an auto-encoder-decoder set up.

heading: 3.1 VIDEO CLASSIFICATION
text: We start by using the method in Yu & Salzmann (2017) which we summarize here. Given a video with dimensions F × 3×H ×W of F frames, 3 color channels and a frame size of H ×W , we can apply a convolution layer to obtain an output of size F × C ×H ′ ×W ′ consisting of C channels of size H ′ ×W ′. We compute the covariance matrix of the channels to obtain a sequence of F symmetric positive (semi) definite matrices of size C × C.
Covariance Matrices
C × C C × C C × C C × C
...
[ , , … , ] ∈y1 y2 yn R n
   
= d( , FM({ }))yi xi xi
wFM({ }, { })xi wi
.
.
Text
C × C C × C C × C
C × C C × C
form very well on video classification tasks while using very few parameters, and runs efficiently due to the wFM structure.
We tested the SPD-TCN on the Moving MNIST dataset (Srivastava et al., 2015). Recently, in Chakraborty et al. (2018) authors developed a manifold valued recurrent network architecture, dubbed SPD-SRU, which produced state-of-the-art classification results on Moving
MNIST dataset in comparison to LSTM (Hochreiter & Schmidhuber, 1997), SRU (Oliva et al., 2017), TT-LSTM and TT-GRU (Yang et al., 2017) networks. For the LSTM and SRU networks, convolution layers are also used before the recurrent unit. We will compare directly with the results presented in Chakraborty et al. (2018). For details of the various architectures used please see section 5 of Chakraborty et al. (2018). We will also compare with Huang & Van Gool (2017) , which proposes a set of layers for learning SPD matrices, dubbed SPDNet. When using SPDNet we first downsample
the videos to 20× 20 frames and compute the covariance matrix of the frames, which is then fed to an architecture similar to that of the emotion recognition experiment of Huang & Van Gool (2017) Specifically, we use three blocks of BiMap/ReEig layers, since such a configuration gave the best validation performance. The transformation sizes for these blocks are set to 400× 200, 200× 100 and 100× 50 respectively. The Moving MNIST data generated in Srivastava et al. (2015) consists of 1000 samples, each of 20 frames. Each sample shows two randomly chosen MNIST digits moving within a 64× 64 frame, with the direction and speed of movement fixed across all samples in a class. The speed is kept the same across different classes, but the digit orientation will differ across two different classes. We summarize the 10-fold cross validation results for several orientation differences between classes in Table 1. For this experiment the SPD-TCN will consist of a single wFM layer with kernel size 5 and stride 3 returning 8 channels, making for an 8 × 8 covariance matrix. We then apply three temporal SPD wFM layers of kernel size 3 and stride 2, with the following channels 1 → 4 → 8 → 16, i.e. after these three temporal SPD wFMs we have 16 temporal channels. This 16× 8× 8 is used as an input to the invariant final layer to get a 16 dimensional output vector, which is transformed by a FC+SoftMax layer to obtain the output.
3.2 DIMENSIONALITY REDUCTION Gr(k;n)
wFM
wFM
Recon. using principal subspaces,
ffig
of ffig
n efi o
denoted by {efi }
DecoderEncoder
Weighted FM
Figure 3: Pictorial description of autoencoder+iFME
Here we present experiments demonstrating the applicability of the theory layed out in Section 2 to the case of linear dimensionality reduction, specifically principal component analysis (PCA), which is the workhorse of many machine learning algorithms. In Chakraborty et al. (2017), authors presented an online subspace averaging algorithm for construction of principal components via in-
trinsic averaging on the Grassmannian. In this section, we achieve the intrinsic Grassmann averaging process in the framework of ManifoldNet to compute the principal subspaces and achieve the dimensionality reduction. In the context of DNNs, dimensionality reduction is commonly achieved via an autoencoder architecture. More recently, DNNs have shown promising results when the data manifold is intrinsically non-linear, as in the case of natural images. In the deep learning community this has become a field in its own right, known as representation learning or feature learning (Bengio et al., 2013) with works including Vincent et al. (2010), Kingma & Welling (2013) Oord et al. (2016), Van Den Oord et al. (2016)), Kingma & Dhariwal (2018). Many of these architectures are modifications of the traditional autoencoder network, which involves learning an identity map through a small latent space. In our application, we modify the traditional autoencoder model by adding a ManifoldNet layer to perform a learned linear dimensionality reduction in the latent space, although in principal, our techniques can be applied to most autoencoder based models such as the variational autoencoders. To compute a linear subspace in the ManifoldNet framework we use an intrinsic averaging scheme on the Grassmannian. A point on the Grassmannian Gr(k, n) corresponds to k-dimensional subspace of Rn and thus can be specified by an orthonormal basis X . Chakraborty et al. (2017) proposed an efficient intrinsic averaging scheme on Gr(k, n) that converges to the k-dimensional principal subspace of a normally distributed dataset in Rn. In the ManifoldNet framework we can modify this technique to learn a wFM of points on the Grassmannian that corresponds to a subspace of the latent space which minimizes the reconstruction error by using a Grassmannian averaging layer that learns the weights in the wFM. This essentially will give us a lower dimensional representation of the samples after projecting them on to the learned subspace. Note that combining the convergence proof in Chakraborty et al. (2017) and Proposition 2 (ii), we claim that the wFM learned using the ManifoldNet asymptotically converges to the principal subspace. Now, we give a detailed description of our experimental setup to show the applicability of ManifoldNet to dimensionality reduction.

heading: 3.2.1 VIDEO RECONSTRUCTION EXPERIMENT
text: A traditional convolutional autoencoder performs non-linear dimensionality reduction by learning an identity function through a small latent space. A common technique used when the desired latent space is smaller than the output of the encoder is to apply a fully connected layer to match dimensions. We replace this fully connected layer by a weighted subspace averaging and projection block, called the Grassmann averaging layer. Specifically, we compute the wFM of the output of the encoder to
get a subspace in the encoder output space. We then project the encoder output onto this space to obtain a reduced dimensionality latent space. In general this offers a significant parameter reduction while also increasing the reconstruction error performance of the autoencoder and giving realistic reconstructions. We call an autoencoder with the Grassmann averaging block
an autoencoder+iFME network, as shown in Fig. 3. In the experiments we compare this to other dimensionality reduction techniques, including regular autoencoders that use fully connected layers to match encoder and latent space dimensions. We begin by testing on a 1000 frame color sample of video from the 1964 film “Santa Clause Conquers the Martians” of frame size 320 × 240. Here we use an 8 layer encoding-decoding architecture with Conv → ELU → Batchnorm layers, with the final layer applying a sigmoid activation to normalize pixel values. The encoder returns a feature video consisting of 128 channels of size 120 for a dimen-
sion of 1000 × 15360. We compare a fully connected layer to a Grassmann averaging layer, both mapping to a desired latent space of dimension 1000× 20. The per pixel average reconstruction error for the Grassmann block network is 0.0110, compared to 0.0122 for the fully connected network,
representing an improvement of 10.9%. We also observe a parameter reduction of 46%, which can be attributed to the number of parameters in the large fully connected layer. In Fig. 5, the computation time is plotted against error tolerance for the autoencoder and the iFME+autoencoder. We can see that iFME+autoencoder achieves faster convergence than autoencoder. It is possible to obtain a low reconstruction error on autoencoding tasks and still observe low visual quality reconstructions. To ensure this is not the case we run the same experiment on 300 frames of the 1280× 720 short film 1, with a latent space frame dimension of 300x50. In Fig. 4 we compare the visual quality of our autoencoder to that
of PCA with 50 principcal components, i.e., we reduce the dimension from 1280× 720× 3 to 50. The entire sample reconstruction is shown in 2 (in the same order as in Fig. 4).

heading: 4 CONCLUSIONS
text: In this paper, we presented a novel deep network called ManifoldNet suited for processing manifoldvalued data sets. Inputs to the ManifoldNet are manifold-valued and not real or complex-valued functions defined on non-Euclidean domains. Our key contributions are: (i) A novel deep network to be perceived as a generalization of the CNN to the case when the input data are manifold-valued using purely intrinsic operations on the manifold where the data reside. (ii) Analogous to convolutions in vector spaces – which can be computed using the weighted mean – we present wFM operations on the manifold and prove the equivariance of the wFM to natural group operations admitted by the manifold. This equivariance allows us to share the learned weights within a layer of the ManifoldNet. (iii) An efficient recursive wFM estimator that is provably convergent is presented. (iv) Experimental results demonstrating the efficacy of the ManifoldNet for, (a) video classification and (b) principal component computation from videos and reconstruction are also presented.
1https://www.youtube.com/watch?v=t1hMBnIMt5I 2https://streamable.com/3yqrx

heading: 5 APPENDIX
text: Proposition 2. (i) supp (pX) = supp (p̃X). (ii) wFE (X, w) = FE ( X̃ ) .
Proof. Let X ∈ supp (pX), then, pX (X) > 0. Since, w(X) > 0, hence, p̃X (X) > 0 and thus, X ∈ supp (p̃X). On the other hand, assume X̃ to be a sample drawn from p̃X . Then, either pX ( X̃ ) = 0 or pX ( X̃ ) > 0. If, pX ( X̃ ) = 0, then, p̃X ( X̃ ) = 0 which contradicts our
assumption. Hence, pX ( X̃ ) > 0, i.e., X̃ ∈ supp (pX). This concludes the proof of part (i).
Let X and X̃ be theM valued random variable following pX and p̃X respectively. We define the weighted Fréchet expectation (wFE) of X as:
wFE (X, w) = argmin Y ∈M ∫ M w(X)d2(X,Y )pX(X)ω M(X)
Observe,
Ew [ d2(, Y ) ] := ∫ U w(X)d2(X,Y )pX(X)ω M(X)
= C ∫ U d2(X,Y )p̃X(X)ω M(X)
= C Ẽ [ d2(, Y ) ] . (4)
Hence, we get FE ( X̃ )
= wFE (X, w), as C is independent of the choice of Y , which concludes the proof of part (ii).
Proposition 3. Using the notations and assumptions used in the paper, let {Xi}Ni=1 be i.i.d. samples drawn from pX onM. Let the wFE be finite. Then, MN converges a.s. to wFE as N →∞. Proof. Using Proposition 2, we know that ∃ p̃X such that, wFE (X, w) = FE ( X̃ )
. Thus, it is enough to show the consistency of our proposed estimator when weights are uniform. In order to prove the consistency, we will split the proof into two cases namely, manifolds with (i) non-positive sectional curvature and (ii) non-negative sectional curvature. The reason for doing this split is so that we can use existing theorems in literature for proving the result. We will use the theorems proved in Sturm (2003) and Bonnabel (2013) for manifolds with non-positive and non-negative sectional curvatures respectively. Note that the proof holds only for manifolds with a uniform sign of sectional curvatures.
Theorem 6 (M has non-negative sectional curvature). Using the above notations, if ∃A > 0 such that, d (Mn, Xn+1) ≤ A for all n. Then, MN converges a.s. to wFE as N → ∞ (see Bonnabel (2013) for the proof).
Theorem 7 (M has non-positive sectional curvature). Using the above notations MN converges a.s. to wFE as N →∞ (see Sturm (2003) for the proof).
Proposition 5. The last layer with {Zi}di=1 inputs and {yi} c i=1 outputs is group invariant.
Proof. Using the above construction, let W ∈ Rc×d and b ∈ Rc be the weight matrix and bias respectively of the FC layer. Then,
y = F ( WTo + b ) = F ( WT d (Mu, Z) + b ) , (5)
where, F is the softmax function. In the above equation, we treat d (Mu, Z) as the vector [d (Mu, Z1) , · · · , d (Mu, Zd)]t. Observe that, g.Mu = FM ( {g.Zi}di=1 ) . As each of the d channels
is group equivariant, Zi becomes g.Zi. Because of the property of the distance under group action, d (g.Mu, g.Zi) = d (Mu, Zi). Hence, one can see that if we change the inputs {Zi} to {g.Zi}, the output y will remain invariant.

Review summaries:
This paper introduces a generalization of convolution operations to manifold-valued data using  the computation of the weighted Frechet mean (wFM). Without applying any non-linear units and pooling layers, the paper proposes to merely stack suching generalized convolutional layers to construct a deep network for the data residing on Riemannian manifolds. The evaluations on video classification and reconstruction tasks show the advantages of the introduced network over some baselines.