Title: MANIFOLDNET: A DEEP NEURAL NETWORK FOR MANIFOLD-VALUED DATA

Abstract: Developing deep neural networks (DNNs) for manifold-valued data sets has gained much interest of late in the deep learning research community. Examples of manifold-valued data include data from omnidirectional cameras on automobiles, drones etc., diffusion magnetic resonance imaging, elastography and others. In this paper, we present a novel theoretical framework for DNNs to cope with manifold-valued data inputs. In doing this generalization, we draw parallels to the widely popular convolutional neural networks (CNNs). We call our network the ManifoldNet. As in vector spaces where convolutions are equivalent to computing the weighted mean of functions, an analogous definition for manifold-valued data can be constructed involving the computation of the weighted Fréchet Mean (wFM). To this end, we present a provably convergent recursive computation of the wFM of the given data, where the weights makeup the convolution mask, to be learned. Further, we prove that the proposed wFM layer achieves a contraction mapping and hence the ManifoldNet does not need the additional non-linear ReLU unit used in standard CNNs. Operations such as pooling in traditional CNN are no longer necessary in this setting since wFM is already a pooling type operation. Analogous to the equivariance of convolution in Euclidean space to translations, we prove that the wFM is equivariant to the action of the group of isometries admitted by the Riemannian manifold on which the data reside. This equivariance property facilitates weight sharing within the network. We present experiments, using the ManifoldNet framework, to achieve video classification and image reconstruction using an auto-encoder+decoder setting. Experimental results demonstrate the efficacy of ManifoldNet in the context of classification and reconstruction accuracy.

1. INTRODUCTION:

Convolutional neural networks (CNNs) have attracted enormous attention in the past decade due to their significant success in Computer Vision, Speech Analysis and other fields. CNNs were pioneered by LeCun et al. (1998) and gained much popularity ever since their significant success on Imagenet data reported in Krizhevsky et al. (2012) . CNNs have traditionally been restricted to dealing with data residing in vector spaces. However, in the past few years, there is growing interest in generalizing the CNNs and deep networks in general to data that reside on smooth non-Euclidean spaces. In this context, at the outset, it would be useful to categorize problems into 1) those that involve data as samples of real-valued functions defined on a manifold and 2) those that are simply manifold-valued and hence are sample points on a manifold.

In the context of input data being samples of functions defined on smooth manifolds, recently there has been a flurry of activity in developing methods that can cope specifically with samples of functions on a sphere that are encountered in many applications such as, omnidirectional cameras on drones, robots etc., meteorological data and many others. The key property that allows learned weight sharing in CNNs is the equivariance to translations. The simplest technique to achieve equivariance is via data augmentation (Krizhevsky et al., 2012; Dieleman et al., 2015) . Cascade of wavelet transforms to achieve equivariance was shown in Bruna & Mallat (2013) ; Oyallon & Mallat (2015) . In Gens (2014) , authors describe 'Symnet', which achieves invariance to symmetry group actions. Equivariance to discrete group actions was achieved through parameter sharing in Ravanbakhsh et al. (2017) . For the case of data on a spherical domain, one considers exploiting equivariance to the rotation group. Several research groups recently reported spherical-CNNs (SCNNs) to accommodate such an equivariance in defining the convolution of functions (Worrall et al., 2017; Cohen & Welling, 2016; Cohen et al., 2018) . In another recent work Esteves et al. (2017) , authors describe a polar transformer network, which is equivariant to rotations and scaling transformations. By combining this with a spatial transformer (Jaderberg et al., 2015) , they achieve equivariance to translations as well. More generally, equivariance of convolution operations to group actions admitted by the underlying manifold is what is needed to this end and most recent work reported in Chakraborty et al. (2018) ; Kondor & Trivedi (2018) achieves this for Riemannian homogeneous spaces.

In this paper we will consider the second problem, namely, when the input data are sample points on known Riemannian manifolds for example, the manifold of symmetric positive definite matrices, SP D(n), the special orthogonal group, SO(n), the n-sphere, S n , the Grassmannian, Gr(p, n), and others. To be precise, the domain of interest is an n-dimensional field of points sampled from a Riemannian manifold. There is very little prior work that we are aware of on DNNs that can cope with input data samples residing on these manifolds with the exception of Huang et al. (2016) ; Huang & Van Gool (2017) . In Huang et al. (2016) , authors presented a deep network architecture for classification of hand-crafted features residing on a Grassmann manifold that form the input to the network. In Huang & Van Gool (2017) , authors presented a deep network architecture for data on SP D(n). In both of these works, the architecture does not involve the use of any convolution or equivalent operations on Gr(p, n) or SP D(n). Further, it does not use the natural invariant metric or intrinsic operations on the Grassmannian or the SP D(n) in the network blocks. Using intrinsic operations within the layers guarantees that the result remains on the manifold and hence does not require any projection (extrinsic) operations to ensure the result lies in the same space. Further, using extrinsic operations can yield results that are susceptible to significant inaccuracies when the data variance is large (Salehian et al., 2015) . Moreover, since there are no convolution type operations defined for data on these manifolds in their network, it can not be considered a generalization to the CNN and as a consequence does not consider equivariance property to the action of the group of isometries denoted by I (M), admitted by the manifold M.

There are several deep networks reported in literature to deal with cases when data reside on 2manifolds encountered in Computer Vision and Graphics for modeling shapes of objects. Some of these are based on graph-based representations of points on the surfaces in 3D and a generalization of CNNs to graphs (Henaff et al., 2015; Defferrard et al., 2016) . There is also recent work in Masci et al. (2015) where the authors presented a deep network called geodesic CNN (GCNN), where convolutions are performed in local geodesic polar charts constructed on the manifold. For more literature on deep networks for data on 2-manifolds, we refer the interested reader to a recent survey paper Bronstein et al. (2017) and references therein.

In this paper, we present a novel DNN framework called the ManifoldNet. This is a potential analog of a CNN and can cope with input data sampled from a Riemannian manifold. The intuition in defining the analog relies on the equivariance property. Note that convolution of functions in vector spaces are equivariant to translations in the spatial domain and in the pixel domain. I.e. if the input pixels are all translated by a fixed amount, then the output pixels are equally translated. Further, it is easy to show that traditional convolutions are equivalent to computing the weighted mean (Goh et al., 2011) . Hence, for the case of manifold-valued data, we can define the analogous operation of a weighted Fréchet mean (wFM) and prove that it is equivariant to the action of I (M). This will be achieved in a subsequent section. Our key contributions in this work are: [presented in section 2](i) we define the analog of convolution operations for manifold-valued data to be one of estimating the wFM for which we present a provably convergent, efficient and recursive estimator. (ii) A proof of equivariance of wFM to the action of I (M). This equivariance allows the network to share weights within the layers. (iii) A novel deep architecture involving the Riemannian counterparts to the conventional CNN units. [presented in section 3] (iv) Several real data experiments on classification and reconstruction demonstrating the performance of the ManifoldNet.

2. GROUP ACTION EQUIVARIANT NETWORK FOR MANIFOLD-VALUED DATA:

In this section, we will define the equivalent of a convolution operation on Riemannian manifolds. As mentioned in the introduction, the domain of interest is an n-dimensional field of manifold valued points. Before formally defining such an operation and building the DNN for the manifold-valued data, dubbed a ManifoldNet, we first present some relevant concepts from differential geometry that will be used in the rest of the paper.

Preliminaries. Let (M, g M ) be a orientable complete Riemannian manifold with a Riemannian metric g M , i.e., (∀x ∈ M) g M x : T x M × T x M → R is a bi-linear symmetric positive definite map, where T x M is the tangent space of M at x ∈ M. Let d : M × M → [0, ∞) be the metric (distance) induced by the Riemannian metric g M . With a slight abuse of notation we will denote a Riemannian manifold (M, g M ) by M unless specified otherwise. Let ∆ be the supremum of the sectional curvatures of M. Definition 1. Let p ∈ M, r > 0. Define B r (p) = {q ∈ M|d(p, q) < r} to be a open ball at p of radius r. Definition 2. (Groisser, 2004) The local injectivity radius at p ∈ M, r inj (p), is defined as r inj (p) = sup r|Exp p : (B r (0) ⊂ T p M) → M is defined and is a diffeomorphism onto its image}. The injectivity radius Manton (2004) of M is defined as r inj (M) = inf p∈M {r inj (p)}.

Within B r (p), where r ≤ r inj (M), the mapping Exp -1 p : B r (p) → U ⊂ T p M, is called the inverse Exponential/ Log map. Definition 3. (Kendall, 1990) An open ball B r (p) is a regular geodesic ball if r < r inj (p) and r < π/ 2∆ 1/2 .

In Definition 3 and below, we interpret 1/∆ 1/2 as ∞ if ∆ ≤ 0. It is well known that, if p and q are two points in a regular geodesic ball B r (p), then they are joined by a unique geodesic within B r (p) (Kendall, 1990) . Definition 4. (Chavel, 2006 ) U ⊂ M is strongly convex if for all p, q ∈ U, there exists a unique length minimizing geodesic segment between p and q and the geodesic segment lies entirely in U. Definition 5. (Groisser, 2004) Let p ∈ M. The local convexity radius at p, r cvx (p), is defined as

r cvx (p) = sup {r ≤ r inj (p)|B r (p) is strongly convex}. The convexity radius of M is defined as r cvx (M) = inf p∈M {r cvx (p)}.

For the rest of the paper, we will assume that the samples on M lie inside an open ball U = B r (p) where r = min {r cvx (M), r inj (M)}, for some p ∈ M, unless mentioned otherwise. Now, we are ready to define the operations necessary to develop the ManifoldNet.

2.1. WFM ON M AS A GENERALIZATION OF CONVOLUTION:

We will now define a convolution type operation on points sampled from M. This convolution operation will perform an averaging operation over a moving window, but will replace weighted sums with weighted intrinsic averages. Let {X i } N i=1 be the manifold-valued samples on M. We define the convolution type operation on M as the weighted Fréchet mean (wFM) (Maurice Fréchet, 1948) of the samples {X i } N i=1 . Also, by the aforementioned condition on the samples, the existence and uniqueness of FM is guaranteed (Afsari, 2011) . As mentioned earlier, it is easy to show (see Goh et al. (2011) ). that convolution ψ * = b a of two functions a :

X ⊂ R n → R and b : X ⊂ R n → R can be formulated as computation of the weighted mean ψ * = argmin ψ a(u)(ψ -b u ) 2 du, where, ∀x ∈ X, b u (x) = b (u + x) and a(x)dx = 1.

Here, f 2 for any function f is defined pointwise. Further, the defining property of convolutions in vector spaces is the linear translation equivariance. Since weighted mean in vector spaces can be generalized to wFM on manifolds and further, wFM can be shown (see below) to be equivariant to group actions admitted by the manifold, we claim that wFM is a generalization of convolution operations to manifold-valued data.

Let {w i } N i=1 be the weights such that they satisfy the convexity constraint, i.e., ∀i, w i > 0 and i w i = 1, then wFM, wFM ({X i } , {w i }) is defined as:

EQUATION

Analogous to the equivariance property of convolution translations in vector spaces, we will now proceed to show that the wFM is equivariant under the action of the group of isometries of M. We will first formally define the group of isometries of M (let us denote it by G) and then define the equivariance property and show that wFM is G-equivariant. Definition 6 (Group of isometries of M (I (M))). A diffeomorphism φ : M → M is an isometry if it preserves distance, i.e., d (φ (x) , φ (y)) = d (x, y). The set I(M) of all isometries of M forms a group with respect to function composition. Rather than write an isometry as a function φ, we will write it as a group action. Henceforth, let G denote the group I(M), and for g ∈ G, and x ∈ M, let g.x denote the result of applying the isometry g to point x.

Clearly M is a G set (see Dummit & Foote (2004) for the definition of a G set). We will now define equivariance and show that wFM, is G-equivariant. Definition 7 (Equivariance). Let X and Y be G sets. Then, F :

X → Y is said to be G-equivariant if ∀g ∈ G, ∀x ∈ X, F (g.x) = g.F (x).

Let U ⊂ M be an open ball inside which FM exists and is unique, let P be the set consists of all possible finite subsets of U . Theorem 1. Given {w i } satisfying the convex constraint, let F : P → U be a function defined by

{X i } → wFM ({X i } , {w i }). Then, F is G-equivariant. Proof. Let g ∈ G and {X i } N i=1 ∈ P , now, let M * = wFM ({X i } , {w i }), as g.F ({X i }) = g.M * , it suffices to show g.M * is wFM ({g.X i } , {w i }) (

assuming the existence and uniqueness of wFM ({g.X i } , {w i }) which is stated in the following claim). Claim: Let U = B r (p) for some r > 0 and p ∈ M. Then, {g.X i } ⊂ B r (g.p) and hence wFM ({g.X i } , {w i }) exists and is unique.

Let M be wFM ({g.X i } , {w i }). Then, N i=1 w i d 2 g.X i , M = N i=1 w i d 2 X i , g -1 . M . Since, M * = wFM ({X i } , {w i }), hence, M * = g -1 . M , i.e., M = g.M * . Thus, g.M * = wFM ({g.X i } , {w i }), which implies F is G-equivariant.

Now we give some examples of M with the corresponding group of isometries G. Let M = SPD(n) (the space of n × n symmetric positive-definite matrices). Let d be the Stein metric on SPD(n). Then, the group of isometries G is O(n) (the space of n × n orthogonal matrices). A class of Riemannian manifolds on which G acts transitively are called Riemannian homogeneous spaces. We can see that on a Riemannian homogeneous space M, wFM is G-equivariant. Equipped with a G-equivariant operator on M, we can claim that the wFM (defined above) is a valid convolution operator since group equivariance is a unique defining property of a convolution operator. The rest of this subsection will be devoted to developing an efficient way to compute wFM. Let ω M > 0 be the Riemannian volume form. Let p X be the probability density of a U -valued random variable X with respect to

ω M on U ⊂ M, so that Pr (X ∈ A) = A p X (Y )ω M (Y ) for any Borel-measurable subset A of U. Let Y ∈ U , we can define the expectation of the real valued random variable d 2 (, Y ) : U → R by E d 2 (, Y ) = U d 2 (X, Y )p X (X)ω M (X). Now, let w : U → (0, ∞) be an integrable function and U w (X) ω M (X) = 1. Then, observe that, E w d 2 (, Y ) := U w(X)d 2 (X, Y )p X (X)ω M (X) = C U d 2 (X, Y ) p X (X) ω M (X) = C E d 2 (, Y ) .

Here, p X is the probability density corresponding to the probability measure Pr defined by, Pr (X

∈ X) = X p X (Y )ω M (Y ) := X 1 C p X (Y )w(Y )ω M (Y )

, where, X lies in the Borel σ-algebra over U and C = U p X (Y )w(Y )ω M (Y ). Note that the constant C > 0, since p X is a probability density, w > 0 and M is orientable. Thus, E w d 2 (, Y ) with respect to p X is proportional to E d 2 (, Y ) with respect to p X . Now, we will state the following proposition (the proof is in the appendix section).

Proposition 2. (i) supp (p X ) = supp ( p X ). (ii) wFE (X, w) = FE X .

Let {X i } N i=1 be samples drawn from p X and X i N i=1 be samples drawn from p X . In order to compute wFM, we will now present an online algorithm (inductive FM Estimator -dubbed iFME).

Given, {X i } N i=1 ⊂ U and {w i := w (X i )} N i=1 such that ∀i, w i > 0, the n th estimate, M n of wFM ({X i } , {w i }) is given by the following recursion:

EQUATION

In the above equation, Γ Y X : [0, 1] → U is the shortest geodesic curve from X to Y . Observe that, in general wFM is defined with N i=1 w i = 1, but in above definition, N i=1 w i = 1. We can normalize {w i } to get { w i } by w i = w i / ( i w i ), but then Eq. 2 will not change as w n / n j=1 w j = w n / n j=1 w j . This gives us an efficient inductive/recursive way to define convolution operation on M. Now, we state that the proposed wFM estimator is consistent (the proof is in the appendix).

Proposition 3. Using the above notations and assumptions, let {X i } N i=1 be i.i.d. samples drawn from p X on M. Let the wFE be finite. Then, M N converges a.s. to wFE as N → ∞.

2.2. NONLINEAR OPERATION BETWEEN WFM-LAYERS FOR M-VALUED DATA:

In the traditional CNN model, we need a nonlinear function between two convolutional layers similar to ReLU and softmax. As argued in Mallat (2016) , any nonlinear function used in CNN is basically a contraction mapping. Formally, let F be a nonlinear mapping from U to V . Let assume, U and V are metric spaces equipped with metric d U and d V respectively. Then, F is a contraction mapping iff ∃c < 1 such that, d V (F (x), F (y)) ≤ c d U (x, y). F is a non-expansive mapping (Mallat, 2016) = max i,j d (X i , Y j ). The choice of weights for wFM is said to be trivial if one of the weights is 1 and hence the rest are 0.

iff d V (F (x), F (y)) ≤ d U (x, y).

Proposition 4. For all nontrivial choices of {α i } N i=1 and {β j } M j=1 satisfying the convexity constraint , ∃c < 1 such that,

d wFM {X i } N i=1 , {α i } N i=1 , wFM {Y j } M i=1 , {β j } M i=1 ≤ c d ι {X i } N i=1 , {Y j } M j=1

(3)

2.3. THE INVARIANT (LAST) LAYER:

We will form a deep network by cascading multiple wFM blocks each of which acts as a convolutiontype layer. Each convolutional-type layer is equivariant to the group action, and hence at the end of the cascaded convolutional layers, the output is equivariant to the group action applied to the input of the network. Let d be the number of output channels each of which outputs a wFM, hence each of the channels is equivariant to the group action. However, in order to build a network that yields an output which is invariant to the group action, we now seek the last layer (i.e., the analogue to a linear classifier) to be invariant to the group action. The last layer is thus constructed as follows:

Let {Z 1 , • • • , Z d } ⊂ M be the output of d channels and M u = FM {Z i } d i=1 = wFM {Z i } d i=1 , { 1 /d} d 1 be the unweighted FM of the outputs {Z i } d i=1

. Then, we construct a layer with d outputs whose i th output o i = d (M u , Z i ). Let c be the number of classes for the classification task, then, a fully connected (FC) layer with inputs {o i } and c output nodes is build. A softmax operation is then used at the c output nodes to obtain the outputs {y i } c i=1 . In the following proposition we claim that this last layer with {Z i } In Fig. 1 we present a schematic of ManifoldNet depicting the different layers of processing the manifold-valued data as described above in Sections 2.1-2.3.

3. EXPERIMENTS:

In this section we present performance of the ManifoldNet framework on several standard computer vision problems. The breadth of application coverage here includes classification and reconstruction problems. We begin with a video classification problem and then present a reconstruction problem using an auto-encoder-decoder set up.

3.1. VIDEO CLASSIFICATION:

We start by using the method in Yu & Salzmann (2017) which we summarize here. Given a video with dimensions F × 3 × H × W of F frames, 3 color channels and a frame size of H × W , we can apply a convolution layer to obtain an output of size F × C × H × W consisting of C channels of size H × W . We compute the covariance matrix of the channels to obtain a sequence of F symmetric positive (semi) definite matrices of size C × C. , and for the weights w i of any output channel we add a weight penalty of the form ( w i -1) 2 to the loss function to ensure that we obtain a proper wFM. We then reshape this to F K × C × C and pass it through an invariant final layer (section 2.3) to obtain a vector of size F K. Finally, a single FC+SoftMax layer is applied to produce a classified output. We call this the SPD temporal convolutional architecture network (SPD-TCN). Figure 2 illustrates the network architecture described above. In general, the SPD-TCN tends to perform very well on video classification tasks while using very few parameters, and runs efficiently due to the wFM structure. (Hochreiter & Schmidhuber, 1997) , SRU (Oliva et al., 2017) , TT-LSTM and TT-GRU (Yang et al., 2017) networks. For the LSTM and SRU networks, convolution layers are also used before the recurrent unit. We will compare directly with the results presented in Chakraborty et al. (2018) . For details of the various architectures used please see section 5 of Chakraborty et al. (2018) . We will also compare with Huang & Van Gool (2017) , which proposes a set of layers for learning SPD matrices, dubbed SPDNet. When using SPDNet we first downsample the videos to 20 × 20 frames and compute the covariance matrix of the frames, which is then fed to an architecture similar to that of the emotion recognition experiment of Huang & Van Gool (2017) Specifically, we use three blocks of BiMap/ReEig layers, since such a configuration gave the best validation performance. The transformation sizes for these blocks are set to 400 × 200, 200 × 100 and 100 × 50 respectively. The Moving MNIST data generated in Srivastava et al. (2015) consists of 1000 samples, each of 20 frames. Each sample shows two randomly chosen MNIST digits moving within a 64 × 64 frame, with the direction and speed of movement fixed across all samples in a class.

Covariance Matrices:

C × C C × C C × C C × C ... [ , , … , ] ∈ y 1 y 2 y n R n = d( , FM({ })) y i x i x i wFM({ }, { }) x i w i . . Text C × C C × C C × C C × C C × C

The speed is kept the same across different classes, but the digit orientation will differ across two different classes. We summarize the 10-fold cross validation results for several orientation differences between classes in Table 1 . For this experiment the SPD-TCN will consist of a single wFM layer with kernel size 5 and stride 3 returning 8 channels, making for an 8 × 8 covariance matrix. We then apply three temporal SPD wFM layers of kernel size 3 and stride 2, with the following channels 1 → 4 → 8 → 16, i.e. after these three temporal SPD wFMs we have 16 temporal channels. This 16 × 8 × 8 is used as an input to the invariant final layer to get a 16 dimensional output vector, which is transformed by a FC+SoftMax layer to obtain the output. Here we present experiments demonstrating the applicability of the theory layed out in Section 2 to the case of linear dimensionality reduction, specifically principal component analysis (PCA), which is the workhorse of many machine learning algorithms. In Chakraborty et al. (2017) , authors presented an online subspace averaging algorithm for construction of principal components via intrinsic averaging on the Grassmannian. In this section, we achieve the intrinsic Grassmann averaging process in the framework of ManifoldNet to compute the principal subspaces and achieve the dimensionality reduction. In the context of DNNs, dimensionality reduction is commonly achieved via an autoencoder architecture. More recently, DNNs have shown promising results when the data manifold is intrinsically non-linear, as in the case of natural images. In the deep learning community this has become a field in its own right, known as representation learning or feature learning (Bengio et al., 2013) 2016)), Kingma & Dhariwal (2018) . Many of these architectures are modifications of the traditional autoencoder network, which involves learning an identity map through a small latent space. In our application, we modify the traditional autoencoder model by adding a ManifoldNet layer to perform a learned linear dimensionality reduction in the latent space, although in principal, our techniques can be applied to most autoencoder based models such as the variational autoencoders. To compute a linear subspace in the ManifoldNet framework we use an intrinsic averaging scheme on the Grassmannian. A point on the Grassmannian Gr(k, n) corresponds to k-dimensional subspace of R n and thus can be specified by an orthonormal basis X. Chakraborty et al. (2017) proposed an efficient intrinsic averaging scheme on Gr(k, n) that converges to the k-dimensional principal subspace of a normally distributed dataset in R n . In the ManifoldNet framework we can modify this technique to learn a wFM of points on the Grassmannian that corresponds to a subspace of the latent space which minimizes the reconstruction error by using a Grassmannian averaging layer that learns the weights in the wFM. This essentially will give us a lower dimensional representation of the samples after projecting them on to the learned subspace. Note that combining the convergence proof in Chakraborty et al. (2017) and Proposition 2 (ii), we claim that the wFM learned using the ManifoldNet asymptotically converges to the principal subspace. Now, we give a detailed description of our experimental setup to show the applicability of ManifoldNet to dimensionality reduction.

3.2.1. VIDEO RECONSTRUCTION EXPERIMENT:

A traditional convolutional autoencoder performs non-linear dimensionality reduction by learning an identity function through a small latent space. A common technique used when the desired latent space is smaller than the output of the encoder is to apply a fully connected layer to match dimensions. We replace this fully connected layer by a weighted subspace averaging and projection block, called the Grassmann averaging layer. Specifically, we compute the wFM of the output of the encoder to get a subspace in the encoder output space. We then project the encoder output onto this space to obtain a reduced dimensionality latent space. In general this offers a significant parameter reduction while also increasing the reconstruction error performance of the autoencoder and giving realistic reconstructions. We call an autoencoder with the Grassmann averaging block error tolerance plot comparison between autoencoder and iFME+autoencoder representing an improvement of 10.9%. We also observe a parameter reduction of 46%, which can be attributed to the number of parameters in the large fully connected layer. In Fig. 5 , the computation time is plotted against error tolerance for the autoencoder and the iFME+autoencoder. We can see that iFME+autoencoder achieves faster convergence than autoencoder. It is possible to obtain a low reconstruction error on autoencoding tasks and still observe low visual quality reconstructions. To ensure this is not the case we run the same experiment on 300 frames of the 1280 × 720 short film1 , with a latent space frame dimension of 300x50. In Fig. 4 we compare the visual quality of our autoencoder to that of PCA with 50 principcal components, i.e., we reduce the dimension from 1280 × 720 × 3 to 50. The entire sample reconstruction is shown in2 (in the same order as in Fig. 4 ).

4. CONCLUSIONS:

In this paper, we presented a novel deep network called ManifoldNet suited for processing manifoldvalued data sets. Inputs to the ManifoldNet are manifold-valued and not real or complex-valued functions defined on non-Euclidean domains. Our key contributions are: (i) A novel deep network to be perceived as a generalization of the CNN to the case when the input data are manifold-valued using purely intrinsic operations on the manifold where the data reside. (ii) Analogous to convolutions in vector spaces -which can be computed using the weighted mean -we present wFM operations on the manifold and prove the equivariance of the wFM to natural group operations admitted by the manifold. This equivariance allows us to share the learned weights within a layer of the ManifoldNet. (iii) An efficient recursive wFM estimator that is provably convergent is presented. (iv) Experimental results demonstrating the efficacy of the ManifoldNet for, (a) video classification and (b) principal component computation from videos and reconstruction are also presented. is group equivariant, Z i becomes g.Z i . Because of the property of the distance under group action, d (g.M u , g.Z i ) = d (M u , Z i ). Hence, one can see that if we change the inputs {Z i } to {g.Z i }, the output y will remain invariant.

:

https://www.youtube.com/watch?v=t1hMBnIMt5I

https://streamable.com/3yqrx

Figures and Tables:

Figure 1: Schematic diagram of ManifoldNet

i=1 inputs and {y i } c i=1 outputs is group invariant. Proposition 5. The last layer with {Z i } d i=1 inputs and {y i } c i=1 outputs is group invariant.

Figure 2: SPD-TCN Network ArchitectureFrom here we can apply a series of temporal ManifoldNet wFMs to transform the F × C × C input to a temporally shorter F × K × C × C output, where K is the number of wFM channels. Within the temporal ManifoldNet wFMs we use a simple weight normalization to ensure that the weights are within [0, 1], and for the weights w i of any output channel we add a weight penalty of the form ( w i -1) 2 to the loss function to ensure that we obtain a proper wFM. We then reshape this to F K × C × C and pass it through an invariant final layer (section 2.3) to obtain a vector of size F K. Finally, a single FC+SoftMax layer is applied to produce a classified output. We call this the SPD temporal convolutional architecture network (SPD-TCN). Figure2illustrates the network architecture described above. In general, the SPD-TCN tends to perform very well on video classification tasks while using very few parameters, and runs efficiently due to the wFM structure.

Figure 3: Pictorial description of autoencoder+iFME

with works including Vincent et al. (2010), Kingma & Welling (2013) Oord et al. (2016), Van Den Oord et al. (

Figure 4: Reconstruction of select movie frames (a) original frame (b) using PCA (c) using iFME+autoencoder an autoencoder+iFME network, as shown in Fig. 3.In the experiments we compare this to other dimensionality reduction techniques, including regular autoencoders that use fully connected layers to match encoder and latent space dimensions. We begin by testing on a 1000 frame color sample of video from the 1964 film "Santa Clause Conquers the Martians" of frame size 320 × 240. Here we use an 8 layer encoding-decoding architecture with Conv → ELU → Batchnorm layers, with the final layer applying a sigmoid activation to normalize pixel values. The encoder returns a feature video consisting of 128 channels of size 120 for a dimension of 1000 × 15360. We compare a fully connected layer to a Grassmann averaging layer, both mapping to a desired latent space of dimension 1000 × 20. The per pixel average reconstruction error for the Grassmann block network is 0.0110, compared to 0.0122 for the fully connected network,

Figure 5: Computation time vs. error tolerance plot comparison between autoencoder and iFME+autoencoder

Comparison results on Moving MNIST

Table content: <table/>

We compare a fully connected layer to a Grassmann averaging layer, both mapping to a desired latent space of dimension 1000 × 20. The per pixel average reconstruction error for the Grassmann block network is 0.0110, compared to 0.0122 for the fully connected network,

Table content: <table><tr><td>In the experiments we com-</td></tr><tr><td>pare this to other dimensionality reduc-</td></tr><tr><td>tion techniques, including regular autoen-</td></tr><tr><td>coders that use fully connected layers to</td></tr><tr><td>match encoder and latent space dimen-</td></tr><tr><td>sions. We begin by testing on a 1000</td></tr><tr><td>frame color sample of video from the 1964</td></tr><tr><td>film "Santa Clause Conquers the Martians"</td></tr><tr><td>of frame size 320 × 240. Here we use</td></tr><tr><td>an 8 layer encoding-decoding architecture</td></tr><tr><td>with Conv → ELU → Batchnorm lay-</td></tr><tr><td>ers, with the final layer applying a sigmoid</td></tr><tr><td>activation to normalize pixel values. The</td></tr><tr><td>encoder returns a feature video consisting</td></tr><tr><td>of 128 channels of size 120 for a dimen-</td></tr><tr><td>0.02 autoencoder+iFME 0.03 sion of 1000 × 15360. 0.01 0.04 0.05 0.06 0.07 0.08 0.09 0 2000 4000 6000 8000 10000 12000 14000 Time(s) autoencoder</td></tr><tr><td>Reconstruction error/pixel</td></tr></table>

Peer review (iclr2019):
 This paper proposes a method to use weighted Frechet Mean (wFM) for the operation on Manifold valued data for CNN. The novel point is to view wFM as a convolutional layer. Overall, this paper is mathematically well written, however, how each theory improves CNN and the model used in experiments are not clear enough. Pros + The use of wFM instead of a convolutional layer is an interesting idea. + This paper is mathematically well written. Cos - It is hard to understand how each theory presented in this paper helps to improve CNN. For example, the invariance to group operation. Some experimental results would help to understand the advantage of the group invariance. - It is also unclear why the authors constructed the invariant last layer although the inputs of the last layer are invariant under group operations. - In the introduction section, the authors raised the omnidirectional camera, diffusion magnetic resonance imaging, elastography as examples of manifold-valued data. However, experiments are limited to standard video sequences. - It is unclear how to obtain the weights {w_i} of wFM by backpropagation. - Since the contribution of this paper is to to use wFM instead of a convolutional layer, it is more interesting to visualize the weights {w_i}. - More explanation needs for the model used for experiments. Especially in dimensional reduction experiments, I could not understand how each subspace is obtained and averaged. If each frame is a subspace, by averaging frames, the reconstruction would be blurred.