Neural Architecture Search of SPD Manifold Networks
Authors: Rhea Sanjay Sukthanker
Zhiwu Huang
Suryansh Kumar
Erik Goron
Yan Wu
Luc Van Gool
Venue:  (2021)
Keywords: Neural Architecture Search
AutoML

Abstract: In this paper, we propose a new neural architecture search (NAS) problem of Symmetric Positive Definite (SPD) manifold networks. Unlike the conventional NAS problem, our problem requires to search for a unique computational cell called the SPD cell. This SPD cell serves as a basic building block of SPD neural architectures. An efficient solution to our problem is important to minimize the extraneous manual effort in the SPD neural architecture design. To accomplish this goal, we first introduce a geometrically rich and diverse SPD neural architecture search space for an efficient SPD cell design. Further, we model our new NAS problem using the supernet strategy, which models the architecture search problem as a one-shot training process of a single supernet. Based on the supernet modeling, we exploit a differentiable NAS algorithm on our relaxed continuous search space for SPD neural architecture search. Statistical evaluation of our method on drone, action, and emotion recognition tasks mostly provides better results than the state-of-the-art SPD networks and NAS algorithms. Empirical results show that our algorithm excels in discovering better SPD network design and providing models that are more than 3 times lighter than searched by state-of-the-art NAS algorithms.

Under review as a conference paper at ICLR 2021
NEURAL ARCHITECTURE SEARCH OF SPD MANIFOLD NETWORKS
Anonymous authors
ABSTRACT
In this paper, we propose new neural architecture search (NAS) problem of Symmetric Positive Definite (SPD) manifold networks. Unlike the conventional NAS problem, our problem requires to search for unique computational cell called the SPD cell. This SPD cell serves as a basic building block of SPD neural architectures. An efficient solution to our problem important to minimize the extraneous manual effort in the SPD neural architecture design. To accomplish this goal, we first introduce a geometrically rich and diverse SPD neural architecture search space for an efficient SPD cell design. Further, model our new NAS problem using the supernet strategy, which models the architecture search problem one-shot training process of a single supemee.. Based on the supernet modeling, we exploit a differentiable NAS algorithm on our relaxed continuous search space for SPD neural architecture search. Statistical evaluation of our method drone, action, and emotion recognition tasks mostly provides better results than the state- of-the-art SPD networks and NAS algorithms. Empirical results show that our algorithm excels in discovering better SPD network design and providing models that are more than times lighter than searched by state-of-the-ar NAS algorithms.
INTRODUCTION
Designing a favorable neural network architecture for given application requires a lot of time, effort, and domain expertise. To mitigate this issue, researchers in the recent years have started developing algorithms to automate the design process of neural network architectures (Zoph & Le, 2016; Zoph et al., 2018; Liu et al., 2017; 201 8a; Real et al., 201 9; Liu et al., 8b; Tian et al., 2020). Although these neural architecture search (NAS) algorithms have shown great potential to provide an optimal architecture for given application, it is limited to handle architectures with Fuclidean operations and representation. To deal with non-euclidean data representation and corresponding set of operations, researchers have barely proposed any NAS algorithms -to the best of our knowledge.
is well-known that manueold-valued data representation such as symmetric positive definite (SPD) matrices have shown overwhelming accomplishments in many real-world applications such as pedestrian detection (Tuzel et al., 2006; 2008), magnetic resonance imaging analysis (Pennec et al., 2006), action recognition (Harandi al., 2014 face recognition (Huang et al., 2014; 2015), brain- computer interfaces (Barachant et al., 2011), 1), structure from motion (Kumar et al., 2018; Kumar, 2019), etc. Also, in applications like diffusion tensor imaging the brain, drone imaging, samples are collected directly as SPD's. As a result, neural network usage based on Euclidean data representation becomes inefficient for those applications. Consequently, this has led the development of the SPD neural network (SPDNet) architectures for further improvements in these areas of research (Huang & Van Gool, 2017; Brooks et al.. 2019). However, these architectures are handcrafted, so the operations or the parameters defined for these networks generally change as per the application. This motivated us to propose a new NAS problem of SPD manifold networks. solution to this problem can reduce unwanted efforts in SPDNet design. Compared to the traditional NAS problem, our NAS problem requires a new definition of computation cell and proposal for diverse SPD candidate operation set. In particular, model the basic architecture cell with specific directed acyclic graph (DAG), where each node a latent SPD representation, and each edge corresponds to a SPD candidate operation. Here, the intermediate transformations between nodes respect the geometry of the SPD manifolds.
For solving the suggested NAS problem, we exploit a supernet search strategy which models the architecture search problem one-shot training process of a supernet that comprises of mixture

---

Under review as a conference paper at ICLR 2021
of SPD neural architectures. The supernet modeling enables to perform a differential architecture search on a continuous relaxation of SPD neural architecture search space, and therefore, can be solved using gradient descent approach. Our evaluation validates that the proposed method can build a reliable SPD network from scratch. We show the results of our method on benchmark datasets that clearly show results better than handcrafted SPDNet. Our work makes the following contributions:
We introduce a NAS problem of SPD manifold networks that opens up a new direction of research in automated machine learning and SPD manifold learning. Based on a supernet modeling, we propose a novel differentiable NAS algorithm for SPD neural architecture search. Concretely, we exploit a sparsemax based Fréchet mixture of SPD operations to introduce sparsity that is essential for an effective diffentiable search, and bi-level optimization with manifold- update and convexity-basee update to jointly optimize architecture parameters and network kernel weights. Besides well-studied operations from exiting SPDNets (Huang & Van Gool, 2017; Brooks et al., 2019; Chakraborty et al., 2020), we follow Liu et al. (201 to further introduce some new SPD layers, i.e., skip connection, none operation, max pooling and averaging pooling. Our introduced additional set of SPD operations make the search space more diverse for the neural architecture search algorithm to obtain more generalized SPD neural network architectures.
Evaluation on three benchmark datasets shows that our searched SPD neural architectures can outperform the existing handcrafted SPDNets (Huang & Van Gool, 2017; Brooks et al., 2019; Chakraborty et al., 2020) and the state-of-the-arr NAS methods (Liu et al., 2018b; Chu et al., 2020). Notably, our searched architecture is more than times lighter than those searched by the traditional NAS algorithms.
2 BACKGROUND
In recent years, plenty of research work has been published in the area of NAS (Gong et al., 2019; Liu et al., 2019; Nayman et al., 2019; Guo et al., 2020). This is probably due to the success of deep learning for several applications which has eventually led to the automation of neural architecture design. Also, improvements in the processing capabilities of machines has influenced the researchers to work out this computationally expensive yet an important problem. Computational cost for some of the well-known NAS algorithms is in thousands of GPU days which has resulted in the development of several computationally efficient methods (Zoph et al., 2018; 8;; Real et al., 2019; Liu et al., 201 2017; Baker et al., 2017; Brock et al., 2017: Bender, 2019; Elsken et al., 2017: Cai et al., 2018; Pham et al., 2018; Negrinho & Gordon, 2017; Kandasamy et al., 2018; Chu et al., 2020). In this work, we propose new NAS problem of SPD networks. We solve this problem using supernet modeling methodology with one-shot differentiable training process of an overparameterized supernet. Our modeling is driven by the recent progress in supernet methodology. Supernet methodology has shown great potential than other NAS methodologies in terms of search efficiency. Since our work is directed towards solving a new NAS problem, we confine our discussion to the work that have greatly influenced our method i..., one-shot NAS methods and SPD networks.
To the best of our knowledge. there are mainly two types of one-shot NAS methods based on the architecture modeling (Elsken et al., 2018) (a) parameterized architecture (Liu et al., 2018b; Zheng et al., 2019; Wu et al., 2019; Chu et al., 2020), and (b) sampled architecture (Deb et al., 2002; Chu et al., 2019). In this paper, we adhere to the parametric modeling due to its promising results on conventional neural architectures. A majority of the previous work on NAS with continuous search space fine-tunes the explicit feature of specific architectures (Saxena & Verbeek, 2016; Veniat & Denoyer, 2018; Ahmed & Torresani, 2017; Shin et al., 2018). On the contrary, Liu et al. (2018b); Liang et al. (2019); Zhou et al. (2019): Zhang al. (2020); Wu al. (2020): Chu al. (2020) provides architectural diversity for NAS with highly competitive performances. The other part of our work focuses on SPD network architectures. There exist algorithms to develop handcrafted SPDNet (Huang & Van Gool, 2017; Brooks et al., 2019; Chakraborty et al., 2020). To automate the process of SPD network design, in this work, we choose the most promising approaches from these fields (NAS (Liu et al., 2018b) (b)..). SPD networks (Huang & Van Gool, 2017)) and propose a NAS algorithm for SPD inputs. Next, we summarize the essential notions of Riemannian geometry of SPD manifolds, followed by an introduction of some basic SPDNet operations and layers. As some of the introduced operations and layers have been well-studied by the existing literature, we applied them directly to define our SPD neural architectures" search space.
2

---

Under review as a conference paper at ICLR 2021
Representation and Operation: We denote
real SPD as X ST+. real SPD matrix X E ES"+ satisfies the property that for any non-zero R R=, xx 0 (Harandi al., 2017). We denote T as the tangent space of the manifold M at X E E and log corresponds to matrix logarithm. Let X1, X2 be any two points on the SPD manifold then the distance between them is given by
( X2) = 0.5|| log x2X, )|
There are other efficient methods to compute distance between two points on the SPD manifold (Gao ct al., 2019; Dong et al., 2017b), however, their discussion is beyond the scope of our work. Other property of the Riemannian manifold of our interest is local diffeomorphism of geodesics which is one-to-one mapping from the point on the tangent space of the manifold to the manifold (Pennec, 2020; Lackenby, 2020). To define such notions, let X S be the base point and, Y Sx then (2) associates Y E S to a point on the manifold (Pennec, 2020).
expx (Y) x exp( X y 1) x)x² E ST++, VY Tx
(2)
Similarly, inverse map is defined logx (Z) = log(X 22 x)x+ E Tx, A E S
1) Basic operations of SPD Network: is well-known that operations such as mean centralization, normalization, and adding bias batch of data are inherent performance booster for most neural networks. In the same spirit, existing works like Brooks et al. (2019); Chakraborty (2020) use the notion of these operations for the SPD or general manifold data to define analogous operations on manifolds. Below we introduce them following the work of Brooks et al. (2019).
Batch mean, centering and bias: Given batch of N SPD matrices {X;) we can compute its Riemannian barycenter (z) as B argmin Σ (X, Xp)). It is sometimes referred
x,
as Fréchet mean (Moakher, 2005; Bhatia & Holbrook, 2006). This definition can be extended to compute the weighted Riemannian Barycenter also known as weighted Fréchet Mean (wFM).
argmin X, ESP
(XX,,(X, s.t. wA 2 0 and Σ =1
E3:(3) can be approximated using Karcher flow (Karcher, 1977; Bonnabel, 2013; Brooks et al., 2019) or recursive geodesic mean (Cheng et al., 2016; Chakraborty et al., 2020).
Analogous to standard CNN, methods like Huang & Van Gool (2017); Brooks et al. (2019); Chakraborty et al. (2020) designed SPD layers to perform operations that respect SPD manifold constraints. Assuming X E $ be the input SPD matix to the ktth layer, the SPD network layers are defined as follows:
2) Basic layers of SPD Network:
BiMap layer: This layer corresponds to a dense layer for SPD data. The BiMap layer reduces the dimension of input SPD matrix via transformation matrix Wt X₂ W WT. To ensure the matrix X to be an SPD matrix, the Wj matrix must be of full row-rank.
Batch normalization layer: To perform batch normalization after each BiMap layer, we first compute the Riemannian barycenter of the batch of SPD matrices followed by running mean update step, which is Riemannian weighted average between the batch mean and the current running mean, with the weights (1 0) and (0) respectively. Once mean is calculated, centralize and add bias to each SPD sample of the batch using Eq:(4) (Brooks et al., 2019), where 30 is the notation used for parallel transport:
F xx 1, is the identity matrix Bias the batch Bias towards G: X = f.-c(())) = G G is the identity matrix
Batch centering: Centering the 3 X =
(4)
ReEig layer: The ReFig layer is analogous to ReLU like layers present in the classical ConvNets. It aims to introduce non-linearity to SPD network. The ReEig for the ttt layer is defined as: X U max Σn U where, X U2~1 1 - I is the identity matrix, and > 0 is rectification threshold value. UK-1, Σ are the orthonormal matrix and singular-value matrix respectively which are obtained via matrix factorization of X
Following (Tuzel et al. (2006; 2008); Brooks et al. (2019)), we focus the estimate of wFM with Karcher flow, and the thorough study the general wFM's existence and uniqueness is beyond the facus of this paper.
3

---

Under review as a conference paper at ICLR 2021
(a) SPD Cell
(b) Mixture of Operations
(c) Optimized SPD Cell
Figure I: (a) SPD cell structure composed of SPD nodes, input node and output node. Initially edges are unknown (h) Mixture candidate SPD operations between nodes (c) Optimal architecture obtained after solving the relaxed continuous search space under a bi-level optimization formulation.
LogEig layer: To map the manifold representation of SPD to flat space so that Buclidean operation can be performed, LogÉig layer introduced. The LogEig layer is defined as: X₁,- = Uk-1 1oo(ooog(E-)1) 1 where, X UK-1 UR-IEA-1UT-1 The LogEig layer is used with fully connected layers to solve tasks with SPD representation.
ExpEig layer: This layer maps the corresponding SPD representation from flat space back SPD manifold space. It is defined as X Un-1 exp (es-)(E--- where, X Uk Ex- U
Weighted Riemannian pooling layer: It uses wFM definition to compute the output of the layer. Recent method use recursive geodesic mean algorithm calculate the mean (Chakraborty et al., 2020), in contrast, we use Karcher flow algorithm to compute it (Karcher, 1977) as it is simple and widely used in practice.
NEURAL ARCHITECTURE SEARCH OF SPD MANIFOLD NETWORK
As alluded before, to solve the suggested problem, there are a few key changes that must be introduced. Firstly, definition of the computation cell is required. In contrast to the computational cells designed by regular NAS algorithms like Liu et al. (2018b); Chu et al. (2020), our computational cell which call as SPD cell, additionally incorporate the notion of SPD manifold geometry so that SPD representations can be treated properly. On the other hand, like the regular NAS cell design, our SPD cell can either be normal cell that returns SPD feature maps of the same width and height or, reduction cell in which the SPD feature maps are reduced by a certain factor in width and height. Secondly, to solve our new NAS problem will require an appropriate and diverse SPD) search space that can help NAS method to optimize for an effective SPD cell, which can then be stacked and trained to build an efficient SPD neural network architecture.
Concretely, SPD cell is modeled by directed asyclic graph (DAG) which is composed of nodes and edges. In our DAG each node an latent representation the SPD manifold valued data i.e. an intermediate SPD feature map and, each edge corresponds to a valid candidate operation on SPD manifold (see Fig. (a)). Each edge of a SPD cell is associated with a set of candidate SPD manifold operations (0M) that transforms the SPD valued latent representation from the source node (say x ) to the target node (say X XM) We define the intermediate transformation between the nodes in (i.)) (x. x x()) where dm denotes the geodesic
our SPD cell as: x XM = argmin Eie; 87m (
x
distance (q)( Generally, this transformation result corresponds to the unweighted Fréchet mean of the operations based the predecessors, such that the mixture of all operations still reside SPD manifolds. Note that our definition of SPD cell ensures that each computational graph preserves the appropriate geometric structure of the SPD manifold. Equipped with the notion SPD cell and its intermediate transformation, we are prepared to propose our search space ($3.1) followed by the solution to our SPDNet NAS problem ($3.2) and its results (G4).
3.1 SEARCH SPACE
Our search space consists of a set of valid SPD network operations which is defined for the supernet search. First of all, the search space includes some existing SPD operations', e.g., BiMap, batch normalization, ReEig, LogEig, ExpEig and weighted Riemannian pooling layers, all of which are
Our search space include other exiting SPD operations like manifokdnorm Chakmahory (2020). However, comprehensive
study on them focus, which studying algorithm given promising search space.
4

---

Under review as a conference paper at ICLR 2021
Table 1:
Search space for the proposed SPD architecture search method.
Operation
Definition
Operation
Delinition
BlMap.U
{ MBiMap. Batch Normalization
WoightedRemanna Poolingnoonnaall normal
{WFM on SPD multiple times)
BiMap_1
[BiMap.Barc Batch Normslization. ReEig)
AveragePoollig rrduuced
(oogEig, AveragePooling ExpEig
BiMap 2
RRFFgg, BiMap, Barch Normalization)
MaxPooling reduced
(LogEig, MaxPooling, ExpÉig]
Skip_normal
(Quiput same as input)
Skippreducdd = (Ci BiMap(X,.)) [U,. Da. --] svd((C.) in = 1, 2),, Cuu U,D,D where, U2 ding(U,, U2) and D, diag(D,, Dy)
None_nornal
Return identity matrix)
introduced in Sec.2. Though those individual operations (e.g., BiMap, LogEig, ExpEig) have been explored well by existing works, different aggregations on them are still understudied, which are essential to enrich our search space. To be specific to enrich the search space, following Liu et al. (2018b); Gong et al. (2019) traditional NAS methods, we apply the SPD batch normalization to every SPD convolution operation (i.e., BiMap), and design three variants of convolution blocks including the one without activation (i.e., ReEig), the one using post-activation and the one using pre-activation (see Table I). In addition, we introduce five new operations analogous to DARTS (Liu et al., 2018b) to enrich the search space in the context of SPD networks. These are skip normal, none normal, average pooling, max pooling and skip reduced. The effect of such diverse operation choices have not been fully explored for SPD networks. All the candidate operations are illustrated in Table (1), and the definitions of the new operations are detailed as follows:
(a) Skip normal: It preserves the input representation and is similar to skip connection. (b) None normal: It corresponds to the operation that returns identity as the output i.e, the notion of zero in the SPD space. (c) Max pooling: Given set of SPD matrices, max pooling operation first projects these samples to a flat space via a LogEig operation, where a standard max pooling operation is performed. Finally, an ExpEig operation is used to map the sample back to the SPD manifold.
(d) Average pooling: Similar to Max pooling, the average pooling operation first projects the samples to the flat space using a LogEig operation, where a standard average pooling employed. To map the sample back to SPD manifold, an ExpEig operation is used. (e) Skip reduced: It is similar to akip-normal' but in contrast, it decomposes the input into small matrices to reduces the nnterddeendeeccy between channels. Our definition of reduce operation is in line with the work of Liu et al. (2018b).
The newly introduced operations allow us to generate a more diverse discrete search space. As presented in Table 2, the randomly selected architecture (generally consisting of the newly introduced SPD operations) shows some improvement over SPDNet and SP both of which only contain conventional SPD operations. This establishes the effectiveness of the introduced rich search space.
3.2 SUPERNET SEARCH
To solve the suggested new NAS problem, one of the most promising NAS methodologies is supernet modeling. While we can resort to some other NAS methods to solve the problem like reinforcement learning based method (Zoph & Le, 2016) or evolution based algorithm (Real et al., 2019). in general, the supernet method models the architecture search problem as a one-shot training process of a single supernet that consists of all architectures. Based on the supernet modeling, we search for the optimal SPD neural architecture either using parameterization of architectures or sampling of single-path architectures. In this paper, we focus on the parameterization approach that is based on the continuous relaxation of the SPD neural architecture representation. Such an approach allows for an efficient search of architecture using the gradient descent approach. Next, we introduce our supernet search method, followed by a solution to our proposed bi-level optimization problem. Fig.1(6 and Fig. (c) illustrates an overview of our proposed method.
To search for an optimal SPD architecture (a)), we optimize the over parameterized supernet. In essence, it stacks the basic computation cells with the parameterized candidate operations from our search space in one-shot search manner. The contribution of specific subnets to the supernet helps in deriving the optimal architecture from the supernet. Since the proposed operation search space is discrete in nature, we relax the explicit choice of operation make the search space continuous. To do so, we use wFM over all possible candidate operations. Mathematically,
No
(M)) = argmin ā 8282 (0/00) (X)))XM)) x%): (XM),xM)) subject to: a 155 (5)
X M
5

---

Under review as a conference paper at ICLR 2021
Algorithm 1:
The proposed Neural Architecture Search of SPD Manifold Nets (SPDNeINAS)
Require:
Mixed Operation L. which is parameterized by of for each edge & E Ne;
while not converged do
Stepl: Update a (architecture) using (8) solution by satisfying an additional strict convex constraint.
and (Eq:(9), Eq: (10) should follow the gradient descent on SPD manifold; Step2: Update w by solving V Etraso (w, a); Ensure SPD manifold gradient to update w (Absil et al.,
Note that updates
2009; Huang & Van Gool, 2017; Brooks ct al., 2019);
end
Ensure: Final architecture based on a. Decide the operation at an edge &s using ngmmax(ab)
M
where, 0%1y is the 1tth candidate operation between nodes, X is the intermediate SPD manifold mean (Eq.3) and, .N. denotes number of edges. We can compute wFM solution either using Karcher flow (Karcher, 1977) or recursive geodesic mean (Chakraborty et al., 2020) algorithm. Nonetheless, we adhere to Karcher flow algorithm as it is widely used to calculate wFM To impose the explicit convex constraint on å we project the solution onto the probability simplex as
minimize ||a - = subject to: 1" 1, 0 a
(6)
(q:(6) enforces the explicit constraint on the weights to supply for our task and can easily be added as a convex layer in the framework (Agrawal et al., 2019). This projection is likely to reach the boundary of the simplex, in which case a becomes sparse (Martins & Astudillo, 2016). Optionally, softmax, sigmoid and other regularization methods can be employed to satisfy the convex constraint. However, Chu al. (2020) has observed that the use of softmax can cause performance collapse and may lead to aggregation of skip connections. While Chu et al. (2020) suggested sigmoid can overcome the unfairness problem with softmax, it may output smoothly changed values which is hard to threshold for dropping redundant operations with non-marginal contributions to the supernet. Also, FairDARTS (Chu et al., 2020) regularization, may not preserve the summation equal to I constraint. Besides, Chakraborty et al. (2020) proposes recursive statistical approach to solve wFM with convex constraint, however, the definition proposed do not explicitly preserve the equality constraint and it requires re-normalization of the solution. In contrast, our approach composes of the sparsemax transformation for convex Fréchet mixture of SPD operations with the following two advantages: I) It can preserve most of the important properties of softmax such as, it is simple to evaluate, cheaper to differentiate (Martins & Astudillo, 2016). 2) It is able to produce sparse distributions such that the best operation associated with each edge is more likely to make more dominant contributions to the supernet, and thus more optimal architecture can be derived (refer Figure 2(a)) and 84).
From Eq:(5-6), the mixing of operations between nodes is determined by the weighted combination of alpha's (a*) and the set of operations. This relaxation makes the search space continuous and therefore, architecture search can be achieved by learning set of alpha (a fa², A k E .N.)). To achieve our goal, we must simultaneously learn the contribution of several possible operation within all the mixed operations (w) and the corresponding architecture a. Consequently, for given w, we can find a and vice-versa resulting in the following bi-level optimization problem.
minimize E (w opt (a), a); subject to: w (a) = argmin El (w. a:)
The lower-level optimization E. Erain corresponds to the optimal weight variable learned for a given a i... wont (a) using a training loss. The upper-level optimization E solves for the variable a given the optimal w using validation loss. This li-le search method gives optimal mixture of multiple small architectures. To derive each node in the discrete architecture, we maintain top-k operations i.e, with the 1.th highest weight among all the candidate operations associated with all the previous nodes.
Bi-level Optimization: The bi-level optimization problem proposed in Fq:(7) is difficult to solve. Following Liu et al. (2018b) work, we approximate w (a) in the upper- optimization problem to skip inner-opiimization as follows:
Ey (w (a))(())))) V E. (w- V El (w, a), a)
In Appendis, we provide some comparison between Karcher flow and recursive geodesic mean method, A comprehensive study on this actually heyond the scope our paper.

---

Under review as a conference paper at ICLR 2021
Sigmeld
Sparsemuxx
Bdges
Openations
Openations
Operations
Neraa Collurudee RARAR
Redavid Collanadti RARAR
(a) Distribution of edge weights for operation selection
(b) Derived architecture
Figure 2: (a) Distribution of edge weights for operation selection using softmax, sigmoid, and sparsemax on Frechet mixture of SPD operations. (b) Derived sparsemax architecture by the proposed SPDNetNAS. Better sparsity leads to less skips and poolings compared those other NAS solutions shown in Appendix Fig.5.
Here, 7/ is the learning rate and V is the gradient operator. Note that the gradient based optimization for w must follow the geometry of SPD manifold to update the structured connection weight, and its corresponding SPD matrix data. Applying the chain rule to (q:(8) gives
first term
second
Boat (i 0.) 1) V. Prrai (10 a V i E (ú, (10 a)
where, w - V.(w V/ Errai (w) denotes the weight update on the SPD manifold for the forward model. V u. D symbolizes the Riemannian gradient and the retraction operator respectively. The second term in the Eq:(9) involves second order differentials with very high computational complexity, hence, using the finite approximation method the second term of Eq:(9) reduces to:
Va. Eiratn (w, a) V Elal (i0, ax) = (V. (V.,Eirain(ue aa) V. E.rein .2))
(10)
where, w (r + V. E (wē, a)) and 8 is small number set to 0.01 /Il V E (tē, a)l2 Though the structure of bi-level optimization the same as the DARTS Liu et al. (2018b), there some key differences. Firstly, the updates on the manifold-valuee kernel weights are constrained on manifolds, which ensures that the feature maps at every intermediate layer are SPDs. For concrete derivations on back-proe preckk-pppggtiion for SPD network layers. refer to Huang & Van Gool (2017) work. Secondly, the update on the aggregation weights of the involved SPD operations needs to satisfy an additional strict convex constraint, which is enforced as part of the optimization problem. The pseudo code of our method is outlined in Algorithm(I).
EXPERIMENTS AND RESULTS
To keep the experimental evaluation consistent with the previously proposed SPD networks (Huang & Van Gool, 2017; Brooks et al., 2019), we used RADAR (Chen et al., 2006), HDMO5 (Müller et al., 2007), and AFEW (Dhall et al., 2014) datasets. For SPDNeINAS, we first optimize the supernet on the training/validation sets, and then prune it with the best operation for each edge. Finally, we train the optimized architecture from scratch to document the results. For both these stages, we consider the same normal and reduction cells. A cell receives preprocessed inputs which is performed using fixed BiMap_2 to make the input of same initial dimension. All architectures are trained with a batch size of 30. Learning rate (1) for RADAR, HDM05, and AFEW dataset set to 0.025, 0.025 and 0.05 respectively. Besides, we conducted experiments where we select architecture using a random search path (SPDNeINAS (R)). to justify whether our search space with the introduced SPD operations can derive meaningful architectures. We refer to SPDNet (Huang & Van Gool, 2017), SPDNetBN (Brooks al., 2019), and ManifoldNet (Chakraborty et al., 2020) for comparison against handcrafted SPD networks. SPDNet and SPDNetBN are evaluated using their original implementations. We follow the video classification setup of (Chakraborty et al., 2020) to evaluate ManifoldNet on AFEW. It is non-trivial to adapt ManifoldNet to RADAR and HDM05, as ManifoldNet requires SPD features with multiple channels and both of the two datasets can hardly obtain them. For comparing against Euclidean NAS methods, used DARTS (Liu et al., 2018b) and FairDARTS (Chu et al., 2020) by treating SPD's logarithm maps as Euclidean data in their official implementation with default setup. We observed that using SPD's as input to Euclidean NAS algorithms degrades its performance.
a) Drone Recognition: For this task, we used the RADAR dataset from (Chen et al., 2006). The synthetic setting for this dataset is composed of radar signals, where each signal is split into windows
7

---

Under review as a conference paper at ICLR 2021
of length 20 resulting in a 20x20 covariance matrix for each window (one radar data point). The synthesized dataset consists of 1000 data points per class. Given 20 x 20 input covariance matrices, our reduction cell reduces them to 10 x 10 matrices followed by normal cell to provide complexity to our network. Following Brooks et al. (2019), we assign 50%. 25%, and 25% of the dataset for training, validation, and test set respectively. The Fuclidean NAS algorithms are evaluated on the euclidean map of the input. For direct SPD input the performance of darts(95.866 (566 and fairdarts (92.26%) are worse as expected. For this dataset, our algorithm takes 1 CPU day of search time to provide the SPD architecture. Training and validation take 9 CPU hours for 200 epochs'. Test results on this dataset are provided in Table (2) which clearly shows the benefit of our method. Statistical performance show that our NAS algorithm provides an efficient architecture with much fewer parameters (more than 140 times) than state-of-the-art Euclidean NAS on the SPD manifold valued data. The normal and reduction cells obtained on this dataset are shown in Fig. 2(b).
b) Action Recognition: For this task, we used the HDM05 dataset (Müller et al., 2007) which contains 130 action classes, yet, for consistency with previous work (Brooks et al., 2019), we used 117 class for performance comparison. This dataset has 3D coordinates of 31 joints per frame. Following the previous works (Harandi et al., 2017; Huang & Van Gool, 2017), we model an action for a sequence using 93 x 93 joint covariance matrix. The dataset has 2083 SPD matrices distributed among all 117 classes. Similar to the previous task, split the dataset into 50%, 25%, and 25% for training, validation, and testing. Here, our reduction cell is designed to reduce the matrices dimensions from 93 30 for legitimate comparison against Brooks et al. (2019). To search for the best architecture, we ran our algorithm for 50 epoch (3 CPU days). Figure 2(b) show the final cell architecture that got selected based on the validation performance. The optimal architecture is trained from scratch for 100 epochs which took approximately 16 CPU hours. The test accuracy achieved on this dataset is provided in Table (2). Statistics clearly show that our models despite being lighter performs better than the NAS models and the handcrafted SPDNets. The NAS models' inferior results show that the use of SPD layers for respecting SPD geometries is crucial for SPD data analysis.
Table 2: Performance comparison of our method against existing SPDNets and TraditionalNAS on drone and action recognition. SPDNeINAS (R): randomly select architecure from our search space, DARTS/FairDARTE accepts logarithm forms of SPDs. The search time of our method RADAR and HDMO5 noted to be I CPU days and CPU days respectively. And the scarch cost DARTS and FairDARTS on RADAR and HDMO5 are about GPU hours, #RADAR and #HDM05 show model parameter comparison on the respective datasct.
Datasct
DARTS
FarDARTS
SPDNet
SPDNetBN
SPDNGINAS (R)
SPDNetNAS
RADAR
98.21% 0,23
98.51 + 0.09
93.21% = 0.39
92.13% +: 0.77
95.49% + 0.08
97.75% + 0.30
IRADAR
2.6383
2.6614
0.0014
0.0018
0,0185
MB 69.87% +. 0.31
HDMOS
53.93% 1.42
47.71% 1,46
61 60%
1,35
1,15 65,20%
66.92% 0.72
#HDM0S
3.6800MB
5,1355 MB
0.1082 MB
0,1091 MB
1.0557 MB
L064MB MB
c) Emotion Recognition: We used AFEW dataset (Dhall al., 2014) to evaluate the transferability of our searched architecture for emotion recognition. This dataset has 1 1345 videos of facial expressions classified into 7 distinct classes. To train on the video frames directly, we stack all the handcrafted SPDNets and our searched SPDNet on top of a covolutional network Meng et al. (2019) with its official implementation. For ManifoldNet, we compute 64 x 64 spatial covariance matrix for each frame on the intermediate CNN features of 64 x 56 x 56 (channels, height, width). We follow the reported setup of Chakraborty et al. (2020) to first apply a single wFM layer with kernel size 5, stride and 8 channels, followed by three temporal wFM layers of kernel size 3 and stride 2, with the channels being 1, 4, 8 respectively. We closely follow the official implementation of ManifoldNet $ for the wFM layers and adapt the code to our specific task. Since SPDNet, SPDNetBN and our SPDNetNAS require single channel SPD matrix as input, we use the final 512 dimensional vector extracted from the covolutional network, project it using dense layer to a 100 dimensional feature vector and compute a 100 x 100 temporal covariance matrix. To study the transferability of our algorithm, we evaluate its searched architecture on RADAR and HDM05. In addition, we evaluate DARTS and FairDARTS directly on the video frames of AFEW. Table (3) reports the evaluations results. As we can observe, the transferred architectures can handle the new dataset quite convincingly, and their test accuracies are better than those of the existing SPDNets and the Euclidean NAS algorithms. In Appendix, we present results of competing methods and our searched models on the raw SPD features of AFEW.
*For details on the choice of CPU rather than GPU, see appendix http://ggiin...miiiibaaaaaaaaddd
8

---

Under review as a conference paper at ICLR 2021
Table Performance comparison our transferred architectures on AFEW against handcrafted SPDNets and Euclidean NAS. SPDNGUMSS/RADARHHDDDU architectures searched on RADAR and HDMOS respectively.
DARTS
FarDARTS
ManifoldNet
SPDNet
SPDNe1BN
SPDNeINAS (RADAR)
SPDNeINAS (HDM05)
26.88 %
22.31%
28.84%
34.06%
37.80%
40.80%
40.64%
d) Ablation study:
Lastly, we conducted some ablation study realize the effect of probability simplex con-
Table 4: Ablations study on different solutions sug
gested Fréchet mixture of SPD operations.
straint (sparsemax) on our suggested Fréchet mixture of SPD operations. Although in Fig.
Dutuset
soltmax
sigmoid
sparsemax
RADAR
96.47%
0.10
97.70%
0.23
97.75%
0.30
2(a) we show better probability weight distri- bution with sparsemax, Table(4) shows that it performs better empirically as well on both RADAR and HDM05 compared to the softmax and sigmoid cases. Therefore, SPD architectures derived using the sparsemax is observed to be better.
HDMOS
68.74% = 0.93
68.64% = 0.09
69.87% % + 0.31
e) Statistical comparison under same model complexity: We compare the statistical performance of our method against the other competing methods under similar model sizes. Table 5 show the results obtained on the RADAR dataset. One key point to note here is that when we increase the observe very severe degradation in the
number of parameters in SPDNet and SPDNetBN, performance accuracy mainly because the network starts overfitting rapidly. The performance degradation is far more severe for the HDM05 dataset with SPDNet (1.047MB) performing 0.7619% and SPDNetBN (1 (1.082MB) performing 1 1.45% and hence, is not reported in the table below. That further indicates the ability of SPDNeINAS generalize better and avoid overfitting despite the larger model size.
Similarly, we experimented on the AFEW dataset. To have a fair comparison against the related method like ManifoldNet, whose model size is about (76MB), we must reduce the model size accordingly. ManifoldNet model size is large mainly due to multiple final dense fully connected layers. Hence, to reduce the model size, we decreased the number of FC layers. The performance result with comparable model sizes on the AFEW dataset is shown in Table Again, we can infer that our SPDNeINAS achieves significant performance improvement over the others.
Table 5: Performance of our model against ManifoldNet. SPDNet and SPDNetBN with comparable model sizes the RADAR and AFEW datasets.
Dataset
Manifoldnet
SPDNet
SPDNeiBN
SPDNe
RADAR
73.066%
87.866%
97.75%
#RADAR
NA
0.01838 MB
0.01838 MB
0,01840 MB
AFEW
23.8%
34.06%
37.80%
40.64%
#AFEW
11,6476
11,2626 MB
11,2651
11,700
5 CONCLUSION AND FUTURE DIRECTION
In this work, we present neural architecture search problem of SPD manifold networks. To solve it, a SPD cell representation and corresponding candidate operation search space is introduced. A parameterized supernet search method is employed to explore the relaxed continuous SPD search space following a bi-level optimization problem with probability simplex constraint for effective SPD network design. The solution to our proposed problem using back-propagation is carefully crafted, so that, the weight updates follow the geometry of the SPD manifold. Quantitative results on the benchmark dataset show commendable performance gain over handcrafted SPD networks and Euclidean NAS algorithms. Additionally, we demonstrate that the learned SPD architecture is much lighter than other NAS based architecture and, it transferable to other datasets as well.
Our work provides an architecture search methodology for the scenarios where the acquired input data are SPD's, for example, diffusion tensor imaging for medical applications, drone recognition, etc. In addition, our method offers paradigm to automate the neural architecture design for the scenarios that require the second-order www.w Wang al. (2017); Engin et al. (2018); Wang al. (2019)). Accordingly, we encourage more future works to pursue these two directions. Also, it is fairly interesting to extend our proposed method to sequential manifold valued data (Zhen et al., 2019; Chakraborty et al., 2018).
representations/poolings for robust visual recognition (e.g..

---

Under review as a conference paper at ICLR 2021
REFERENCES
P-A Absil, Robert Mahony, and Rodolphe Sepulchre. Optimization algorithms on matrix manifolds. Princeton University Press, 2009.
Akshay Agrawal, Brandon Amos, Shane Barratt, Stephen Boyd, Steven Diamond, and Zico Kolter. Differentiable convex optimization layers. In Advances in neural information processing systems, pp. 9562-9574, 2019.
Karim Ahmed and Lorenzo Torresani. Connectivity learning in multi-branch networks. arXiv preprint arXiv: 170:::7090099900 795889 2017.
Bowen Baker, Otkrist Gupta, Ramesh Raskar, and Nikhil Naik. Accelerating neural architecture search using performance prediction. arXiv preprint arXiv: 10823, 2017.
Alexandre Barachant, Stéphane Bonnet, Marco Congedo, and Christian Jutten. Multiclass brain- computer interface classification by riemannian geometry. IEEE Transactions on Biomedical Engineering. (9(:9920 -288 2011.
Gabriel Bender.
Understanding and simplifying one-shot architecture search. 2019.
Rajendra Bhatia and John Holbrook. Riemannian geometry and matrix geometric means. Linear algebra and applications, 413(2-3):594-615 2006.
Silvere Bonnabel. Stochastic gradient descent on riemannian manifolds. IEEE Transactions on Automatic Control, 58(9):2217-2229) 2013.
Andrew Brock, Theodore Lim, James M Ritchie, and Nick Weston. Smash: one-shot model architecture search through hypernetworks. arXiv preprint arXiv: 10XX::/708.05344,, 2017.
Daniel Brooks, Olivier Schwander, Frédéric Barbaresco, Jean- Schneider, and Matthieu Cord. Riemannian batch normalization for spd neural networks. In Advances in Neural Information Processing Systems, pp. 15463-15474, 2019.
Han Cai, Tianyao Chen, Weinan Zhang, Yong Yu, and Jun Wang. Efficient architecture search by network transformation. In Thirty-Second AAAI conference on artificial intelligence, 2018.
Rudrasis Chakraborty. Manifoldnorm: Extending normalizations on riemannian manifolds. arXiv
preprinl arXiv:2003. 2020.
Rudrasis Chakraborty, Chun-Hao Yang, Xingjian Zhen, Monami Banerjee, Derek Archer, David Vaillancourt, Vikas Singh, and Baba Vemuri. A statistical recurrent model on the manifold of symmetric positive definite matrices. In Advances in Neural Information Processing Systems, pp. 8883-8894, 2018
Rudrasis Chakraborty, Jose Bouza, Jonathan Manton, and Baba Vemuri. Manifoldnet: A deep neural network for manifold-valued data with applications. IEEE Transactions Pattern Analysis and Machine Intelligence, 2020.
Victor C Chen, Fayin Li, S-S Ho, and Harry Wechsler. Micro-doppler effect in radar: phenomenon, model, and simulation study. IEEE Transactions Aerospace and electronic systems, 42(1):2-21, 2006.
Guang Cheng, Jeffrey Ho. Hesamoddin Salehian, and Baba C Vemuri. Recursive computation of the fréchet mean on non-pose non-positively curved riemannian manifolds with applications. In Riemannian Computing in Computer Vision, pp. 21-43. Springer, 2016.
Xiangxiang Chu, Bo Zhang, Jixiang Li, Qingyuan Li, and Ruijun Xu. Scarletnas: Bridging the gap between scalability and fairness in neural architecture search. arXiv preprint arXiv:1908. 1902200022 2019.
Xiangxiang Chu, Tianbao Zhou, Bo Zhang, and Jixiang Li. Fair DARTS: Eliminating Unfair Advantages in Differentiable Architecture Search. In 16th Europoean Conference On Computer Vision, 2020. URL https: /tttttp::////iieeeeeeeeee //arxiv...v///////aa//aaa/aaaa org/
hwwgg//////// abs 911 12126. 26......... pdf.
10

---

Under review as a conference paper at ICLR 2021
Kalyanmoy Deb, Amrit Pratap, Sameer Agarwal, and TAMT Meyarivan. A fast and elitist multiob- jective genetic algorithm: Nsga-ii. IEEE transactions on evolutionary computation, 6(2):182-197, 2002.
Abhinav Dhall, Roland Goecke, Jyoti Joshi, Karan Sikka, and Tom Gedeon. Emotion recognition in the wild challenge 2014: Baseline, data and protocol. In Proceedings of the 16th international conference on multimodal interaction, pp. 461-466, 2014.
Tingxing Dong, Azzam Haidar, Stanimire Tomov, and Jack Dongarra. Optimizing the svd bidiago- nalization process for a batch small matrices. In ICCS, pp. 1008-1018, 2017a
Zhen Dong, Su Jia, Chi Zhang, Mingtao Pei, and Yuwei Wu. Deep manifold learning of symmetric positive definite matrices with application to face recognition. In Thirty-First AAAI Conference on Artificial Intelligence, 2017b.
Thomas Elsken, Jan-Hendrik Metzen, and Frank Hutter. Simple and efficient architecture search for convolutional neural networks. arXiv preprint 11Xi://11..05266 2017.
Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. Neural architecture search: A survey. arXiv preprint (80.::1800005377 2018.
Melih Engin, Lei Wang, Luping Zhou, and Xinwang Liu. Deepkspd: Learning kernel-matrix-based spd representation for fine-grained image recognition. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 612-627, 2018.
Zhi Gao, Yuwei Wu, Mehrtash Harandi, and Yunde Jia. A robust distance measure for similarity- -base classification on the spd manifold. IEEE Transactions on Neural Networks and Learning Systems, 2019.
Mark Gates, Stanimire Tomov, and Jack Dongarra. Accelerating the svd two stage bidiagonal reduction and divide and conquer using gpus. Parallel Computing, 74:3-18, 2018
Xinyu Gong, Shiyu Chang, Yifan Jiang, and Zhangyang Wang. Autogan: Neural architecture search for generative adversarial networks. Proceedings of the IEEE International Conference on Computer Vision, pp.
2019.
Zichao Guo, Xiangyu Zhang, Haoyuan Mu, Wen Heng. Zechun Liu, Yichen Wei, and Jian Sun. Single path one-shot neural architecture search with uniform sampling. In European Conference on Computer Vision, pp. 540- Springer, 2020.
Mehrtash Harandi, Mathieu Salzmann, and Richard Hartley. Dimensionality reduction on spd manifolds: The emergence of geometry-aware methods. IEEE transactions on pattern analysis and machine intelligence, 40(1):48 ((1)):48-6)) 62, 2017.
Mehrtash T Harandi, Mathieu Salzmann, and Richard Hartley. From manifold to manifold: Geometry- aware dimensionality reduction for spd matrices. In European conference computer vision, pp. 17-32. Springer, 2014.
Zhiwu Huang and Luc Van Gool. A riemannian network for spd matrix learning. In Thirty-First AAAI Conference on Artificial Intelligence, 2017.
Zhiwu Huang. Ruiping Wang, Shiguang Shan, and Xilin Chen. Learning euclidean-to-riemmmian metric for point-to-set classification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1677-1684, 2014.
Zhiwu Huang, Ruiping Wang. Shiguang Shan, Xianqiu Li, and Xilin Chen. learning symmetric positive definite manifold with application image set classification. In International conference on machine learning, pp. 720-729. 2015.
metric
Kirthevasan Kandasamy, Willie Neiswanger, Jeff Schneider, Barnabas Poczos, and Eric P Xing. Neural architecture search with bayesian optimisation and optimal transport. In Advances in Neural Information Processing Systems, pp. 2016-2025, 2018
11

---

Under review as a conference paper at ICLR 2021
Hermann Karcher. Riemannian center of mass and mollifier smoothing. Communications on pure and applied mathematics, 30(5):509 541, 1977.
Suryansh Kumar. Jumping manifolds: Geometry aware dense non-rigid structure from motion. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 5346-5355, 2019.
Suryansh Kumar, Anoop Cherian, Yuchao Dai, and Hongdong Li. Scalable dense non-rigid structure- from-motion: A grassmannian perspective. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 254-263, 2018.
Mare Lackenby. Introductory chapter on riemannian manifolds. Notes, 2020.
Hanwen Liang, Shifeng Zhang, Jiacheng Sun, Xingqiu He, Weiran Huang, Kechen Zhuang, and Zhenguo Li. Darts+: Improved differentiable architecture search with early stopping. arXiv
preprint arXiv: 06035, 2019.
Chenxi Liu, Barret Zoph, Maxim Neumann, Jonathon Shlens, Wei Hua, Li-Jia Li, Li Fei-Fei, Alan Yuille, Jonathan Huang, and Kevin Murphy. Progressive neural architecture search. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 19-34, 2018a.
Chenxi Liu, Liang-Chia Chich Chen, Florian Schroff, Hartwig Adam, Wei Hua, Alan L Yuille, and Li Fei- Fei. Auto-deeplab: Hierarchical neural architecture search for semantic image segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 82-92, 2019.
Hanxiao Liu, Karen Simonyan, Oriol Vinyals, Chrisantha Fernando, and Koray Kavukcuoglu. Hi- erarchical representations for efficient architecture search. arXiv preprint 2017.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. arXiv
preprini arXiv::8066 1806.0905, 2018b.
Andre Martins and Ramon Astudillo. From softmax to sparsemax: A sparse model of attention and multi-label classification, In International Conference on Machine Learning, pp. 1614-1623, 2016.
Debin Meng, Xiaojiang Peng, Kai Wang, and Yu Qiao. Frame attention networks for facial expression recognition in videos. In 2019 IEEE International Conference on Image Processing (ICIP), pp. 3866-3870. IEEE, 2019.
Maher Moakher. A differential geometric approach the geometric mean of symmetric positive- definite matrices. SIAM Journal on Matrix Analysis and Applications, 26(3):735-747, 2005. 2005.
Meinard Müller, Tido Röder, Michael Clausen, Bernhard Eberhardt, Björn Krüger, and Andreas Weber. Documentation mocap database hdm05. 2007.
Niv Nayman, Asaf Noy, Tal Ridnik, Itamar Friedman, Rong Jin, and Lihi Zelnik. Xnas: Neural architecture search with expert advice. In Advances in Neural Information Processing Systems, pp. 1977-1987, 2019.
Renato Negrinho and Geoff Gordon. Deeparchitect: Automatically designing and training deep architectures. arXiv preprint /rXiv: 2017.
Xavier Pennec. Manifold-valur valued image processing with spd matrices. Riemannian Geometric Statistics in Medical Image Analysis, pp. 75-134. Elsevier, 2020.
Xavier Pennec, Pierre Fillard, and Nicholas Ayache. A riemannian framework for tensor computing. International Journal of computer vision, 66(1):41-66, 2006.
Hicu Pham, Melody Guan, Barret Zoph, Quoe Le. and Jeff Dean. Efficient neural architecture search via parameter sharing. arXiv preprint arXiv: / 008.002.3326 2018.
Esteban Real, Alok Aggarwal, Yanping Huang, and Quoe V Le. Regularized evolution for image classifier architecture search. In Proceedings of the aaai conference on artificial intelligence. volume 33, pp. 4780-4789, 2019.
12

---

Under review as a conference paper at ICLR 2021
Shreyas Saxena and Jakob Verbeek. Convolutional neural fabrics. In Advances in Neural Information Processing Systems, pp. 4053-4061, 2016.
Richard Shin, Charles Packer, and Dawn Song. Differentiable neural network architecture search. 2018.
Yuan Tian, Qin Wang, Zhiwu Huang, Wen Li, Dengxin Dai, Minghao Yang, Jun Wang, and Olga Fink. Off-policy reinforcement learning for efficient and effective gan architecture search. In Proceedings of the European Conference on Computer Vision, 2020.
Oncel Tuzel, Fatih Porikli, and Peter Meer. Region covariance: A fast descriptor for detection and classification. In European conference on computer vision, pp- 019-5599-90 600. Springer, 2006.
Oncel Tuzel, Fatih Porikli, and Peter Meer. Pedestrian detection via classification on riemannian manifolds. IEEE transactions on pattern analysis and machine intelligence, 30(10):1133-1727, 2008.
Tom Veniat and Ludovic Denoyer. Learning time/memory-efficien deep architectures with bud- geted super networks. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp. 3492-3500, 2018.
Qilong Wang. Peihua Li, and Lei Zhang. G2denet: Global gaussian distribution embedding network and its upplication to visual recognition. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 2730-2739, 2017.
Qilong Wang, Jiangtao Xie, Wangmeng Zuo, Lei Zhang, and Peihua Li. Deep cnns meet global covariance pooling: Better representation and generalization. arXiv preprint arXiv:1904.0683 2019.
Ruiping Wang, Huimin Guo, Larry S Davis, and Qionghai Dai. Covariance discriminative learning: A natural and efficient approach to image set classification. In 2012 IEEE Conference on Computer Vision and Pattern Recognition, pp. 2496-2503. IEEE, 2012.
Bichen Wu, Xiaoliang Dai, Peizhao Zhang, Yanghan Wang, Fei Sun, Yiming Wu, Yuandong Tian, Peter Vajda, Yangqing Jia, and Kurt Keutzer. Fbnet: Hardware-aware efficient convnet design via differentiable neural architecture search. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 10734-10742, 2019.
Yan Wu, Aoming Liu, Zhiwu Huang, Siwei Zhang, and Lue Van Gool. Neural architecture search as sparse supernet. arXiv preprint arXiv:2007.161 2020.
X. Zhang, Z. Huang, N. Wang, S. XIANG, and C. Pan. You only search once: Single shot neural architecture search via direct sparse optimization. IEEE Transactions Pattern Analysis and Machine Intelligence, pp. 1-1, 2020.
Xingjian Zhen, Rudrasis Chakraborty, Nicholas Vogt. Barbara B Bendlin, and Vikas Singh. Dilated convolutional neural networks for sequential manifold-valued data. In Proceedings of the IEEE International Conference Computer Vision, pp. 10621-10631, 2019.
Xiawu Zheng, Rongrong Ji, Lang Tang, Baochang Zhang, Jianzhuang Liu, and Qi Tian. Multino- mial distribution learning for effective neural architecture search. In Proceedings of the IEEE International Conference Computer Vision, pp. 1304-1313, 2019.
H Zhou, M Yang, J Wang, and W Pan. Bayesnas: A bayesian approach for neural architecture search. In 36th International Conference on Machine Learning, ICMI. 2019, volume 97, pp. 7603-7613, Proceedings of Machine Learning Research (PMLR), 2019.
Barret Zoph and Quoe V Le. Neural architecture search with reinforcement learning, arXiv preprint arXiv:161 1.0157877 2016.
Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V Le. Learning transferable architectures for scalable image recognition. In Proceedings of the IEEE conference on computer vision and pattem recognition, pp. 8697-8710, 2018
13

---

Under review as a conference paper at ICLR 2021
A ADDITIONAL EXPERIMENTAL ANALYSIS
A.1 EFFECT OF MODIPYING PREPROCESSING LAYERS FOR MULTIPLE DIMENSIONALITY REDUCTION
Unlike Huang & Van Gool (2017) work on the SPD network, where multiple transformation matrices are applied at multiple layers to reduce the dimension of the input data, our reduction cell presented in the main paper one step. For example: For HDM05 dataset (Müller al., 2007), the author's of SPDNet (Huang & Van Gool, 2017) apply 93 x 70, 70 x 50, 50 x 30, transformation matrices to reduce the dimension of the input matrix, on the contrary, we reduce the dimension in one step from 93 to 30 which is inline with Brooks et al. (2019) work.
To study the behaviour of our method under multiple dimesionality reduction pipeline on HDM05, we use the preprocessing layers to perform dimensionality reduction. To be precise, we consider a preprocessing step to reduce the dimension from 93 70 to 50 and then, a reduction cell that reduced the dimension from 50 to 24. This modification has the advantage that it reduces the search time from CPU days to 2.5 CPU days, and in addition, provides a performance gain (see Table (6)). The normal and the reduction cells for the multiple dimension reduction are shown in Figure (3).
Table 6: Results of modifying prepracessing layers for multiple dimentionality reduction on HDM05
Prepracess dim reduction
Coll dim reduction
SPDNetNAS
Search time
NA
93 - 30
68.74% 0.93
3 CPU days
93-000-50
50-+24
69.41 +
CPU days
AvgPooling2 roluved
(k-2)
MusPooling
WagselPorieg somal
ArgPoollngg2 redused
c_(k)
01:3
Skguaamal
c_ 1)
AugPooling2 reduned
01+1)
(a) Normal cell
(b) Reduction cell
Figure 3: (a)-(b) Normal cell and Reduction cell for multiple dimensionality reduction respectively
A.2 EFFECT OF ADDING NODES TO THE CELL
Experiments presented in the main paper consists of N = 5 nodes per cell which includes two input nodes, one output node, and two intermediate nodes. To do further analysis of our design choice, we added nodes to the cell. Such analysis can help us study the critical behaviour of our cell design i.e, whether adding an intermediate nodes can improve the performance or not?, and how it affects the computational complexity of our algorithm? To perform this experimental analysis, we used HDM05 dataset (Müller al., 2007). We added one extra intermediate node (N 6) the cell design. We observe that we converge towards an architecture design that is very much similar in terms of operations (see Figure 4). The evaluation results shown in Table (7) help us deduce that adding more intermediate nodes increases the number of channels for output node, subsequently leading to increased complexity and almost double the computation time.
Table 7: Results for multi-node experiments HDM05
Number nodes
SPDNetNAS
Search
5
68.74%
0.93
CPU days
6
67.96% + 0.67
CPU days
A.3 EFFECT OF ADDING MULTIPLE CELLS
In our paper we stack normal cell over reduction cell for all the experiments. For more extensive analysis of the proposed method, conducted training experiments by stacking multiple cells which
14

---

Under review as a conference paper at ICLR 2021
Step Liplacii
Sepp redase
StaPodig
Normal cell
(b) Reduction cell
Figure 4: (a)-(b) Optimal Normal cell and Reduction cell with 6 nodes the HDM05 dataset
in-line with the experiments conducted by Liu et al. (2018b). We then transfer the optimized architectures from the singe cell search directly to the multi-cell architectures for training. Hence, the search time for all our experiments is same as for a single cell search i.c. 3 CPU days. Results for this in the table shows the performance for single cell
experiment are provided in Table 8. The first model, while the second and third rows show the performance with multi-cell stacking. Remarkably. by stacking multiple cells our proposed SPDNeiNAS outperforms SPDNetBN Brooks et al. (2019) by a large margin (about 8%, ie., about 12% for the relative improvement).
Table 8: Results for multiple cell search and training experiments on HDM05: reduction corresponds to reduction cell and normal corresponds to the normal cell.
type sequence
SPDNeINAS
Search Time
single cell
93
46
sedection-normal
68.74% + 0.93
CPU days
mulii-cell
93 > 46
normal normal-reductie caamaallrddctionnnoneaee
71.48% + 0.42
3 days
mulii-cll
93 +
16
+ 22
73.59 0.33
days
A.4 AFEW PERFORMANCE COMPARISON ON RAW SPD FEATURES
In addition to the evaluation on CNN features in the major paper, also use the raw SPD features (extracted from gray video frames) from Huang & Van Gool (2017); Brooks et al. (2019) to compare the competing methods. To be specific, each frame is normalized to 20 20 and then represent each video using a 400 x 400 covariance matrix (Wang et al., 2012: Huang & Van Gool, 2017). Table summarizes the results. As convincingly. The test accuracy is comparable to the best SPD network method for RADAR model transfer. For HDM05 model transfer, the test accuracy is much better than the existing SPD networks.
see, the transferred architecture can handle the new dataset quite
Table 9: Performance of transferred SPDNetNAS Network architecture comparison existing SPD Networks on the AFEW dataset Dhall et al. (2014). RAND symbolizes random architecture from our search space. DARTS/FairDARTS: accepts the logarithms of raw SPDs, and the other competing methods receive the SPD
features.
DARTS
FarDARTS
ManioolNeet
SPDNet
SPDNeIBN
Ous<RAADAR)
(HDMUS)
25.87%
23.34%
23.98%
33.17%
35.22%
32.88%
35.31 %
38.01% %
A.5 DERIVED CELL ARCHITECTURE USING SIGMOID ON FRÉCHET MIXTURE OF SPD OPERATION
Figure 5(a) and Figure 5(b) show the cell architecture obtained using the softmax and sigmoid respectively on the Fréchet mixture of SPD operation. It can be observed that it has relatively more skip and pooling operation than sparsemax ((sce Figure 2(b))). In contrast to softmax and sigmoid, the SPD cell obtained using sparsemax is composed of more convolution type operation in the architecture, which in fact is important for better representation of the data.
15

---

Under review as a conference paper at ICLR 2021
Sapp peree
WeMiig mikked
Suy
Peeling
SAV
(a) Normal Call Insenndd en HID0005
(b) Reduued Call Iearned on HDDMS
Ca) Nommal Call learned HIMMS
(b) Redacced loourned en HIDMES
MasTwekk
(O lear an RADAR
(d) learned RADAR
Nermal Call karmnd en RADAIR
Redaced Call kaamndd on RADAR
(a) Derived cell architecture using softmax
(b) Derived cell architecture using sigmoid
Figure 5: (a), (h) Derived architecture by using softmax and sigmoid on the Fréchet mixture of SPD operations. These the normal cell and reduced cell obtained on RADAR and HDMOS dataset.
A.6 COMPARISON BETWEEN KARCHER FLOW AND RECURSIVE APPORACH FOR WEIGHTED FRECHET MEAN
The proposed NAS algorithm is based on Fréchet Mean computations. From the weighted mixture of operations between nodes to the derivation of intermediate nodes, both compute the Fréchet mean of set of points on the SPD manifold. It is well known that there is no closed form solution when the number of input samples is bigger than 2 (Brooks et al., 2019). We can only compute an approximation using the famous Karcher flow algorithm (Brooks et al., 2019) or recursive geodesic mean (Chakraborty et al., 2020). For comparison, we replace our used Karcher flow algorithm with the recursive approach under our SPDNeINAS framework. Table 10 sumarizes the comparison between these two algorithms. We observe considerable decrease in accuracy for both the training and test set when using the recursive methods, showing that the Karcher flow algorithm favors our proposed algorithm more.
Table 10: Test performance the proposed SPDNeINAS using the Karcher flow algorithm and recursive algorithm to compute Fréchet means.
Dataset Method
Karcher flow
Recursive algoritbm
RADAR
96.47 % +: 0.08
68. 13% +: 0.64
HDMUS
68.74% 0.93
56.85% 0.17
A.7 CONVERGENCE CURVE ANALYSIS
Figure 6(a) shows the validation curve which almost saturates at 200 epoch demonstrating the stability of our training process. First column bar of Figure (6(b) show the test accuracy comparison when only 10% of the data is used for training our architecture which demonstrate the effectiveness of our algorithm. Further, we study this for our SPDNetNAS architecture by taking 10%, 33%, 80% of the data for training. Figure 6(b)) clealy show our superiority of SPDNeINAS algorithm than handcrafted SPD networks.
Figure (7(a) and Figure (7(b) show the convergence curve of our loss function on the RADAR and HDM05 datasets respectively. For the RADAR dataset the validation and training losses follow similar trend and converges 200 epochs. For the HDM05 dataset, we observe the training curve plateaus after 60 epochs, where as the validation curve takes 100 epochs to provide a stable noticed a reasonable gap between the training loss and validation loss
performance. Additionally, for the HDM05 dataset (Müller et al., 2007). A similar pattern of convergence gap between validation loss and training loss has been observed by Huang & Van Gool (2017) work.
A.8 WHY WE PREFERRED TO SIMULATE OUR EXPERIMENTS ON CPU RATHER THAN GPU?
When dealing with SPD matrices, need to carry out complex computations. These computations are performed to make sure that our transformed representation and corresponding operations respect the underlying manifold structure. In our study, we analyzed SPD matrices with the Affine Invariant
16

---

Under review as a conference paper at ICLR 2021
Validation accuracy synthetic RADAR dataset
Test Porformance on synthetic RADAR datasct
93
M
95
% Validaton Aucurae)
50
Accunacy
8:5
85
80-
SPDNet
SPDNet
SPUNetiN
75
SPDNeiBN
SPDNeINAS (Ores)
SPDNetNAS (Ours)
70
70
150
250
750
1000
1250
1500
Number spoch
training sumples
(a)
(b)
Figure 6: (a) Validation accuracy of our method in comparison to the SPDNet and SPDNeIBN on RADAR dataset. Clearly, SPDNeINAS algorithm show steeper validation accuracy curve. (b) Test accuracy 10%. 33%, 80%. 100% total data sample. It can observed that our method exhibit superior performance.
convergence of SPDNeINAS on RADAR
Loss convergence of SPDNeNNAS on HDMO5
0.8
Validation Loss
Validition Loss
Training Loss
Training Loss
0.6
Lass
0.4-
50
100
200
0
20
60
80
100
epoch
epach
(a)
(b)
Figure 7: (a) Loss function curve showing the values over 200 epochs for the RADAR dataset (b) Loss function curve showing the values over 100 epochs the HDM05 dataset.
Riemannian Metrie (AIRM), this induces operations heavily dependent on singular value decom- position (SVD) or eigendecomposition (EIG). Both decompositions suffer from weak support on GPU platforms. Hence, our training did not benefit from GPU acceleration and we decided to train on CPU. As future work, we aim to speedup our implementation on GPU by optimizing the SVD Householder bi-diagonalization process as studied in some existing works like Dong et al. (2017a); Gates et al. (2018).
B DETAILED DESCRIPTION OF OUR PROPOSED OPERATIONS
In this section, we describe some of the major operations defined in the main paper from intuitive point view. We particularly focus on some of the new operations that are defined for the input SPDs, i..., the Weighted Riemannian Pooling, the Average/Max Pooling, the Skip Reduced operation and the Mixture of Operations.
B. WEIGHTED RIEMANNIAN POOLING
Figure provides an intuition behind the Weighted Riemannian Pooling operation. Here, w.ll, w 1, etc., corresponds to the set of normalized weights for each channel (shown as two blue channels). The next channel shown in orange, is then computed as weighted Fréchet mean over these two input channels. This procedure is repeated to achieve the desired number of output channels (here two), and finally all the output channels are concatenated. The weights are learnt as a part of the optimization procedure ensuring the explicit convex constraint is imposed.
17

---

Under review as a conference paper at ICLR 2021
Weighted Frechet Mean
w11
WZ2
Concat
Figure 8: Weighted Riemannian Pooling: Performs multiple weighted Fréchet means on the channels of the input SPD
B.2 AVERAGE AND MAX POOLING
In Figure we show our average and max pooling operations. We first perform LogEig map on the SPD matrices to project them to the Euclidean space. Next, perform average and max pooling these Euclidean matrices similar to classical convolutional neural networks. We further perform an ExpEig map to project the Euclidean matrices back the SPD manifold. The diagram shown in Figure 9 is inspired by Huang & Van Gool (2017) work. The kernel size of AveragePooling reduced and MaxPooling reduced is set 4 for all experiments according the specific dimensionality reduction factors.
Avg/Max
ExpEig
LogEig
Pooling
SPD
Euclidean
SPD
Figure 9: Avg/Max Pooling: Maps the SPD matrix to Euclidean space using LogEig mapping, does avg/max pooling followed by ExpEig map
B.3 SKIP REDUCED
Following Liu et al. (2018b), defined an analogous of Skip operation on a single channel for the reduced cell (Figure 10). We start by using a BiMap layer equivalent to Conv in Liu et al. (2018b), to map the input channel an SPD whose space dimension is half of the input dimension. We further perform an SVD decomposition on the two SPDs followed by concatenating the Us, Vs and Ds obtained from SVD block diagonal matrices. Finally, compute the output by multiplying the block diagonal U, V and D computed before.
BiMap Layer
Block Diagonal Computation
Reassemble
BiMap1
Output-U_op out D_out V_ouu
[U1_D1,V1]
U. Loottdiag(U1 .2)
wooouddag(DD.... D2)
[22]D2.V]
Lot=diag(V1.
BiMap2
Figure 10: Skip Maps input two smaller matrices using BiMaps, followed by SVD decomposition on them then computes the output using a block diagonal form of U's V's
B.4 MIXED OPERATION ON SPDS
In Figure 11 we provide an intuition of the mixed operation we have proposed in the main paper. We consider very simple base case of three nodes, two input nodes (1 and 2) and output
18

---

Under review as a conference paper at ICLR 2021
node (node 3). The goal is compute the output node 3 from input nodes and 2. We perform candidate set of operations on the input node, which correspond to edges between the nodes (here two for simplicity). Each operation has weight az where corresponds the node index and the candidate operation identifier. In Figure below and j E (1,2) and 01 = {01.1, (1,2) part of the bi-level optimization procedure proposed in
Q2 (02,1, 02.2) a's optimized the main paper. Using these alpha's, ss we perform a channel-wise weighted Fréchet mean (wFM) as depicted the figure below. This effectively corresponds Note that the alpha's s corresponding to all channels of single operation are assumed to be the same. Once the weighted Fréchet means have been computed for nodes 1 and 2, we perform channel-wise concatenation on the outputs of the two nodes, effectively doubling the number of channels in node 3.
mixture of the candidate operations.
opl
wFMa,
op2
wFMa
CL
Concat
opl
WFMa,
op2
wFM0,
as.
Figure 11: Detailed overview of mixed operations. We simplify the example taking 3 nodes (two input nodes and one output node) and candidate operations. Input nodes have two channels (SPD matrices), we perform channelwise weighted Fréchet mean between the result of cach operation (edge) where weights a's are optimized during bi-level architecture search optimization. Output node is formed by concatenating both mixed operation outputs, resulting in four channel node.
C DIFFERENTIABLE CONVEX LAYER FOR SPARSEMAX OPTIMIZATION
import evxpy as cp from suxpylayers.tore import Cvxpylayer
def sparsemax__onnex_layer (x, n):
w_ cp. Variabllel
x_ Parantttte
define the objective and constraint objective cp. (cp. (ep, multiply (w_, x)))
10
constraint [cp. sum sw_] 1.0, 0.0 w., w_<=1.0]
11
12
opt_problen = ep. P.poProeem (objective, constraint)
layer = CvxpyLayer (opt_problem, parameters- variables-(w_) Layer (x)
15
return
Listing 1: Function to solve the sparsemax constraint optimization
19


Reviews:
[{"review_id": "yZ2Pz4yTPAR", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces a neural architecture search method for SPD inputs, which extends DARTs from Euclidean space to SPD manifold by defining several SPD cells. Each SPD cell consists of some operations in search space of SPDNets and DARTs. The experiments are conducted on three datasets.\n\nStrengths:\n+: The idea of neural architecture search for SPD inputs seems interesting.\n+: The proposed SPDNetNAS shows superiority to hand-crafted SPDNets/ManifoldNet and Euclidean NAS (i.e., DARTS and FairDARTS).\n+: The paper is clear written.\n\nWeaknesses:\n-: The authors would better carefully reconsider motivation and main contribution of this paper. The current work looks like combination of SPDNets and DARTs (with some minor modifications). Then, the first attempt for NAS problem of SPD manifold networks is not an appropriate motivation for such a combination. The authors should clarify why it is necessary to search neural architecture for SPD inputs. In another words, what are drawbacks of traditional SPDNets? \n\nFor Euclidean NAS, the searched architectures aim to have lower model complexity than hand-crafted architectures, while having comparable or higher accuracies. The proposed SPDNetNAS achieves higher accuracies than hand-crafted counterparts, but has 10x times of parameters number (as shown in Table 2). Therefore, it is hard to say that performance gains come from neural architecture search or parameter increasing. Meanwhile, it is not fair for comparison of SPDNetNAS with Euclidean NAS in terms of parameters number or comparison of SPDNetNAS with SPDNets only in terms of performance. \n\n-: The authors mainly compare with hand-crafted SPDNet and Euclidean NAS. [r1] presents a recurrent model for SPD inputs, which can handle sequential (or temporal) data, e.g., action and emotion recognition. Therefore, what are merits of the proposed SPDNetNAS over [r1] for handling sequential (or temporal) data, except architecture design vs architecture search?\n[r1] A Statistical Recurrent Model on the Manifold of Symmetric Positive Definite Matrices. NIPS, 2018.\n\n-: I wonder that why a section of FURTHER STUDY is located in appendix? Could it be integrated into section of Conclusion?\n\n-: It is well known that two-level optimization algorithm for NAS will lead optimization gap, e.g., overfitting or model collapsing. What are strategies used to solve this issue, e.g., early stopping? If source code is not released or no more details are given, the results will be very hard to reproduce.  \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "cf2wYGadh6u", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "1. \"In recent years, plenty of research work has been published in the area of NAS.\" needs citations.\n2. What is \"best of both these fields (NAS, SPDNets) \" in page 2?\n3. The definition of SPD matrix is wrong, this definition is for positive-definite matrix, not for SPD.\n4. In Eq. (1), log should be matrix logarithm, and need to be mentioned.\n5. In page 3, \"There are other efficient methods to compute distance between two points on the SPD manifold\" needs to be backed by references. \n6. \" Other property of the Riemannian manifoldof our interest is local diffeomorphism of geodesics which is a one-to-one mapping from the pointon the tangent space of the manifold to the manifold\", this is not true, geodesic is a function of two points (start and end) and time point (say between 0 and 1), so it can not be a one-one mapping between tngent space to manifold. Authors here meant to say exponential map as defined in Eq. (2). \n7. X in Eq. (2) is not the \"reference point\", it should be the base point for exponential map (see Chavel's book).\n8. The inverse of exp map is not defined everywhere, as exp is LOCAL diffeo., please clarify.\n9. Before Eq. (3), sometime->sometimes\n10. \"This definition is trivially extended tocompute the weighted Riemannian Barycenter\" not true, the authors should mention existence and uniqueness of wFM, for example, even if FM exists, wFM may not for some choice of w.\n11. What is \"valid SPD\" in Bimap layer, contrary to \"invalid SPD\"!\n12. what is P in Eq. (4)? I think it is parallel transport but needs to be defined.\n13. \\epsilon in ReEig layer should be >0.\n14. In logEig, essentially you are mapping SPD to space of symmetric matrices (which is \"flat\"), then why not use Inverse Exp. map? what is the necessity to define logEig map additionally?\n15. Same goes for expEig map, what is need given that one can use Exp. map!\n16. In weighted pooling layer, why karcher flow is \"simple\" and recursive method is not? Please clarify.\n17. \" such that the mixture of all operations still reside on SPD manifolds\", here the term mixture is ambiguous!\n18. Given that the operations are already defined, definition of SPD cell is NOT novel!\n19. The additional search space operations are all trivial extensions of basic operations, which are not the contribution, hence I think the authors need to tone down \"Most of these operations are not fully explored for SPD networks\" \n20. In 3.2, what the authors meant by \"optimize the over parameterized supernet\"?\n21. In Algorithm 1, do the authors mean Riemannian gradient descent (Absil et al.)?\n22. What the authors meant by \"Note that the gradient based optimizationforwmust follow the geometry of SPD manifold to update the structured connection weight, and itscorresponding SPD matrix data\"? Do the authors mean they need to project the Euclidean gradient found in Eq. (8) to get tangent vectors?\n23. The experimental setup is weak, e.g., SPDNet and ManifoldNet need to be compared with same model complexity, otherwise the comparison is not fair! ", "extra_scores": {}, "extra_text": {}}, {"review_id": "HsQLJQq6FtC", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper considers a generalization of convolutional neural networks (CNNs) to manifold-valued data such as Symmetric Positive Definite (SPD). This paper proposes a neural architecture search problem of SPD manifold networks. A SPD cell representation and corresponding candidate operation search space is introduced. They demonstrate on drone, action and emotion recognition datasets that their method is performed well compared to SPD approaches.\n\n- It is not clear how the computational graph preserves the geometric structure of SPD manifold.\n- It is unclear how to obtain the weights of Frechet Mean w_i of wFM (eq. 3) by backpropagation. \n- Relatively thorough experimental valuations: using 3 datasets comparing with sufficient number of prior\napproaches. However, the comparisons of the experiments results are limited to the SPD methods.  It is hard to understand how the generalization of convolutional neural networks (CNNs) to Symmetric Positive Definite (SPD) presented in this paper helps to improve CNN.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "4-ezZrNk7OR", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The work focus on finding better SPD Manifold Networks from the view of neural architecture search. They model the network parameterized by a set of architecture parameter and take the problem as a bi-level optimization problem, which is similar with DARTS way. I think the most contribution of this paper is the attempt to apply NAS on a new domain.\n\n1.  The authors introduce a lot of concepts about SPD Network. But they are not meaningful in this paper since the authors just consider operations as the node. From the graph view, SPD Network can be seen as a neural architecture directly even though the operation type is different.\n\n2. My main concern is novelty problem.  The authors also treat SPD Network as a graph in Figure 1 such that the DARTS method can be applied directly. As for Algorithm 1, they also use two types parameters: architecture weights $\\alpha$  and operation weight $w$. Then bi-level optimization problem is solved using the same method with DARTS.\n\n ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper tries to find better semi-positive definite (SPD) manifold networks using neural architecture search. However, as pointed out by the reviewers, the paper has a few weaknesses: (a) it lacks in novelty, (b) it lacks in experiments that are mentioned in the SOTA papers, (c) the experiments should be performed with the same model complexity for fairness.


Decision: Reject