Neural Architecture Search of SPD Manifold Networks

1 INTRODUCTION . Designing a favorable neural network architecture for a given application requires a lot of time , effort , and domain expertise . To mitigate this issue , researchers in the recent years have started developing algorithms to automate the design process of neural network architectures ( Zoph & Le , 2016 ; Zoph et al. , 2018 ; Liu et al. , 2017 ; 2018a ; Real et al. , 2019 ; Liu et al. , 2018b ; Tian et al. , 2020 ) . Although these neural architecture search ( NAS ) algorithms have shown great potential to provide an optimal architecture for a given application , it is limited to handle architectures with Euclidean operations and representation . To deal with non-euclidean data representation and corresponding set of operations , researchers have barely proposed any NAS algorithms —to the best of our knowledge . It is well-known that manifold-valued data representation such as symmetric positive definite ( SPD ) matrices have shown overwhelming accomplishments in many real-world applications such as pedestrian detection ( Tuzel et al. , 2006 ; 2008 ) , magnetic resonance imaging analysis ( Pennec et al. , 2006 ) , action recognition ( Harandi et al. , 2014 ) , face recognition ( Huang et al. , 2014 ; 2015 ) , braincomputer interfaces ( Barachant et al. , 2011 ) , structure from motion ( Kumar et al. , 2018 ; Kumar , 2019 ) , etc . Also , in applications like diffusion tensor imaging of the brain , drone imaging , samples are collected directly as SPD ’ s . As a result , neural network usage based on Euclidean data representation becomes inefficient for those applications . Consequently , this has led to the development of the SPD neural network ( SPDNet ) architectures for further improvements in these areas of research ( Huang & Van Gool , 2017 ; Brooks et al. , 2019 ) . However , these architectures are handcrafted , so the operations or the parameters defined for these networks generally change as per the application . This motivated us to propose a new NAS problem of SPD manifold networks . A solution to this problem can reduce unwanted efforts in SPDNet design . Compared to the traditional NAS problem , our NAS problem requires a new definition of computation cell and proposal for diverse SPD candidate operation set . In particular , we model the basic architecture cell with a specific directed acyclic graph ( DAG ) , where each node is a latent SPD representation , and each edge corresponds to a SPD candidate operation . Here , the intermediate transformations between nodes respect the geometry of the SPD manifolds . For solving the suggested NAS problem , we exploit a supernet search strategy which models the architecture search problem as a one-shot training process of a supernet that comprises of a mixture of SPD neural architectures . The supernet modeling enables us to perform a differential architecture search on a continuous relaxation of SPD neural architecture search space , and therefore , can be solved using a gradient descent approach . Our evaluation validates that the proposed method can build a reliable SPD network from scratch . We show the results of our method on benchmark datasets that clearly show results better than handcrafted SPDNet . Our work makes the following contributions : • We introduce a NAS problem of SPD manifold networks that opens up a new direction of research in automated machine learning and SPD manifold learning . Based on a supernet modeling , we propose a novel differentiable NAS algorithm for SPD neural architecture search . Concretely , we exploit a sparsemax-based Fréchet mixture of SPD operations to introduce sparsity that is essential for an effective diffentiable search , and bi-level optimization with manifold-based update and convexity-based update to jointly optimize architecture parameters and network kernel weights . • Besides well-studied operations from exiting SPDNets ( Huang & Van Gool , 2017 ; Brooks et al. , 2019 ; Chakraborty et al. , 2020 ) , we follow Liu et al . ( 2018b ) to further introduce some new SPD layers , i.e. , skip connection , none operation , max pooling and averaging pooling . Our introduced additional set of SPD operations make the search space more diverse for the neural architecture search algorithm to obtain more generalized SPD neural network architectures . • Evaluation on three benchmark datasets shows that our searched SPD neural architectures can outperform the existing handcrafted SPDNets ( Huang & Van Gool , 2017 ; Brooks et al. , 2019 ; Chakraborty et al. , 2020 ) and the state-of-the-art NAS methods ( Liu et al. , 2018b ; Chu et al. , 2020 ) . Notably , our searched architecture is more than 3 times lighter than those searched by the traditional NAS algorithms . 2 BACKGROUND . In recent years , plenty of research work has been published in the area of NAS ( Gong et al. , 2019 ; Liu et al. , 2019 ; Nayman et al. , 2019 ; Guo et al. , 2020 ) . This is probably due to the success of deep learning for several applications which has eventually led to the automation of neural architecture design . Also , improvements in the processing capabilities of machines has influenced the researchers to work out this computationally expensive yet an important problem . Computational cost for some of the well-known NAS algorithms is in thousands of GPU days which has resulted in the development of several computationally efficient methods ( Zoph et al. , 2018 ; Real et al. , 2019 ; Liu et al. , 2018a ; 2017 ; Baker et al. , 2017 ; Brock et al. , 2017 ; Bender , 2019 ; Elsken et al. , 2017 ; Cai et al. , 2018 ; Pham et al. , 2018 ; Negrinho & Gordon , 2017 ; Kandasamy et al. , 2018 ; Chu et al. , 2020 ) . In this work , we propose a new NAS problem of SPD networks . We solve this problem using a supernet modeling methodology with a one-shot differentiable training process of an overparameterized supernet . Our modeling is driven by the recent progress in supernet methodology . Supernet methodology has shown a great potential than other NAS methodologies in terms of search efficiency . Since our work is directed towards solving a new NAS problem , we confine our discussion to the work that have greatly influenced our method i.e. , one-shot NAS methods and SPD networks . To the best of our knowledge , there are mainly two types of one-shot NAS methods based on the architecture modeling ( Elsken et al. , 2018 ) ( a ) parameterized architecture ( Liu et al. , 2018b ; Zheng et al. , 2019 ; Wu et al. , 2019 ; Chu et al. , 2020 ) , and ( b ) sampled architecture ( Deb et al. , 2002 ; Chu et al. , 2019 ) . In this paper , we adhere to the parametric modeling due to its promising results on conventional neural architectures . A majority of the previous work on NAS with continuous search space fine-tunes the explicit feature of specific architectures ( Saxena & Verbeek , 2016 ; Veniat & Denoyer , 2018 ; Ahmed & Torresani , 2017 ; Shin et al. , 2018 ) . On the contrary , Liu et al . ( 2018b ) ; Liang et al . ( 2019 ) ; Zhou et al . ( 2019 ) ; Zhang et al . ( 2020 ) ; Wu et al . ( 2020 ) ; Chu et al . ( 2020 ) provides architectural diversity for NAS with highly competitive performances . The other part of our work focuses on SPD network architectures . There exist algorithms to develop handcrafted SPDNet ( Huang & Van Gool , 2017 ; Brooks et al. , 2019 ; Chakraborty et al. , 2020 ) . To automate the process of SPD network design , in this work , we choose the most promising approaches from these fields ( NAS ( Liu et al. , 2018b ) , SPD networks ( Huang & Van Gool , 2017 ) ) and propose a NAS algorithm for SPD inputs . Next , we summarize the essential notions of Riemannian geometry of SPD manifolds , followed by an introduction of some basic SPDNet operations and layers . As some of the introduced operations and layers have been well-studied by the existing literature , we applied them directly to define our SPD neural architectures ’ search space . Representation and Operation : We denote n × n real SPD as X ∈ Sn++ . A real SPD matrix X ∈ Sn++ satisfies the property that for any non-zero z ∈ Rn , zTXz > 0 ( Harandi et al. , 2017 ) . We denote TXM as the tangent space of the manifoldM at X ∈ Sn++ and log corresponds to matrix logarithm . Let X1 , X2 be any two points on the SPD manifold then the distance between them is given by δM ( X1 , X2 ) = 0.5‖ log ( X − 1 2 1 X2X − 1 2 1 ) ‖F ( 1 ) There are other efficient methods to compute distance between two points on the SPD manifold ( Gao et al. , 2019 ; Dong et al. , 2017b ) , however , their discussion is beyond the scope of our work . Other property of the Riemannian manifold of our interest is local diffeomorphism of geodesics which is a one-to-one mapping from the point on the tangent space of the manifold to the manifold ( Pennec , 2020 ; Lackenby , 2020 ) . To define such notions , letX ∈ Sn++ be the base point and , Y ∈ TXSn++ , then Eq : ( 2 ) associates Y ∈ TXSn++ to a point on the manifold ( Pennec , 2020 ) . expX ( Y ) = X 1 2 exp ( X− 1 2Y X− 1 2 ) X 1 2 ∈ Sn++ , ∀ Y ∈ TX ( 2 ) Similarly , an inverse map is defined as logX ( Z ) = X 1 2 log ( X− 1 2ZX− 1 2 ) X 1 2 ∈ TX , ∀ Z ∈ Sn++ . 1 ) Basic operations of SPD Network : It is well-known that operations such as mean centralization , normalization , and adding bias to a batch of data are inherent performance booster for most neural networks . In the same spirit , existing works like Brooks et al . ( 2019 ) ; Chakraborty ( 2020 ) use the notion of these operations for the SPD or general manifold data to define analogous operations on manifolds . Below we introduce them following the work of Brooks et al . ( 2019 ) . • Batch mean , centering and bias : Given a batch of N SPD matrices { Xi } Ni=1 , we can compute its Riemannian barycenter ( B ) as B = argmin Xµ∈Sn++ ∑N i=1 δ 2 M ( Xi , Xµ ) . It is sometimes referred as Fréchet mean ( Moakher , 2005 ; Bhatia & Holbrook , 2006 ) . This definition can be extended to compute the weighted Riemannian Barycenter 1 also known as weighted Fréchet Mean ( wFM ) . B = argmin Xµ∈Sn++ N∑ i=1 wiδ 2 M ( Xi , Xµ ) ; s.t . wi ≥ 0 and N∑ i=1 wi = 1 ( 3 ) Eq : ( 3 ) can be approximated using Karcher flow ( Karcher , 1977 ; Bonnabel , 2013 ; Brooks et al. , 2019 ) or recursive geodesic mean ( Cheng et al. , 2016 ; Chakraborty et al. , 2020 ) . 2 ) Basic layers of SPD Network : Analogous to standard CNN , methods like Huang & Van Gool ( 2017 ) ; Brooks et al . ( 2019 ) ; Chakraborty et al . ( 2020 ) designed SPD layers to perform operations that respect SPD manifold constraints . AssumingXk−1 ∈ Sn++ be the input SPD matix to the kth layer , the SPD network layers are defined as follows : • BiMap layer : This layer corresponds to a dense layer for SPD data . The BiMap layer reduces the dimension of a input SPD matrix via a transformation matrix Wk as Xk = WkXk−1W Tk . To ensure the matrixXk to be an SPD matrix , theWk matrix must be of full row-rank . • Batch normalization layer : To perform batch normalization after each BiMap layer , we first compute the Riemannian barycenter of the batch of SPD matrices followed by a running mean update step , which is Riemannian weighted average between the batch mean and the current running mean , with the weights ( 1 − θ ) and ( θ ) respectively . Once mean is calculated , we centralize and add bias to each SPD sample of the batch using Eq : ( 4 ) ( Brooks et al. , 2019 ) , where P is the notation used for parallel transport : Batch centering : Centering the B : Xci = PB→I ( Xi ) = B − 1 2XiB − 1 2 , I is the identity matrix Bias the batch : Bias towards G : Xbi = PI→G ( X c i ) = G 1 2XciG 1 2 , I is the identity matrix ( 4 ) • ReEig layer : The ReEig layer is analogous to ReLU like layers present in the classical ConvNets . It aims to introduce non-linearity to SPD network . The ReEig for the kth layer is defined as : Xk = Uk−1 max ( I , Σk−1 ) U T k−1 where , Xk−1 = Uk−1Σk−1U T k−1 , I is the identity matrix , and > 0 is a rectification threshold value . Uk−1 , Σk−1 are the orthonormal matrix and singular-value matrix respectively which are obtained via matrix factorization ofXk−1 . 1Following ( Tuzel et al . ( 2006 ; 2008 ) ; Brooks et al . ( 2019 ) ) , we focus on the estimate of wFM with Karcher flow , and the thorough study on the general wFM ’ s existence and uniqueness is beyond the focus of this paper . • LogEig layer : To map the manifold representation of SPD to flat space so that a Euclidean operation can be performed , LogEig layer is introduced . The LogEig layer is defined as : Xk = Uk−1 log ( Σk−1 ) U T k−1 where , Xk−1 = Uk−1Σk−1U T k−1 . The LogEig layer is used with fully connected layers to solve tasks with SPD representation . • ExpEig layer : This layer maps the corresponding SPD representation from flat space back to SPD manifold space . It is defined asXk = Uk−1 exp ( Σk−1 ) UTk−1 where , Xk−1 = Uk−1Σk−1U T k−1 . • Weighted Riemannian pooling layer : It uses wFM definition to compute the output of the layer . Recent method use recursive geodesic mean algorithm to calculate the mean ( Chakraborty et al. , 2020 ) , in contrast , we use Karcher flow algorithm to compute it ( Karcher , 1977 ) as it is simple and widely used in practice .

Summary: The paper considers a generalization of convolutional neural networks (CNNs) to manifold-valued data such as Symmetric Positive Definite (SPD). This paper proposes a neural architecture search problem of SPD manifold networks. A SPD cell representation and corresponding candidate operation search space is introduced. They demonstrate on drone, action and emotion recognition datasets that their method is performed well compared to SPD approaches.