Matrix Multilayer Perceptron

1 INTRODUCTION . For certain applications , it is desirable to construct a conditional covariance matrix as a function of the input ( the explanatory variable ) . The problem arises , for instance , in spatial ( and spatio-temporal ) statistics , in relation to the heteroscedastic multivariate regression ( e.g. , Pourahmadi , 1999 ; Hoff & Niu , 2012 ; Fox & Dunson , 2015 ) , where we deal with multivariate response measurements for which the typical assumption of homoscedasticity may not be suitable . In such cases , we require models that estimate the covariance matrix that captures the spatial variations in correlations between the elements of the response vector . The covariance matrix is a symmetric positive definite ( SPD ) matrix which can be challenging to estimate due to its non-Euclidean geometry ( Pennec et al. , 2005 ) . The central problem that this work is concerned with is learning SPD matrices using neural networks . To motivate our discussion , consider solving the problem of SPD matrix learning using a multilayer perceptron ( MLP ) as an example of a fully connected neural network ( e.g. , Goodfellow et al. , 2016 ) . To meet the SPD constraint , one would need to tailor the output layer of the MLP so that the estimated covariance matrix satisfies the SPD requirement . One possible approach would be to use the Cholesky decomposition . The main concern is that this approach does not take into account the non-Euclidean geometry of the SPD matrices . Using empirical evaluations , we will show that the use of “ wrong geometry ” results in the poor estimation of the SPD matrices in particular where data are scarce . The primary objective here is to design a nonlinear architecture using neural networks that can effectively perform the task of SPD matrix learning . More specifically , our main contribution is to show how to alter the architecture of the MLP in such a way that it not only respects the SPD constraint , but also makes an explicit use of it . We will achieve this by : 1 ) Explicitly taking the non-Euclidean geometry of the underlying SPD manifolds ( e.g. , Pennec et al. , 2005 ) into account by designing a new loss function , and 2 ) by deriving a new backpropagation algorithm ( Rumelhart et al. , 1986 ) that respects the SPD nature of the matrices . This new model will be referred to as the matrix multilayer perceptron ( mMLP ) 1 . The mMLP makes use of positive-definite kernels to satisfy the SPD requirement across all layers . Hence , it provides a natural way of enabling deep SPD matrix learning . We take a step-by-step approach in the development of the model . We first develop a simplified version of the resulting model that is designed for learning SPD matrices ( Section 3 ) . We then extend this model into its most general form that can be used for joint estimation of the conditional mean function and the conditional covariance function in a mean-covariance regression setting ( Section 3.2 ) . An application of the model is discussed in the context of the heteroscedastic multivariate regression . 1An implementation of this work will be made available via GitHub . 2 RELATED WORK . SPD manifold metric . Earlier approaches for analyzing SPD matrices relied on the Euclidean space . But over the past decade , several studies suggest that non-Euclidean geometries such as the Riemannian structure may be better suited ( e.g. , Arsigny et al. , 2006 ; Pennec et al. , 2005 ) . In this work , we consider the von Neumann divergence ( e.g. , Nielsen & Chuang , 2000 ) as our choice of the SPD manifold metric which is related to the Riemannian geometry . Previously , Tsuda et al . ( 2005 ) used this divergence in derivation of the matrix exponentiated gradients . Their work suggests its effectiveness for measuring dissimilarities between positive definite ( PD ) matrices . SPD manifold learning . There are multiple approaches towards the SPD matrix learning , via the flattening of SPD manifolds through tangent space approximations ( e.g. , Oncel Tuzel , 2008 ; Fathy et al. , 2016 ) , mapping them into reproducing kernel Hilbert spaces ( Harandi et al. , 2012 ; Minh et al. , 2014 ) , or geometry-aware SPD matrix learning ( Harandi et al. , 2014 ) . While these methods typically employ shallow learning , the more recent line of research aims to design a deep architecture to nonlinearly learn target SPD matrices ( Ionescu et al. , 2015 ; Huang & Gool , 2017 ; Masci et al. , 2015 ; Huang et al. , 2018 ) . Our method falls in this category but differs in the problem formulation . While the previous methods address the problem where the input is an SPD matrix and the output is a vector , we consider the reverse problem where the input is a matrix with an arbitrary size and the output is an SPD matrix . Backpropagation . Our extension of the matrix backpropagation differs from the one introduced by Ionescu et al . ( 2015 ) . In their work , the necessary partial derivatives are computed using a two-step procedure consisting of first computing the functional that describes the variations of the upper layer variables with respect to the variations of the lower layer variables , and then computing the partial derivatives with respect to the lower layer variables using properties of the matrix inner product . In contrast , we make use of the concept of α-derivatives ( Magnus , 2010 ) and its favorable generalization properties to derive a procedure which closely mimics the standard backpropagation . 3 MATRIX MULTILAYER PERCEPTRON . Preliminaries : Matrix α-derivative . Throughout this work we adopt the narrow definition of the matrix derivatives known as the α-derivative ( Magnus , 2010 ) in favor of the broad definition , the ω-derivative . The reason for this is that the α-derivative has better generalization properties . This choice turned out to be crucial in the derivation of the mMLP ’ s backpropagation routine which involves derivatives of matrix functions w.r.t . the matrix of variables . Definition : Let F be an m× n matrix function of an n× q matrix of variables X . The α-derivative of F ( X ) is defined as ( Magnus , 2010 , Definition 2 ) DXF : = ∂ vecF ( X ) ∂ ( vecX ) > , ( 1 ) where DXF is an mp × nq matrix which contains all the partial derivatives such that each row contains the partial derivatives of one function with respect to all variables , and each column contains the partial derivatives of all functions with respect to one variable . For convenience , the α-derivative ’ basic properties , including the product rule and the chain rule , are summarized in Appendix B . 3.1 THE BASIC FORM OF THE MMLP . Activation matrix function . Let Z = ( z1 , . . . , zd ) denote a matrix of variables zi ∈ Rd . The activation function K ( Z ) defines a matrix function in the form of [ K ( Z ) ] i , j = κ ( zi , zj ) , ∀i , j ∈ { 1 , . . . , d } , where κ is some differentiable activation function outputting scalar values . In the following , we restrict ourselves to the kernel functions which form PD activation matrix functions . For numerical stability reasons , irrespective of the functional form of κ , we normalize the resulting matrix . This can be achieved by enforcing the trace-one constraint , H ( Z ) = K ( Z ) /tr ( K ( Z ) ) , where H denotes a differentiable SPD activation matrix function of trace one . Without loss of generality , throughout this work , we use the Mercer sigmoid kernel ( Carrington et al. , 2014 ) defined as κ ( zi , zj ) = tanh ( αzi + β ) tanh ( αzj + β ) , ( 2 ) where α and β denote the slope and the intercept , respectively . Furthermore , denotes the dot product . In all experiments , we use default values of α = 1 and β = 0 . The The α-derivative of the Mercer sigmoid kernel is computed in Appendix E. Model construction . Let X ∈ Rp1×p2 indicate the input matrix and Y ∈ Rd0×d0 indicate the corresponding output matrix , an SPD matrix of trace one . The mMLP of L hidden layers is shown as mMLP : X→Ŷ and constructed as Ŷ = H ( Z0 ) , Z0 = W0H1W > 0 + B0 , Hl = H ( Zl ) , Zl = WlHl+1W > l + Bl , ∀ 1 ≤ l ≤ L , HL+1 = H ( ZL+1 ) , ZL+1 = WL+1vecX ( WL+11p1p2 ) > + BL+1 . ( 3 ) The pair of Wl∈Rdl×dl+1 , ∀0 ≤ l≤ L , and WL+1∈RdL+1×p1p2 are the weight matrices , Bl∈Rdl×dl , ∀0≤ l ≤ L+1 , are the bias matrices , Zl ∈ Rdl×dl , ∀0 ≤ l≤ L+1 , are the latent input matrices , and Hl ∈ Rdl×dl , ∀1 ≤ l ≤ L+ 1 , are latent output SPD matrices of trace one . Design choice . In the construction of ( 3 ) , we have ensured that Hl are SPD matrices of trace one across all layers as opposed to only at the output layer . The idea behind this design choice is to propagate the nonlinearities introduced via the SPD activation matrix functions through all layers . This design choice turned out to be more effective than the alternative , and arguably simpler , design where the SPD requirement is met only at the output layer . We will discuss this further in Section 5.1.3 , where we also present an illustrative numerical example . Loss function . We consider the normalized von Neumann divergence ( e.g. , Nielsen & Chuang , 2000 ) , also commonly known as the quantum relative entropy ( QRE ) , as the base for the loss function . For two arbitrary SPD matrices of trace one , Φ and Φ̃ , the normalized von Neumann divergence is defined as : ∆QRE ( Φ̃||Φ ) = tr ( Φ̃logΦ̃− Φ̃logΦ ) , ( 4 ) where log denotes the matrix logarithm ( taking Φ as an example , it is computed using logΦ = Vdiag ( logλ ) V > , where V and λ are the matrix of eigenvectors and the vector of eigenvalues from the eigendecomposition of Φ ) . The von Neumann divergence is asymmetric . However , it can be symmetrized by using the fact that the von Neumann entropy of trace one follows the class of generalized quadratic distances ( Nielsen & Nock , 2007 ) . Hence , we define the loss function as2 ` QRE ( Ŷ , Y ) = 1 2 ( ∆QRE ( Ŷ||Y ) + ∆QRE ( Y||Ŷ ) ) . ( 5 ) Taking α-derivative of ` QRE involves taking partial derivatives through the eigendecomposition . In the following , we derive a method for analytically computing the derivative of ` QRE . The α-derivative of the symmetrized von Neumann divergence . For the loss function defined in ( 5 ) , using the α-derivative ’ s product rule and chain rule ( refer to Appendix B ) , we obtain DŶ ` = 1 2 DŶtr ( ( Ŷ −Y ) logŶ ) − 1 2 DŶtr ( ŶlogY ) , ( 6 ) where the above two terms are computed using DŶ ( tr ( ( Ŷ −Y ) logŶ ) ) = ( vec ( logŶ ) > ) > ︸ ︷︷ ︸ 1×d20 + vec ( Ŷ > −Y > ) vec ( ∂ ∂Ŷ11 logŶ ) vec ( Ŷ > −Y > ) vec ( ∂ ∂Ŷ21 logŶ ) ... vec ( Ŷ > −Y > ) vec ( ∂ ∂Ŷd0d0 logŶ ) > ︸ ︷︷ ︸ 1×d20 , ( 7 ) DŶ ( tr ( ŶlogY ) ) = ( vec ( logY ) > ) > ︸ ︷︷ ︸ 1×d20 . ( 8 ) 2Note that there are multiple ways of symmetrizing the von Neumann divergence . Our choice in ( 5 ) resembles to the α-Jensen–Shannon divergence for α = 1 ( Nielsen , 2010 ) . The remaining part in the computation of ( 6 ) is to evaluate ∂ ∂Ŷij logŶ , for all i , j ∈ { 1 , . . . , d0 } , which involves taking derivatives through the eigendecomposition . In the following , we take a similar approach as in Papadopoulo & Lourakis ( 2000 ) to compute the necessary partial derivatives . Let Ŷ = Υdiag ( λ1 , . . . , λd0 ) Υ > be the eigendecomposition . We can write ∂ ∂Ŷij logŶ = ∂ ∂Ŷij ΥΛΥ > , where Λ = diag ( logλ1 , . . . , logλd0 ) , = ∂Υ ∂Ŷij ΛΥ > + Υ ∂Λ ∂Ŷij Υ > + ΥΛ ∂Υ > ∂Ŷij . ( 9 ) By multiplying ( 9 ) from left and right by Υ > and Υ respectively , we obtain : Υ > ∂ ∂Ŷij logŶ Υ = Υ > ∂Υ ∂Ŷij Λ + ∂Λ ∂Ŷij + Λ ∂Υ > ∂Ŷij Υ = Ξij ( Υ ) Λ + ∂Λ ∂Ŷij −ΛΞij ( Υ ) , ( 10 ) where we have defined Ξij ( Υ ) = Υ > ∂∂Ŷij Υ and used the fact that Ξij ( Υ ) is an antisymmetric matrix , Ξij ( Υ ) + Ξ > ij ( Υ ) = 0 , which in turn follows from the fact that Υ is an orthonormal matrix , Υ > Υ = Id0 ⇒ ∂Υ > ∂Ŷij Υ + Υ > ∂Υ ∂Ŷij = Ξ > ij ( Υ ) + Ξij ( Υ ) = 0 . ( 11 ) Taking the antisymmetric property of Ξij ( Υ ) into account in ( 10 ) , we obtain ∂ ∂Ŷij logλk = ΥikΥjk , ( 12 ) Ξij ( Υkl ) = ΥikΥjl + ΥilΥjk 2 ( logλl − logλk ) , ∀l 6= k. ( 13 ) It is notable that by construction , we do not have repeating eigenvalues , that is λk 6= λl , ∀k 6= l , so there exists a unique solution to ( 13 ) . Once Ξij ( Υ ) is computed , it follows that ∂Υ ∂Ŷij = ΥΞij ( Υ ) , ∂Υ > ∂Ŷij = −Ξij ( Υ ) Υ > . ( 14 ) In summary , the necessary partial derivatives for computing ( 9 ) is given by ( 12 ) and ( 14 ) . Once ( 9 ) is computed for all i , j , we can evaluate ( 7 ) and ultimately evalaute ( 6 ) . Optimization . The remaining steps are feed-forward computation , backpropagation , and learning , as in the standard MLP . However , here , the backpropagation requires taking derivatives with respect to the matrix functions . These steps are described in Appendix C .

Summary: This manuscript proposes a novel formulation of the MLP to address predicting a symmetric positive definite (SPD) matrix from an input vector or matrix.  While the field has had methods for years to estimate SPD matrices (such as the covariance matrix estimate in the reparameterization trick), this manuscript proposes a markedly different approach based on a different layer structure and repeated normalization steps based on Mercer Kernels.  Additionally, the loss used can be modified to use more PSD-specific losses, such as the symmetrized von Neumann divergence, rather than the traditional quadratic loss.  This loss appears to give significantly better solutions on synthetic data.