Graphical Models in Heavy-Tailed Markets

1 Introduction . Graph learning frameworks are often designed based on the assumption that the observed graph signals are Gaussian distributed [ 1–9 ] . While such assumption for graphical models has found great success in many practical areas , which includes brain network analysis [ 10 ] , psychological networks [ 11 ] , and single-cell sequencing [ 12 ] , it inherently neglects scenarios where there may exist outliers or the underlying data is naturally heavy-tailed distributed . As a consequence , those methods often lack robustness and may not succeed in capturing a meaningful representation of the underlying graph [ 13 ] . Data from financial instruments are well-known examples of such scenarios where heavy-tailedness and skewedness are present [ 14–19 ] . In addition , there has been a growing interest in methods for estimating graphical models in financial markets , which hence demands the development of scalable and robust learning algorithms [ 20 ] . Perhaps one of the most prominent applications , clustering financial time-series via graph techniques has been an active research topic [ 20–24 ] . Nonetheless , current techniques rely on the assumption that the underlying graph has a tree structure , which does bring advantages due to its hierarchical clustering properties , but also have been shown to be unstable [ 25–27 ] and not suitable when the data is not Gaussian distributed [ 28 ] . Motivated by practical challenging applications in finance , such as clustering of financial instruments and network estimation , we investigate the problem of learning graph matrices whose structure follow that of a Laplacian matrix of an undirected weighted graph for which the data generating process is assumed to be Student-t distributed . In particular , the main contributions of this paper are as follows : • We propose a novel formulation for learning undirected weighted graphs under the assumption that the data generating process is Student-t distributed . We solve the underlying 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . learning problem via a carefully designed numerical algorithm based on the alternating direction method of multipliers ( ADMM ) , along with the establishment of its theoretical convergence guarantees . We note that the proposed algorithm can be easily extended to account for additional linear constraints on the graph weights . • We extend the proposed framework to account for heavy-tails and k-component graphs simultaneously , which enables a novel method for clustering financial time-series . • We present extensive practical results , with real-world data from the US stock market , foreign exchanges , and cryptocurrencies , that showcase clear advantages of including heavy-tail assumptions into graph learning frameworks when compared to state-of-the-art , Gaussian-based methods . Notation : Matrices ( vectors ) are denoted by bold , italic , capital ( lowercase ) roman letters likeX , x. Vectors are assumed to be column vectors . The ( i , j ) element of a matrixX ∈ Rn×p is denoted as Xij . The i-th element of a vector x is denoted as xi . The i-th row of X is denoted as xi ∈ Rp×1 . Given a symmetric matrixA , λi ( A ) and λmax ( A ) denote the i-th smallest and maximum eigenvalue of A , respectively . The Moore-Penrose inverse of A is denoted as A† . The Frobenius norm of a matrix A is denoted as ‖A‖F = √ tr ( A > A ) . The operator Diag : Rp → Rp×p creates a diagonal matrix with the elements of an input vector along its diagonal . The operator diag : Rp×p → Rp extracts the diagonal of a square matrix . For x ∈ Rp , ‖x‖∞ = maxi|xi| . ( x ) + denotes the projection on to the nonnegative orthant , i.e. , ( x ) + = max ( 0 , x ) . 2 Background & Related Works . An undirected , weighted graph is denoted as a triple G = ( V , E , W ) , where V = { 1 , 2 , . . . , p } is the node set , E ⊆ { { u , v } : u , v ∈ V , u 6= v } is the edge set , that is , a subset of the set of all possible unordered pairs of nodes such that { u , v } ∈ E iff there exists a link between nodes u and v. W ∈ Rp×p+ is the symmetric weighted adjacency matrix that satisfies Wii = 0 , Wij > 0 iff { i , j } ∈ E and Wij = 0 , otherwise . The combinatorial , unnormalized graph Laplacian matrix L is defined , as L , D −W , whereD , Diag ( W1 ) is the degree matrix . A p-dimensional , real-valued , Gaussian random variable x , with mean vector E [ x ] , µ and rankdeficient precision matrix L , is said to form a Laplacian constrained Gaussian Markov random field ( LGMRF ) [ 9 , 29–31 ] of rank p− k , k ≥ 1 , with respect to a graph G , when its probability density function is given as p ( x ) ∝ √ det∗ ( L ) exp { −1 2 ( x− µ ) > L ( x− µ ) } , ( 1 ) where det∗ ( L ) is the pseudo-determinant of L , i.e. , the product of its positive eigenvalues [ 32 ] . Assume we are given n observations of x , i.e. , X = [ x1 , x2 , . . . , xn ] > , X ∈ Rn×p , xi ∈ Rp×1 . The goal of graph learning algorithms is to learn a Laplacian matrix , or equivalently an adjacency matrix , given only the data matrixX , i.e. , often without any knowledge of E . To that end , the penalized maximum likelihood estimator ( MLE ) of the Laplacian constrained precision matrix of x , on the basis of the observed dataX , is : minimize L 0 tr ( LS ) − log det∗ ( L ) + h ( L ) , subject to L1 = 0 , Lij = Lji ≤ 0 , ( 2 ) where S is a similarity matrix , e.g. , the sample covariance ( or correlation ) matrix S ∝X > X , and h is a regularization function to promote certain properties on L such as sparsity or low-rankness . Even though Problem ( 2 ) is convex , provided we assume a convex choice for h , it is not adequate to be solved by disciplined convex programming languages , such as cvxpy [ 33 ] , particularly due to scalability issues related to the computation of the term log det∗ ( L ) [ 6 , 7 ] . Indeed , recently , considerable efforts have been directed towards the design of scalable , iterative algorithms based on block coordinate descent [ 34 ] , majorization-minimization ( MM ) [ 35 , 36 ] , and ADMM [ 37 ] to solve Problem ( 2 ) in an efficient fashion , e.g. , [ 6 ] and [ 7 ] . Estimators based on Gaussian assumptions have been proposed for connected graphs [ 2 , 4–7 ] . Some of their properties , such as sparsity , are yet being investigated [ 9 , 31 ] . The authors in [ 29 ] and [ 38 ] proposed optimization programs for learning the class of k-component graphs , as such class is an appealing model for clustering tasks due to the spectral properties of the Laplacian matrix . However , a major shortcoming in their formulations is the lack of constraints on the degrees of the nodes , which allows for trivial solutions , i.e. , graphs with isolated nodes . Elliptical losses along with linear structural constraints that retain the positive-definiteness of the estimated covariance matrix have been proposed in the literature [ 39 ] . In this work , however , we address the case of Laplacian constraints , which lead to positive-semidefinite precision matrices , and nonconvex k-component structural constraints . 3 Proposed Formulations & Algorithms . In this section , we propose optimization formulations and an iterative algorithm to learn a Laplacian matrix from heavy-tailed assumptions . With that goal , we express the Laplacian matrix via its linear operator , i.e. , L = Lw [ 29 ] , wherew ∈ Rp ( p−1 ) /2 is the vectorized form of the upper triangular part of the adjacency matrix , also known as the vector of graph weights . In addition , we use the fact that , for connected graphs , it follows that det∗ ( Lw ) = det ( Lw + J ) , J , 1p11 > [ 6 ] . In order to address the inherent heavy-tailed nature of financial market data [ 40 ] , we consider the Student-t distribution under the improper Markov random field assumption [ 41 ] with Laplacian structural constraints , that is , we assume the data generating process to be modeled as multivariate zero-mean Student-t distribution , whose probability density function can be written as p ( x ) ∝ √ det∗ ( Θ ) ( 1 + x > Θx ν ) −ν + p 2 , ν > 2 , ( 3 ) where Θ is a positive-semidefinite inverse scatter matrix modeled as a combinatorial graph Laplacian matrix and ν is the number of degrees of freedom , which measures the rate of decay of the tails . This results in a robustified version of the MLE for connected graph learning , i.e. , minimize w≥0 , Θ 0 p+ ν n n∑ i=1 log ( 1 + x > i Lwxi ν ) − log det ( Θ + J ) , subject to Θ = Lw , dw = d , ( 4 ) where d : Rp ( p−1 ) /2 → Rp is the degree operator defined as dw , diag ( Lw ) . The constraint dw = d enables the learning of additional graph structures such as regular graphs and it is crucial for k-component graphs , as discussed in Section 3.2 . From a theoretical perspective , the Student-t model naturally yields sparse graphs . Comparing the objective function in Problem ( 4 ) to that of Problem ( 2 ) , we note that the Student-t contains a log ( · ) term in place of a linear term of the graph weights . The usage of a log function to promote sparsity is closely related to the iteratively reweighted ` 1-norm as an approximation for the ` 0-norm problem [ 42 ] . Problem ( 4 ) is , in general , nonconvex due to the summation of log terms and hence it is challenging to be considered directly . Hence , we design an iterative algorithm based on the ADMM framework . 3.1 ADMM Solution . The partial augmented Lagrangian function of Problem ( 4 ) is given as Lρ ( Θ , w , Y , y ) = p+ ν n n∑ i=1 log ( 1 + x > i Lwxi ν ) − log det ( Θ + J ) + 〈y , dw − d〉+ ρ 2 ‖dw − d‖22 + 〈Y , Θ− Lw〉+ ρ 2 ‖Θ− Lw‖2F , ( 5 ) where Y and y are the dual variables associated with the equality constraints Θ = Lw and dw = d , respectively . Note that we deal with the constraints w ≥ 0 and Θ 0 directly , hence there are no dual variables associated with them . Given wl and Y l , the subproblem for Θ can be written as Θl+1 = arg min Θ 0 − log det ( Θ + J ) + 〈Θ , Y l〉+ ρ 2 ∥∥Θ− Lwl∥∥2 F , ( 6 ) whose closed-form solution is given by Lemma 1 . Lemma 1 The global minimizer of problem ( 6 ) is [ 43 , 44 ] Θl+1 = 1 2ρ U ( Γ + √ Γ2 + 4ρI ) U > − J , ( 7 ) where UΓU > is the eigenvalue decomposition of ρ ( Lwl + J ) − Y l. Given Θl+1 , Y l , and yl , the subproblem for w can be formulated as minimize w≥0 ρ 2 w > ( d∗d + L∗L ) w − 〈 w , L∗ ( Y l + ρΘl+1 ) − d∗ ( yl − ρd ) 〉 + p+ ν n n∑ i=1 log ( 1 + x > i Lwxi ν ) , ( 8 ) where d∗ and L∗ are the adjoint operators of the degree and Laplacian operators , respectively . In general , subproblem ( 8 ) is nonconvex due to the concave nature of the logarithm function . Hence , we resort to the MM method [ 36 ] to find a stationary point of subproblem ( 8 ) . We proceed by constructing a global upper-bound of the objective function of ( 8 ) at point wj ∈ Rp ( p−1 ) /2+ as g ( w , wj ) = g ( wj , wj ) + 〈w −wj , ∇wf ( wj ) 〉+ µ 2 ∥∥w −wj∥∥2 2 , ( 9 ) where f is the objective function in the minimization in ( 8 ) , its gradient is given as ∇wf ( wj ) = aj + bj , where aj = L∗ ( S̃j − Y l − ρ ( Θl+1 − Lwj ) ) , ( 10 ) bj = d∗ ( yl − ρ ( d− dwj ) ) , ( 11 ) where S̃j , 1n ∑n i=1 ( p+ ν ) xix > i 〈wj , L∗ ( xix > i ) 〉+ ν is a weighted sample covariance matrix , and µ = ρλmax ( d ∗d + L∗L ) , and the maximum eigenvalue of d∗d + L∗L is given by Lemma 2 , whose proof is presented in the Supplementary Material . Lemma 2 The maximum eigenvalue of the matrix d∗d + L∗L is given as λmax ( d ∗d + L∗L ) = 2 ( 2p− 1 ) . ( 12 ) The vector of graph weights w can then be updated by minimizing the function g constructed in ( 9 ) , which is tantamount to solving the following nonnegative , quadratic-constrained , strictly convex problem : wj+1 = arg min w≥0 ρ ( 2p− 1 ) ∥∥w −wj∥∥2 2 + 〈w , aj + bj〉 , ( 13 ) whose unique solution can be readily obtained via its KKT optimality conditions and is given as wj+1 = ( wj − a j + bj 2ρ ( 2p− 1 ) ) + , ( 14 ) that is , a projected gradient descent step with learning rate ( 2ρ ( 2p− 1 ) ) −1 . Thus , we iterate ( 14 ) in order to obtain a stationary point , wl+1 , of Problem ( 8 ) . In practice , we observe that a few iterations are sufficient to retrieve wl+1 . The dual variables Y and y are updated via gradient ascent steps . Algorithm 1 summarizes the implementation to find a stationary point of Problem ( 4 ) . We present Theorem 3 , proved in the Supplementary Material , which establishes the convergence of Algorithm 1 . Theorem 3 Algorithm 1 converges subsequently for any sufficiently large ρ , that is , the sequence { ( Θl , wl , Y l , yl ) } generated by Algorithm 1 has at least one limit point , and each limit point is a stationary point of Problem ( 4 ) . Algorithm 1 : Student-t Graph Learning Data : Data matrixX ∈ Rn×p , initial estimate of the graph weights w0 , desired degree vector d , penalty parameter ρ > 0 , degrees of freedom ν > 2 , convergence tolerance > 0 Result : Graph Laplacian estimation : Lw ? 1 initialize Y = 0 , y = 0 2 l← 0 3 while ∥∥rl∥∥∞ > or ∥∥sl∥∥∞ > do 4 . update Θl+1 via ( 7 ) 5 . update wl+1 by iterating ( 14 ) 6 . update Y l+1 = Y l + ρ ( Θl+1 − Lwl+1 ) 7 . update yl+1 = yl + ρ ( dwl+1 − d ) 8 . compute residual rl+1 = Θl+1 − Lwl+1 9 . compute residual sl+1 = dwl+1 − d 10 l← l + 1 11 end 3.2 An extension to k-component graphs The graph learning formulation proposed in ( 4 ) is applicable to learn connected graphs . Learning graphs with k components , k assumed to be known , poses a considerably higher challenge , as the dimension of the nullspace of the Laplacian matrix Lw is equal to the number of components of the graph [ 45 ] . One way to achieve this requirement is by imposing the constraint rank ( Lw ) = p− k , which is nonconvex and nondifferentiable , in the maximum likelihood problem generated by ( 3 ) . We instead relax this rank constraint by noting that via Fan ’ s theorem [ 46 ] , we have k∑ i=1 λi ( Lw ) = minimize V ∈Rp×k , V > V =I tr ( V > LwV ) . ( 15 ) Thus , by using the right hand side of ( 15 ) as a regularization term , we are able formulate the following optimization problem to learn a Student-t k-component graph : minimize w≥0 , Θ 0 , V p+ ν n n∑ i=1 log ( 1 + x > i Lwxi ν ) − log det∗ ( Θ ) + ηtr ( LwV V > ) , subject to Θ = Lw , rank ( Θ ) = p− k , dw = d , V > V = I , V ∈ Rp×k . ( 16 ) The partial augmented Lagrangian function of Problem ( 16 ) can be expressed as Lρ ( Θ , w , V , Y , y ) = p+ ν n n∑ i=1 log ( 1 + x > i Lwxi ν ) − log det∗ ( Θ ) + ηtr ( LwV V > ) + 〈y , dw − d〉+ ρ 2 ‖dw − d‖22 + 〈Y , Θ− Lw〉+ ρ 2 ‖Θ− Lw‖2F . ( 17 ) Given wl and Y l , the subproblem for Θ can be written as Θl+1 = arg min rank ( Θ ) =p−k − log det∗ ( Θ ) + 〈Θ , Y l〉+ ρ 2 ∥∥Θ− Lwl∥∥2 F , ( 18 ) which is nearly the same as ( 6 ) . Its solution is obtained as Θl+1 = 1 2ρ U ( Γ + √ Γ2 + 4ρI ) U > , ( 19 ) except that now UΓU > is the eigenvalue decomposition of ρLwl − Y l , with Γ having the largest p− k eigenvalues along its diagonal and U ∈ Rp× ( p−k ) contains the corresponding eigenvectors . The subproblem to obtainwl+1 is virtually the same as in ( 8 ) except for the additional linear term ηtr ( LwV lV l > ) . Hence , its update is also a projected gradient descent step , alike ( 14 ) where aj , L∗ ( S̃j + ηV lV l > − Y l − ρ ( Θl+1 − Lwj ) ) . ( 20 ) Given wl+1 , we have the following subproblem for V : minimize V ∈Rp×k , V > V =I tr ( V > Lwl+1V ) , ( 21 ) whose closed-form solution is given by the k eigenvectors associated with the k smallest eigenvalues of Lwl+1 [ 47 , 48 ] . Algorithm 2 summarizes the implementation to find a stationary point of Problem ( 16 ) , and its convergence is established through Theorem 4 , whose proof is presented in the Supplementary Material . Theorem 4 Algorithm 2 converges subsequently for any sufficiently large ρ , that is , the sequence { ( Θl , wl , V l , Y l , yl ) } generated by Algorithm 2 has at least one limit point , and each limit point is a stationary point of Problem ( 16 ) . Algorithm 2 : k-component Student-t graph learning Data : Data matrixX ∈ Rn×p , initial estimate of the graph weights w0 , number of graph components k , desired degree vector d , degrees of freedom ν , rank hyperparameter η > 0 , penalty parameter ρ > 0 , tolerance > 0 Result : Laplacian estimation : Lw ? 1 initialize Y = 0 , y = 0 2 l← 0 3 while ∥∥rl∥∥∞ > or ∥∥sl∥∥∞ > do 4 . update Θl+1 via ( 19 ) 5 . update wl+1 as in ( 14 ) with aj given in ( 20 ) 6 . update V l+1 as in ( 21 ) 7 . update Y l+1 = Y l + ρ ( Θl+1 − Lwl+1 ) 8 . update yl+1 = yl + ρ ( dwl+1 − d ) 9 . compute residual rl+1 = Θl+1 − Lwl+1 10 . compute residual sl+1 = dwl+1 − d 11 l← l + 1 12 end

Summary: The paper proposes a framework for learning a graphical model when the data generating distribution is heavy-tailed. More specifically, the author(s) propose a graph learning estimator under Markov Random Field framework when the data is generated from a Student's t-distribution. A scalable algorithm is proposed to handle connected and K-component graphs. The methodology is particularly applicable to financial time series data. The algorithm is shown to be superior in performance compared to existing methods in both simulated and real data applications.