Graphical Models in Heavy-Tailed Markets

1 Introduction . Graph learning frameworks are often designed based on the assumption that the observed graph signals are Gaussian distributed [ 1–9 ] . While such assumption for graphical models has found great success in many practical areas , which includes brain network analysis [ 10 ] , psychological networks [ 11 ] , and single-cell sequencing [ 12 ] , it inherently neglects scenarios where there may exist outliers or the underlying data is naturally heavy-tailed distributed . As a consequence , those methods often lack robustness and may not succeed in capturing a meaningful representation of the underlying graph [ 13 ] . Data from financial instruments are well-known examples of such scenarios where heavy-tailedness and skewedness are present [ 14–19 ] . In addition , there has been a growing interest in methods for estimating graphical models in financial markets , which hence demands the development of scalable and robust learning algorithms [ 20 ] . Perhaps one of the most prominent applications , clustering financial time-series via graph techniques has been an active research topic [ 20–24 ] . Nonetheless , current techniques rely on the assumption that the underlying graph has a tree structure , which does bring advantages due to its hierarchical clustering properties , but also have been shown to be unstable [ 25–27 ] and not suitable when the data is not Gaussian distributed [ 28 ] . Motivated by practical challenging applications in finance , such as clustering of financial instruments and network estimation , we investigate the problem of learning graph matrices whose structure follow that of a Laplacian matrix of an undirected weighted graph for which the data generating process is assumed to be Student-t distributed . In particular , the main contributions of this paper are as follows : • We propose a novel formulation for learning undirected weighted graphs under the assumption that the data generating process is Student-t distributed . We solve the underlying 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . learning problem via a carefully designed numerical algorithm based on the alternating direction method of multipliers ( ADMM ) , along with the establishment of its theoretical convergence guarantees . We note that the proposed algorithm can be easily extended to account for additional linear constraints on the graph weights . • We extend the proposed framework to account for heavy-tails and k-component graphs simultaneously , which enables a novel method for clustering financial time-series . • We present extensive practical results , with real-world data from the US stock market , foreign exchanges , and cryptocurrencies , that showcase clear advantages of including heavy-tail assumptions into graph learning frameworks when compared to state-of-the-art , Gaussian-based methods . Notation : Matrices ( vectors ) are denoted by bold , italic , capital ( lowercase ) roman letters likeX , x. Vectors are assumed to be column vectors . The ( i , j ) element of a matrixX ∈ Rn×p is denoted as Xij . The i-th element of a vector x is denoted as xi . The i-th row of X is denoted as xi ∈ Rp×1 . Given a symmetric matrixA , λi ( A ) and λmax ( A ) denote the i-th smallest and maximum eigenvalue of A , respectively . The Moore-Penrose inverse of A is denoted as A† . The Frobenius norm of a matrix A is denoted as ‖A‖F = √ tr ( A > A ) . The operator Diag : Rp → Rp×p creates a diagonal matrix with the elements of an input vector along its diagonal . The operator diag : Rp×p → Rp extracts the diagonal of a square matrix . For x ∈ Rp , ‖x‖∞ = maxi|xi| . ( x ) + denotes the projection on to the nonnegative orthant , i.e. , ( x ) + = max ( 0 , x ) . 2 Background & Related Works . An undirected , weighted graph is denoted as a triple G = ( V , E , W ) , where V = { 1 , 2 , . . . , p } is the node set , E ⊆ { { u , v } : u , v ∈ V , u 6= v } is the edge set , that is , a subset of the set of all possible unordered pairs of nodes such that { u , v } ∈ E iff there exists a link between nodes u and v. W ∈ Rp×p+ is the symmetric weighted adjacency matrix that satisfies Wii = 0 , Wij > 0 iff { i , j } ∈ E and Wij = 0 , otherwise . The combinatorial , unnormalized graph Laplacian matrix L is defined , as L , D −W , whereD , Diag ( W1 ) is the degree matrix . A p-dimensional , real-valued , Gaussian random variable x , with mean vector E [ x ] , µ and rankdeficient precision matrix L , is said to form a Laplacian constrained Gaussian Markov random field ( LGMRF ) [ 9 , 29–31 ] of rank p− k , k ≥ 1 , with respect to a graph G , when its probability density function is given as p ( x ) ∝ √ det∗ ( L ) exp { −1 2 ( x− µ ) > L ( x− µ ) } , ( 1 ) where det∗ ( L ) is the pseudo-determinant of L , i.e. , the product of its positive eigenvalues [ 32 ] . Assume we are given n observations of x , i.e. , X = [ x1 , x2 , . . . , xn ] > , X ∈ Rn×p , xi ∈ Rp×1 . The goal of graph learning algorithms is to learn a Laplacian matrix , or equivalently an adjacency matrix , given only the data matrixX , i.e. , often without any knowledge of E . To that end , the penalized maximum likelihood estimator ( MLE ) of the Laplacian constrained precision matrix of x , on the basis of the observed dataX , is : minimize L 0 tr ( LS ) − log det∗ ( L ) + h ( L ) , subject to L1 = 0 , Lij = Lji ≤ 0 , ( 2 ) where S is a similarity matrix , e.g. , the sample covariance ( or correlation ) matrix S ∝X > X , and h is a regularization function to promote certain properties on L such as sparsity or low-rankness . Even though Problem ( 2 ) is convex , provided we assume a convex choice for h , it is not adequate to be solved by disciplined convex programming languages , such as cvxpy [ 33 ] , particularly due to scalability issues related to the computation of the term log det∗ ( L ) [ 6 , 7 ] . Indeed , recently , considerable efforts have been directed towards the design of scalable , iterative algorithms based on block coordinate descent [ 34 ] , majorization-minimization ( MM ) [ 35 , 36 ] , and ADMM [ 37 ] to solve Problem ( 2 ) in an efficient fashion , e.g. , [ 6 ] and [ 7 ] . Estimators based on Gaussian assumptions have been proposed for connected graphs [ 2 , 4–7 ] . Some of their properties , such as sparsity , are yet being investigated [ 9 , 31 ] . The authors in [ 29 ] and [ 38 ] proposed optimization programs for learning the class of k-component graphs , as such class is an appealing model for clustering tasks due to the spectral properties of the Laplacian matrix . However , a major shortcoming in their formulations is the lack of constraints on the degrees of the nodes , which allows for trivial solutions , i.e. , graphs with isolated nodes . Elliptical losses along with linear structural constraints that retain the positive-definiteness of the estimated covariance matrix have been proposed in the literature [ 39 ] . In this work , however , we address the case of Laplacian constraints , which lead to positive-semidefinite precision matrices , and nonconvex k-component structural constraints . 3 Proposed Formulations & Algorithms . In this section , we propose optimization formulations and an iterative algorithm to learn a Laplacian matrix from heavy-tailed assumptions . With that goal , we express the Laplacian matrix via its linear operator , i.e. , L = Lw [ 29 ] , wherew ∈ Rp ( p−1 ) /2 is the vectorized form of the upper triangular part of the adjacency matrix , also known as the vector of graph weights . In addition , we use the fact that , for connected graphs , it follows that det∗ ( Lw ) = det ( Lw + J ) , J , 1p11 > [ 6 ] . In order to address the inherent heavy-tailed nature of financial market data [ 40 ] , we consider the Student-t distribution under the improper Markov random field assumption [ 41 ] with Laplacian structural constraints , that is , we assume the data generating process to be modeled as multivariate zero-mean Student-t distribution , whose probability density function can be written as p ( x ) ∝ √ det∗ ( Θ ) ( 1 + x > Θx ν ) −ν + p 2 , ν > 2 , ( 3 ) where Θ is a positive-semidefinite inverse scatter matrix modeled as a combinatorial graph Laplacian matrix and ν is the number of degrees of freedom , which measures the rate of decay of the tails . This results in a robustified version of the MLE for connected graph learning , i.e. , minimize w≥0 , Θ 0 p+ ν n n∑ i=1 log ( 1 + x > i Lwxi ν ) − log det ( Θ + J ) , subject to Θ = Lw , dw = d , ( 4 ) where d : Rp ( p−1 ) /2 → Rp is the degree operator defined as dw , diag ( Lw ) . The constraint dw = d enables the learning of additional graph structures such as regular graphs and it is crucial for k-component graphs , as discussed in Section 3.2 . From a theoretical perspective , the Student-t model naturally yields sparse graphs . Comparing the objective function in Problem ( 4 ) to that of Problem ( 2 ) , we note that the Student-t contains a log ( · ) term in place of a linear term of the graph weights . The usage of a log function to promote sparsity is closely related to the iteratively reweighted ` 1-norm as an approximation for the ` 0-norm problem [ 42 ] . Problem ( 4 ) is , in general , nonconvex due to the summation of log terms and hence it is challenging to be considered directly . Hence , we design an iterative algorithm based on the ADMM framework . 3.1 ADMM Solution . The partial augmented Lagrangian function of Problem ( 4 ) is given as Lρ ( Θ , w , Y , y ) = p+ ν n n∑ i=1 log ( 1 + x > i Lwxi ν ) − log det ( Θ + J ) + 〈y , dw − d〉+ ρ 2 ‖dw − d‖22 + 〈Y , Θ− Lw〉+ ρ 2 ‖Θ− Lw‖2F , ( 5 ) where Y and y are the dual variables associated with the equality constraints Θ = Lw and dw = d , respectively . Note that we deal with the constraints w ≥ 0 and Θ 0 directly , hence there are no dual variables associated with them . Given wl and Y l , the subproblem for Θ can be written as Θl+1 = arg min Θ 0 − log det ( Θ + J ) + 〈Θ , Y l〉+ ρ 2 ∥∥Θ− Lwl∥∥2 F , ( 6 ) whose closed-form solution is given by Lemma 1 . Lemma 1 The global minimizer of problem ( 6 ) is [ 43 , 44 ] Θl+1 = 1 2ρ U ( Γ + √ Γ2 + 4ρI ) U > − J , ( 7 ) where UΓU > is the eigenvalue decomposition of ρ ( Lwl + J ) − Y l. Given Θl+1 , Y l , and yl , the subproblem for w can be formulated as minimize w≥0 ρ 2 w > ( d∗d + L∗L ) w − 〈 w , L∗ ( Y l + ρΘl+1 ) − d∗ ( yl − ρd ) 〉 + p+ ν n n∑ i=1 log ( 1 + x > i Lwxi ν ) , ( 8 ) where d∗ and L∗ are the adjoint operators of the degree and Laplacian operators , respectively . In general , subproblem ( 8 ) is nonconvex due to the concave nature of the logarithm function . Hence , we resort to the MM method [ 36 ] to find a stationary point of subproblem ( 8 ) . We proceed by constructing a global upper-bound of the objective function of ( 8 ) at point wj ∈ Rp ( p−1 ) /2+ as g ( w , wj ) = g ( wj , wj ) + 〈w −wj , ∇wf ( wj ) 〉+ µ 2 ∥∥w −wj∥∥2 2 , ( 9 ) where f is the objective function in the minimization in ( 8 ) , its gradient is given as ∇wf ( wj ) = aj + bj , where aj = L∗ ( S̃j − Y l − ρ ( Θl+1 − Lwj ) ) , ( 10 ) bj = d∗ ( yl − ρ ( d− dwj ) ) , ( 11 ) where S̃j , 1n ∑n i=1 ( p+ ν ) xix > i 〈wj , L∗ ( xix > i ) 〉+ ν is a weighted sample covariance matrix , and µ = ρλmax ( d ∗d + L∗L ) , and the maximum eigenvalue of d∗d + L∗L is given by Lemma 2 , whose proof is presented in the Supplementary Material . Lemma 2 The maximum eigenvalue of the matrix d∗d + L∗L is given as λmax ( d ∗d + L∗L ) = 2 ( 2p− 1 ) . ( 12 ) The vector of graph weights w can then be updated by minimizing the function g constructed in ( 9 ) , which is tantamount to solving the following nonnegative , quadratic-constrained , strictly convex problem : wj+1 = arg min w≥0 ρ ( 2p− 1 ) ∥∥w −wj∥∥2 2 + 〈w , aj + bj〉 , ( 13 ) whose unique solution can be readily obtained via its KKT optimality conditions and is given as wj+1 = ( wj − a j + bj 2ρ ( 2p− 1 ) ) + , ( 14 ) that is , a projected gradient descent step with learning rate ( 2ρ ( 2p− 1 ) ) −1 . Thus , we iterate ( 14 ) in order to obtain a stationary point , wl+1 , of Problem ( 8 ) . In practice , we observe that a few iterations are sufficient to retrieve wl+1 . The dual variables Y and y are updated via gradient ascent steps . Algorithm 1 summarizes the implementation to find a stationary point of Problem ( 4 ) . We present Theorem 3 , proved in the Supplementary Material , which establishes the convergence of Algorithm 1 . Theorem 3 Algorithm 1 converges subsequently for any sufficiently large ρ , that is , the sequence { ( Θl , wl , Y l , yl ) } generated by Algorithm 1 has at least one limit point , and each limit point is a stationary point of Problem ( 4 ) . Algorithm 1 : Student-t Graph Learning Data : Data matrixX ∈ Rn×p , initial estimate of the graph weights w0 , desired degree vector d , penalty parameter ρ > 0 , degrees of freedom ν > 2 , convergence tolerance > 0 Result : Graph Laplacian estimation : Lw ? 1 initialize Y = 0 , y = 0 2 l← 0 3 while ∥∥rl∥∥∞ > or ∥∥sl∥∥∞ > do 4 . update Θl+1 via ( 7 ) 5 . update wl+1 by iterating ( 14 ) 6 . update Y l+1 = Y l + ρ ( Θl+1 − Lwl+1 ) 7 . update yl+1 = yl + ρ ( dwl+1 − d ) 8 . compute residual rl+1 = Θl+1 − Lwl+1 9 . compute residual sl+1 = dwl+1 − d 10 l← l + 1 11 end 3.2 An extension to k-component graphs The graph learning formulation proposed in ( 4 ) is applicable to learn connected graphs . Learning graphs with k components , k assumed to be known , poses a considerably higher challenge , as the dimension of the nullspace of the Laplacian matrix Lw is equal to the number of components of the graph [ 45 ] . One way to achieve this requirement is by imposing the constraint rank ( Lw ) = p− k , which is nonconvex and nondifferentiable , in the maximum likelihood problem generated by ( 3 ) . We instead relax this rank constraint by noting that via Fan ’ s theorem [ 46 ] , we have k∑ i=1 λi ( Lw ) = minimize V ∈Rp×k , V > V =I tr ( V > LwV ) . ( 15 ) Thus , by using the right hand side of ( 15 ) as a regularization term , we are able formulate the following optimization problem to learn a Student-t k-component graph : minimize w≥0 , Θ 0 , V p+ ν n n∑ i=1 log ( 1 + x > i Lwxi ν ) − log det∗ ( Θ ) + ηtr ( LwV V > ) , subject to Θ = Lw , rank ( Θ ) = p− k , dw = d , V > V = I , V ∈ Rp×k . ( 16 ) The partial augmented Lagrangian function of Problem ( 16 ) can be expressed as Lρ ( Θ , w , V , Y , y ) = p+ ν n n∑ i=1 log ( 1 + x > i Lwxi ν ) − log det∗ ( Θ ) + ηtr ( LwV V > ) + 〈y , dw − d〉+ ρ 2 ‖dw − d‖22 + 〈Y , Θ− Lw〉+ ρ 2 ‖Θ− Lw‖2F . ( 17 ) Given wl and Y l , the subproblem for Θ can be written as Θl+1 = arg min rank ( Θ ) =p−k − log det∗ ( Θ ) + 〈Θ , Y l〉+ ρ 2 ∥∥Θ− Lwl∥∥2 F , ( 18 ) which is nearly the same as ( 6 ) . Its solution is obtained as Θl+1 = 1 2ρ U ( Γ + √ Γ2 + 4ρI ) U > , ( 19 ) except that now UΓU > is the eigenvalue decomposition of ρLwl − Y l , with Γ having the largest p− k eigenvalues along its diagonal and U ∈ Rp× ( p−k ) contains the corresponding eigenvectors . The subproblem to obtainwl+1 is virtually the same as in ( 8 ) except for the additional linear term ηtr ( LwV lV l > ) . Hence , its update is also a projected gradient descent step , alike ( 14 ) where aj , L∗ ( S̃j + ηV lV l > − Y l − ρ ( Θl+1 − Lwj ) ) . ( 20 ) Given wl+1 , we have the following subproblem for V : minimize V ∈Rp×k , V > V =I tr ( V > Lwl+1V ) , ( 21 ) whose closed-form solution is given by the k eigenvectors associated with the k smallest eigenvalues of Lwl+1 [ 47 , 48 ] . Algorithm 2 summarizes the implementation to find a stationary point of Problem ( 16 ) , and its convergence is established through Theorem 4 , whose proof is presented in the Supplementary Material . Theorem 4 Algorithm 2 converges subsequently for any sufficiently large ρ , that is , the sequence { ( Θl , wl , V l , Y l , yl ) } generated by Algorithm 2 has at least one limit point , and each limit point is a stationary point of Problem ( 16 ) . Algorithm 2 : k-component Student-t graph learning Data : Data matrixX ∈ Rn×p , initial estimate of the graph weights w0 , number of graph components k , desired degree vector d , degrees of freedom ν , rank hyperparameter η > 0 , penalty parameter ρ > 0 , tolerance > 0 Result : Laplacian estimation : Lw ? 1 initialize Y = 0 , y = 0 2 l← 0 3 while ∥∥rl∥∥∞ > or ∥∥sl∥∥∞ > do 4 . update Θl+1 via ( 19 ) 5 . update wl+1 as in ( 14 ) with aj given in ( 20 ) 6 . update V l+1 as in ( 21 ) 7 . update Y l+1 = Y l + ρ ( Θl+1 − Lwl+1 ) 8 . update yl+1 = yl + ρ ( dwl+1 − d ) 9 . compute residual rl+1 = Θl+1 − Lwl+1 10 . compute residual sl+1 = dwl+1 − d 11 l← l + 1 12 end

Summary: The paper proposes a new estimator for learning graphical models under the t-distribution assumption for data generation process. For the new estimator, an ADMM-based algorithm is designed to solve the associated optimization problem in order to find the graph structures. The new estimator is evaluated extensively on real-world datasets to demonstrate its effectiveness. The paper also establishes the convergence of the proposed algorithm.