Title: GmGM: a fast Gaussian graphical model for multi-modal data
Venue: NeurIPS 2023
Accepted: no
Keywords: bigraphical lasso

network inference

gaussian graphical models

multi-omics

single-cell
Abstract: This paper introduces the Gaussian multi-Graphical Model, a model to construct sparse graph representations of matrix- and tensor-variate data.  We generalize prior work in this area by simultaneously learning this representation across several tensors that share axes, which is necessary to allow the analysis of multimodal datasets such as those encountered in multi-omics.  Our algorithm uses only a single eigendecomposition per axis, achieving an order of magnitude speedup over prior work in the ungeneralized case.  This allows the use of our methodology on large multi-modal datasets such as single-cell multi-omics data, which was challenging with previous approaches.  We validate our model on synthetic data and five real-world datasets.

Abstract

This paper introduces the Gaussian multi-Graphical Model, a model to construct sparse graph representations of matrix- and tensor-variate data. We generalize prior work in this area by simultaneously learning this representation across several tensors that share axes, which is necessary to allow the analysis of multimodal datasets such as those encountered in multi-omics. Our algorithm uses only a single eigendecomposition per axis, achieving an order of magnitude speedup over prior work in the ungeneralized case. This allows the use of our methodology on large multi-modal datasets such as single-cell multi-omics data, which was challenging with previous approaches. We validate our model on synthetic data and five real-world datasets.

1 Introduction

A number of modern applications require the estimation of networks (graphs) exploring the dependency structures underlying the data. In this paper, we propose a new approach for estimating conditional dependency graphs. Two datapoints x, y are conditionally independent (with respect to a dataset D ) if knowing one provides no information about the other that is not already contained in the rest of the dataset: P ( x | y, D \ xy ) = P ( x |D \ xy ) . For normally distributed data, conditional dependencies are encoded in the inverse of the covariance matrix (the 'precision' matrix). Two datapoints are conditionally dependent on each other if and only if their corresponding element in the precision matrix is not zero. If our dataset were in the form of a vector d , we could then model it as d ∼ N ( 0 , Ψ -1 ) for precision matrix Ψ . This is a Gaussian Graphical Model (GGM); Ψ encodes the graph.

However, datasets are often more structured than vectors. For example, single-cell RNA sequencing datasets (scRNA-seq) come in the form of a matrix of gene expression counts whose rows are cells and columns are genes. Video data naturally requires a third-order tensor of pixels to represent it - rows, columns, and frames. Furthermore multi-omics datasets such as those including both scRNA-seq and scATAC-seq may require two or more matrices to be properly represented; one for each modality.

We could assume that each row of our matrix is an i.i.d. sample drawn from our model. However, independence is a strong and often incorrect assumption. If we wanted to make no independence assumptions, we could vectorize the dataset D and estimate Ψ in vec[ D ] ∼ N ( 0 , Ψ -1 ) . However, this produces intractably large Ψ , whose number of elements is quadratic in the product of the lengths of our dataset's axes.

Thankfully, tensors are highly structured, and we are often interested in the dependency structure 33

of each axis individually - i.e. the dependencies between samples or the dependencies between 34

features - rather than the dependencies between the elements of the tensor themselves. To model this, 35

we can represent Ψ as some deterministic combination of the axis-wise dependencies: vec[ D ] ∼ 36

Species

People

People

Metagenomics

Metabolomics

N ( 0 , ζ ( Ψ row , Ψ col ) -1 ) , for some function ζ . The strategy is to estimate Ψ row , Ψ col directly, without 37 computing the intractable ζ ( Ψ row , Ψ col ) -1 . While there are multiple choices for ζ , this paper 38 considers only the Kronecker sum. 39

40

41

42

43

44

45

46

47

48

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

1.1 Prior work

The Kronecker sum BiGraphical Lasso (BiGLasso) model was first considered by Kalaitzis et al. [14]. BiGLasso is the multi-axis analog to graphical lasso methods [10], which are used to estimate covariance matrices of data drawn from a multivariate Gaussian distribution. The Kronecker sum of two matrices, A ⊕ B , can be expressed in terms of Kronecker products: A ⊗ I + I ⊗ B . When the matrices A , B are adjacency matrices of graphs, the Kronecker sum has the interpretation as the Cartesian product of those graphs. This sum is one choice ζ to combine the per-axis precision matrices into the precision matrix of the vectorized dataset, vec[ D ] ∼ N ( 0 , ( Ψ row ⊕ Ψ col ) -1 ) .

Other choices for ζ have been considered, such as using the Kronecker product [23, 8], or the square of the Kronecker sum [24, 25]. Each method has its strengths; the benefits of a Kronecker sum structure are its interpretability as a graph product, stronger sparsity, and its allowance of inter-task transfer [14].

The original BiGLasso model was very slow to converge to a solution, in large part due to its nonoptimal space complexity of O ( n 2 p 2 ) . This prohibited its use on large datasets (measuring in a couple hundred samples and/or features). Numerous modifications have been made to the algorithm to improve its speed and achieve an optimal space complexity of O ( n 2 + p 2 ) , such as scBiGLasso [17], TeraLasso [12], and EiGLasso [27]. Of these, TeraLasso is notable in that it generalizes to an arbitrary number of axes, i.e. ζ ( Ψ 1 , ..., Ψ k ) = Ψ 1 ⊕ ... ⊕ Ψ k . TeraLasso and EiGLasso, the fastest prior work, both rely on computing an eigendecomposition every iteration.

All of these algorithms and models, including our own, rely on a normality assumption. We are most interested in the case of omics data, in which case a log-transform renders our dataset sufficiently Gaussian-like for our algorithm to achieve good performance. An overview of the use of GGMs in omics data is given by Altenbuchinger et al. [2].

1.2 Unmet need

Many datasets, especially those in multi-omics, are representable as a collection of matrices or tensors. 64 As a case study, we consider (a subset of) the Lifelines-DEEP dataset from Tigchelaar et al. [22], 65 which is summarized graphically in Figure 1. 66

In this dataset, two different modalities of data were gathered from the same people: counts of 67 microbial species found in their stools (metagenomics) and counts of metabolites found in their 68 blood plasma (metabolomics). While different matrices, each modality shares an axis. If we were to 69 estimate a graph of people on each modality independently, they would likely yield different graphs. 70 This is not ideal; if our aim is to estimate the true graph of conditional dependencies, there should be 71 only one resultant graph. To estimate it, we should be considering both modalities simultaneously. 72

Species

People

People

Metagenomics

Metabolomics

Elements

Structure

Cells from

Patient 1

Genes

Transcriptomics

Genes

Transcriptomics

(A)

(B)

Cells from

Patient N

One way to do this would be to concatenate the modalities, producing a matrix of people by 73 "species+metabolites". This could yield interesting results, if one is interested in connections between 74 individual species and a metabolite. However, it would increase the size of the output graph, which 75 grows quadratically in the length of the axis. Furthermore, it is not always possible or feasible; some 76 datasets may not be concatenatable. We visually demonstrate some cases where concatenation fails 77 in Figure 2. 78

79

1.3 Our contributions

We introduce a novel method to extend the use of Gaussian Graphical Models to multi-tensor datasets. 80 This extension is essential to model conditional dependencies in multimodal datasets such as those 81 frequently occurring in multi-omics. We present an efficient algorithm to estimate these conditional 82 dependencies. When restricted to the single-tensor case, our algorithm is much faster than previous 83 algorithms that estimated conditional dependency graphs for each axis, such as TeraLasso[12] and 84 EiGLasso[27]. 85

86

87

88

89

2 Methods

2.1 Notation

In prior work, a single-tensor dataset D is modelled as vec [ D ] ∼ N ( 0 , ( ⊕ ℓ Ψ ℓ ) -1 ) , also written as D ∼ N KS ( { Ψ ℓ } ℓ ) .

Our model considers multiple tensors, each with their own (potentially shared) axes. We aim to 90 estimate the precision matrices Ψ ℓ for each axis ℓ of each tensor D γ , indexed by γ ∈ N . To describe 91 that an axis ℓ is one of the axes of a tensor D γ , we will write ℓ ∈ γ . Some values will be indexed 92 by both an axis and a tensor; for consistency we will use subscripts to denote axes (typically ℓ ) and 93 superscripts to denote tensors (typically γ ). d γ ∀ will represent the number of elements in D γ , and 94 d ∀ = ∑ d γ . 95

An important concept is the Gram matrix S γ ℓ . In the single-tensor case, this is a sufficient statistic; all 96 prior work first computes these matrices as the first step in their algorithm. Let mat ℓ [ D γ ] represent 97 the "matricization" of D γ along axis ℓ , then S γ ℓ = mat ℓ [ D γ ] mat ℓ [ D γ ] T . The matricization of a 98 tensor picks one axis, ℓ , to index the rows, and flattens the rest out into columns. Note that for 99 a matrix M , mat columns [ M ] = M T . Rather than S γ ℓ , we consider the "effective Gram matrices" 100 S ℓ = ∑ γ | ℓ ∈ γ S γ ℓ , as these fulfill the role of the Gram matrices in the multi-tensor case. 101

102

2.2 The model

To properly handle sets of tensors, we propose modelling each tensor as being drawn independently 103 from a Kronecker-sum normal distribution. If the tensors share an axis ℓ , then they will still be drawn 104 independently - but their distributions will be parameterized by the same Ψ ℓ . For an arbitrary set of 105 tensors, the model is: 106

We call this model the "Gaussian multi-Graphical Model" (GmGM) as it extends Gaussian Graphical 107 Models to estimate multiple graphs from a set of tensors. In this paper, we will make the assumption 108 that no tensor in our set contains the same axis twice - notably, covariance matrices would violate 109 this assumption. Any tensor with a repeated axis would naturally be interpretable as a graph - such 110 datasets are rare, and if one already has a graph the need for an algorithm such as this is diminished. 111

As an example, we model the LifeLines-DEEP dataset D metagenomics and D metabolomics indepen112 dently as: 113

114

115

116

117

2.3 The algorithm

Here, we present an algorithm to compute the maximum likelihood estimate (MLE) jointly for all parameters Ψ ℓ of the GmGM. The general idea is to produce an analytic estimate for the eigenvectors of Ψ ℓ , and then iterate to solve for the eigenvalues; this is summed up graphically in Figure 3.

In the supplementary material, we derive the following: 118

From this, we can observe that the effective Gram matrices S ℓ form a set of sufficient statistics for 119 our distribution. Furthermore, the log-likelihood is the sum of log-likelihoods in the single-axis case, 120 thus preserving convexity of the loss function. 121

122

123

Theorem 1. Let V ℓ e ℓ V T ℓ be the eigendecomposition of S ℓ (where V ℓ ∈ R d ℓ × d ℓ and e ℓ ∈ R d ℓ × d ℓ is a diagonal matrix). Then V ℓ are the eigenvectors of the maximum likelihood estimate of Ψ ℓ .

Theorem 1 is critical to allowing efficient estimation of Ψ ℓ , as it not only allows us to extract the 124 computationally intensive eigendecomposition operation from the iterative portion of the algorithm, 125 but also reduces the number of parameters to be linear in the length of an axis. 126

To find the eigenvalues Λ ℓ of Ψ ℓ , we produce the second theorem: 127

128

129

130

131

132

133

134

135

T

T

Theorem 1

Compute Gram

Matrices

Eigendecompose

Eigen-recompose

Threshold

Theorem 2: Iterate until Convergence

For each dataset

, compute intermediate values

And use this to update our estimate of the eigenvalues

Theorem 2. Let { G γ ℓ } be matrices such that the expression ⊕ ℓ ∈ γ G γ ℓ is the best Frobenius-norm approximation of ( ⊕ ℓ ∈ γ Λ t ℓ ) -1 . Then, for a learning rate µ t , gradient descent can be performed with the update equation Λ t +1 ℓ = Λ t ℓ -µ t [ e ℓ -∑ γ | ℓ ∈ γ G γ ℓ ] . As Ψ ℓ is positive definite, µ t must be chosen to prevent Λ t ℓ from becoming negative.

While the definition of G γ ℓ is technical, it is analogous to the notion of the blockwise-trace from Kalaitzis et al. [14] and proj K from Greenewald, Zhou, and Hero III [12]. Proofs of Theorems 1 and 2, along with a method to compute G γ ℓ , are given in the supplementary material. Overall, our algorithm is described in the pseudocode at the top of the next page.

For regularization, one can choose to either keep the top p %of edges, or keep the top k edges per 136 vertex (for parameters p, k ). The incorporation of more advanced regularizers, such as Lasso, would 137 require an eigen-recomposition on each iteration, which would be much slower. As we demonstrate 138 empirically in Section 3, it is not necessary to use advanced regularizers to recover the graph structure 139 to the same precision as prior work. 140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

The GmGM algorithm

3 Results

Wetested our algorithm on synthetic data and five real-world datasets. Explanations of data generation, collection, preprocessing, and regularization are given in the supplementary material.

3.1 Synthetic Data

We verified that our algorithm was indeed faster on matrix-variate data compared to prior work (Figure 4) on our computer (Ubuntu 20.04 with Intel Core i7 Processor and 8GB RAM). Our results on matrix data are encouraging - extrapolating the runtimes, datasets up to size 16,000 by 16,000 could have their graphs estimated in less than an hour. Larger datasets would require more than 6GB of memory for our algorithm to run, pushing the limits of RAM. Our algorithm was not significantly faster on higher-order tensor data (see the supplementary material). This is due to the complexity of computing the Gram matrices, which grows exponentially with the number of axes.

In addition to these speed improvements, we show that we perform equivalently to state-of-the-art on matrix data (Figure 5a). On higher-order tensor data, we are outperformed by TeraLasso, which is able to achieve near-perfect recovery of the graphs. We believe this is due to our algorithm's use of thresholding rather than a more advanced regularization technique. Since our speed gains are not significant relative to TeraLasso, on higher-order tensor data without shared axes one should prefer TeraLasso to GmGM. Finally, we demonstrate that taking into account shared axes does indeed improve performance (see blue line, Figure 5b). Prior work could not take this into account.

3.2 Real Data

We tested our method on various real datasets. These include two video datasets (COIL-20 [19] and EchoNet-Dynamic [20]), a transcriptomics dataset (E-MTAB-2805 [5]), and two multi-omics datasets (LifeLines-DEEP [22] and a 10x Genomics dataset [1]).

(a)

(b)

(a)

(b)

(a)

(b)

MouseEmbryoStemCells

EcnocaraiogramFrarmes

-0.1

0.0

0.1

0.2

100

1000

10000

Number of Edges

Assortativities

linetype

GmGM

ziln

Phylum

Class

Order

Family

Genus

(a)

-0.1

0.0

0.1

0.2

0.3

100

1000

10000

Number of Edges

Assortativities

linetype

GmGM

ziln

Phylum

Class

Order

Family

Genus

(b)

The E-MTAB-2805 dataset consists of transcriptomics data for individual cells split into three groups 163 by their stage in the cell cycle (G, S, and G2/M). If our estimated precision matrices had a 3x3 164 block-diagonal structure, this would indicate that it had recreated this grouping. This is not what we 165 see, but we do see a 3x3 block matrix structure (Figure 6a). We found that cells in the DNA synthesis 166 stage (S) had few connections between them, and that there were many connections between the G1 167 and G2/M stages. This result is biologically plausible, as cells in the synthesis stage are the most 168 variable. 169

170

171

172

173

174

The results on EchoNet-Dynamic (Figure 6b) are much more encouraging, as we would expect a periodic structure due to the beating of the heart. A precision matrix with repeating diagonals is what we would expect to see in this case, which is what our algorithm produces. In the supplementary material, we further verify that this corresponds to a heartbeat by using the repetition to accurately predict the opening of the mitral valve in the video.

175

176

177

178

179

180

181

182

183

184

185

186

187

188

The duck video in the COIL-20 dataset was considered in the original BiGLasso paper [14], in which they showed that their algorithm could recover the ordering of the frames of the video. To do this they had to heavily downsample the image (to a 9x9 image with half the frames), and flatten the rows and frames into a single axis. Due to the speed improvements of our algorithm, and its ability to handle tensor-variate data, we were able to run our algorithm on the raw, unprocessed data and achieve a similar result in negligible time. Specifically, the reconstruction of the frames had an accuracy of 99%.

Prior work by Prost, Gazut, and Brüls [21] used assortativity to assess their validity of the species graph estimated by their model on the LifeLines-DEEP metagenomics dataset. Assortativity represents the tendency of related species to cluster together in the graph. A random graph would have an assortativity of zero, but we would expect moderate assortativity in the true network as similar species may fulfill similar roles in the gut microbiome. Our assortativity is comparable to prior work (Figure 7). We also found that our graphs were more robust to noise than prior work; we analyze this in the supplementary material.

Finally, we tested our approach on a 10x Genomics single-cell (RNA+ATAC) dataset taken from a 189 B Cell lymphoma tumour. We demonstrate that the clusters we find (using Louvain clustering[3]) 190 on the graph remain visually cohesive when projected into lower-dimensional space by UMAP[18] 191 (Figure 8). In particular, the disconnected 'islands' in UMAP correspond to their own cluster on the 192 graph as well. As these island-clusters were arrived at independently through two methods, UMAP 193 and our algorithm, it increases our confidence in the validity of the clustering. In the supplementary 194

(a)

(b)

material, we verify that these clusters do represent distinct groups via a GO term enrichment analysis. 195 Our overall approach has been implemented in Python. All of the code to run the algorithm and 196 recreate the experiments has been made publicly available on GitHub; https://github.com/NeurIPS197 GmGM-Paper/GmGM. 198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

4 Limitations

Our method uses thresholding rather than more sophisticated regularizers. However, there is no fundamental barrier preventing our algorithm from allowing regularizers at the cost of an eigenrecomposition per iteration. This would increase the asymptotic complexity of the iterative portion of our algorithm, making it questionable whether any gains in precision would be worth the loss in efficiency.

Our method assumes that no tensor has a repeated axis (i.e. a matrix of people by people rather than people by species). If there is a repeated axis, one can no longer analytically find the eigenvectors of the MLE, at least by our methods. This is not a substantial issue, as such datasets are uncommon and already represent graphs. Rather than extending the algorithm to work with repeated-axis tensors, it would be more fruitful to extend it to work with priors.

When considering multi-tensor datasets, it may be the case that two axes only partially overlap. For example, the full LifeLines-DEEP dataset contains a second (follow-up) metagenomics dataset for a third of the study participants; two thirds of the patients are missing from this dataset. We do not make an attempt to handle this type of missing data, even though missing data shows up in many applications. The lack of ability to handle missing data is a major limitation of our algorithm. It is nontrivial to extend the algorithm to handle this case, as it renders Theorem 1 ineffective and hence removes the speed advantage we attained. Prior work has not addressed this problem, as it only exists in multi-tensor datasets and we are the first to consider this case.

5 Conclusion

We have created a novel model, GmGM, which successfully generalizes Gaussian graphical models to the common scenario of multi-tensor datasets. Furthermore, we demonstrated that our algorithm is significantly faster than prior work focusing on Gaussian tensor-graphical models such as EiGLasso and TeraLasso while still preserving state-of-the-art performance. These speed improvements allow tensor-graphical models to be applied to datasets with axes of length in the thousands. Finally, we demonstrated the application of our algorithm on five real-world datasets to prove its efficacy.

225

References

10x Genomics. Flash-Frozen Lymph Node with B Cell Lymphoma (14k sorted nuclei) . en. May 226 2021. URL: https://www.10xgenomics.com/resources/datasets/fresh-frozen227 lymph-node-with-b-cell-lymphoma-14-k-sorted-nuclei-1-standard-2-0-0 228 (visited on 05/10/2023). 229

230

231

232

233

234

235

Michael Altenbuchinger et al. 'Gaussian and Mixed Graphical Models as (multi-)omics data analysis tools'. en. In: Biochimica et Biophysica Acta (BBA) Gene Regulatory Mechanisms . Transcriptional Profiles and Regulatory Gene Networks 1863.6 (June 2020), p. 194418. ISSN: 1874-9399. DOI: 10.1016/j.bbagrm.2019.194418 . URL: https: //www.sciencedirect.com/science/article/pii/S187493991930224X (visited on 02/24/2023).

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

270

271

Vincent D. Blondel et al. 'Fast unfolding of communities in large networks'. In: Journal of Statistical Mechanics: Theory and Experiment 2008.10 (Oct. 2008). arXiv:0803.0476 [condmat, physics:physics], P10008. ISSN: 1742-5468. DOI: 10.1088/1742-5468/2008/10/ P10008 . URL: http://arxiv.org/abs/0803.0476 (visited on 05/16/2023).

Danila Bredikhin, Ilia Kats, and Oliver Stegle. 'MUON: multimodal omics analysis framework'. In: Genome Biology 23.1 (Feb. 2022), p. 42. ISSN: 1474-760X. DOI: 10.1186/s13059021-02577-8 . URL: https://doi.org/10.1186/s13059-021-02577-8 (visited on 05/16/2023).

Florian Buettner et al. 'Computational analysis of cell-to-cell heterogeneity in single-cell RNA-sequencing data reveals hidden subpopulations of cells'. en. In: Nature Biotechnology 33.2 (Feb. 2015). Number: 2 Publisher: Nature Publishing Group, pp. 155-160. ISSN: 15461696. DOI: 10.1038/nbt.3102 . URL: https://www.nature.com/articles/nbt.3102 (visited on 05/10/2023).

Thomas A. Caswell et al. matplotlib/matplotlib: REL: v3.7.1 . Mar. 2023. DOI: 10.5281/ zenodo.7697899 . URL: https://zenodo.org/record/7697899 (visited on 05/16/2023).

Gabor Csardi and Tamas Nepusz. 'The Igraph Software Package for Complex Network Research'. In: InterJournal Complex Systems (Nov. 2005), p. 1695.

Andy Dahl et al. Network inference in matrix-variate Gaussian models with non-independent noise . arXiv:1312.1622 [stat]. Dec. 2013. DOI: 10.48550/arXiv.1312.1622 . URL: http: //arxiv.org/abs/1312.1622 (visited on 02/27/2023).

Pan Du, Warren A. Kibbe, and Simon M. Lin. 'Improved peak detection in mass spectrum by incorporating continuous wavelet transform-based pattern matching'. In: Bioinformatics 22.17 (Sept. 2006), pp. 2059-2065. ISSN: 1367-4803. DOI: 10.1093/bioinformatics/btl355 . URL: https://doi.org/10.1093/bioinformatics/btl355 (visited on 05/11/2023).

Jerome Friedman, Trevor Hastie, and Robert Tibshirani. 'Sparse inverse covariance estimation with the graphical lasso'. In: Biostatistics 9.3 (July 2008), pp. 432-441. ISSN: 14654644. DOI: 10.1093/biostatistics/kxm045 . URL: https://doi.org/10.1093/ biostatistics/kxm045 (visited on 05/09/2023).

Thomas M. J. Fruchterman and Edward M. Reingold. 'Graph drawing by forcedirected placement'. en. In: Software: Practice and Experience 21.11 (1991). _eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1002/spe.4380211102, pp. 1129-1164. ISSN: 1097024X. DOI: 10.1002/spe.4380211102 . URL: https://onlinelibrary.wiley.com/ doi/abs/10.1002/spe.4380211102 (visited on 05/16/2023).

Kristjan Greenewald, Shuheng Zhou, and Alfred Hero III. Tensor Graphical Lasso (TeraLasso) . arXiv:1705.03983 [stat]. Sept. 2019. URL: http://arxiv.org/abs/1705.03983 (visited on 02/24/2023).

Charles R. Harris et al. 'Array programming with NumPy'. en. In: Nature 585.7825 (Sept. 272 2020). Number: 7825 Publisher: Nature Publishing Group, pp. 357-362. ISSN: 1476-4687. DOI: 273 10.1038/s41586-020-2649-2 . URL: https://www.nature.com/articles/s41586274 020-2649-2 (visited on 05/16/2023). 275

Alfredo Kalaitzis et al. 'The Bigraphical Lasso'. en. In: Proceedings of the 30th International 276 Conference on Machine Learning . ISSN: 1938-7228. PMLR, May 2013, pp. 1229-1237. URL: 277 https://proceedings.mlr.press/v28/kalaitzis13.html (visited on 02/24/2023). 278

Tamara Kolda. Multilinear operators for higher-order decompositions. en. Tech. rep. 279 SAND2006-2081, 923081. Apr. 2006, SAND2006-2081, 923081. DOI: 10.2172/923081 . 280 URL: https://www.osti.gov/servlets/purl/923081/ (visited on 05/05/2023). 281

Tamara G. Kolda and Brett W. Bader. 'Tensor Decompositions and Applications'. en. In: 282 SIAM Review 51.3 (Aug. 2009), pp. 455-500. ISSN: 0036-1445, 1095-7200. DOI: 10.1137/ 283 07070111X . URL: http://epubs.siam.org/doi/10.1137/07070111X (visited on 284 02/26/2023). 285

Sijia Li et al. Scalable Bigraphical Lasso: Two-way Sparse Network Inference for Count 286 Data . arXiv:2203.07912 [cs, stat]. Mar. 2022. URL: http://arxiv.org/abs/2203.07912 287 (visited on 02/24/2023). 288

Leland McInnes, John Healy, and James Melville. UMAP: Uniform Manifold Approximation 289 and Projection for Dimension Reduction . arXiv:1802.03426 [cs, stat]. Sept. 2020. DOI: 10. 290 48550/arXiv.1802.03426 . URL: http://arxiv.org/abs/1802.03426 (visited on 291 05/10/2023). 292

Sameer A Nene, Shree K Nayar, and Hiroshi Murase. 'Columbia Object Image Library 293 (COIL-20)'. en. In: (). 294

David Ouyang et al. 'Video-based AI for beat-to-beat assessment of cardiac function'. en. In: 295 Nature 580.7802 (Apr. 2020). Number: 7802 Publisher: Nature Publishing Group, pp. 252-256. 296 ISSN: 1476-4687. DOI: 10.1038/s41586-020-2145-8 . URL: https://www.nature.com/ 297 articles/s41586-020-2145-8 (visited on 05/10/2023). 298

299

300

301

302

303

Vincent Prost, Stéphane Gazut, and Thomas Brüls. 'A zero inflated log-normal model for inference of sparse microbial association networks'. en. In: PLOS Computational Biology 17.6 (June 2021). Publisher: Public Library of Science, e1009089. ISSN: 1553-7358. DOI: 10. 1371/journal.pcbi.1009089 . URL: https://journals.plos.org/ploscompbiol/ article?id=10.1371/journal.pcbi.1009089 (visited on 02/24/2023).

304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

Ettje F. Tigchelaar et al. 'Cohort profile: LifeLines DEEP, a prospective, general population cohort study in the northern Netherlands: study design and baseline characteristics'. en. In: BMJ Open 5.8 (Aug. 2015). Publisher: British Medical Journal Publishing Group Section: Epidemiology, e006772. ISSN: 2044-6055, 2044-6055. DOI: 10.1136/bmjopen-2014-006772 . URL: https://bmjopen.bmj.com/content/5/8/e006772 (visited on 04/30/2023).

Theodoros Tsiligkaridis, Alfred O. Hero III, and Shuheng Zhou. 'Convergence Properties of Kronecker Graphical Lasso Algorithms'. In: IEEE Transactions on Signal Processing 61.7 (Apr. 2013). arXiv:1204.0585 [stat], pp. 1743-1755. ISSN: 1053-587X, 1941-0476. DOI: 10.1109/TSP.2013.2240157 . URL: http://arxiv.org/abs/1204.0585 (visited on 02/27/2023).

Yu Wang and Alfred Hero. 'SG-PALM: a Fast Physically Interpretable Tensor Graphical Model'. en. In: Proceedings of the 38th International Conference on Machine Learning . ISSN: 2640-3498. PMLR, July 2021, pp. 10783-10793. URL: https://proceedings.mlr. press/v139/wang21k.html (visited on 02/27/2023).

Yu Wang, Byoungwook Jang, and Alfred Hero. The Sylvester Graphical Lasso (SyGlasso) . arXiv:2002.00288 [cs, stat]. Feb. 2020. DOI: 10.48550/arXiv.2002.00288 . URL: http: //arxiv.org/abs/2002.00288 (visited on 02/27/2023).

F. Alexander Wolf, Philipp Angerer, and Fabian J. Theis. 'SCANPY: large-scale single-cell 321 gene expression data analysis'. In: Genome Biology 19.1 (Feb. 2018), p. 15. ISSN: 1474-760X. 322 DOI: 10.1186/s13059-017-1382-0 . URL: https://doi.org/10.1186/s13059-017323 1382-0 (visited on 05/16/2023). 324

Jun Ho Yoon and Seyoung Kim. 'EiGLasso: Scalable Estimation of Cartesian Product of Sparse 325 Inverse Covariance Matrices'. en. In: Proceedings of the 36th Conference on Uncertainty 326 in Artificial Intelligence (UAI) . ISSN: 2640-3498. PMLR, Aug. 2020, pp. 1248-1257. URL: 327 https://proceedings.mlr.press/v124/ho-yoon20a.html (visited on 02/24/2023). 328

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces the Gaussian multi-Graphical Model (GmGM) as a novel method to construct sparse graph representations of matrix- and tensor-variate data. It simultaneously learns the representation across several tensors that share axes. The authors demonstrate that GmGM outperforms previous methods in terms of speed when applied to matrix data.\n\nSTRENGTHS:\n1.\tGmGM extends the application of Gaussian Graphical Models to multi-tensor datasets, presenting a novel approach in the field.\n\n2.\tGmGM exhibits significantly improved speed compared to previous methods when dealing with matrix data.\n\n3.\tThe results of GmGM on five real datasets are well explained.\n\n4.\tEspecially, I appreciate the comprehensive discussion provided in this paper. The authors present cases where the results are excellent, as well as cases where the results are not as impressive, such as the performance on higher-order tensor data (fig 4b) and the E-MTAB-2805 dataset (fig 6a). This in-depth analysis helps readers gain a better understanding of the method and be aware of the situations in which it should be employed.\n\nWEAKNESSES:\nOne major concern I have relates to the evaluation. Although the authors present many intriguing findings on the datasets, it would be beneficial to include some more quantitative analysis.\n\nQUESTIONS:\n1.\tCould the authors show the results on the COIL-20 dataset and provide quantitative comparisons with baselines in terms of both efficiency and accuracy?\n2.\tThe two multi-omics datasets, LifeLines-DEEP and 10x, are analyzed from different perspectives. It would be advantageous if the authors could also present UMAP consistency analysis results for the LifeLines-DEEP dataset. Additionally, conducting quantitative comparisons with baselines on the 10x dataset would also be informative for readers.\n\nLIMITATIONS:\nYes. It is well discussed in the study.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-16"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose Gaussian multi-Graphical Model (GmGM), a novel approach to constructing sparse graph representations of matrix- and tensor-variate data. It stands out from previous models by learning representations across multiple tensors that share axes simultaneously, a feature crucial for analyzing multimodal datasets, particularly in multi-omics scenarios. The GmGM algorithm utilizes a single eigendecomposition per axis, which results in a significant speedup over previous models. This efficiency enables the application of the methodology on large multi-modal datasets, such as single-cell multi-omics data, a task that was challenging with previous approaches.\n\nSTRENGTHS:\n1. Fair and Interesting Motivation: \nThe paper's motivation on model multi-tensor decomposition with shared axis is rooted in the real-world need for handling multi-omics scenarios, which often involve multi-tensor data with shared axes. The GmGM is introduced as a solution, addressing a significant gap in existing data analysis methodologies and providing a fair and interesting motivation for the study.\n\n2. Reasonable solution and impressive improvements in Efficiency\nThe GmGM model stands out for its impressive efficiency improvements, achieved through the use of the KS decomposition of the precision matrix and transiting it to the eigen-decomposition over each dim. This approach results in a substantial speedup over previous models, enabling the handling of large multi-modal datasets, This efficiency, coupled with the model's ability to maintain state-of-the-art performance, underscores the strength of the paper.\n\nWEAKNESSES:\n1. **Limited Technical Contribution**\n\nWhile the problem setting proposed in the paper is reasonable, the algorithm's strict assumptions about data integrity (no missing data) and quality (no noise) somewhat limit its potential for broader application. The authors are encouraged to consider relaxing these assumptions or proposing strategies to handle missing data and noise, which are common issues in real-world datasets. Addressing these issues could significantly enhance the model's practical utility and broaden its applicability.\n\n2. **Improvements Needed in Representation and Flow**\n\nThe paper could benefit from substantial improvements in its representation and flow. The omission of important concepts and content significantly hinders reader comprehension. Some sentences appear casual and can lead to confusion. The overall logical flow of the paper is not clear, making it difficult to follow. This is particularly evident in the following areas:\n\n   - Concepts such as the Kronecker product and Gram matrix are not clearly introduced.\n   - Many notations and their subscripts and superscripts in the algorithm table are not clearly defined.\n   - The task setting and metric definition in the experimental section are vague, reducing the persuasiveness of the validation part.\n\nOverall, the authors are encouraged to make a concerted effort to reorganize and polish the paper's presentation, improve the flow, and highlight the key points of the work and problem. This could significantly enhance the readability and impact of the paper.\n\nQUESTIONS:\nSee weakness parts\n\nLIMITATIONS:\nSee weakness parts", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes the Gaussian multi-Graphical Model, a novel method to extend the use of Gaussian Graphical Models to multi-tensor datasets. It generalizes Gaussian graphical models to the common scenario of multi-tensor datasets. For the single-tensor case,  the proposed algorithm is faster than prior work while still preserving state-of-the-art performance.\n\nSTRENGTHS:\nThe paper considers an interesting and still challenging topic, extending conventional Gaussian graphical models (GGM) for complex systems like multi-modal data models. The paper has been generally well-written and the problem has been clearly defined. Indeed, the theoretical parts that extend the GGM to multi-tensor datasets have proper quality.  This algorithm is significantly faster on lower-order tensor data (reported for the synthetic data sets) and its efficacy is slightly better in the real-world data sets.\n\nWEAKNESSES:\n- Some parts of the paper should be checked again. For instance, line 52 starts to explain the computational costs of the state-of-the-art methods. The parameters n and p have not been defined before. It seems it uses the defined parameter in the main reference paper (Kalaitzis et. al. 2013), where n and p are the numbers of observations and features, respectively. Indeed, the computational costs of the other baselines need a piece of clarification. For instance, O(n^2 * p^2) in BIGLasso represents the number of non-zeros in the Kronecker-sum (KS) structure. It would be better if the authors consider the full cost of the algorithm for the proposed method and available baselines.\n\n- The paper models each tensor as being drawn independently\n from a Kronecker-sum normal distribution. It makes sense to see this assumption reduces the computational cost at least in small-order data sets. However, it does not describe how this strong assumption still preserves state-of-the-art performance.\n\n- As has been reported in the paper, the proposed solution can not improve the complexity of higher-order tensor data sets (fig. 4b). Indeed, its performance can not significantly outperform the other baseline (Fig. 5a). By decreasing the sparsity, the performance of the model suffers and it seems it works properly only on high sparse graphs (Fig 7).\n\nQUESTIONS:\n- See the Weaknesses part.\n\n-  A question about the scope of the proposed model: It has been designed for multi-modal data sets. Another problem that has a similar structure is distributed learning when the entire data set is divided into several partition and each partition provides local inference. The partitions are clusters of multi-dimensional data sets, and the features are the same for all clusters. Can the proposed method be used for estimating the conditional dependencies between the features and also dependencies between local partitions?\n\nLIMITATIONS:\nThe authors addressed the limitation of the work in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-01"}