MoReL: Multi-omics Relational Learning
Authors: Arman Hasanzadeh
Ehsan Hajiramezanali
Nick Duffield
Xiaoning Qian
Venue: ICLR 2022 Poster (2022)
Keywords: relational learning
data integration
multi-view learning
Bayesian generative model

Abstract: Multi-omics data analysis has the potential to discover hidden molecular interactions, revealing potential regulatory and/or signal transduction pathways for cellular processes of interest when studying life and disease systems. One of critical challenges when dealing with real-world multi-omics data is that they may manifest heterogeneous structures and data quality as often existing data may be collected from different subjects under different conditions for each type of omics data. We propose a novel deep Bayesian generative model to efficiently infer a multi-partite graph encoding molecular interactions across such heterogeneous views, using a fused Gromov-Wasserstein (FGW) regularization between latent representations of corresponding views for integrative analysis. With such an optimal transport regularization in the deep Bayesian generative model, it not only allows incorporating view-specific side information, either with graph-structured or unstructured data in different views, but also increases the model flexibility with the distribution-based regularization. This allows efficient alignment of heterogeneous latent variable distributions to derive reliable interaction predictions compared to the existing point-based graph embedding methods. Our experiments on several real-world datasets demonstrate enhanced performance of MoReL in inferring meaningful interactions compared to existing baselines.

Published as a conference paper at ICLR 2022
MOREL: MULTI-OMICS RELATIONAL LEARNING
Arman Hasanzadeh, Ehsan Hajiramezanali, Nick Duffield & Xiaoning Qian Department of Electrical and Computer Engineering. Texas A&M University {a Armannnhe rm,anih,,a ehsanr, duffieldng xqi an @t.mu.
ABSTRACT
Multi-omics data analysis has the potential to discover hidden molecular inter- actions, revealing potential regulatory and/or signal transduction pathways for cellular processes of interest when studying life and dísease systems. One of critical challenges when dealing with real-world multi-omics data is that they may manifest heterogeneous structures and data quality as often existing data may be collected from different subjects under different conditions for each type of omics data. We propose a novel deep Bayesian generative model to efficiently infer a multi-partite graph that encodes molecular interactions across such heterogeneous views, using a fused Gromov- Wasserstein (FGW) regularization between latent representations of corresponding views for integrative analysis. With such an optimal transport regularization in the deep Bayesian generative model, it not only allows incorporating view- new-speciiic side information, either with graph strrutuuud unstructured data in different views, but also increases the model flexibility with the distribution-ban based regularization. This allows efficient alignment of heterogeneous latent variable distributions to derive reliable interaction predictions compared to the existing point-based graph embedding methods. Our experiments several real-world datasets demonstrate the enhanced performance of MoReL in inferring meaningful interactions compared to existing baselines.
INTRODUCTION
Multi-view learning tries to fully leverage the information from multiple sources (i.e. different types of omics data in molecular biology) and represents them in shared embedding space, which beneficial for many downstream tasks with limited number of training samples. In biomedical applications, the shared embedding space also enables better understanding of the underlying biological mechanisms by discovering interactions between different types of molecules, which is our focus in this paper.
Existing multi-omic data integration methods are limited in their applicability. First, most of them attempt to derive low-dimeniional embeddings of the input samples and are not designed to infer a multi-partite graph that encodes the interactions across views. unsupervised settings, matrix factorization based methods, such as Bayesian Canonical Correlation Analysis (BCCA) (Klami et al.) 2013) and Multi-Omics Factor Analysis (MOFA) (Argelaguet et al., 2018), can achieve the similar goal of cross-view relational learning but often through two-step procedures, in which the factor loading parameters are used for downstream interaction analyses across views. Second, a very recent relational inference for multi-view data integration, BayRel (Hajiramezanali et al.) 2020). is built on three strict assumptions, which may limit its practical application, including in multi-omics data integration: A graph of dependency between features of each view is available; 2) The input dataset is complete on all views with no missing samples; 3) The samples in different views are well-paired. While the first limitation might be solved by learning a graph using an ad-hoe technique. the last two issues are common in many multi-omic data integration problems. Integrated samples commonly have one or more views with various missing patterns. This is mostly due limitations of experimental designs or compositions from different data platforms. In addition, data might be collected in different laboratories or the sample IDs are not available due to patient identification or privacy/security concerns, leading to unpaired datasets. Apart from these, we might not have access to a priori graph structured data in some view(s) as the nature of data might not be structured, or we only have incomplete or very noisy prior knowledge. For such multi-omics data, leaving out such a view may lose some complementary information while enforcing graph structures may lead to degraded performances.

---

Published as a conference paper at ICLR 2022
In this work, we propose new Multi- Relational Learning method, MoReL, based on the fused Gromov- in (FGW) regularization, mitigating the dependency of multi-view learning on the aforementioned two assumptions. The proposed method contains four major contributions: 1) MoRel. provides a new Bayesian multi-omics relational learning framework with efficient variational inference and is able to exploit non-linear transformations of data by leveraging deep learning models for either unstructured or graph-structured data; 2) MoReL learns multi-partit, graph across different features from multiple views using FGW-based decoder, facilitating meaningful biological knowledge discovery from integrative multi-omic data analysis while accounting for arbitrarily permutation and/or transformation caused by processing features with different deep functions across the views; 3) MoReL can flexibly integrate both structured and unstructured heterogeneous views in one framework, in which only confident constraints need to imposed to improve the model performance; 4) MoRel. is able to integrate multiple views with unpaired samples and/or arbitrary sample-missing patterns.
RELATED WORKS
Optimal transport. There have been extensive efforts to utilize Gromov- Wasserstein (GW) dis-
crepancy to solve the alignment problems in shape and object matching (Mémoli [2009] 2011). A
similar attempt has been made recently to investigate its potential for more diverse applications,
such as aligning vocabulary sets between different languages (Alvarez-Melis & Jaakkola 2018).
and graph matching (Chowdhury & Mémoli, 2019 Vayer et al. [0118) Xu et al. 2011bb) Peyré
al. (016)) have proposed a fast Sinkhorn projection-basee algorithm (Cuturi 2013) to compute the retuoy-regularized GW distance. Following this direction, Xu et al. (2019b) have replaced the entropy regularizer with a Bregman proximal term. To further reduce the computational complexity. the recursive GW distance (Xu et al. (0199) and the sliced GW distance (Vayer et al. 2019) have been proposed. In Bunne et al. (2019), a pair of generative models are learned for incomparable spaces by defining an adversarial objective function based on the GW discrepancy. It imposes an orthogonal assumption on the transformation between the sample and its latent space. However, can not incorporate the graph structured data. Similar to our model in this paper, Vayer et al. (2018a) and Xu et al. 2020) have proposed to impose the fused GW regularization in their objective functions by combining GW and Wasserstein discrepancies.
Graph CCA (gCCA). To utilize a priori known information about geometry of the samples, gCCA methods (Chen al., 2019 2018) have been proposed to construct a dependency graph between samples and directly impose it into a regularizer. Similar to classical CCA, gCCA learns an unstruc- tured shared latent representation. Unlike our MoReL., though, they can neither take advantage of the dependency graph between features, nor explicitly model relational dependency between features across views. Therefore, they rely on ad-hoc prot-procsssing procedures to infer inter-relations.
Graph representation learning.
Graph neural network architectures have been shown to be ef- fective for link prediction (Hamilton al.] 2017 Kipf & Welling 2016 Hasanzadeh et al. 2019
Hajiramezanali et al. 2019 Hasanzadeh et al. (0200) as well as matrix completion for recommender
systems (Berg et al. 2017 Monti et al.] 2017 7 Kalofolias et al., 2014 Ma et al., 2011). The first group of models is dealing with a single graph and is not able to deal with heterogeneous graphs, with multiple types of nodes and edges, and node attributes (Zhang et al. 2019). The second group utilizes the known item-item and user-user relationships and their attributes to complete the user-item rating matrix. However, they rely on two strict assumptions: 1) the inter-relation matrix is partially observed; and 2) both views have structured information. The proposed MoReL achieves robust multi-view learning without these assumptions, making it more practical in multi-omics data integration.
PRELIMINARIES
3.1 WASSERSTEIN DISTANCE
Wasserstein distance (WD) quantifies the geometric discrepancy between two probability distributions
by measuring the minimal amount of "work" needed to move all the mass contained in one distribution onto the other (Solomon et al. 2015). More specifically, given two probability measures A E P(X)
and A P(Y), and a transportation cost e X x Y + R4, WD is the solution the following
2

---

Published as a conference paper at ICLR 2022
DEC
Hlg
Z
pat
FGW
A
992
Z
c(u)
H.u
Figure I: Graphical illustration of MoReL's generative flow with structured and unstructured views. DEC stand for decoder. The rest of variables and abbreviations are defined in the manuscript.
optimization problem:
inf =El((XX)) / o(x, y) dx y); F(x,y)>> (c(x)) y)l
inf =EI(XxF)
where и is the transport map and I1 x Y) = f E P(xx (x, A [x y) dx
(y)] is the set of all admissible couplings. Assuming that the probability distributions are discrete, with probability mass functions Σ 0;0x, and b, WD optimization could be simplified as follows:
m Tú, Yx),
Dw A)
min Tell(a, b)
j=1
where Tig is an element of the transport matrix T whose row-wise and column-wise sums equal to [a.]=. and [6,]5=1, respectively.
3.2 GROMOV- WASSERSTEIN DISTANCE
-esser-Wsssestteinn distance (GWD) has been proposed as natural extension of WD when meaningful transportation cost between the distributions cannot be defined. For example, when two distributions are defined in Euclidean spaces with different dimensions or more generally when X and Y are unaligned, i.e. when their features are not in correspondence (Vayer et al.) 2019). Instead of measuring inter-domain distances, GWD measures the distance between pairs of samples in one domain and compares it to those in the other domain. More specifically, given two probability measures A P(X) and A P(Y), well two domain-specifee transportation costs ((x) X x x + R4 and c()) Y X Y + R2,, GWD is the solution to the following optimization
problem:
/ | / L(xx X'') d y) dr (x', y'),
inf x
F(x,))~x,(xx [xxx x y. y')l
inf =ET((xx))
where L(x, y. y') =|| -x) y') 1l. is the transport map, and x)
P(X 8) [ yx) dy A(x), (xx dx ( is the set of all admissible couplings. Likewise, this be derived for discrete distributions with probability mass functions > ax,x, and ΣØ bydy, follows:
m
Dew (A, A)
min TelI(a, 6)
Σ
Σ
Toi Te. L(x,, X Ye,y
(,==1
where Tiy is an element of transport matrix T whose row-wise and column- sums equal to [a:]i= and [b,] respectively.
METHOD
4.1 PROBLEM FORMULATION AND NOTATIONS
We propose a novel hierarchical generative model for multi-omics data integration that incorporates view-specific structure information when it is available. Given observations from structured and
3

---

Published as a conference paper at ICLR 2022
unstructured views, our model, Multi-omics Relational Learning (MoReL), aims to infer the inter- relations among entities, ie. features, across all of the views. More specifically, assume that multiple views, V. of data are given. Without loss of generality, provided as graph, is available for some of the views V, C V, and the remaining views V. = V V. unstructured. We note that every structure could be represented a graph. For example, image and sequential data could be represented over grid and directed path graphs, respectively.
assume that the structure information,
We represent the set of graphs for structured views by 6 = hee,, and their adjacency matrices by 21, = {A (r) Joe, We also define X, = (X( joev, as the set node attributes for structured views, and Xr ) number of nodes in structured views and number of features for unstructured views. MoReL infers the interactions among the nodes in 6g and features X. We represent these inter-relations by multi-partite graph with Evey N₂ nodes and a multi-adjacency tensor A = {A where A (ev') is the N, Nor bi-adjacency matrix between views v and v'.
as the set data for unstructured views. Moreover, N, denotes the
(vv")) Jo,
4.2 MoREL GENERATIVE MODEL
We define a hierarchical Bayesian model for MoReL with three sets of latent variables: 1) H. H. U Hh = (H(*) /eV,,v, which captures the (hidden) structural information; 2) A. which encodes the interaction among features across views; and 3) 2 2, U 2,, = ) which summarizes the feature/atrribute specific information. The joint probability of observations and latent variables factorizes as follows:
Po (o X, 21 H. A, 2)
(2)
pe. (Ne Z.) po, (X, 2.) Pa, (2l, H.,) pe. (2 H. AA (A (A]H)p((H)
Figure ! depicts the generative model of MoReL with structured and unstructured views. In the following subsections, we define different parts of the generative and inference model.
4.2.1 OPTIMAL TRANSPORT FOR MULTI-PARTITE GRAPH DECODER
In this subsection, we define generative distribution of the multi-adjacency tensor, A. We note that inferring A is the main goal of our model. Given the structural latent variables H. introduce fused Gromov- Wasserstein (FGW) distance based mapping to generate AA. FGW refers to distance metrics defined by combining WD and GWD, which has been proposed compare structured distributions (Vayer et al. 2018b Chen et al., 2020). Considering graphs with node attributes as structured distributions, WD compares node distributions in graphs (i.e, node similarity), GWD measures the distance between pairs of nodes in one graph and compares it to those in the other (i.e., edge/path similarity).
FGW distance. Given two structured probability distributions, A E P(X) and A E P(Y), FGW defined as follows:
DFaw(A, A, A) a (w A) + 3 Dow(A, A)
inf To EXI((
E(x, (x0)) (x, y)]
(3)
+ +
inf Tau EI((X(
[(x, x), y' xx) y') ll].
where a, E [0, 1] are scalar hyper-parameters, IT(X Y) is the set of all admissible couplings between A and A, and c(XY) c(x), and c00 are corresponding transportation cost functions. Drew can be further simplified by choosing T.e to be equal to Tgw (Chen et al. 2020).
Relational learning via FGW. We are interested in aligning the nodes/features in every pair of views, i.c. (u, o'). Hence, we will have FGW distance based decoder for every pair of views, in which each view independently belongs to either structured or unstructured views, i.e. (v, o') E V. To that end, we first define the transportation cost functions c(vn') and c(*), and then approximate Drow. We define the (inter- cost function for the first term of FGW, i.c. Dw, as follows:
(vo')
(e")) = - H ())
v, v E V,
(4)
4

---

Published as a conference paper at ICLR 2022
where denotes the sigmoid function, and H represents the structural latent variable of node/feature i in view v. To calculate the Dew, we define two different transportation costs based on the nature of the imputs. For the structured views, we define the cost function combination of the shortest path distance from graph and the distance between structural latent variables. More specifically, given the normalized shortest path distance matrix between every pair of nodes in the input graph (
D(w) (1-o H (H ())) for E V.
where @ denotes the Hadamard product. This construction ensures both graph and attributes informa- tion are incorporated in the distance function. For unstructured views, we define the cost function between two features as follows:
-
for E Vn.
Noting the definitions of WD and GWD in Section with shared transport matrixas follows:
we rewrite DFGW between two views of data
Draw (p(H(")), ((H)))
N. N,
Σ
(ac
min
(5)
j=
T
(eo')
ell
H
[I]
To approximate the FGW distance, we first deploy GW algorithm in equation to obtain T (ve')
gw
and Daw, and then utilize T along with the defined transportation cost clvv') to calculate Wasserstein distance term in DEGW (Chen et al. (020). The pseudo-code in [
A.1 provides the details of the FGW distance calculation procedure. Please note that we use the same Sinkhorn solver in Chen al. (2020) and Alvarez-Melis & Jaakkola (2018).
further can generate A for every pair of views based on T(pe") as follows:
II V aight
p(A p(Atev)s (ev')
(D)) H(e") = II
Ber (vee)) 1 T(tee) g1o
///m/aaxt) / (6)
gw
V
where 7 E [0, 1] is normalizing hyper-parameter. and Ber is short for Bernoulli. We note that the sum of the elements each the transport matrices T(ve") equals one. Hence each of its elements has small value. Therefore, normalize the transport matrices (as TT (on') //max(TTww (ver)) ')) to avoid very sparse and trivial solutions. To use the reparametrization trick during training, we sample from conerete relaxation of Bernoulli (Gal et al. (017). We emphasize that our proposed FGW-hased decoder is the key in aligning features/nodes across structured and unstructured views via accurate and efficient distribution matching scheme.
4.2.2 PRIOR CONSTRUCTION AND LIKELIHOODS
Prior. We impose independent aero-mean unit-variance Gaussian priors elements of H. The prior for Z is a multivariate Gaussian distribution whose mean and diagonal covariance matrix constructed from the inferred multi-partite graph and the struetural latent variable H. We use two graph neural networks (GNNs) gps and gos to map H. and .A to the parameters of po_ (2). Specifically.
Po, (3
II II Po. (Z(1)) H.
pa_ (Z 11) H. A) = =N ,0 (o.)).
(u.1)
VEV, 1=1
with
[ (1(v.)) Jo.s 9f11) (H, A),
a ((.1)) (v.3)
u, = 9/.7 (H..

---

Published as a conference paper at ICLR 2022
We note that in this setting, H. is considered as node attributes of the multi-partite interaction graph. Likelihood observations. To reconstruct the input graphs in the structured views, we assume that the views and edges are conditionally independent. More specifically, employ inner-produc decoder as follows:
N I po f,1=1
Pog (2l,
P(.)
II
(A
(0)
VEV,
Ber (a(H)
()))
po,
To generate the features in unstructured views and node attributes in structured views, we assume that views are conditionally independent. Hence we can expand the feature reconstruction terms in the the equation 2 follows:
po. (N 2.)= II po. (x))) (()),
po_ (X, 12) = II po. (x Z()).
DEV.
VEV.
We note that po, could also be view specific depending on whether the node attributes/features in view discrete continuous. In our experiments, have deployed the Gaussian likelihood with the unit variance. The mapping from Z to the parameters of pe_ (X),, in our case, the mean of the Gaussian distribution, be any highly expressive function such as neural networks. We denote these functions by f(x-*) and p(2.)
4.3 INFERENCE NETWORK AND LEARNING
Posterior. We model the posterior of the structural latent variables Gaussian distribution and infer its parameters independently for each view. More specifically,
qah, (H, Xx) = II
H X ())..
4eh, (H, Ru. 21,) = II go.. (H (v) (v)
A (o)).
4o.
DEV.
VEV.
We use two GNNs for each structured view, (g (X (x)) A(()), (a.)) (x A (0) Juer, and two 9gh
, to map inputs to the mean and variance of the posteriors. We consider the variational distribution of Z to be a multivariate Gaussian distribution, and it is factorized as follows:
fully connected neural networks per unstructured view, (f( (x(*)), fla.t) (X (() Jeen.
)
9s. (Z, X.a) = II 3
X ()),
(o, Xu, 21.) = II
Jo. (Z(*) (0)
X (0)
A (*)).
VEV.
VE
We use two GNNs per structured view, {94= (u.0) (x) A (1)), (a.0) (x A (1) hoev, and two fully (x()),, flg.v) (x(*)) eev, in the
Iq=
connected neural networks for each unstructured view, (f(..) same fashion Go. to infer parameters of qo.-
Objective function. Having defined the prior and posterior distributions as well as the likelihood, we write the overall loss function as the sum of the negative variational ELBO and FGW regularization terms. Specifically,
L = ELBO + LFGW
E Egg, (Z..7h. x..) log pa (z, A, H) + E 36, (2, 1, 2.) log pe (Z. |A, H)
Ego. (2, (x,.) log 9o. (z,) xe..) - Lgg, (2, X. a.) log qo. (Z, X2. 2.)
+ Ego,, (H. (.x.) log (H..) + E0,, (H. Xx, 2.) logp(HH)
Egsy, (Hw (x)) log 4o., (H,, Xx)) Eg,, (Hel |cc M. log 4o, (Hg Nu, 21,))
(7)
+ Ego, (2,, (x.) log Po. (Xu, 2.) Ego, (2, NC, a, log Po. (X, 2.)
Ego,, (H., IX. a.) log pa, (2l, H.)) + Σ Σ
DEGW (p(HI p(H)
N'EV v'4v
While, mentioned previously, the Sinkhorn algorithm to caleulate the DrGw, the overall loss is optimized using stochastic gradient descent based optimization algorithms such as Adam (Kingma & Ba 2014).

---

Published as a conference paper at ICLR 2022
5 EXPERIMENTS
5.1 DATASETS AND EVALUATION METRICS
Datasets. We use the same datasets as BayReL (Hajraamzzaaai et al. 2020). i.c. microbiome- metabolite interactions in cystic fibrosis (CF) and gene-drug interactions in precision medicine. Dataset description and graph construction procedure are detailed in Appendix |A.2 We want to emphasize that although these datasets have structured views, have missing samples and their samples are completely paired, in many real-world cases these assumptions are not satisfied. These datasets were chosen merely get better understanding of the advantages of MoReL specially compared to BayReL. We evaluate MoReL in different settings. More specifically, we demonstrate the performance of MoReL when: 1) one or both views are unstructured, 2) there are missing samples, and 3) samples are not paired. Furthermore, we have comprehensive comparison with BayReL when both views are structured.
Evaluation metrics.
To quantify the performance of the methods, we use the same evaluation metrics as the ones introduced in BayReL. Since in these datasets, the true negatives, i.e. non-interactions, are not known; and there are only small subset of true positives, i.e. true interactions, well-known classification metrics cannot be used for evaluation. Therefore, positive accuracy and negative experiments. Positive accuracy refers
accuracy have been defined to evaluate to the accuracy of identifying validated interactions with P. aeruginosa. Negative accuracy exploits the fact that there should not be any common metabolite targets between known anaerobic microbes (Veillonella, Fusobacterium, Prevotella, and Streptococcus) and notable pathogen P. aeruginosa. Let B denote the set of all microbes and AA and A2 represent two disjoint sets of metabolites. Negative accuracy is defined as 1 EiE,, EjE,, Eies 1i and are connected to where 11() is the indicator |Ai| Aa]x
function. Having both higher positive and negative accuracy is desired.
For precision medicine, we compare the prediction sensitivity of identifying known interactions in the test sets while tracking the average density of the overall constructed graphs. We note that inferring very dense graphs would lead to high prediction sensitivity as it will includes most of the possible interactions. Therefore, tracking the sparsity of the inferred graphs is the key to properly evaluate the models' capability in predicting meaningful interactions.
5.2 BASELINES AND EXPERIMENTAL. SETUPS
Baselines. We compare MoReL with three baselines including Spearman's Rank Correlation Analysis (SRCA), BCCA (Klami et al. 2013), and BayReL. While SRCA applies to raw data, BCCA first finds dimenimnnii latent representations of views via matrix factorization and then the interactions are discovered based on the correlation between representations. BCCA and SRCA could not incorporate the structure of data and need a two-step procedure to infer the interaction between features across the views. In contrast, BayReL is able to use the structure of data and infer the relations without any ad-hoc post-pree postt-rocessing procedure. However, BayReL suffers from three strict assumptions: 1) All views of data structured; 2) There
missing samples in any views; and 3) Samples are paired, i.e. the ID of samples are known. We emphasize that MoReL. is the very first model that not only can infer interactions across structured and unstructured views but also is able to handle missing and unpaired samples in different domains, making it more applicable in real-world multi-omics data integration. A widely used method for multi-omics data integration is MOFA (Argelaguet et al. 2018). The mathematical modeling of MOFA is the same as BCCA except for the data likelihood part. While BCCA only supports continuous data, MOFA can have discrete likelihoods. Since our datasets do not have discrete features, we are only reporting BCCA results.
Hyper-parameters. In all of our experiments, to have fair comparison, architectural hyper- parameters (i.e. number of layers and number of neurons) were set to be the same as in BayReL. Other hyper-parameters that are unique to MoReL were tuned using the validation set. More specifically, the number of hidden layers as well as their dimensions are the same for the corresponding functions in both structured and unstructured views. We use graph convolutional layers (Kipf & Welling 2017) for structured views and fully connected layers for unstructured views except for reconstructing X from Z, for which we use fully connected layers in all of the views. The mapping from inputs to the mean and variance parameters of H are two 2-layer neural networks (16 and 8 dimensional layers)
7

---

Published as a conference paper at ICLR 2022
Table Comparison of positive accuracy (in %) on CF dataset at negative accuracy of > 97%.
SRCA
BCCA
Mo un
MoReL us
Positive accuracy
26.41
28.30
3.21
56.16 = 1.85
63.77 + 1.11
with a shared first layer for each view. We use two 2-layer neural networks (16 and dimensional layers) with a shared first layer for each view for the mapping from H. to the mean and variance of Z. We use a 3-layer fully connected neural network (8 and 16 dimensional hidden layers) for each view as the reconstruction function mapping Z to X'. The temperature for relaxed Bernoulli distribution is set to 0.3. The normalizing parameter in equation i 0.9 while a and 33 in Draw are set to and 0.5, respectively. We used the exponential decaying learning rate with the decay rate of 0.01 and initial learning rate of 0.01. All of our results are averaged over multiple runs with different random seeds. We have implemented MoReL and all the competing methods in Tensorflow (Abadi al. 2015). All the experiments are performed on workstation with single NVIDIA P100 GPU.
5.3 DISCUSSION, DATASETS WITH UNSTRUCTURED VIEWS
Table shows the performance of three variants of MoReL and competing methods for microbiome- metabolite data integration with the CF data. In these experiments, we assume that the samples are paired and all are available both views. In MoRel. we report the results when both views are unstructured. In MoReL us- we have the graph of interactions between microbiomes while the metabolite view is assumed to be unstructured. Comparing MoReL mu and baselines that do not incorporate any graph-structured data input, we observe almost 30% improvement in positive accuracy while maintaining higher than 97% negative accuracy. This demonstrates that proposed MoReL, even without any structural information, is effective in inferring meaningful interactions. Further incorporating the network between microbiomes (i.e. MoRel. leads to a 37% and 7% improvement compared to the baselines and MoRel- uns respectively. This shows not only the importance of incorporating view-specific side information, but also the effectiveness of FGW-based decoder in aligning structured and unstructured views. Further results on interpretability and robustness of MoReL. on CF dataset is provided in Appendix A.3
The results for prediction sensitivity of two variants of MoReL and competing methods in the precision medicine experiments are shown in Table We observe that both MoReL uas where both views are unstructured, and MoReL ux. where the graph structure between genes is given, consistently outperform the baselines by significant margin in graphs with different densities. This proves that MoRel. is able learn meaningful relations both sparse and dense graphs. Comparing the results for MoReL us and BCCA, the difference between their performance increases as the the density of the bipartite graph increases, showing that MoReL can identify gene-drug interactions more robustly.
5.4 COMPARISON WITH BAYREL
While the primary goal of experiments far was showing the effectiveness of MoReL in integrating unstructured and structured views, here we investigate the advantages of MoReL over BayReL.
All structured. As mentioned earlier in the manuscript, BayReL assumess that all of the views structured. To show the expressive power of MoRel., we train it in the same setting as BayRel. where all of the views are structured. Particularly, and microbiome network are observed in the microbiome- metabolle experiment. Also, in precision medicine experiment both drug network and gene regulatory network are known priori. For fair
assume that for CF dataset both metabolites network
Table 2: Comparison of prediction sensitivity (in %)
the precision medicine experiment.
Avg. degree
0.10
0.15
0.20
0.25
SRCA
8.03
12.00
17.15
20.70
26.85
BCCA
9.65 = 0.75
14.34 0.06
18.96
+ 0.42
23.29 + 0.52
28.22 = 0.66
38.02 +: 2.15
46.88 + 1.88
MoRel.go
11.29 + 0.16
15.74
0.62
21.21
+ 0.81
26.20 +: 1.10
30.47 = 1.07
39.05 +: 0.75
50.19 + 0.19
MoKel.
12.79 + 0.39
17.51
2.21
22.82 + 1.01
29.58 + 1.08
35.05 + 1.27
45.74 1.75
53.16 + 0.96
8

---

Published as a conference paper at ICLR 2022
comparison, we set the number of layers as well hidden dimensions be the same in both models. We train MoReL with the exponential decaying learning rate with the initial rate of 0.01 and decay rate of 0.001 for 120 training epochs. For BayReL, use the setting reported in Hajiramzzanalii et al. 2020). The results for CF and precision medicine are summarized in Tables 3 [nd We see that MoReI. outperforms BayRel. CF dataset by a margin of 7% which indicates that knowing the metabolic pathways can greatly improve interaction learning. In precision medicine experiment, we
observe a consistent 2% improvement by MoReL compared to BayReL.
We emphasize that the declined performance of MoReL un and MoReL us (shown in Tables I and 2 compared to BayReL is expected, as they uses less information than BayRel.. In- corporating this extra information in MoReL. enhances its performance substantially. Note that BayReL is bound to use the same set of functions for all views to account for arbitrar- ily rotations and transformations, which limits its expressive power. However, the FGW based decoder in MoRel. allows to have different pro- cessing functions for each view. We argue that this increases the expressive power and plays the key role in enhancing the performance.
Table 3: Positive aceuracy (%) on CF dataset.
BayReL.
MoReL ss
Positive Acc.
82.70 - 4.70
89.50 = 3.29
Table 4: Prediction sensitivity (%) in the precision medicine experiment.
Avg. degree
BayReL
MoReL
0.4
47.90 = 0.43
49.24 + 1.64
0.5
56.76 + 0.50
58.92 + 0.40
Paired vs. unpaired. To show that MoReL can handle unpaired input samples, we perform an ablation study on CF dataset. We reverse the order of samples in metabolite view while keeping the order of samples in microbiome. We report the performance of BayReL and MoReL us where we don't ton't use the structure of metabolite view. The results are shown in Table 5 While MoReL performed virtually the same as completely paired scenario (shown in Table D. BayReL's performance drastically declined. We note that the reported negative accuracy is the best one achieved by BayRel..
Missing samples. We should again point out that in setting where all views are structured but the number of node attributes are not the same in differ- ent views, BayReL cannot be deployed (as it uses the same processing functions for all views). To see how MoRel us performs in such a scenario, domly remove 10% of samples in metabolite view of CF dataset. MoRel. achieves positive accuracy 61.36 16.66 3.74 with negative accuracy of 97%. This again shows the robustness of FGW-based decoder in aligning nodes with different number of samples.
Table 5: Positive accuracy (%) on CF dataset with unpaired samples.
BayReL
MoReL us
Positive Acc.
31.56
63.24 + 2.13
Negative Acc.
72
97
(in %) of 61.
Computational complexity. We have also benchmarked computational complexity of MoReL and BayReL by tracking their runtime on CF dataset on the same hardware. While BayReL takes 0.6 seconds per training epoch, MoReL takes 2.7 seconds per training epoch. This is due to the computational overhead caused by deploying FGW-based decoder. Considering the model flexibility and significant prediction performance improvement, such computational overhead is acceptable.
CONCLUSIONS
We have proposed MoReL, a novel Bayesian deep generative model that efficiently infers hidden molecular relations across heterogeneous views of data. By using a fused Wasseerseess based decoder, MoReL addresses several main shortcomings of the state-of- the--ar omics data integration model. Specifically, MoReL. can: 1) integrate both structured and unstructured omics datasets while accounting for arbitrarily permutation and/or transformation caused by processing features with different deep functions across the views; 2) handle unpaired samples across the views of data; 3) combine multiple views from different data sources with any number of missing samples. Our experiments on two real-world datasets have demonstrated substantial improvement in inferring meaningful relations as well improving prediction sensitivity compared to the competing methods. MoReL has shown the promising potential for multi-view learning, in particular multi-omics data integration for biological knowledge discovery, when facing heterogeneous data from different views.

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGMENTS
The presented materials are based upon the work supported in part by the National Science Foundation under Grants CCF-1553281 CCF-1934904, DMR-2111103, ECCS-1839816, IIS- 118-1812641, IIS- 1848596, and OAC-1835690.
REFERENCES
Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving. Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dandelion Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Mike Schuster, Jonathon Shlens, Benoit Steiner, llya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. TensorFlow: Large-scale machine learning on heterogeneous systems, 2015. URL https: /www///www////////////////wwt www.:tensorffogg org/ Software available from tensorflow.org.
Olah,
David Alvarez-Melis and Tommi S Jaakkola. Gromov-wasserstein alignment of word embedding spaces. arXiv preprint 30Xiv:/009.000133 2018.
Ricard Argelaguet, Britta Velten, Damien Arnol, Sascha Dietrich, Thorsten Zenz, John C Marioni, Florian Buetmer, Wolfgang Huber, and Oliver Stegle. Multi-omics factor analysis a framework for unsupervised integration of multi-omics data sets. Molecular systems biology, 14(6):e8124 2018.
Jordi Barretina, Giordano Caponigro, Nicolas Stransky, Kavitha Venkatesan, Adam A Margolin, Sungjoon Kim, Christopher Wilson, Joseph Lehár, Gregory Kryukov, Dmitriy Sonkin, et al. The cancer cell line encyclopedia enables predictive modelling of anticancer drug sensitivity. Nature, 483(7391)):033 001660006776-0000 2012.
Yoav Benjamini and Yosef Hochberg. Controlling the false discovery rate: a practical and powerful approach to multiple testing. Journal of the Royal statistical society: series B (Methodological), 57(1):289-300, 1995.
Rianne van den Berg, Thomas N Kipf, and Max Welling, Graph convolutional matrix completion. arXiv preprint arXiv: 02263322.......... 2017.
Charlotte Bunne, David Alvarez-Melis, Andreas Krause, and Stefanie Jegelka. Learning generative models across incomparable spaces. In International Conference on Machine Learning, pp. 851-861. PMLR, 2019.
Jia Chen, Gang Wang. Yanning Shen, and Georgios B Giannakis. Canonical correlation analysis of datasets with a common source graph. IEEE Transactions on Signal Processing, 66( 66(16):4398 4408, 2018.
Jia Chen, Gang Wang, and Georgios Giannakis. Multiview canonical correlation analysis over graphs. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 2947-2951, IEEE, 2019.
Liqun Chen, Zhe Gan, Yu Cheng, Linjie Li, Lawrence Carin, and Jingjing Liu. Graph optimal transport for cross-domain alignment. In International Conference on Machine Learning, pp. 1542-1553, PMLR, 2020.
Samir Chowdhury and Facundo Mémoli. The gromov-wassersein distance between networks and stable network invariants. Information and Inference: Journal of the IMA, 8(4):757-787, 2019.
Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. Advances in neural information processing systems, 26:2292-2300 2013.
Yarin Gal, Jiri Hron, and Alex Kendall, Concrete dropout. Advances in neural information processing systems, pp. 3581-35900 2017.
10

---

Published as a conference paper at ICLR 2022
Ehsan Hajiramezanali, Arman Hasanzadeh, Krishna Narayanan, Nick Duffield, Mingyuan Zhou, and Xiaoning Qian. Variational graph recurrent neural networks. In Advances in neural information processing systems, volume 32, 2019.
Ehsan Hajiramezanali, Arman Hasanzadeh, Nick Duffield, Krishna Narayanan, and Xiaoning Qian. Bayrel: Bayesian relational learning for multi-omics data integration. Advances in Neural Informa- lion Processing Systems, 33: 13:19251-1926, 2020.
Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. In Advances in neural information processing systems, pp. 1024-1034, 2017.
Arman Hasanzadeh, Ehsan Hajiramezanali, Krishna Narayanan, Nick Duffield, Mingyuan Zhou, and Xiaoning Qian. Semi-implicit graph variational auto-encoders, Advances in neural information processing systems, 32, 2019.
Arman Hasanzadeh, Ehsan Hajiramezanali, Shahin Boluki, Mingyuan Zhou, Nick Duffield, Krishna Narayanan, and Xiaoning Qian. Bayesian graph neural networks with adaptive connection sam- pling. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 4094-4104. PMLR, 13-18 Jul 2020.
Vassilis Kalofolias, Xavier Bresson, Michael Bronstein, and Pierre Vandergheynst. Matrix completion on graphs. arXiv preprint arXiv: 1717, 2014.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXiv:1412.6980 2014.
Thomas N Kipf and Max Welling, Variational graph auto-encoders. arXiv preprint arXiv:1611.07308. 2016.
Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In International Conference on Learning Representations, 2017.
Arto Klami, Seppo Virtamen, and Samuel Kaski. Bayesian canonical correlation analysis. Journal of Machine Learning Research, 14(A 2013.
Su-In Lee, Safiye Celik, Benjamin A Logsdon, Scott M Lundberg, Timothy Martins, Vivian G Oehler, Elihu H Estey, Chris P Miller, Sylvia Chien, Jin Dai, et al. A machine learning approach to integrate big data for precision medicine in acute myeloid leukemia. Nature communications, 9(1): 1-13, 2018
Hao Ma, Dengyong Zhou, Chao Liu, Michael Lyu, and Irwin King. Recommender systems with social regularization. In Proceedings of the fourth ACM international conference Web search and data mining, pp. 287-296, 2011.
Facundo Mémoli. Spectral gromov wnsserssssn distances for shape matching, In 2009 IEEE 12th International Conference on Computer Vision Workshops. ICCV Workshops, pp. 256-263. IEEE, 2009.
Facundo Mémoli. Grosov-wasserstein distances and the metric approach to object matching. Foundations of computational mathematics, 114):417-487, 2011.
Federico Monti, Michael Bronstein, and Xavier Bresson. Geometric matrix completion with recurrent multi-graph neural networks. In Advances Neural Information Processing Systems, pp. 3697- 2017.
James Morton, Alexander A Aksenov, Louis Felix Nothias, James R Foulds, Robert A Quinn, Michelle H Badri, Tami L Swenson, Mare W Van Goethem, Trent R Northen, Yoshiki Vazquez- Baeza, al. Learning representations of microbe- interactions. Nature methods, 16 (12):1306-1314, 2019.
Gabriel Peyré, Mareo Cuturi, and Justin Solomon. Grosov-wasserstein averaging of kernel and distance matrices. In International Conference on Machine Learning, pp. 2674-2672. PMLR, 2016.
11

---

Published as a conference paper at ICLR 2022
Robert A Quinn, Katrine Whiteson, Yan-Wei Lim, Peter Salamon, Barbara Bailey, Simone Mienardi, Savannah Sanchez, Don Blake, Doug Conrad, and Forest Rohwer. A winogradsky baasedd culture system shows association between microbial fermentation and cystic fibrosis exacerbation. The ISME journal, 1(4):102411038, 2015.
Justin Solomon, Fernando De Goes, Gabriel Peyré, Marco Cuturi, Adrian Butscher, Andy Nguyen, Tao Du, and Leonidas Guibas. Convolutional wasserstein distances: Efficient optimal transportation on geometric domains. ACM Transactions on Graphics (TOG), 34(4):1-11, 2015.
Alexandre Irrthum Vân Anh Huynh-Thu, Louis Wehenkel, and Pierre Geurts. Inferring regulatory networks from expression data using tree-based methods. PloS one, 5(9), 2010.
Titouan Vayer, Laetita Chapel, Rémi Flamary, Romain Tavenard, and Nicolas Courty. Fused gromov- wasserstein distance for structured objects: theoretical foundations and mathematical properties. 2018a.
arXiv preprint
Titouan Vayer, Lactitia Chapel, Rémi Flamary, Romain Tavenard, and Nicolas Courty. Optimal transport for structured data with application on graphs. arXiv preprint 1rXiv:1805.001 14, 2018b.
Titouan Vayer, Rémi Flamary, Romain Tavenard, Laetitia Chapel, and Nicolas Courty. Sliced
gromov-wasserstein. arXiv preprint 1IXiv:1905.10144 2019.
Alex H Wagner, Adam C Coffman, Benjamin Ainscough, Nicholas C Spies, Zachary L Skidmore, Katie M Campbell, Kilannin Krysiak, Deng Pan, Joshua McMichael, James Eldred, et al. Dgidh 2.0: mining clinically relevant drug-gene interactions. Nucleic acids research, 44(D1): D1036-D1044, 2016.
Hongteng Xu, Dixin Luo, and Lawrence Carin. Scalable gromov-wassestein learning for graph partitioning and matching. arXiv preprint urXiv:1905.0766 2019a.
Hongteng Xu, Dixin Luo, Hongyuan Zha, and Lawrence Carin Duke. Gromov-wasserstiin learning for graph matching and node embedding. In International conference machine learning, pp. 6932 6941. PMLR, 2019b.
Hongteng Xu, Dixin Luo, Ricardo Henao, Svati Shah, and Lawrence Carin. Learning autoencoders with relational regularization. In International Conference on Machine Learning, pp. 10576-10586. PMLR, 2020.
Chuxu Zhang, Dongjin Song, Chao Huang, Ananthram Swami, and Nitesh V Chawla. Heterogeneous graph neural network. Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 793-803, 2019.
12

---

Published as a conference paper at ICLR 2022
A APPENDIX
A.1 FUSED BROMOV-WASSERSS (FGW)
The algorithm to calculate the fused Gromov-Wassen distance between two views in our decoder provided as pseudo-code in Algorithm | The algorithm takes C(v), C(r), and C(ve', which are intra/inter-costs between nodes in matrix form, as well p. which is hyper-parameter. It returns the Wasserstein distance, Kromov-Wassersee distance, as well as the transport matrix.
Algorithm 1: Computing fused Gromov-Wasserstenn distance.
Input: Crf)) C(ee) Chuv') "xn' mxm" nxm"
Definitions: C Hadamard product, () = Frobenius dot-product
// Cross-view similarity:
((ve') (C 1
1m ((C())))))
Initializing variables:
T = 1, T
a= 2. 1m
Bus= = exp(((())))
for f = 1,.....
((ve)) TC((O(
for t2 = 1,2..... do M = B
10
for t3 1,2,... do
11
AMo
nM 5
T = diag(o) M diag(o))
14 Dw ((C(oO)) T,
15 Dew (CT, ,T)
16 Return T, Dw, Dow
A.2 DATA DESCRIPTION
Microbiome-meetbbolom interactions. The goal studying this dataset is to detect the microbe- metabolite interactions patients with Cystic Fibrosis (CF). This dataset includes the 16S ribosomal RNA (rRNA) sequencing and metabolomics for 172 patients diagnosed with CF. We follow the same preprocessing steps as in Morton et al. (2019); Hajiramezanali et al. (2020), and filter out microbes that appear in less than ten samples, which results in 138 unique microbial taxa and 462 metabolite features. To construct the microbiome network, we perform a taxonomic enrichment analysis using Fisher's test and calculating p- p-values for each pairs of microbes as in Hajiramezanal et al. (2020). More specifically, the Benjamini- Sunjamini-Hochberg procedure (Benjamini for multiple test correction and an edge is added between two microbes if the adjusted p-value is lower than 0.01, The microbiome graph has 984 edges with the graph density of 0.2. For the metabolomics network, there are 1185 edges in total, with each edge representing a connection between metabolites via same chemical construction (Morton et al. 2011)). The graph density of the metabolite network is 0.01 We use 80% of the reported target molecules of P. aeruginosain studies in Quinn et al. 2015)) and Morton et al.] 2019) metabolome interactions. The remaining 20% of the reported molecules are considered validation set and are only used for the early stopping purpose.
Hochberg] 1995) is adopted
test set to evaluate the predicted microbiome-
Precision medicine. Here we aim to identify genetic markers of cancer drug responses. This is very challenging task due to the very limited number of observations with respect to the system complexity and huge number of biological and experimental confounders, which often leads to significant false positive associations (Barretina al. 2012 We consider dataset from 30 acute myeloid leukemia (AML) patients that contains gene expression and drug sensitivity data of 160 chemotherapy drugs and targeted inhibitors (Lee et al. 2018 For gene expression, we preprocessed the RNA-Seq data resulting in 9073 genes (Lee et al. 2018) Following Hajiramezanali et al.] (2020), we construct the gene regulatory network based on the publicly available expression data of the 14 AML cell lines from
13

---

Published as a conference paper at ICLR 2022
Pssudomonas
Pscutiomonazis fragi
Pocudommnas
Pseudomonasia fra
Anaerobic
Anaerobic microbes
9_ Paeudomonas
Pseutommnns metabolites
a_Pseudoemonas
Psoudomonas metabolites
Figure 2:
A sub-network of the relational graph consisting of P. aeruginosa microbes, their validated targets, and anaerobic microbes, inferred using MoReI.gs (Left) and MoRel.gs (Right) with sub- network negative accuracy of 100%.
the Cancer Cell Line Encyclopedial (CCLE) using R package GENIE3 (Vân Anh Huynh-TThu et al. 2010). Moreover, We construct drug-drug interaction networks based on their action mechanisms. Specifically, the selected 53 drugs categorized into 20 broad pharmacodynamies classes (Lee) ct al. 2018); 14 classes contain more than one drugs. Only 16 out the 53 drugs are shared across two classes. We consider that drugs interact if they belong to the same class. We use the under the drug response curve reported the CCLE dataset to indicate drug sensitivity across a range of drug concentrations (Barretina al. 2012 Lee et 2018). Following [ee al. (2018), only consider the drugs that have less than 50% cell viability in at least half of the samples, resulting in 53 drugs. We 797 reported drug-gene interactions in The Drug-Gene Gene Interaction Database (DGIdb) (Wagner et al. (016) in order to evaluate different models. We note that our test and validation sets only include the interactions for 43 of the 53 drugs in the dataset. We use 20% of the evaluation set as the validation set. Please note that the validation set has been only used for early stopping.
A.3 ADDITIONAL RESULTS FOR CF DATASET
In this section, provide additional results for CF dataset demonstrating interpretability and robustness of MoReL.
Figure shows sub-networks of the inferred bipartite relational graphs by MoRel and MoRel-us consisting P. aeruginosa, anaerobic microbes, and validated target nodes of P. aeruginosa and all of the inferred interactions between them. Based on the biology knowledge, the expected interactions in these sub-networks should be that the four highlighted nodes in the bottom row are connected to all of the nodes in the top row, and any other nodes in the bottom row are not connected any of the top nodes. At the sub-network negative accuracy of 100% (i.e. any nodes in the bottom row other than the four highlighted ones are not connected to any of the top nodes). while MoReL,s identifies 70% of the validated edges of P. aeruginosa, MoRel.s identifies 86.8% of the edges. We note that BayReL. identifies 78% of the validated interactions (Hajiramezanali et al.) 2020). This clearly shows the effectiveness of our proposed FGW-based decoder and interpretability of MoReL to identify inter-relations.
In the main manuscript, we only reported the results for one specific threshold value of negative accuracy (97%). Here we provide additional results with other threshold values, which show similar improvements competing methods and similar trends MoReL ss and MoReL uss clearly observed in Figure 3 We note that there a trade-off between positive and negative accuracy, and the optimal point be chosen depending the application.
14

---

Published as a conference paper at ICLR 2022
MoReLss BayReL
MoReLys SRCA
BCCA
90
Positive Accuracy
70
50
30
10
80
85
90
95
100
Negative Accuracy
Figure 3: Positive accuracy negative accuracy of various models in CF data.
15


Reviews:
[{"review_id": "f-944taGjK", "reviewer": "Reviewer_wNNw", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "On the positive side: the approach deals with problems that could not be previously easily tackled, such as 1) having access to a graph of dependencies between features for each view; 2) dealing with missing instances on some of the views; 3) having pairing information across each view. \nOn the negative side: a more thorough empirical investigation of how well the approach can deal with those cases is lacking. Specifically, 1) a study of the robustness when the graph of dependencies between features is noisy is missing (one should expect that some edges are wrong and study the dependency of the performance w.r.t. the number of wrong edges); 2) the authors show a single example where 10% of data is removed, however a study that reports the performance w.r.t. the fraction of removed data is needed to substantiate the claim that the approach is indeed successful in dealing with missing data; moreover the fraction of missing instances should be different for different views; 3) the authors report a single example of lack of alignment (reversing the order for one view but not the other), while a more systematic analysis with several random permutations is what should be reported.\nThe computational complexity is reported for a single dataset, while it should be possible to give a sense of the time dependency of the proposed approach  w.r.t. the training set and test set size (does it scale linearly?).\nSome comments would be appreciated to clarify how the model deals with the case of several structured or unstructured views: arguably the  graph of dependencies between features would form disconnected components when multiple structured views are considered and hence the shortest path distance cannot be computed in that case. \nWith regards to the case with multiple views, it would be of interest to study the sample complexity as a function of the number of available views, since, as the authors have stated in the introduction, leveraging the information from multiple sources should improve the performance on tasks with a limited number of training samples.  \n", "summary_of_the_paper": "The authors propose a Bayesian model to infer relations across heterogeneous views of data, which can be of structured and unstructured type. ", "summary_of_the_review": "The approach deals with problems that could not be previously easily tackled, however  a more thorough empirical investigation of how well the approach can deal with those cases is lacking. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "XSHRyTl6or8", "reviewer": "Reviewer_cXWm", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\nI generally find the paper quite well written. The paper may be interesting to a much broader community rather than just the biomedical use cases the experiments have shown, as the methodology is presented for any multi-omic data in a very general way. \n\nThe overall methodology is similar to BayRel. Both papers formulate the problem of multi-omics data integration by modeling it as multi-view link prediction. However, this paper does have several nontrivial improvements over BayRel: it could handle scenarios that have no known structure information, and the samples could be unpaired across views. \n\nWeaknesses:\nMy main concern is with the experimental part. \nThe experiments are not extensive and cannot fully demonstrate the advantages of the proposed approach. First, the datasets are restrictive. Microbiome-metabolite interactions in CF may not be the most commonly used benchmark datasets in the biomedical multi-omic datasets, with many other more usual datasets not considered. Second, many of the advantages the proposed method claims cannot be fully demonstrated by the two datasets in experiment, though the authors do manipulate the structure of datasets manually to show that the method still works when some key assumptions of BayRel do not hold. It should be more telling if the authors could conduct a more comprehensive experimental studies, including more common datasets and datasets that are naturally with missingness and pairing issues. Also as the computational overhead compared to BayRel, knowing whether the methods could handle large-scale data with heterogeneous dependency structures would be helpful.\n", "summary_of_the_paper": "The authors propose a Bayesian framework to learn relations among multi-omic datasets. The main unique advantage over existing methods is the proposed method is able to learn without a priori dependency structure and it allows a certain degree of missingness and mismatching. Experiments on two biomedical multi-omics data sets partially demonstrate the effectiveness of the proposed method.", "summary_of_the_review": "The proposed method is convincing with respect to an important problem. The paper has nontrivial improvement over existing methods. Though it lacks comprehensive experimental validation, it is overall a good complement to existing literature.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "XHkI-vxVKWK", "reviewer": "Reviewer_dcdZ", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is dense and terse making it difficult to follow in order to pick out the novelty of the proposed approach. The definition of the experiments and the data used in the Experiments section is not self-sufficient. Without consulting the Appendix it is hard to know what are the problems being solved, the preprocessing and representation of the data to be used as input to the method. Although the approach is aimed at multiomic integration, there is no real connection to the domain of application. The evaluation is focused on the comparison to BayReL a related approach mentioned throughout the paper but not described in the related work.\n\nMulti-omics integration is of interest in computational biomedicine. However, focusing on BayReL only, in the related work the authors miss a large body of work, old and new. To name some: SNF, iCluster, LIGER, embedding propagation. They also miss out on a large amount of data being generated with increasing pace within the field.\n\nJudging from the results, MoReL seems to offer some performance improvement over BayReL on structured problems. The authors demonstrate the ability of MoReL to learn from unpaired data and in a setting with missing data. It all comes at a cost of 4.5 fold increase in computational time. \n", "summary_of_the_paper": "The authors propose a hierarchical Bayesian generative model for multiview learning aimed at application to integration of multiple modalities of omics data.\n", "summary_of_the_review": "While there is a limited novelty and demonstrated improvement in performance over a single related approach, from the application and result interpretation point of view, the advantages of using MoReL are not clear.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "F2RpGBoQMh", "reviewer": "Reviewer_2ULW", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The proposed model allows to integrate of both structured and unstructured heterogeneous views which often happens in the context of omics data. In order to align nodes/features in every pair of views, the Gromov-Wasserstein (FGW) has been used. The experimental results show a significant improvement of the proposed model to the baselines in several downstream tasks. However, the paper can be improved in the following aspects:\n  - It is not clear what is the key difference between BayReL and the proposed besides the FGW regularization term. \n  - Can we apply BayReL and FGW into a pipeline to learn heterogeneous views? And how is it compared to the proposed models if possible?\n  - What is the square node FGW in the graphical model in Figure 1. It is difficult to follow the generative process of the proposed models.", "summary_of_the_paper": "The paper proposed a deep Bayesian model for heterogeneous multi-omics data integration. The Gromov-Wasserstein (FGW) regularization between latent representations of heterogeneous views is used to align nodes/features in every pair of views. The experimental results have demonstrated improvement in inferring meaningful relations.", "summary_of_the_review": " In short, the paper is trying to solve an important challenge of heterogeneous multi-omics data integration by proposing an advanced model compared with BayReL. However, what is the key difference and improvements compared with BayReL should be discussed in detail.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: A deep Bayesian generative model is presented for multi-omics
integration, using fused Gromov-Wasserstein regularization between
latent representations of the data views. The method removes several
non-trivial and practically important restrictions from an earlier
method BayRel, enabling application in new setups, while still
performing well.

Reviewers discussed the paper with the authors, resolving
misunderstandings of the differences from earlier work
(esp. BayReL). The authors reported more extensive experiments in the
rebuttal, though not comparisons. The main remaining weakness is that
the contributions are in a very narrow field, or at least aplications
have only been demonstrated in the narrow field of multi-omics data
analysis. And even within that field, only in a narrow subfield. In a
machine learning venue that is restrictive. Another issue is
computational efficiency. The final decision then depends on how much
weight we place on the novel contributions vs these weaknesses.
Decision: Accept (Poster)