Bayesian Relational Generative Model for Scalable Multi-modal Learning
Authors: Ehsan Hajiramezanali
Talip Ucar
Lindsay Edwards
Venue: ICLR 2022 Submitted (2022)
Keywords: Multi-modal Learning
Bayesian Learning
Neural Processes
Variational Inference

Abstract: The study of complex systems requires the integration of multiple heterogeneous and high-dimensional data types (e.g. multi-omics). However, previous generative approaches for multi-modal inputs suffer from two shortcomings. First, they are not stochastic processes, leading to poor uncertainty estimations over their predictions. This is mostly due to the computationally intensive nature of traditional stochastic processes, such as Gaussian Processes (GPs), that makes their applicability limited in multi-modal learning frameworks. Second, they are not able to effectively approximate the joint posterior distribution of multi-modal data types with various missing patterns. More precisely, their model assumptions result in miscalibrated precisions and/or computational cost of sub-sampling procedure. In this paper, we propose a class of stochastic processes that learns a graph of dependencies between samples across multi-modal data types through adopting priors over the relational structure of the given data modalities. The dependency graph in our method, multi-modal  Relational Neural Process (mRNP), not only posits distributions over the functions and naturally enables rapid adaptation to new observations by its predictive distribution, but also makes mRNP scalable to large datasets through mini-batch optimization. We also introduce mixture-of-graphs (MoG) in our model construction and show that it can address the aforementioned limitations in joint posterior approximation. Experiments on both toy regression and classification tasks using real-world datasets demonstrate the potential of mRNP for offering higher prediction accuracies as well as more robust uncertainty estimates compared to existing baselines and state-of-the-art methods.

Under review as a conference paper at ICLR 2022
BAYESIAN RELATIONAL GENERATIVE MODEL FOR SCALABLE MULTI-MODAL LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
The study of complex systems requires the integration of multiple heterogeneous and high-dimensional data types (c.g. multi-omics), However, previous generative approaches for mudti-modal inputs suffer from two shortcomings. First, they are not stochastic processes, leading to poor uncertainty estimations over their predictions. This mostly due the computationally intensive nature of traditional stochastic processes, such as Gaussian Processes (GPs), that makes their applicability limited in multi-modal learning frameworks. Second, they are not able to effectively approximate the joint posterior distribution of multi-modal data types with various missing patterns. More precisely, their model assumptions result in miscalibrated precisions and/or computational cost of sub-sampling procedure. In this paper, we propose a class of stochastic processes that learns a graph of dependencies between samples across multi-modal data types through adopting priors over the relational structure of the given data modalities. The dependency graph in our method, multi-modal Relational Neural Process (mRNP), not only posits distributions over the functions and naturally enables rapid adaptation to new observations by its predictive distribution, but also makes mRNP scalable to large datasets through mini-batch optimization. We also introduce mixture-of-gee of-graphs (MoG) in our model construction and show that it can address the aforementioned limitations in joint posterior approximation. Experiments on both toy regression and classification tasks using real-world datasets demonstrate the potential of mRNP for offering higher prediction accuracies well as robust uncertainty estimates compared to existing baselines and state-of-the methods.
INTRODUCTION
Many prominent methods for multi-modal learning (Argelaguet et al., 2018 Andrew et al. 2013 Klami et al. 2013 Zhao et al.) 2016) are based on the canonical correlation analysis (CCA), which extracts shared components across multiple views. The main idea is that, given two vectors of random variables, the method finds the linear projections a shared latent space, in which the projected vectors maximally correlated (Thompson) 1984). This can help understand the overall dependency structure between these two vectors. However, the classical CCA cannot handle non-linearity (Andrew al. (2013) and suffers from lack probabilistic interpretation when applied to high dimensional data (Klami et al., (2013). To address these issues, probabilistic CCA (PCCA) has been proposed and extended to non-linear settings using kernel methods and neural networks (Bach & Jordan 2005).
More recently, Wu & Goodman 2018 introduced the multimodal variational autoencoder (m VAE) that models the joint posterior as product-of-exxe cross-modal generation test time without requiring additional inference networks and multi-stage training regimes. However, training PoE is difficult and the following techniques are needed in order to ensure that the individual views are learnt faithfully; i) using artificial sub-sampling of the observed views (Wu & Goodman, 2018). ii): applying variants of contrastive divergence (Hinton 2002), and iii): utilizing information bottleneck on the marginal representations of each view (Lee & Schaar 2021). Furthermore, each expert holds the power of veto and low density of only one marginal posteriors among a given set of observations leads to the low density of joint distribution (Shi et al. 2019). In the case of Gaussian experts, different levels of complexity of modalities or
(PoE) over the marginal posteriors, enabling

---

Under review as a conference paper at ICLR 2022
sub-optimal initialisation might result in the miscalibrated precisions that potentially lead to biased overall mean prediction (Shi et al. 2019))
Despite the success of the existing multi-modal generative methods, they suffer from inducing stochasticity in their inferred functions. This leads to; i) poor adaptation to new observation ii inefficient generalizability in few-shot settings and, iii) inaccurate uncertainty estimation, all of which are important in many safety critical applications such as healthcare. On the other hand, stochastic processes, such Gaussian processes (GPs), are exchangeable models that posit distributions over possible functions, and are updated in light of data through the probabilistic inference. Despite these advantages, GPs are computationally expensive and their underlying model is not flexible for nigh-dimensional inputs, making them infeasible for multi-modaa learning settings with multiple heterogeneous and high-dimensional data types.
Apart from the aforementioned issues, existing generative models for multi-modal learning focus on latent representation, but do not fully incorporate the label information. In order to facilitate predictions on the target labels, they need to apply two-step procedures, in which the shared latent spaces are used for downstream classification. While information relevant to the reconstruction of views will be well-captured in the shared representations. discriminative information relevant to the target task may be discarded, leading to poor prediction performance of such models (Lee & Schaar 202I). Furthermore. the shared representations of these methods in high-dimensional space are often difficult to interpret and need an extra downstream analyses such as PCA or t-SNE (Van Der Maaten 2014) that may cause biased results (Lötsch & Ultsch] 2020).
In this paper, instead of following the existing multi-modal generative methods that rely on determin- istic predictive functions, we propose the first multi-modal stochastic processes family that learns distributions over functions for inputs with any missing patterns. Like GPs, it naturally provides robust uncertainty estimates and can encode inductive biases. Unlike GPs, though, mRNP is computa- tionally efficient during training and evaluation since it can take advantage of batih optimization, and learns to adapt their priors to data. The inferred dependency graph in our model construction can be used to visualise the high dimensional latent representation, obviating the need for ad-hoc post-procet post-processing steps as required in most of the multi-modal learning methods.
The presented work makes four major contributions: 1) We develop a novel multi-modal Relational Neural Process, mRNP, that defines a distribution over functions of multiple data types by employing local latent variables, and learns dependency structure among the samples of the given modalities. 2) We theoretically prove exchangcability and consistency of mRNP, two necessary conditions that have to be satisfied during the construction of such model, showing that mRNP is valid stochastic process. 3) We further show that the local latent variable structure in mRNP is able to encode inductive biases and demonstrate this by designing an mRNP model that behaves similarly to a GP with an RBF kernel (an ablation study). 4) We introduce mixture-of-grape of-graphss (MoG) in our model construction that can address the issues such as computational complexity and miscalibrated precisions observed in the previous approaches in multi-modal learning.
METHOD
We propose a new graph-structure multi-modal learning method, referred as multi-modal Relational Neural Process (mRNP), that combines the benefits of neural networks with that of stochastic processes. In the supervised multi-modal setting, our dataset contains sets of feature-label pairs, (x,y) from |V] different modalities with various missing patterns, where | is number of views, x= (x modality present eeev E X is the input modalities and y E y is the given label. One of the key motivation of mRNP is that it gives us the ability to define a distribution over functions rather than learning single static function when fitting the data. In this framework, inspired by the idea of inducing points in sparse Gaussian processes (Titsias) 2009 Damianou & Lawrence, 2013), we first select reference set of samples that consists of functions f : + y that are sampled from some underlying distribution H. We then establish a probability distribution over fa around those samples. We define the reference set, X' [X XR,1,..... X XRy C X,, where X R. E RN,.D, and
Rapid adaption to new observations is what distinguishes a model that can learn a whole distribution over functions by seeing different functions in each batch from the one that learns single underlying function by observing several batches during the training. At the test time, the former will narrow down what the current. function is hy seeing a few context observations.
2

---

Under review as a conference paper at ICLR 2022
o e
Xn
MaG
Figure 1: Left:: Venn diagram of the sets where X' and Xas are colored as pink and teal, respectively; and the samples enclosed in the dashed circle are Dz- O is the complement of Xn and X = O U Xn. Right: Graphical model for the proposed mRNP.
N₂ and D., are the number samples and dimension of features in the domain v, respectively. We also define O X') \ AR. which is the set of all possible samples that are not in Xx. Please note that two sets Xx and O includes both labeled and unlabeled samples. We also denote our labeled samples D, :....... &N) a set of any finite random set from X,, as well defining sets X'M = D2 RR and X'=- = Xx U XM. The Venn diagram of the sets are provided in Figure [ We infer the parameters of probabilistic model using variational inference. In the following, first introduce each of the latent variables in our model as well as their corresponding prior and posterior distributions, and then prove that it corresponds to an infinitely exchangeable stochastic process. The graphical model of mRNP is illustrated in Figure
-peccit latent spaces. The first step to construct coddality-speciffe latent representations, denoted by 21 = modality (i.e. N₂) into a D,, dimensional latent space as follows:
a set of N₂ x D, matrices, which independently embed samples of each
f (UB, XB) = II
pa (U,, XB.)) dX BEV
II
/
po (UB, XB.)) XB.))(XB..) d.
-E
We further parameterize the distribution over the samples u independently:
Po(UB, B X B.+)=
II po (ur, Xi,,);
iEB.
where pa (u., xi..) can be any distribution that derived from the observed input data. For simplicity, we use a diagonal Gaussian, the parameters of which are function of the observed input. More specifically, we use two functions denoted by emmb, (emb,"(XB,.)) and pemb."(XBB.v ).) to infer the mean and variance of the distribution for each modality. Depending the nature of the modality, these functions can be implemented using any highly expressive functions such as many variants of neural networks (NNs) and graph NNs.
directed graph among the reference samples. The next step is to construct a directed graph across the samples in Xr using different modalities. Given the latent embedding Un. we first construct a set of directed acyclic graphs (DAGs), G = {G. hvev, where G, is a random binary adjacency matrix between reference samples present in the 7h modality. Then we combine the graphs G from multiple views to construct a common graph G. Inspired by the concept of stochastic orderings (Shaked & Shanthikumar 2007), we impose topological ordering over the vectors in Ug to avoid cycles in each individual G. The distribution of the adjacency matrices are defined as follows:
p( G UR.) = II II RER. jERu.jii
Bernoulli (Gij.v [(((),,)) > ((ug,v) yp sim (uiv. Ug,0)
(2)
Similar Louizos al. (2019), we employ a parameter free scalar projection ((u,,.) = Ex (usk.)) where (-) is a monotonic function. In practice, we use the log cumulative distribution func- tion of a standard normal distribution. Given the graphs G.,, construct directed graph G, in which the weight of each edge is average of its weights in different modalities, i.e. when sample i
G G=Engy B. /.GG/.. Evey m. where m, is binary vector and m,, is observed in the modality v, and 3, = 1/ assuming that the all modalities are equally impor- tant. The proposed formulation, referred as mixture-of-graphs (MoG), is used to learn the shared structured representation of multi-moda inputs by using the graphs of individual modalities.
relational graph of dependencies from reference samples to Xas- In addition to the di- rected graph G used for the reference samples, we apply MoG to construct bipartite graph
3

---

Under review as a conference paper at ICLR 2022
A= Dvy that is bi-adjacency matrix from AR to A's- Given the latent embedding UB as described previously, we model the elements of each individual bi-adjacency matrix by Bernoulli distribution follows:
p( A UM. Un.o)
II II Bernoulli (Aiy 1 (ur uz,0))
(3)
EE M, JER,
where cptim (.)-) is a score function measuring the similarity between the latent representations of the input samples in each modality v, and a₂ 1/IVI, assuming the all modalities equally important. Depending on the desired relational inductive biases, we can appropriately define the
function g (..)). The Bernoulii- Poisson link ب (u,.. uz,0) = - exp( Σk= Th Usk,v Uzk,o) and yptii (u uن,0) = exp(-3 Ilu., u1,+l33) are two examples of such potential score functions. Please note that use the latter one throughout this paper.
For both A = [A. and G, we use the concrete relaxation (Maddison et al. 2016 Gal et al. 2017) during training while sample from Bernoulli distributions in the testing phase.
shared latent variable. Having obtained the sample representations 2 and the dependency graphs A and G. construct shared latent variable, denoted by N x D. matrix Z, which can be used to predict target variable distributions yi- We parameterize the distributions of the predictive targets as follows:
f Poly ZB| A, G. GR
/
po ZR G. XR) Pa Pa(yM: Zm A, dZ dZM
= =II [z para, (Kn. (y z;) dz; II 'ER"
/ po(z) par (nn. Po |z,) dzj.
JEM
where para,() (-), par (.) are functions that return the parents of the points and j according to G and A, respectively. We summarize the information from the parent samples and their targets in X'R through the local latent variable z, allowing the distribution y; be explicitly dependent on the available graph dependency structures A and G. We set the distribution over z; as an independent Gaussian distribution whose parameters are function of either A or although any distribution with a permutation invariant probability density with respect to the parents can be used. More formally, the following distribution is adopted:
D.
Dj = II
(z par A, (R)) =
II pa (a( par (ER,YR))
(
prior
(4)
k=1
Mik
prior
Ci
Σ
A A
prior, (x,,R= Vj.R),
o
exp(c
prion (x Uj.n)). JENR
Σ
Aijih
JEN2
where X,, (uy,, pptior, and prioo transform functions with co-domain Rial, and C; = =E, =E,Ar, Aij +) is normalization constant. Given input (XB,ehvev, we factorize the variational posterior distribution as
4s Z XB = II (z(z, X,,),
(5)
In practice, we also parameterize the priors over the latent z; in terms of the posterior distribution of the reference samples, i.e. (z; Xx,R). More precisely, we define parior, and parior. in (Ui..r) and yo (x.,,t + Plabel (y,,n). respectively; where spPost and emb provide the means and variances of qu (z: X.,R) and the linear embedding of the labels of reference samples, respectively.
equation (4) as popost ((x,n) + cmb.t (Ui.))
We should point out that in this formulation, the prediction of y/ indirectly depends on the input samples X, through the graphs G and A, which are functions of U, This leads to an uninformative standard normal prior over z; for the samples with very small probability of being connected to the reference set through A, and hence the prediction of y: will be constant. This can be seen encoding inductive bias similar to a GP with an RBF kernel.
MoG vs POE. PoE is a prime choice to learn the variational joint posteriors; it can learn under any combination of missing modalities (Shi et al., 2019). Despite this advantage, it also has two main limitations: 1) the underlying model suffers from the overconfident experts since experts with
4

---

Under review as a conference paper at ICLR 2022
greater precision will have more influence over the combined prediction than experts with lower precision; 2) training and inference can be costly due to its artificial sub-sampling procedure. Given the aforementioned limitations of PoE, propose to build a graph of dependencies among local latent variables as a mixture of modality-speciffe graphs, i.e. mixxture-of-graple (MoG), and then parameterize the joint latent space Z conditioned on those graph. Unlike PoE, MoG effectively spreads its density over all the individual modalities by imposing a weight to each individual graphs. When prior knowledge is available, terms a² and 8g in the MoG construction can be used encourage certain modalities, which is the case in multi-omics data integration where individual modalities often contain complementary information of target task. A more detailed discussion can be found in Section D of the supplement.
We also should point out that mRNP not graph learning model. Rather, we propose novel multi-view NPs that is able to solve pitfall in utilizing stochastic processes for multi-view setting by learning a graph of dependencies. We can also consider graph learning in mRNP as kind of cross-attention in the form of a dependency graph among local latent variables. From this viewpoint, mRNP learning attention mechanism to combine modalities. The current framework in which mRNP learns the individual graphs might not be computationally efficient, especially if training set is large. We may improve the scalability issue of graph construction part of our mRNP by e.g. considering hierarchical structure similar hFGW (Xu et al. 2020). We leave this for future study.
Overall likelihood and learning. Putting everything together, the marginal likelihood is
( B XB)
(O XB,)) p(G, A (uB) (y ZB A, G, XR) dUB dZg. (6) NEV
II
G.A
We should point out that, for the sake of simplicity, we choose the reference set to be part of the training set Dr throughout this paper. More specifically, assume XB Dz. In case of existing reference samples that are not part of the training set, we need to marginalize out over them in equation (6) in addition to the marginalizations can also use the marginalization technique to incorporate unlabelled data in order to learn better representation and/or to impute the missing labels. We leave this for future study.
the latent variables and structured graphs. We
We deploy variational inference to optimize the model parameters and variational parameters & by minimizing the following derived Evidence Lower Bound (ELBO):
L = -= (gg(Z [R) P(G (2x)) Po XR.v) [log Rv ZR G. - log g0 Kx)]
Ez P(A) (Mg) 1bev (U X [log pe aM
Zm)+
log (Z
par (R). log qs (ZM [xa)]:
Please note that we can use mini-batches for the second expectation term where the size of the batches scale according to the size of the reference set Xx.. Due to the DAG structure in the first expectation term, we cannot decompose it to independent sums.
Predictive distribution. After optimizing the model parameters, we derive the predictive distribution for unseen samples (xr heev with missing modalities based on Bayes theorem follows:
po(u" (x:
Σ a²"
/
II Po (u UR x X R,,)
DEV
p(a" (ut (dR) pe(z" |prr (X yR)) Pe(y") 2z") cl dutey dz",
where ut is the embedding representation of the observed modality x through the neural network bemb and a* is binary vector similar row of A that denotes reference parents of the new samples. This is similar to the predictive distribution in few-shot learning (Sung et al. 2018).
Proposition I. are valid, permutation invariant stochastic processes.
mRNP corresponds to Bayesian models as the distributions defined in Equation (6)
proof. The proof is deferred to the supplement.
3 RELATED WORKS
Multi-modal VAE. There has been a long line of research in using variants of VAEs to explicitly
model the joint distribution over latents and data (Suzuii et al.) 2016 Vedantam al. 2017 Wu
5

---

Under review as a conference paper at ICLR 2022
& Goodman 2018 Shi et al. 2019). Most of these works are not able to efficiently explore more than two modalities and/or suffer from multi-stage training. Using PoE, mAE (Wu & Goodman 2018) addresses the aforementioned problems and learns joint embedding under any combination of missing modalities. However, 1) it suffers from the miscalibration of the precision of the experts due to differences in complexity of input modalities, and 2) given complete dataset with missing modalities, it requires artificial sub-sampling in order to faithfully learn the individual modalities.
Closely related the VAE, DeepIMV (Lee & Schaar 2021 applies PoE in a supervised framework for multi-omics data integration with missing views. Our mRNP is similar to this method in terms of using supervised framework. But, unlike mRNP, DeepIMV predictive function is deterministic and therefore is not suitable for out-of of-distriitttinn prediction and/or few-shot settings. It also includes a set of [V] view-specific predictors in its construction and apply information bottleneck (IB) principle on the marginal representations of individual views in order to allow each encoder to build view-specific expertise for predicting the target. Depending on the number of input modalities, computational complexity might be issue.
Neural process (NP). NPs (Garnelo al. 2018b have been recently proposed to learn an approximation of a stochastic process by using a neural network-based formulation. Traditional stochastic processes such GPs are usually computationally expensive and the available kernels usually restricted in their functional form and are not very flexible for high dimensional problems (Garnelo et al. 2018b). On the other hand, NPs are data-driven models that impose stochasticity in the function realizations and are able to define a flexible class of stochastic processes well suited for highly non-trivial functions(Lee et al. 2020). The main idea behind NPs is to define explicit global latent variables in their construction which can capture functional uncertainty(Garee Gaanelo al.) (018b).
Closely related to our mRNP, Louizos et al. (2019) discards the idea the global latent variables and instead builds a graph of dependencies among local latent variables. making it more suitable for modeling high-dimensions improve the performance of NPs. However, none of these works addresses multi-modal learning problem. To the best of our knowledge, mRNP is the first NP-based model to explore multi-modal in computationally efficient way.
data. There are other NP variants (Kim et al. 2019; Lee et al. 2020) to
4 EXPERIMENTS
We evaluate the effectiveness of mRNP based two different multi-modal experiments: 1) On two real-world datasets, we measure the performance of mRNP by using its prediction scores and the quality of uncertainty quantification in multi-modal classification tasks under any combination of missing modalities; 2) study by visualizing the predictive distributions in two toy examples of 1 I-dimensional multi-modal regression tasks. We compare its performance with those of three state-of-the -rr methods; MOFA (Argelaguet al., 2018), VAE (Wu & Goodman 2018). and DeepIMV (Lee & Schaar 2021). We should emphasize that MOFA and VAE models are unsupervised embedding models and suffer from task specific training. Specifically, these models derive low-dimensional (non-)linear embedding of the input samples, and hence classification model has to be trained for downstream analysis task. We should also point out that only autoencoder-baid the observed samples, for which the optimized parameters can be used for unseen samples. So, for MOFA, we need to combine training and test data order learn the -iw-dimnsioona embedding.
We investigate the inductive biases that we can encode in mRNP in an ablation
methods are able to be trained using
We implemented our model in PyTorch (Pazkke et al. 2019). Throughout the classification experi- ments, we use two fully connected layers with ReLu as the activation function each hidden layer to obtain an intermediate hidden representation h, of the inputs x., and then parameterize linear output layer to learn the parameters of po (u, x,), which we consider as Gaussian. We also factorize
the posterior qo [Z] Xg) ILEB (p(z.) Ilvv (v) (z; (x..,.)) where p(z:) = AN En). and
% (e) (z, xx, (H,, E.) is the underlying inference network for the modality We use linear layer as an encoder for each modality. For fair comparison, the results of the competing methods are obtained based on their original implementations with the same architectures as mRNP For the regression tasks, we used one fully connected layer with ReLu as the activation function to transfer x,, to h, followed by two separate linear layers to learn the parameters of po (u, xx), and (z(2,) X)), respectively. We train mRNP for a maximum of 50 (5000) epochs using Adam (Kingma

---

Under review as a conference paper at ICLR 2022
Table 1: Accuracy and uncertainty HW from 100 posterior predictive samples. The first column is the average predictive entropy whereas for the o.o.d. datasets the second is the AUC/AP and for the in-distribution is the test accuracy %.
MOFA
mVAF
DespIMV
mRNP
HW
NA. 78.77800000
4.07 66.9843.11
2.28 47.47742.66
0.40/ 52.5/8200-008
Gaussian
4.04/ 50. 1..19988-50
2:30 / / 87.27
1,40 /92.27 89.77
Uniform
NA
4.00 51 54161 78
2.30 94.81 993.22
1.633 99..9/ 94,15
Paisson
NA
3.48/ / 43.39/47.52 52
2,26 / 27.59 36.54
1.86/ 95.558 / 96.16
Average
NA
3.8440.3/ 18.374455761.2555
2.29:50.0/ 37.44 72.5-31. 1.6::0.2/ 29002/999.222/3344455
& Ba 2014) with a learning rate of 0.001 for classification (regression). More implementation details are included in the supplement.
4.1 PREDICTION PERFORMANCE AND UNCERTAINTY QUALITY
For the classification task, we consider two publicly available real-world, multi-modal datasets: the HandWrittin Writing (HW) (Dua & Graff) 2017) and the Cancer Cell Line Encyclopedia (CCLE) (Barretina) ct al. 2012). Throughout the experiments, the dimensions {D.. Dz. Dr } are {32, 64, }00} and
{32, 32, 32} for HW and CCLE, respectively. We use 300 (150) random samples from the training set as Xx for HW (CCLE). We train mRNP for 50 (10) epochs for HW (CCLE) and use the validation set for the early stopping. All of our results are averaged over 100 runs, ten randomly drawn dataset splits and runs with different random seeds on each split.
In order to capture uncertainty, we choose the predictive entropy that combines both epistemic and aleatoric uncertainties (Mukhoti & Gal 2018). In particular, we assume that epistemic uncertainty of our Bayesian model will increase in areas where we have no data, i.c. out of distribution (o.o.d.) datasets, while the predictive entropy of in-distribution can be used to compare different models in terms of combination of both uncertainties. In order to determine whether a sample is in or out of distribution based on its predictive entropy, we also report the area under both the receiver operating characteristic (ROC) and precision-recal (PR) curves. We should point out that, by using ROC and PR, we can make sure that the improvement predictive entropy is not due to a trivial model/learning since the model must have low entropy on the in-distribution test set but high entropy on the o.o.d. datasets in order to perform well in both ROC and PR. For the HW dataset, we consider Gaussian (0,, 1). uniform U|0,, 1], and Poisson(). X max(x)) noises as o.o.d. datasets. For CCLE, we use three real-wort cancer datasets COAD, KIRC, and SKCM from The Cancer Genome Atlas (TCGA)
(Tomezak et al. 2015) well Poisson noise as o.o.d datasets.
Multi-view handwriting data with missing views. First, we consider the classification of multi- view handwriting data. This dataset consists of features of handwritten numerals, i.e. {0..... .9. extracted from a collection of Dutch utility maps (van Breukelen et al. 1998). The dataset includes 76 Fourier coefficients of the character shapes, 47 Zernike moments and 6 morphological features for a total of 2,000 samples (200 samples per class). We follow the same data pre-processing as in Lee & Schaar 2021), D, and choose 20% and 16% of samples as test and validation sets.
The summary of the results are reported in T The pro- posed method outperforms the state-of-the-e (SOTA) methods by a significant margin in terms of both classification and o.o.d prediction accuracies. Our Bayesian mRNP model performs significantly better than DeepIMV, which is the only supervised baseline, in terms of uncertainty quantification. This might be due to the fact that the softmax entropy in DeepIMV is only able to capture aleatoric uncertainty while features with high epistemic uncertainty are ignored, resulting a relatively high number of inaccurate but certain samples. mRNP, on the other hand, captures both aleatoric and epistemic uncertainties.
Table 2: Comparison of the joint representation learning of HW data. Method
mVAE
DecpIMV
mRNP
Accuracy NMI
66.9
67.5
0.78
0.2
0.60
0.78
We also compare the shared (structured) representations of different modalities from mRNP with the joint latent representations learned by DeepIMV and VAE. Figure 2 showw an example of the graph G that mRNP lears,, as well PCA projections of latent representations from DeepIMV and VAE. First, it shows that mRNP learns a meaningful G by connecting samples with same class to each other. Second, few samples that are not in the same community as their labeled class, have
7

---

Under review as a conference paper at ICLR 2022
Figure 2: Comparison of the relational graph G on HW, inferred using mRNP (left) with PCA projections of latent representations from DeepIMV (middle) and mVAE (right). Nodes' colors refer to their class labels, i. 1-10.
lower degrees compared to the other nodes in the same community. This indicates that the model is less confident about the class of such a sample. The proposed mRNP provides more interpretable shared representation than traditional non-structured representations while obviating the need for post-procsssing step to visualize a high-dimensional embedding space in a or 3-dimensional one. Furthermore, we evaluate the joint representations of these models on the HW dataset through two downstream tasks: clustering and classification. For the first task, we apply K-means clustering and for the second one train linear SVM. The normalized mutual information (NMI) and classification accuracies are reported in T
We further perform ablation study HW dataset and provide some additional insights re- garding the sensitivity of mRNP to the number of reference samples. Table 3 shows the test accuracy, entropy, o.o.d entropy and AUC for o.o.d predic-
Table 3: Comparison of test accuracy and un- certainty quality in HW for different number of samples in Xn-
# R
Accuracy
entropy
0.0.d. entropy
AUC
50
0.26
81.91
12.2
tion. Note that the mean and standard error across
100
80.75
+ 1.7
0141
1.4 +4++00 0:15
94.04
2.0
200
82.40
1.3
0.37
155
95.13
+ 3.2
all of the o.o.d. datasets are reported for both o.o.d. entropy and AUC. We observe that the choice of samples as a reference set is important for mRNP
300
0:23
95.27
400
81.47
+ 1.5
0.39
1.5
0:26
95.56
2.16
500
81.72 + 1.35
0.45
14 0.22
91.72 3.1
to fit the data well and that the performance does not always improve with more samples. While the accuracy of mRNP reduced less than 1% in very high sample sizes, its performance degraded more than 30% with small sample size of reference set, e.g. 50 samples. Moreover, the model is less robust and shows higher in-distribution entropy this case. We should also point out that we only explore the effect of reference size, not the quality of the samples. Investigating the automatic selection of reference samples may improve scalability and alleviate the dependence of mRNP on acquiring a reasonable X'. We leave this for future study.
Multi-view CCLE dataset. We further apply mRNP to explore sensitivity of 472 heterogeneous cell lines to the drug Punobinostat. We study the integration of three omics, NicroRNA sequence (mRNA-seq), reverse phase protein array (RPPA), and metabolites. Following Lee & Schaar 202T), we use ActArea as an indicator of drug sensitivity and then label a response as sensitive when this value is higher than median of all responses, and otherwise assign to the nonsensitive group. Note that for TCGA datasets, which we used as o.o.d samples, only two of these modalities are available, Le. mRNA-seq and RPPA, so consider metabolite
missing modality.
The summary of the results are reported in Table 4 mRNP outperforms SOTAS in terms of F1 score while having higher average AUC and AP on the o.o.d. datasets. Lower in-distribution predictive entropy of mRNP compared to DeepIMV and mAE indicates that mRNP is a better
Table 4: Fl score and uncertainty on CCLE from 100 posterior predictive samples. This is analogous table to Table [. with different dataset.
MOFA
mVAE
DepiMV
mRNP
CCLE
/ 56.6915.7
638/66.7011..0
0.67 0667.1.541...7
KIRC
NA
0.67 50 33/51.98
0.68 67.60 64.01
0.37 68.49 63,95
SKCM
NA
0.67 44.44//5555
0.68 66.08 64.75
11.7// 70.65
Poisson
NA
0.53 0353/3399538.8
0.67 50.35 51,76
0.64 90.72 84.24
Average|
NA
0.63100043.188
43..6 0.66.1..0
62.2118..2 60.59.1..0 0.46.10.1 21..0../776....//22......
8

---

Under review as a conference paper at ICLR 2022
choice to deal with high- dige--imeniional data of small sample size, which is often the case in multi-omic data integration when studying complex discase. We further compare the models, based on their robustness to the missing views in the test set. In this ab- lation study, we train mRNP and two other baselines with training set of CCLE, in which only 3% of the views was missing in the main dataset. At the test time, we artificially introduce missing views by defining a probability of not observing an individual view for each sample. Please note that each view of the samples are treated independently. Then, we increase the probability of views missing from 10% to 50% and measure how different mod- els perform in terms of prediction accuracy. The classification errors of the basclines increase monotonically as the percentage of missing views is increased. mVAE has the worst overall per- formance, followed by DeepIMV. Using 70% of all the views, the average F1 score of mRNP reduces less than 0.5%, while DeepIMV and mAE performances degraded around 3%. In the worst case scenario (missing 50% of the views), the mRNP outperforms the DeepIMV and m mAE by 6% and 7%, respectively. This clearly shows the robustness of mRNP to the missing views at test time, and proves that it can faithfully learn individual modalities without any sub- sub-sampinng procedure.
Desplit
10
20
40
Mising Percentage (S)
Figure 3: Performance compar- ison of different methods under various missing probability.
4.2 ABLATION STUDY TO EXPLORE THE INDUCTIVE BIASES IN MULTI-MODAL REGRESSION
To validate whether mRNP can encode the inductive biases, we consider two toy multi-modal 1-d regres- sion tasks. Following the single view regression tasks in Louizos et al.] (019) and Osband et al. (2016). we first draw 12 samples from U0, 0.6) and 8 sam- ples from U00.8, 1] as the common space between two modalities, and then parameterize the target by y = s e sin(4(s; + e)) + sin (s,, + t)), where e (O, 0.032). Next, we generate observations for the first and second modality as xs, s and Xag 6s + e, respectively. Please note that we im- pose 10% chance that the views are missing. Figure shows the results of both modalities, in which mRNP has very similar behaviour to the GP. Similar to GP, it reports high uncertainty in the areas where there are no observed samples. For the second regression study, we draw 20 samples from U (7-4,44 4] for the first modal- ity and simulate the target labels as y x + e, where e ~ (O, 92). Then, we generate samples for the second modality as nonlinear function dig, = 20 + 1) if Fi,, 2 O else 0.7 where NO0 0. Similar to GP, mRNP shows the tendency to quickly move towards a flat prediction outside the areas with observed samples (section D in the supplement).
Modality
Modility
Figure 4: Predictive distributions for the first toy multi-modal regression task based on mRNP (top) and GP (bottom). Shaded areas correspond to +3 standard deviations.
5 CONCLUSION
We have proposed a novel Bayesian relational generative method, tailored for multi-modal neural processes with any missing patterns in their input modalities mRNP. The mRNP learns the dependency graph among the samples of the given modalities and takes advantage of it to define distributions over the functions of multi-modal data types. Unlike traditional stochastic processes, mRNP can be optimized through mini-batch. By introducing the mixture-of g-gaphss (MoG), we also could address the issues such as e.g. precision miscalibration of experts and computational complexity observed in the available multi-modal generative models. We have evaluated mRNP on two different real-world multi-modal classification tasks, which demonstrate that not only mRNP substantially outperforms SOTA methods in terms of both prediction accuracy and uncertainty quantification. but also it captures a meaningful joint (structured) representation across views. Furthermore, visualization results on two toy multi-modal regression tasks show that the predictive distributions of mRNP is similar to GP with an RBF kernel.
9

---

Under review as a conference paper at ICLR 2022
REFERENCES
Galen Andrew, Raman Arora, Jeff Bilmes, and Karen Livescu. Deep canonical correlation analysis. In International conference on machine learning, pp. 1247-1255, PMLR, 2013.
Ricard Argelaguet, Britta Velten, Damien Arnol, Sascha Dietrich, Thorsten Zenz, John C Marioni, Florian Buettner, Wolfgang Huber, and Oliver Stegle. Multi-omics factor analysis-a framework for unsupervised integration of multi-omics data sets. Molecular systems biology, 14(6):e8124, 2018.
Francis Bach and Michael I Jordan. A probabilistic interpretation of canonical correlation analysis. 2005.
Jordi Barretina, Giordano Caponigro, Nicolas Stransky, Kavitha Venkatesan, Adam Margolin, Sungjoon Kim, Christopher Wilson, Joseph Lehár, Gregory V Kryukov, Dmitriy Sonkin, et al. The cancer cell line encyclopedia enables predictive modelling of anticancer drug sensitivity. Nature, (
2012.
Xi Chen, Diederik Kingma, Tim Salimans, Yan Duan, Prafulla Dhariwal, John Schulman, llya Sutskever, and Pieter Abbeel. Variational lossy autoencoder. arXiv preprint arXiv:1611.021 2016.
Andreas Damianou and Neil D Lawrence. Deep gaussian processes. In Artificial intelligence and statistics, pp. 207-215. PMLR, 2013.
Dheeru Dua and Casey Graff. UCI machine learning repository, 2017.
Yarin Gal, Jiri Hron, and Alex Kendall. Concrete dropout. In Advances in neural information processing systems, pp. 3581-3590, 2017.
Marta Garnelo, Dan Rosenbaum, Christopher Maddison, Tiago Ramalho, David Saxton, Murray Shanahan, Yee Whye Tch, Danilo Rezende, and SM Ali Eslami. Conditional neural processes. In International Conference Machine Learning, pp. 1704-1713, PMLR, 2018a.
Marta Garnelo, Jonathan Schwarz, Dan Rosenbaum, Fabio Viola, Danilo J Rezende, SM Eslami, and Yee Whye Teh. Neural processes. arXiv preprint arXiv:1807.016622 2018b.
Gregory Gundersen, Bianca Dumitrascu, Jordan T Ash, and Barbara E Engelhardt. End-to-end training of deep probabilistic eca on paired biomedical observations. In 35th Conference on Uncertainty in Artificial Intelligence, UAI 2019, 2019.
Geoffrey E Hinton. Training products of experts by minimizing contrastive divergence. Neural computation, 14(8):1771-1800, 2002.
Hyunjik Kim, Andriy Mnih, Jonathan Schwarz, Marta Garnelo, Ali Eslami, Dan Rosenbaum, Oriol Vinyals, and Yee Whye Teh. Attentive neural processes. arXiv preprint arXiv:1901.0576/1 2019.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint UrXiv:1412.6980 2014.
Arto Klami, Seppo Virtanen, and Samuel Kaski. Bayesian canonical correlation analysis. Journal of Machine Learning Research, 14(4), 2013.
Achim Klenke. Probability theory: comprehensive course. Springer Science & Business Media, 2013.
Changhee Lee and Mihaela Schaar. A variational information bottleneck approach to multi-omics data integration. In International Conference on Artificial Intelligence and Statistics, pp. 1513-1521. PMLR 2021.
Juho Lee, Yoonho Lee, Jungtack Kim, Eunho Yang. Sung Ju Hwang, and Yee Whye Teh. Bootstrap- ping neural processes. arXiv preprint 02X05 2020.
10

---

Under review as a conference paper at ICLR 2022
Jöm Lötsch and Alfred Ultsch. Current projection cetthdd-innducee biases at subgroup detection for maachin--learingg based data-analysis of biomedical data. International journal of molecular sciences, 21(1):79, 2020.
Christos Louizos, Xiahan Shi, Klamer Schutte, and Max Welling. The functional neural process. 2019.
arXiv preprint
Chris J Maddison, Andriy Mnih, and Yee Whye Tch. The concrete distribution: A continuous relaxation of discrete random variables. arXiv preprint (xXi::6/11.007// 2016.
Jishnu Mukhoti and Yarin Gal. Evaluating bayesian deep learning methods for semantic segmentation. arXiv preprint arXiv: 1811:188,,, 2018.
Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped dqn. arXiv preprint arXiv:1602.04400 2016.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary De Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F, d' d'Alché-Buse E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems 32, pp. 8024-8035. Curran Associates, Inc., 2019.
Moshe Shaked and George Shanthikumar. Stochastic orders. Springer Science & Business Media, 2007.
Yuge Shi, Narayanaswamy Siddharth, Brooks Paige, and Philip HS Torr. Variational mixture-of- experts autoencoders for multi-modal deep generative models. arXiv preprint arXiv:1911.03393, 2019.
Kihyuk Sohn, Honglak Lee, and Xinchen Yan. Learning structured output representation using deep conditional generative models. Advances neural information processing systems, 28:3483-3491. 2015.
Flood Sung, Yongxin Yang, Li Zhang, Tao Xiang, Philip HS Torr, and Timothy M Hospedales. Learning to compare: Relation network for few-shot learning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 119--208, 2018.
Masahiro Suzuki, Kotaro Nakayama, and Yutaka Matsuo. Joint multimodal learning with deep generative models. arXiv preprint arXiv:1611.0mm 161899, 2016.
Bruce Thompson. Canonical correlation analysis: Uses and interpretation. Number 47. Sage, 1984.
Michalis Titsias. Variational learning of inducing variables in sparse gaussian processes. In Artificial intelligence and statistics, pp. 567-574. PMLR, 2009.
Katarzyna Tomezak, Patrycja Czerwińska, and Maciej Wiznerowicz. The cancer genome atlas (tcga): an immeasurable source of knowledge. Contemporary oncology, 19(1A):A68, 2015.
Martijn van Breukelen, Robert PW Duin, David MJ Tax, and JE Den Hartog. Handwritten digit recognition by combined classifiers. Kybernetika, 34(4):381-386, 1998.
Laurens Van Der Maaten. Accelerating t-sne using tree-based algorithms. The Journal of Machine Learning Research, 15(1):3221-324 2014.
Ramakrishna Vedantam, Ian Fischer, Jonathan Huang, and Kevin Murphy. Generative models of visually grounded imagination. arXiv preprint arXiv:170s. 10762, 2017."
Seppo Virtanen, Arto Klami, and Samuel Kaski. Bayesian cca via group sparsity. In Proceedings of the 28th International Conference on International Conference on Machine Learning, pp. 457-464, 2011.
Weiran Wang. Raman Arora, Karen Livescu, and Jeff Bilmes. On deep multi view representation learning. In International conference on machine learning, pp. 1083-1092, PMLR, 2015.
11

---

Under review as a conference paper at ICLR 2022
Mike Wu and Noah Goodman. Multimodal generative models for scalable weakly supervise learning. arXiv preprint arXiv: 2018.
Hongteng Xu, Dixin Luo, Ricardo Henao, Svati Shah, and Lawrence Carin. Learning autoencoders with relational regularization. In International Conference on Machine Learning, pp. 10576-10586. PMLR, 2020.
Shiwen Zhao, Chuan Gao, Sayan Mukherjee, and Barbara E Engelhardt. Bayesian group factor analysis with structured sparsity. The Journal of Machine Learning Research, 17(11) ((1))6868 6914, 2016.
A DETAILS ON THE EXPERIMENTAL SETUPS, RESOURCES, AND RUNTIME
In both classification tasks, we used early stopping, based on the accuracy on the validation set and no other regularization was employed. In order to avoid sub-optimal initialization in CCLE dataset, we also used the validation set for initialization tuning of the parameters. We also employed soft-free bits (Chen et al. 2016) modification of the bound to help with the optimization. More precisely, we allowed one free bit on average across all dimensions and batch elements throughout training of mRNP.
During the regression experiments, we randomly select 10 samples from training set as X' and consider the dimensions (D.,, D2) as (3, 80) and {3, 10} for the first and second tasks, respectively. In both cases we use one fully connected layer with ReLu as the activation function to transfer x, to he, following with linear layer learn the parameters of pa (u,, x,). and linear layer to learn the parameters of (z( X). The dimension of the hidden layer is set to be 100.
All of the experiments
single GPU node Tesla K80. Each training epoch of mRNP for HW, CCLE, and toy regression took 1.08, 0.53, and 0.02 seconds on average, respectively.
B OTHER RELATED WORKS
Multi-modal VAE (continued). Shi et al. (2019) introduces mixture-of-experts multimodal VAE (mm VAE) that does not suffer from potentially overconfident expert. However, is not able to handle missing modalities during the training, and thus limited in its applicability, especially for biomedical data of small sample sizes with missing views. Our mRNP combines benefits of both mAEE and mm VAE, which is especially critical to deal with high- -igh-imensional data integration. Compared to these approaches, mRNP can general be more scalable since does not require sub-sampling as m VAE does. Instead, it effectively takes a vote amongst the graphs and spreads its density over all the individual graph through a, and 3 While the aforementioned models attempt to learn low-dimensional latent variables for multiple views, the focus of mRNP is to introduce distribution over the functions in supervised setting.
Conditional VAE (CVAE). CVAE (Sohn et al., 2015) are conceptually similar to NPs and has been used to train multi-modal generative models. Unlike the global latent variables that are present in the NPs, the latent variables of CVAES are local and their decoder functions are deterministic. Hence, needs to be separately sampled for each of the target predictions y," This is in contrast to the latent variable of an NP that is only sampled once and used to predict multiple values of yf More precisely, conditioning the labels of training samples is done by adding the dependency both in the prior p(2" Ue) and decoder p(y" 22" Ye) leading to a deterministic function of the training samples.
CCA based models. There is a rich literature on the extension of CCA to generative modelling (Bach & Jordan 2005) Virtanen et al.] 2011]] Klami et al., 2013) Wang et al., 2015) Argelaguet et al. 2018). These extensions can deal with high dimensional data small sample size (Argelaguet et al. 2018) and outperform classical CCA in terms of interpretability and expressive power. Some of them, such as (Bach& & Jordan 2005| Klami et al. 2013). are generic factor analysis models that decompose the data into shared and view-specific components, and impose additional constraint to extract the statistical dependencies between the views. Most of the generative methods such MOFA (Argelaguet et al. 2018) retain the linear nature of CCA while providing more robust.
12

---

Under review as a conference paper at ICLR 2022
inference compared the classical solutions. There are also number of recent models based on VAE that incorporate non -ineariiy and keep the probabilisticc interpretability of CCA (Virtanen et al.. 2011 Gundersen al. 2019). However, they require additional computation to handle missing modalities(Wu & Goodman 2018).
C MRNP AS STOCHASTIC PROCESS
mRNP corresponds to Bayesian models as the distributions defined in Equation (7) are valid, permutation invariant stochastic processes.
Proposition I.
proof. Similar Garnelo et al. (0188) and Louizos et al. (019)). rely on de Finetti's 's and Kolmogorov Extension Theorems Klenke (013) that states exchangeability and consistency as two
sufficient conditions define stochastic process.
Exchangeability. This condition requires p(yB XB) to be permutation invariant. More precisely, each probability density of the model be permutation invariant, then the overall probability will be permutation equivariant. Based (x.), where i; = (u: juth modality present heev and x; = {x e modality present Therefore, we can show that the latent variables are permutation equivariant with respect X' as
equation (1) in the main paper, can consider
fulo(XB)) = (Xu
where (-) a permutation function. Then, mRNP defines each elements of the shared graphs A and G, denoted as at,j and gi.j. as a function of (ū: ú,). Therefore, we also have permutation equivariance for the rows and columns of these two graphs. Based on equation (5) in the main paper, mRNP defines the distribution over 2; an independent distribution whose parameters are function of either or G, therefore, the distribution of n, is invariant to parents' permutations. Therefore, the permutation of X' will result in the same re-ordering of the Zg as follows
o (Z) (a((((X)),,
where fz XB - ZB. Let's also define fy 2i = = f((((())))
yi. As result, we can also show
The product, integral, and summation operators are all permutation invariant. And since we show all random variables our mRNP model are permutation equivariant to XB, mRNP model is permutation invariant.
Consistency. This condition requires to show that if we marginalise out a part of p(y [g), the resulting marginal distribution is the same as that defined on the original space. To this end, we follow Louizos et al, (2019) and define XB = XB U {Xo, yo Then, we need to show p(yB (B)) = J á XB)) dyo in two cases; where (Xo, yo belongs to either the training set Dr or the reference set that
part of the training set, i.e. Xx \D,
In the first case, where (Ro, 30 E DD., the new sample will be added to a leaf in the dependency graph. Therefore, it will not affect any of the samples in Xn, hence can marginalize it out as follow:
f pllynllpp dyo Σ
/
XB.,) (B.)) PGG. (o Zn A, G. XR)
G.A,a0
DEV
po( (xan) üg UR) po (zol par (RR. (ʃ pooouol ) dllg dZg dü, dzn-
As po(vol 2o)dy0 = 1, we can re-write above equation as:
f plyB En) dyo Σ
/ II
po (O n, Xn.) PG, po Zn A, G, XR)
G.A.a0
NEV
(f palzol para (KR: dzo) dll dZg düo. (8)
Po( X0,v)p(ag
(R)
13

---

Under review as a conference paper at ICLR 2022
where
J po( par (Kn, dzo 1. We further can summarize the equation 8
I ply& (B)) dyo Σ
II
Po (U,, B, XB,)) (()))((G.AA A (UB) poly Zz A, G.
G.A
NEV
Ep(au p(au (w,, AMg dZ duo.
Po Xo,²)
(9)
Hence, can further marginalize dug as:
J ply& K3) dyo
=E/
/
II NEV
(o XB,) (G,)) polyB, Zz A, GR
G.A
(f po( xo.o)dūo dUB dZg. (10)
Finally, f pe(uu,e Ido 1, we can drive equation (7) in the main paper. More preciscly, we will have
f ply& RB) dyo -
/
Σ G.A
II
pa UB. XB,)) p(G, (4B) polyB, A, G. Xn) dMl dZB
"EV
p Xg))
In the second care, where {8o, yo} E XR Dz, {8o, yo belongs to the reference set, but not included the training set. As we discussed in the main paper, the sets Xas and Dr will be same across Xg and L'.. Hence we need to marginalize out in terms all samples in XR ADz as follow
i ply (B) dyo
Σ
/
II
Po BU, XB.) xn.))(G,AA Un) po Zn A, dl)
G.A
BEV
pyyB B XB)..
(11)
Therefore, mRNP consistent under marginalization in both scenarios.
D ADDITIONAL EXPERIMENTS
In addition to Figure 4 in the main paper, we further demonstrate that mRNP shows similar behaviour GP to encode the inductive biases. In this regression study, draw 20 samples from U1-4, 4] for the first modality and simulate the target labels as y/s F,,.. + where e ~ AN, 92) Then, generate samples for the second modality
nonlinear function Tg,0, 2x + if Ti,m 2
0 else 0.7 where (O, 0.1²). Figure SI shows the results of both modalities. Similar to GP. mRNP shows the tendency to quickly move towards flat prediction outside the areas with observed samples.
For this regression task, we train mRNP for maximum of 1500 epochs using Adam (Kingma Ba 2014) with a learning rate of 0.001. We would also like point out that separately train an RBF kernel GP for each modality.
E MoG vs POE
As stated overconfident predictions by expert in PoE can be detrimental to the whole model. By contrast, MoG does not suffer from potentially overconfident graph, since it effectively takes a vote amongst the graphs, and spreads its density over all the individual graphs. Let us assume that have two input modalities, and the prediction the first expert (graph) is noisy and overconfident. We would like to compare the effect of this two modeling perspectives.
14

---

Under review as a conference paper at ICLR 2022
True functon fanction
Observadirs Reference
Trus feectin
100
50
25
25
25
50
-50
-15
-75
302
100
-25
-25
-75
-10
-100
Modality
Modality
Figure S1: Predictive distributions for the second toy multi-modal regression task based on mRNP
(top) and GP (hottom).
Effeet of overconfident predictions of expert in PoE model. PoE assumes p(z x1)x2) =
p(z] x1) p(z x2).
Considering we have normal distribution in both experts, i.e. p(z x1) = NNoo σo) and p(z x2) = af
(H)0))) Therefore, p(z X1, x2) = (Mz,, 0z), where μı= Mon +u1o and o? og af+oî
Let us
consider that the first expert is overconfident, i.e. do 0. As result, p(z x1, X2) = N og) This means that the second expert does not matter in this situation and if the first expert be potentially overconfident due difference in complexity of input modalities or initialisation conditions, second one does not have a chance to correct it.
Effect of overconfident learning graph in MoG model. MoG assumes p(z) A1, A2) p(z aA + (1 A2),, where A₁ and A2 are the graphs in two view.
According to equation (4) p(z; |AA1 (1 a)A2)
IP pZz aA + (1 A2) = IP: Nu (i) where pig Σ, (aAin + (1 - A) pp prior,a and o
exp (aA1,, (1 22,))) sph ) If assume A₁ is completely noisy, still Ag is able to correct it. In addition, we can incorporate prior knowledge through a. If we have a priori information that the first view is noisy, can consider & (1 a)) that leads to reduce the negative effect of the first view.
15


Reviews:
[{"review_id": "jPp7sWBUHN", "reviewer": "Reviewer_bvzL", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n\n- The problem raised in this paper is interesting. Due to the presence of missing modalities at test time, estimating the uncertainty is important in practice.\n\n- While the neural process has been applied in a wide range of ML tasks, it is the first time to be applied to multimodal learning.\n\n\nWeakness:\n\n- The writing of this paper is terrible. It is difficult to understand to the main content of this paper by simply going through the paper from top to bottom. I spend lots of time to figure out what’s the main idea of this paper, but failed. Then, I came to one of the reference of this paper, Functional Neural Process, then everything becomes clear. The idea in Functional Neural Process is introduced much better and clearer than this submission. \n\n- This submission looks strikingly similar to the paper of Functional Neural Process. I can say that the underlying ideas of the two paper are almost the same, including the introduction of directed acyclic graph in the model. A slight improvement is to mix multiple graph into one (MoG) to handle the modality missing problem. However, the content relevant to this part only account for a very small proportion. Most of the techniques used in the paper have been discussed in the previous FNP paper. It is better to pay more attention to how to adapt FNP on multimodal learning. Besides, how does the performance look like with FNP in figure 4 and s1?\n\n\n- To address modality missing problems, the authors propose a MOG method to model the relationship among different modalities. However, the construction of graphs A and G exactly confuses me. You first construct A_v and G_v for each modality separately, and then the whole graph A and G is the sum of all subgraphs (A_v, G_v) on different modalities. That means, graphs A and G are shaping in a block-like, and there are no edges between different views. From this perspective, there are no interactions among different modalities. The whole training process is equal to the way that trains multi models on different modalities separately, but they share the same parameters. This is a quite different training scheme compared with previous multimodal learning models, and also confusing.\n\n- Uncertainty estimation is an important issue in multimodal learning. Ideally, the more modalities missed at test time, the higher uncertainty it is. However, there are no corresponding experiments to evaluate this hypothesis. I think it is necessary to explore how does the missing rate at test time influence the uncertainty degrees.\n\n- Another question is how do you evaluate the model. If I understand correctly, given a training data {\\bar{x}, y}, you assign each data point x_v \\in \\bar{x} the same label, i.e., y, and use it to maximize the ELBO. At test time, according to the predictive distribution, you predict a label y_v for each test data x_v \\in \\bar{x}. Since there is a unique label for \\bar{x}, my question is, how do you predict the label for \\bar{x}, according to the predictions y_v on different views? Maybe I miss something, but I think it needs to be clarified more clear.\n", "summary_of_the_paper": "This paper proposed to use neural processes for supervised multi-modal learning, which have the abilities to estimate the uncertainty of prediction and to handle missing modalities. Specifically, a directed acyclic graph is learned for each modality in the neural process, which are then used to construct a mixture-of-graphs (MoG) to sidestep the modalities missing problem. The introduction of inducing points makes the predictive distribution tractable. Experimentally, on the label prediction and uncertainty estimation tasks, the proposed model performs well compared to the recent multimodal learning methods.", "summary_of_the_review": "The problem investigated in this paper is. But I think the contribution of this paper is not clear and the writing of this paper should be improved. I recommend the authors to take a thorough review of the field of neural process and think about how effectively adapt the it to address the modalities missing problems.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "MA7-IPShuNd", "reviewer": "Reviewer_fVe5", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths:** The integration of a dependency graph is a novel contribution to\nmultimodal learning and overall the empirical results look promising. The paper\nis transparent about the limitations of the proposed approach, for instance,\nthat it is fully supervised and that the results can dependent on the\nchoice of a good reference set.\n\n**Weaknesses:** First, the paper makes strong claims about previous work that\nare not sufficiently well supported. In particular, claims about the\ncomputational efficiency and the improved approximation of the joint posterior\nrequire a better formalization and/or empirical support. Second, the\nexperiments include mostly unsupervised baselines, but most of the reported\nmetrics are based on labels, which puts the unsupervised models at a clear\ndisadvantage. Hence, the experiments would benefit from having more supervised\nbaselines.\n\n**Computational complexity:**\nThe paper makes strong claims about the computational inefficiency of previous\napproaches (mVAE and DeepIMV), but it is not clear how the proposed method\nresolves these issues. Since these arguments are stated very prominently in the\nabstract, conclusion, and throughout the paper, a formal comparison of the\ncomputational complexities would be helpful.\n\n**Approximation of the joint posterior:**\nThe paper claims that the proposed method effectively approximates the joint\nposterior of multi-modal data types, in contrast to previous methods.  However,\nthe unsupervised baselines do not model the same posterior and therefore the\ncomparison is arguably not very fair. Instead, one could compare to other\nsupervised or semi-supervised approaches (see separate paragraph). \n\n**Miscalibration of the precision of experts:**\nThe claim about the miscalibration of experts is positioned very prominently\n(e.g., in the abstract and conclusion), but there might not be sufficient\nevidence to justify using this explanation to argue in favor of the proposed\napproach. I am aware that the claim originates from previous work, but there,\nthe statement was less central and the results did not depend on this\nexplanation. However, the present paper seems to convert a hypothetical\nexplanation to a fact that is heavily relied upon throughout the paper.\n\n**Insufficient supervised and semi-supervised baselines:** \nAlmost all of the baselines are unsupervised methods, whereas the proposed\napproach is fully supervised (a semi-supervised extension is discussed as an\nopportunity for future work). It is questionable to frame the requirement of\nlabeled data as an advantage of the proposed approach compared to existing\nmultimodal generative models, which are unsupervised and could potentially be\nextended to handle semi-supervised data (Joy et al. 2021). Hence, the\nexperiments seem to be lacking supervised and semi-supervised baselines and the\npaper has to explain why it would suffice to consider only DeepIMV as a\nsupervised baseline.\n\n\n**Questions and comments:**\n\n- Why do you refer to the proposed approach as a generative model?  Is the\n  proposed approach able to *generate* samples from the respective modalities\n  modalities? If not, the naming makes the comparison to multimodal generative\n  models (e.g., the mVAE) confusing.\n- Since you mention CCA-based methods, for which there are many semi-supervised\n  and multi-view or multi-modal extensions, did you consider comparing to these\n  approaches?\n- In the introduction it is written that \"training a PoE\" is difficult, which\n  is confusing. It requires an algorithm to train a model that integrates a\n  PoE.\n- The authors frequently write that the proposed method \"learns individual views\n  faithfully\". This statement should be made more formal, especially since the\n  quality of an encoding is relative to a downstream task.\n- In the introduction, it is not clear why stochasticity is a problem for\n  multimodal generative models. The statement is not backed up by an\n  explanation or reference.\n- The relation to attention mechanisms needs to be formalized. In its current\n  form, the explanation of the relationship is too superficial.\n- Table 2: mRNP accuracy seems to be on a different scale.\n- Figure 2: consider using t-SNE or UMAP instead of PCA for the baselines.\n- Figure 3: please add values for 0% on the x-axis and standard deviations for\n  all methods. Table 4 suggests a significant overlap, if we consider standard\n  deviations.\n\n\n**References**\n- Joy, T., Shi, Y., Torr, P. H., Rainforth, T., Schmon, S. M., and Siddharth,\n  N.  (2021). Learning Multimodal VAEs through Mutual Supervision. arXiv\n  preprint arXiv:2106.12570.\n", "summary_of_the_paper": "The paper proposes a Bayesian model that learns a predictive distribution given\nmultimodal data with labels. The novelty of the proposed method is that it\nlearns a dependency graph between samples from different modalities and\nintroduces a new mixture-of-graphs (MoG) method to aggregate different\ndependency graphs.  Experimental results on several datasets demonstrate that\nthe proposed method, which is fully supervised, improves the predictive\nperformance compared to two unsupervised approaches and one supervised\nbaseline. Qualitative results corroborate the quality of the learned\nrepresentations, which can be visualized as relational graphs, whereas the\nbaselines require additional dimensionality reduction techniques for\nvisualization.", "summary_of_the_review": "The paper introduces an interesting approach to a relevant problem and reports\npromising empirical results. However, the paper makes strong claims in its\ncomparison to previous work, some of which are not sufficiently well supported.\nFurther, for a fair comparison, the experiments would benefit from more\nsupervised and semi-supervised baselines. Therefore, I tend towards rejecting\nthe paper in its current form.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "JPOa379RtJC", "reviewer": "Reviewer_Vnks", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This work proposed a multi-modal relational neural process to provide a predictive distribution. Meanwhile, MoG is introduced to integrate information from different modalities. Generally, this paper is easy to read.\n\nI have the following concerns about the paper:\n\n\n   1. A literature review is lacking, especially for Gaussian processes-based multi-modal / multi-view classification methods. The discussion of the multi-view Gaussian process method should be conducted. The following papers are very relevant to this paper.\n\nMultiview learning with variational mixtures of Gaussian processes. Knowl. Based Syst. 2020\n\nMulti-view representation learning with deep gaussian processes.\n\nMultimodal similarity gaussian process latent variable model.\n\n\n2. Some statements in the article are biased. For example, the introduction states that “existing generative models for multi-modal learning focus on latent representation, but do not fully incorporate the label information.”  Generative models are only one of the branches of multimodal learning, and many multimodal supervised models have been developed, e.g., DeepIMV in the paper.\nThe proposed MoG is not convincing and novel, which just averages the graphs from different modalities. In my opinion, this contribution is weak. \n\n3. The experiment of the paper is not sufficient. The paper only compared the two baseline methods on two real-world datasets. At the same time, the experiment did not use large-scale datasets to illustrate the author’s claim \"makes mRNP scalable to large datasets through mini-batch optimization\". \n\nIn addition, I found some problems in the writing. \nIncorrect use of semicolon “;” and colon “:” in the introduction.\nThe meaning of the symbol is not introduced when it first appears (e.g., φ$_{sim}$ in Eq 2).\nThe title of the paper is multi-modal, but multi-view is used many times in the paper.", "summary_of_the_paper": "This paper proposes a multi-modal Relational Neural Process, which learns a dependency structure among the samples.  To integrate information from multiple modalities, mixture-of-graphs (MoG) is introduced. ", "summary_of_the_review": "The article should conduct more thorough literature research, method design, and improvement of writing.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "8tNVoc0VIA0", "reviewer": "Reviewer_VW19", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The method is interesting, and the introduction is attractive to read the following. But the part for methods are a little confusing. Moreover, it seems the authors are not clear about variational inference. The paper in page 5 shows Evidence Lower Bound (ELBO) is minimized. In fact, it should be maximized.", "summary_of_the_paper": "This paper tries to use Bayesian relational generative model for scalable multi-modal learning. They propose a class of stochastic processes that learns a graph of dependencies between samples across multi-modal data types through adopting priors over the relational structure of the given data modalities. The so-called mRNP method can address the limitations in joint posterior approximation. ", "summary_of_the_review": "It may not be so fit for iclr.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper extends the FNP model to multimodal settings using the mixture of graphs. However, there are legitimate concerns about the quality of experiments, such as baselines, as the reviewers mention. For example, mRNP is supervised, and comparison to DeepIMV is not fair. I encourage the authors to address them appropriately in the next version of the paper. 

The authors can significantly improve the presentation of ideas. Please avoid making hyperbole and excessively bold statements, as the reviewers have pointed out. This way, there will be room for a better demonstration of the novel parts of the paper. For example, the authors misuse the term "generative" for the proposed mRNP. There are multiple hand-waving statements about the role of uncertainty that are not well-supported in the current draft. I believe this paper can be a good paper by addressing the reviewers' comments.
Decision: Reject