Multi-View Representation Learning via Total Correlation Objective
Authors: HyeongJoo Hwang
Geon-Hyeong Kim
Seunghoon Hong
Kee-Eung Kim
Venue: NeurIPS 2021 Poster (2021)
Keywords: Multi-View Representation Learning
Multimodal Generative Models

Abstract: Multi-View Representation Learning (MVRL) aims to discover a shared representation of observations from different views with the complex underlying correlation. In this paper, we propose a variational approach which casts MVRL as maximizing the amount of total correlation reduced by the representation, aiming to learn a shared latent representation that is informative yet succinct to capture the correlation among multiple views. To this end, we introduce a tractable surrogate objective function under the proposed framework, which allows our method to fuse and calibrate the observations in the representation space. From the information-theoretic perspective, we show that our framework subsumes existing multi-view generative models. Lastly, we show that our approach straightforwardly extends to the Partial MVRL (PMVRL) setting, where the observations are missing without any regular pattern. We demonstrate the effectiveness of our approach in the multi-view translation and classification tasks, outperforming strong baseline methods.

Multi-View Representation Learning via Total Correlation Objective
HyeongJoo Hwang, Geon-Hyeong Kim, Seunghoon Hong, Kee-Eung Kim
KAIST
Ohjhwang, ghkim)@ai kaist. ac.kr, .r, [seunghoon hong, Akaist.
Abstract
Multi- Representation Learning (MVRL) aims to discover shared representa- tion of observations from different views with the complex underlying correlation. In this paper, we propose a variational approach which casts MVRL maximizing the amount of total correlation reduced by the representation, aiming to learn a shared latent representation that is informative yet succinct to capture the correla- tion among multiple views. To this end, we introduce a tractable surrogate objective function under the proposed framework, which allows our method to fuse and cali- brate the observations in the representation space. From the information theoretic perspective, we show that our framework subsumes existing multi-view generative models. Lastly, we show that our approach straightforwardly extends to the Partial MVRL (PMVRL) setting, where the observations are missing without any regu- lar pattern. We demonstrate the effectiveness of our approach in the multi-view translation and classification tasks, outperforming strong baseline methods.
Introduction
Multi- View Representation Learning (MVRL) aims to learn shared representation of multiple observations from different types of views. In MVRL, it is important encourage the shared representation to be complete enough to capture the correlation across views without losing view- specific information so that the learned representation tasks. For example, in sensor fusion for the multi-sensor system [51] and in clinical diagnosis based patients' various types of medical records 47 491. would like to aggregate all the information from various observations in order to uncover the true underlying factors under the correlation. Although using views many as possible seems be always beneficial for the task of learning a good representation, it can make the problem itself harder depending on the complexity of correlations across all views and the scalability of handling number of views.
be readily applied to rich set of downstream
MVRL becomes even more challenging when the model does not always have access to complete observations from all views for every data instance during training, which call Partial Multi- View Representation Learning (PMVRL). PMVRL closer to the practical setting since it is unrealistic to expect that observations from all different kinds of views are always available. For example, it unlikely for all the sensors a sensor-based system to have the same frequency to update their measurements or for all kinds of medical records be available for any patient.
Considering those difficulties, any desirable MVRL methods are encouraged to satisfy three desiderata. The first one is scalability to the number of input views. The method should handle multiple observations in way computationally scalable arbitrary many views in both training and testing time. In addition, the method needs to be robust to partial observability. The method should be able to combine any combination of available observations in the representation space in test time. This is important for PMVRL because we have to handle observations arbitrarily missing not only in testing data but also in training data. Lastly, the method should discover cross-view association,
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

which can be learned by identifying both shared and view-specific factors of variation of each view in the representation space. Associating views correctly in the representation space allows the method to utilize any additional observations from different views to improve identification of the shared factors without determining any unobserved ones.
Combining multiple of Variational Auto-Encoder (VAE) [22] for each view several generative models [30 33 34 46 have been recently proposed to address MVRL. Since naive optimization of the evidence lower bound (ELBO) on the joint likelihood of multiple views does not address any of the desiderata, those methods impose different structural bias on the joint representation encoders whose strengths and weaknesses are complementary (see Section[.4] Although they showed encouraging performance on multiple tasks such as predicting common attributes and inferring missing views, they often fail to simultaneously capture both the shared and view-specific factors of variation and turn out to poorly associate views in our experiments.
In this paper, we address the problem of MVRL with a principled approach grounded in information theory. Specifically, we formulate the representation learning task as maximizing the reduction in Total Correlation 12 38 39). Based on our formulation, we derive a novel objective function that not only offers tractable optimization but also introduces multiple types of variational information bottlenecks which successfully associate views. We then show that our method naturally extends to the PMVRL setting via inverse variance weighting, classical approach used in sensor fusion. We demonstrate the validity and effectiveness of our method in the multi-view translation and classification downstream tasks. Our contributions are three-fold:
1. Measuring the informativeness of the multi-view representation by Total Correlation, we propose general information-thhoeeee framework to learn a complete representation, which encompasses existing multi-view generative models.
2. Under the proposed framework, we identify drawbacks of optimizing ELBO and derive a novel objective function that resolves them. Specifically, our method yields a representation that correctly associates views by capturing not only the common factors of variation but also the view-specific ones.
3. We conducted extensive evaluation with comparing methods and ours in translation and classification tasks in both MVRL and PMVRL settings, showing that our method is the most reliable method to obtain the latent representation agnostic to the downstream tasks.
2 Approach
Let Ø {o_) be the observation of a data instance composed of V different views, which is sampled from an unknown joint distribution pD (ō), where we emphasize that this is a data distribution using the subscript D. Given these observations, MVRL is the task of learning complete representation z across the views o. Following [50], we define a complete representation as follows.
Definition 1 (Completeness for Multi-Vier Representation (50) A multi-view representation complete if each observation, i.e., Ov from o, can be reconstructed from a mapping fo()), i.e o, = fz)..
The definition directly indicates that a complete representation describes all factors of variations in o. since every view can be reconstructed solely from the complete representation. While MVRL considers complete observations for training data, PMVRL assumes otherwise, i.e. some views being missing in the training data. This poses a unique challenge of (1) learning produce the complete representation with partial views (2) whose availability varies per instance in training and testing.
We first relate informativeness of the representation measured in terms of Total Correlation (TC) and the goal of MVRL (Section 2.1). We then show that the TC-based MVRL objective function encompasses existing multimodal generative models and analyze its limits based on straightforward variational lower bound (Section 2.2). To resolve those limits, we derive an alternative variational lower bound which suits better for MVRL (Section 2.3) Lastly, we finalize our formulation by proposing the representation aggregation model that naturally extends to PMVRL (Section 2.4
We follow the conventional terminology in the related literature 11 14 24 41
44 47 48 50 53 but any
choice among views, modalities [30 methods, long there is common latent representation that explains different views / modalities / domains.
33 34 46). and domains [19 can be suitable for our paper and our baseline

---

2.1 Multi-View Representation Learning with Total Correlation
complete representation Z should be informative enough to explain the correlation among V different views. Total correlation (TC) [42], defined as the Kullback-Leible divergence of the joint distribution from the factored marginals, measures the amount of information shared among finite set of random variables. In our MVRL. context, TC. is defined
V
TC(Ô) & DKL PD (ō)
II PD (ow) ==1
(1)
We aim to find the encoder pe (2|ō) such that the knowledge of z would reduce TC as much as possible. This be formulated by maximizing the objective
TC; Z) e TC(Ô) TC,( 2):
where the conditional TC in the last term is given by
TCo(olZ) & Exp()) DEL po (o)z)
II po (og
which is the expected Kullback-Leible divergence of the joint conditional from the factored condi- tionals. The parameterized distributions in the above formula involve the encoder po (=|0) as follows: Pe (z) f Pe (2|0) PD (Ø) dō, pe (Ø]z) = pe (zlō) PD (o)/pe (z), and (ow12) = pe (ō]z) dō n-
Intuitively, minimization of Eq. (3) (i.e., maximization of Eq. 2 suits well for MVRL, since (1)
any complete representation Z would minimize Eq. 3 (e. 7 o), and (2) it accords with the
theoretical result that complete representation should factorize the generative distribution 44 50
Further decomposition of (2 reveals that it encourages Z to encode the correlation across views [2] desirable property for MVRL:
OO: Σ Io -6;;
(4)
The first term in Eq. (4). Mutual Information (MI) between each observation and the representation, enforces the representation to be informative for every observation. On the other hand, the second term takes the role of Information Bottleneck (IB), which encourages the encoder to learn minimal sufficient representation
(37]. Consequently, simultaneous optimization of both terms naturally allows the representation Z to capture correlations among views while being minimally sufficient. Note that when V=2, Eq. 4 coincides with Interaction Information 4 25 36), which quantifies the amount of information shared among two views and their joint representation (see Section A.4).
2.2 Limitations of VAE models for multi-view data
Unfortunately, direct optimization of MIs in Eq. 44 intractable 12 A straightforward approach would be employing approximate distributions o (o,|z) R Ps (O 2) and (2) R (z),
and optimize a variational lower bound of Eq. 4 as follows (see Section A. in the supplementary material for full derívation):
(Ō: Σ H (O.) + + (p(()))pp [In [Ing (o.l2)]] Epp [KL [po(2|0)][r( (5)
VIB
where the entropy terms can be dropped from optimization since they are determined by the true data distribution PD (Ø). Without the entropy terms, we note that Eq. (5) is essentially identical to evidence lower bound (ELBO) of the variational auto-encoder (VAE) models for multi-view data [30 33 34 46 by switching the notations, for p for encoder and q for decoder. This notation
switch follows the convention in [
Although this lower bound contains the variational information bottleneck term that encourages the representation to be minimally sufficient for generalizing well even with small training data 22 has the following fundamental limitations:
3

---

1. Unbalanced representation: If subset of views is overwhelmingly informative enough to reconstruct the others, the encoder Pa (2|0) may learn rely on those views while ignoring the rest, yielding degenerate solution of Eq. (4) that fails cross-view association. This is problematic MVRL since such views may not be available test time.
2. Missing views: The encoder po (=lØ) requires complete observations not only in training but also in testing phases. while MVRL requires the model to encode incomplete observa- tions Ō S Ō test time. Furthermore, PMVRL requires handle incomplete observations even in training.
In order to overcome these challenges, prior methods impose special structures (i.e. inductive hypotheses) for pe (zlō), such as Product of Experts (PoE) [67] Mixture of Experts (MoE) [0), or Mixture of Product of Experts (MoPoE) [4]. In our work, we present a more principled approach this problem by deriving alternative lower bound, described in the next section.
2.3 Conditional Variational Information Bottleneck
To resolve the first issue raised in the previous section, we start from Eq. 4 and reformulating it as follows:
V
1 lo (O.; 2) + (Or: 1 (ō:
TC,(0;2) = Σ
v=l
Σ 1) - -00,;
where the last equality is due the chain rule of MI (see Section A.5 in the supplementary material for details). Interestingly, Eq. (6) transforms IB in Eq. (4) into multiple conditional MIs between the latent representation and V 1 other views given every view, each of which penalizes the extra information of the representation not inferable from the given view. Although Eq. (6) is essentially equal Eq. 4.. its conditional information constraints give us intuition derive new tractable lower bound on (Ō; Z) that regularizes unbalanced representation, which we present below.
Since the conditional MIs in Eq. (6) involve po(=|o,) = po (= ( compute intractable integration, we use the variational upper bound of those terms by introducing (=
which requires to
approximate distributions r (zloz) = (2lo,) as follows (see Section A.2 in the supplementary material for the full derivation and analysis):
TCo 2)
H (O+) + Ppot [In (ool=)]]
==1
E E
[DRr. [p((=l0]]r
Conditional VIB
This lower bound equipped with conditional VIBs which provide number of benefits over Eq. (5) in handling challenges mentioned in the previous section. First, conditional VIBs, which upper bounds condtional MIs Eq. (6) by introducing the view-specific encoder ră (zlo,) for each view, regularize pe (z)ō)) to encode representation inferable from v (zlo,) of every view. Consequently, the joint representation is enforced be balanced rather than be prone to uneven dependency on some subset of views. Second, each of them uses forward KL divergence D [po (zlo) IIr (z(o.)] to calibrate each encoder x (zlo,) to the joint encoder po (=lØ), encouraging r (=(o,) to cover all extract the representation even when
the supports or modes of po (zlō). As consequence, some views are missing in the observation. This property is critically important in (P)MVRL, where needs to infer the complete representation from the partially available views without being overly confident on any unobserved factors. We remark that mmJSD [3] adopts reverse KL divergence, which is not ideal
later demonstrate in the experiments.
Finally, although Eq. has aforementioned desirable properties for MVRL, is prone to overfitting when the size of training data is limited. This is because r optimize the conditional VIB by

---

simply memorizing instead of learning to infer the representation of pe (zl0). In order to prevent overfitting, we found that VIB in Eq. ( is an effective regularization it favors the minimal sufficient encoding of the representation, which will be demonstrated in Section 4.2.1] Therefore, we formulate our objective function as convex combination of Eq. ( and Eq. 7 so that we regularize the training via VIB (see Section A.3 in the supplementary file for full derivation):
V V-a
(C;
H (O.) + (pl())pp [ (ool=)]]
E (o) [DE [po(2]0]]]](((]o]]]]]
a) Epo(d) [DL (20())))|)r])]] (8)
Conditional VIB
VIB
where is the hyperparmeter that trades off learning minimal sufficient representation in favor of calibrating r.. For simplicity, we model the encoder, decoder, and approximate marginal distributions using the parameterized Gaussians with the diagonal covariance matrix, i.e. r (zlow) N (Ne, o2 & (o.lz) N (p., I), and (z) N (0, I), respectively.
While the view-specific encoders r. allow us to extract the representation from any available view individually, combining any subset of these representations (fusion) still remains as a problem. In addition, the of joint-view encoder p(=lō) makes our method limited to complete observations ø during training, which needs to be addressed for PMVRL. In the next section, we show that these issues can be effectively resolved by simple model design for po (zlØ).
2.4 Models for Joint Representation Encoder
We review the models adopted by prior methods that make the joint representation encoder Po (z|ō) amenable to missing views and discuss their strengths and weaknesses.
(PoE) [7 combines multiple probability distributions by their product. MVAE 46) models the joint representation encoder as a PoE, treating view-specific encoders as experts. The PoE can produce sharper distribution as we increase the number of input views, thus an effective method for aggregating information across any subset of view-specific encoders. Assuming each of view-specific encoders as Gaussian distributions such that r (=/o.)) N (Mo: I) ), the PoE joint encoder is obtained with computation linearly scales to the number of views by
PoE Product-of-
EV Mo/0? and p
Pe (zlō)= ف N ofT). where Mp
ΣV 1/0
E 1/0
Eq. (9) is also the formula of Inverse- Variance Weighted (IVW) method 7. classical method in statistics for aggregating multiple random variables, such as sensor fusion.
Unfortunately, naive application of the PoE to the ELBO formulation (Eq.(5) may fail to optimize the individual encoders, which is important in learning the balanced representation. MVAE [46), as an example, randomly samples subsets of views among 2 combinations and jointly optimizes their ELBOs in order ensure that all the view-specific encoders are optimized under PoE. However, such treatment may result in precision miscalibration of view-specific encoders [30].
MoE The Mixture-off ofre--of-Esee offExxert (MoE) takes an arithmetic mean of probability distributions, which is computationally scalable to the number of views as well. MMVAE (3) and mmJSD 333 adopt MoE of (2)o,) as the model for the joint representation encoder. In MMVAE, the MoE is trained by pair-wise optimization in such a way that the latent representation from view-specific encoder can reconstruct the observation in other views well as its own view. However, this does not necessarily imply that the latent representation successfully aggregates the information across views. mmJSD addresses this issue by adopting a common learnable prior across views.
We remark that, in Eq. 8 modeling Pe (z(0) as MoE of r (2lo,) and setting a = 0 yields the ELBO objective version of MMVAE. Assuming the same model for po (2 ()) and a = O, and modeling r(z) as the PoE of + (=lo,,) yields the objective function of mmJSD. For tractable optimization of the KL term involving MoE, mmJSD derives lower bound the ELBO by decomposing it into multiple KL terms. This bound can be also obtained from Eq. (8) by setting = and using the reverse KL for the conditional VIB terms. However, empirically show that minimization of the reverse KL is not helpful with learning complete representation in Section
5

---

A2
A
PV
O reconstructed data dacader
latent verlab e
IWW method
Mp
Op
encoder
Input cltaa
F11
μ02
D72
MA
0g
Figure 1: The architecture of Multi-View Total Correlation Auto-Encoder.
(MoPoE) a mixture of 2 combination of PoE experts, which is used as the model for joint encoder in MoPoE-VAE 34) Since the MoPoE joint encoder takes into account all possible combinations of views, it naturally learns to aggregate information across any given views while optimizing every view-specific encoder. Similar to mmJSD, MoPoE-VAD derives lower bound on the objective to decompose the KL term into 2 KL terms with analytic solutions. However, this would render the method intractable for tasks with many views.
MoPoE
The Cecture-ofPrroduc--fff
Multi- View Total Correlation Auto-Encoder Since our model uses conditional VIBs that explic- itly calibrate all the representations encoded by view-specific encoders, we can safely choose PoE. as the model for the joint representation encoder without suffering from the precision miscalibration. We call our resulting model the Multi-View Total Correlation Auto-Encoder (MVTCAE), depicted in Figure D Thanks to PoE joint representation encoder, MVTCAE linearly scales to the number of input views. Furthermore, when training with partial observations, MVTCAE simply treats the covariance matrix of 4% (o.12) to be ooI for any missing observation 0₂ o, so that it does not contribute to the reconstruction loss for missing views, similar to 500 As result, MVTCAE can be naturally extended PMVRL. We show that MVTCAE successfully associates views Section
3 Related Work
Information-Theorees Representation Learning Information Bottleneck (IB) was intro- duced as regularization method to obtain minimal sufficient encoding by constraining the amount of information captured by the latent variable from the observed variable, Deep Variational IB (VIB) [2] extended IB by parameterizing it with neural network, which results in simple yet effective method to achieve a representation that generalizes well. Furthermore, using VIB in unsupervised learning has been revealed to have close relationships among VIB, VAE (2 and 3-VAE [16]. A number of follow-up works 27 31 propose encouraging the encoders to learn representation invariant to any attribute given in advance. Similarly, a modified version of VIB was introduced [ for learning view-invariant representation across two views. While IB and VIB are concerned with computing MI, two similar but distinct generalizations of MI have been applied learning disentangled rep- resentation, which are TC [42] and Interaction Information (II) 4 25 36). The TC quantifies the dependency among all dimensions of the single latent variable, which motivated many works that learn disentangled representation in single view 10, [2] 20 21]. II was used for disentangling shared representation from view-specific representation in cross- views [19].
Multi- View Representation Learning Canonical Correlation Analysis (CCA) 18 and its vari- are classical approaches for unsupervised cross-view view representation learning. CCA projects two different views into one common latent space in a way that those two views are max- imally correlated in the latent space. KCCA D uses kernels and DCCA 3 uses neural networks to learn the common representation. Similarly, DCCAE 41 trains autoencoders to obtain common representations. More recently, a number of notable MVRL methods have been proposed to support more than 2 views. DMF-MVC [53 extracts a common representation of multiple views through deep matrix factorization. MDcR [48 maps each view to a lower-dimensional space and applies the kemne matching to regularize the dependence across multiple views. ITML 9 learns Mahalanobis distance function by Bregman optimization, whereas LMNN 43 learns a Mahalanobis distance metric optimize the k-nearest neighbors classifier using labeled data. CPM-Nets [50] gives formal definition of complete representation in PMVRI. and proposes to learn it without encoders.
ants I 3 41

---

Unear Cissiitation Accuracy
1.00
Conditional Ooherence Accuracy
100
0.95
0.90
140
0.85
160
0.60
0.-
0.75
Nehvl
200
0.70
2.3
2.3,4
2,3.4,5
Number of Input views
Number of Input views
Input views used to reconstruct view
Figure 2: Performance evaluation on the Poly MNIST dataset.
Multi- View Generative Models (VAEs) MVAE [46] and its variants 30 33 34] are multi-view generative models that learn shared representation by maximizing the log-likelihood of joint views via latent variables. We compared these methods to ours in Section 2.2 and 2.4
Experiments
4.1 Multi-View Representation Learning
To verify that our method successfully learns complete representation capturing both common factors and view-specific factors, we evaluate our method on the following two datasets used as evaluation benchmarks in the MVRL literature.
4.1.1 Multi- Classification Translation on PolyMNIST
We employ PolyMNIST dataset (34) composed of tuples with 5 different MNIST images, which have different backgrounds and writing style but share the same digit label. The background of each view is randomly cropped from one image which is not used by other views. Thus, the digit identity is the common factor of variation while the background and writing style are view-specific factors. There are 60K tuples of training samples and 10K of test samples. Although the digit ID should be observable in any view, it is hard to identify in some images depending on the background and the writing style. Therefore, aggregating information across views essential for predicting the label.
evaluate the learned representation, we follow the protocol in K4 Specif- ically, after training all the models in an unsupervised manner using complete observations, evaluate the learned representation with three different metrics, which are linear classification accu- racy and conditional coherence accuracy. We also evaluate the quality of conditional generation with FID score [15]. We compare our method with various state-of-the-ar multi-view generative models which are MVAE [46], MMVAE [30], mmJSD [33, and MoPoE- VAE (34).
Evaluation protocol
To apply our method to classification, we fix the encoders and train a linear classifier predict labels using the joint representation extracted from (=(=)ō) feeding complete observations in training set. We then classify the representations of all subsets and compute the average classification accuracy over all subsets with the same subset size.
To measure the conditional coherence accuracy, we extract the representation every subset of views using pa and generate views that are absent in the subset using gá- Those generated views are fed into the pretrained CNN-based classifier and if the prediction from the classifier matches the label of the given subset. The results are averaged over all subsets with the same size.
Finally, evaluate the sample quality of the first view images generated from different combination of input views ([2), (2.3), (2.3,4), (2.3.4.5 in terms of FID score. Since FID compares statistics of two sets (one is the of samples generated by the models and the other is the first view images in the training set in our context), it takes into account not only the quality of generated samples but previous evaluation metrics, marking lower (better)
also the diversity of them. Thus, unlike FID score requires the model to learn the representation that captures not only the common factors of variation but also view-specific factors to express diversity within the view.
Results The left plot in Figure 2 shows the result of linear classification. Using the PoE as a joint representation encoder same as M MVAE, our method clearly outperforms all the baseline methods in the classification task, reaching 94% and 99% accuracy even when only one view and two views are given respectively. Comparing to MVAE, this result implies that conditional VIBs in our method calibrate every view-specific encoders according the information shared across views so that each of them successfully captures the digit identity, resolving the issue of unbalanced representation.
7

---

MVAE
MMVAB
RmJSD
MoPoE
MVTCAE
6
Figure 3: Conditionally generated images of the view given images from the view 2 (top row) and images from the views 2 and 3 (bottom row).
On the other hand, while all the methods show monotonic improvement on its performance as the number of given views increases, the accuracy of MMVAE does not show any noticeable change. It remarkable that our method even outperforms mmJSD and MoPoF- VAE with simpler aggregation model for the joint encoder, although the prior methods both PoE and MoE.
The middle plot Figure summarizes the result of conditional coherence. Our method outperforms all the baseline methods except when only one view is given. MMVAE and mmJSD fail leverage additional input views, showing the performance staying flat or even degrading, due to the incapability of aggregating information inherent in MoE as we discussed in Section 2.4 In contrast, our method monotonically improves the coherence accuracy with more input views, implying that the view- specific encoders well calibrated by conditional VIBs so that the joint representation aggregated by PoE produces accurate encoding of the digit identity.
The right plot in Figure shows the result of FID scores. Our method achieves the best performance in any combination of input views, which indicates that our method generates more diverse samples in better quality. Figure presents qualitative results of conditional generation from baseline methods and ours. In each row, images above the green line are input observations (views [2] and [2.3] for top and bottom rows respectively) in the test set, whereas images below the line are generated images view 1. The result shows that our method successful cross-view association, especially being much better than any comparing methods at (1) improving the identification of the shared factors using any additional views and (2) expressing the view-specific factors in the target view. Providing more views improves the results of our method, which can be found in Section B.2.1 Compared to ours, MMVAE, mmJSD, and MoPoE hardly express view-specific diversity in the target view. We hypothesize that their joint representation encoders such MoE MoPoE not sharp enough to discover all the view-specific factors of variation correctly. In contrast, although MVAE uses PoE joint encoder, it poorly preserves the shared factors due to the miscalibration of view-specific encoders. Considering that our method also uses PoE encoder, the performance gap between ours and MVAE shows the effectiveness of conditional VIBs.
4.1.2 Multi-View Translation on Caltech-101
We also evaluate our method on the multi-view dataset used in (24 where six visual features are extracted from images in Caltech-101 dataset. In this dataset, each image is associated with six features, which are of Gabor filter 229]. Wavelet Moments (WM) (26). CENTRIST [45). Histogram of Oriented Gradients (HOG) 8 GIST 29), and Local Binary Pattern (LBP) 28 We treat each feature as a view, varying from 40 up to 1984 dimensions. We remark that each feature can be considered as a lossy compression of the original image, where the extracted information from view may not be necessarily inferrable for other views, thus making it nontrivial to learn the correlation across views.
Evaluation protocol Given the partial observations at test time, we extract the joint representation by Eq. (9, and reconstruct the missing view og ő using the output of the decoder gă (o.12)
8

---

(a) Result of training with complete observations
(b) Result of training with partial observations
Plan MMAAC
32-
MAG
1904 Figure 4: The multi-view translation performance of training with the complete and partial observa- tions, measuring the reconstruction eтror of the HOG feature by incrementally adding features.
1:257
1.0
Gezor
CCCT
ICST
HDS
dedicated to view v. We then evaluate the L2 reconstruction error. We investigate the impact of missing views on the completeness of the representation. We train our model with complete observations, and apply the model to reconstruct the HOG feature missing in the test time using partial observations o E HOGG The performance is averaged over the results of 10 independent runs. We compare to the same baseline methods employed in Section 4.1
Results Figure 4a shows the impact of missing views test time. We observe that method effectively decreases the reconstruction error as the input views accumulates in the observation, only with slightly higher error in reconstruction with view. Interestingly, we notice that the reconstruction error reduces significantly at some point (i.e., when adding GIST feature to the observation). It implies that some views contain more information than the others, and our method is able to learn to utilize this property. On the other hand, MoPoE VAE is less effective in utilizing GIST or LBP features, while mmJSD and VM hardly show monotonic performance improvement. Remarkably, MVAE performs worst, although the joint encoder model is same as ours, i.c. PoE. As an ablation study, we explore the impact of weighting parameter a, which trades off between VIB and conditional VIB (Eq. (8) in Section B.1.2 in the supplementary material.
4.2 Partial Multi View Representation Learning
To verify that our method can be effectively generalized to the partial observation setting, we conduct experiments on two tasks: multi-view translation and classification. In both tasks, we aim to show that the representation learned by our method is complete enough to infer the missing views given the incomplete observations (translation task) and useful for downstream tasks (classification task). To simulate the partial observations in both tasks, we follow the protocol of Zhang et al. [50] and generate random view-missing patterns with missing rate 1j = ΣV U./(V S) (0 S 17 < 1), where S is the number of entire samples and U, is the number of samples missing in the v-th view.
4.2.1 Partial Multi-Viewy Translation on Caltech-100 101
In this experiment, we investigate if our model can infer complete representations from partial obser- vations. To quantify the completeness of the representation, we employ the multi-view translation task. The goal is to reconstruct a missing view using the representation extracted from other view(s). In this case, the reconstruction error serves
direct mcasurement of the completeness.
Evaluation protocol Following the same procedures of Section 4.1.2 we repeat the same exper- iments but using the model trained with incomplete observation (7) = 0.5). This experiment can demonstrate the robustness our model to missing views in both inference and training time.
Results Figure 4b summarizes the experimental results using the incomplete training data (1- 0.5). We observe that our method exhibits similar trends with complete observation training data, exhibited by monotonic performance improvement in the number of available views, while other methods get negatively affected by additional views when trained with incomplete data (MMVAE, mmJSD, MoPoEE-VAR VAE). Interestingly, we observe that MVAE performs better when trained with incomplete data. This is because the missing views in the training data serve as regularization similar to Dropout (32 or sub-sampled training paradigm [46). Although MVAE shows monotonic improvement ours, its performance is turned out to be not comparable to ours.
We also note that our method achieves the best performance with a 0.8 when trained with incomplete data, while the best is achieved at a = 0.9 when trained with complete data. We suspect
We choose the HOG since it has the largest dimension thus the reconstruction is most nontrivial.

---

(a)
YaleB
Handerittio
62.5
95.0
05.3-
42
92.5
02.5
250.0
60.0
50.0
Mind
67.5
87
85.0
85.0
62.5
00.0
0.2
0.5
0.5
0.5
0.5
Musiing rite
Figure 5: The partial multi-view classification performance under various view missing rate.
that this is because optimizing conditional VIB with partial observation becomes more difficult than with complete observation, and the model tends to solve it via memorization. In this case, the VIB can be useful to improve the generalization. To summarize, our method achieves the best performance for intermediate 0 < a < I, which implies that simultaneous optimization of VIB and conditional VIBs in Eq. (8) is effective to solve MVRL and PMVRL. Qualitative results can be found in Section B.2.2 in the supplementary material.
4.2.2 Partial Multi- View Classification on Six Datasets [50]
The previous section suggests that our method is able to learn to calibrate representations across views, even when the training data is composed of incomplete observations. In this section, we evaluate the effectiveness of our approach as an unsupervised pre-training of the multi-view representation. investigating if the learned representation is useful for downstream tasks such as classification.
Datasets We evaluate our method on six feature- based image classification datasets used in multi- view learning [50], which are ORL,PIE, YaleB, CUB, Animal and Handwritten. Each dataset is associated with 2 and up to visual features. Similar to the previous section, we treat each feature as a view of data. For all datasets, we follow the same preprocessing and training/test splits used in (50) See Section D in the supplementary material for a comprehensive description of the datasets.
Evaluation protocol We follow (50) to evaluate our model trained with partial observations with various missing rates 77 = {0, 0.1, 0.2, 0.3, 0.4, 0.5}. Similar to the experiment Section 4.1 we first train our model in an unsupervised manner using the observable views. Then we fix the encoders, and train the classifier with labels using the learned representation. To isolate the impact of the additional classifier, we employ the simplest classifier, i.e. logistic regression with the learned representation as input. We report the performance by averaging the results from 10 independent runs.
Result Figure 5 summarizes the results under varying 7) datasets ORL, PIE, YaleB, and Hand- written, which are with at least three views and thus in our primary interest. Due to the space limit, we present the results CUB and Animal in Section [B1.3] in the supplementary material. In addition to the baseline methods compared in previous sections, we include two strong baselines, CPM-Nets(S) and CPM-Nets(U), which address the PMVRL with and without label information in the representation learning stage, respectively. Compared to the supervised baseline (CPM-Nets(S)), our method is clearly outperforming in all datasets, even though our model is trained in purely unsupervised manner. Compared to the unsupervised baseline methods (CPM-Nets(U), MVAE, MMVAE, mmJSD, and MoPoE), our method achieves noticeable improvements, especially when the missing rate is reasonably high (0.3 < 17 S 0.5). It shows that our method is robust in learning the vrews correlation under partial observations, and the learned representation is informative enough to be useful in downstream tasks, even though we do not use label information in the data or adopt a sophisticated aggregation model in the joint encoder.
5 Conclusion
We presented an information theoretic model for unsupervised partial multi-view representation learning. Based on Total Correlation (TC), we derived a novel variational lower bound that allows us to train the model that encodes complete latent representation from partial-view observations. Strictly trained in an unsupervised manner, we also demonstrated that the learned representation is highly effective in downstream tasks, such as multi-view classification and multi-view translation. Although we demonstrated that our method can even learn from partial multi-view data, it still has room for improvement such as learning from unaligned view data, and investigating more sophisticated representation aggregation models for the joint encoder, which we leave as future work.
10

---

Acknowledgments and Disclosure of Funding
This work was supported by the National Research Foundation (NRF) of Korea (NRF- 2019RIA2C1086633 red and NRF-202IRICIC10123 communication Technology (MSIT) of Korea (ITTP No. 2019-0-00075, ITTP No. 2020-0-00940, IITP No. 2017-0-01779 (XAI), IITP No. 0021-0-00537, and IITP No. 2021-0-02068), the ETRI
the Ministry of Science and Information
(Contract No. 212S1100) and Samsung Electronics.
References
[1] S. Akaho. kernel method for canonical correlation analysis. International Meeting of Psychometric Society, 2001.
[2] A. Alemi, I. Fischer, J. Dillon, and K. Murphy. Deep variational information bottleneck. In International Conference on Learning Representations, 2017.
[3] G. Andrew, Arora, Bilmes, and Livescu. Deep canonical correlation analysis. In International Conference Machine Learning, pages 1247-1255. PMLR, 2013.
[4] A. Bell. The co-information lattice, 921-926. In Proceedings of the 4th International Symposium on Independent Component Analysis and Blind Source Separation (ICA2003), Nara, Japan, 2003.
[5] R. Q. Chen, X. Li, R. Grosse, and D. Duvenaud. Isolating sources of disentanglement in variational autoencoders. In Advances in Neural Information Processing Systems, 2018.
[6] W. G. Cochran. The combination of estimates from different experiments. Biometrics, 10(1): 101-129, 1954.
[7] W. G. Cochran and S. Carroll. A sampling investigation of the efficiency of weighting inversely as the estimated variance. Biometrics, ((4):447455) 1953.
[8] N. Dalal and B. Triggs. Histograms of oriented gradients for human detection. In 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 005), volume I, pages 386-893. IEEE, 2005.
[9] Davis, B. Kulis, P. Jain, Sra, and I. S. Dhillon. Information--theortiic metric learning. In Proceedings of the 24th International Conference on Machine Learning, pages 209-216, 2007.
[10] B. Esmaeili, H. Wu, S. Jain, A. Bozkurt, N. Siddharth, B. Paige, D. H. Brooks, J. Dy, and J.-W. van de Meent. Structured disentangled representations. In K. Chaudhuri and M. Sugiyama, editors, Proceedings of the Eventy-Secone International Conference on Artificial Intelligence and Staristics, volume 89 of Proceedings of Machine Learning Research, pages 2525 25344. PMLR, 16-18 Apr 2019.
[11] M. Federici, A. Dutta, P. Forré, N. Kushman, and Z. Akata. Learning robust representations via multi-view information bottleneck. In International Conference on Learning Representations, 2020.
[12] S. Gao, R. Brekelmans, G. Ver Steeg, and A. Galstyan. Auto-encoding total correlation explanation. In The 22nd International Conference on Artificial Intelligence and Statistics, pages 1157-1166. PMLR, 2019.
[13] R. Gross, I. Matthews, J. Cohn, T. Kanade, and S. Baker. Multi-pie. Image and Vision Computing, 28(5):807-813, 2010.
[14] Z. Han, C. Zhang. H. Fu, and J. т. Zhou. Trusted multi-view classification. In International Conference on Learning Representations, 2021. URL https::/ http://pppenrrriieeeeeefroorrr wtt.ieeruwee/for 1d=0DsR8BzCn15
[15] M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. arXiv preprint arXiv: 1706.0000000 08500, 2017.
11

---

[16] I. Higgins, L. Matthey, A. Pal. C. Burgess, X. Glorot, M. Botvinick, S. Mohamed, and A. Ler- chner. 3-VAE: Learning basic visual concepts with a constrained variational framework. International Conference on Learning Representations, 2(5):6, 2017.
[17] G. E. Hinton. Training products of experts by minimizing contrastive divergence. Neural Computation, 14(8):1771-1800, 2002.
[18] H. Hotelling. Relations between two sets of variates. In Breakthroughs in statistics, pages 162-190. Springer, 1992.
[19] H. Hwang. G.-H. Kim, S. Hong, and K.-E. Kim. Variational interaction information maximiza- tion for cross-domain disentanglement, Advances in Neural Information Processing Systems, volume 33, 2020.
[20] Y. Jeong and H. O. Song. Learning discrete and continuous factors of data via alternating disentanglement. In International Conference on Machine Learning (ICML), 2019.
[21] H. Kim and A. Mnih. Disentangling by factorising. In Dy and A. Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 2649-2658, Stockholmsmässan, Stockholm Sweden, 10-15 Jul 2018. PMLR.
[22] D. P. Kingma and M. Welling. Auto-encoding variational bayes. In Y. Bengio and Y. LeCun, editors, 2nd International Conference on Learning Representations, ICLR 2014, Banff. AB, Canada, April 14- 16-16, 2014, Conference Track Proceedings, 2014.
[23] F.-.. Li, R. Fergus, and P. Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In 2004 conference computer vision and pattern recognition workshop, pages 178-178. IEEE, 2004.
[24] Y. Li, F. Nie, H. Huang, and J. Huang. Large-scale multi-view spectral clustering via bipartite graph. In Proceedings of the Twenty-Nimth AAAI Conference on Artificial Intelligence, 2015.
[25] W. McGill. Multivariate information transmission. Transactions of the IRE Professional Group on Information Theory, 4(4):93 111, 1954.
[26] H. A. Moghaddam, T. T. Khajoie, approach for image indexing and retrieval. Pattern Recognition, 38(12):2506-25118 2005.
Rouhi, and M. S. Tarzjan. Wavelet correloggam: a new
[27] D. Moyer, S. Gao, R. Brekelmans, A. Galstyan, and G. Ver Steeg. Invariant representations without adversarial training. In Advances in Neural Information Processing Systems, pages 9084 9093,9093,200 2018
[28] T. Ojala, M. Pietikainen, and T. Maenpaa. Multiresolution gray-scale and rotation invariant texture classification with local binary patterns. IEEE Transactions on pattern analysis and machine intelligence, 24(7):971-987 2002.
[29] A. Oliva and A. Torralba. Modeling the shape of the scene: A holistic representation of the spatial envelope. International journal of computer vision, 42(3)::45-175, 2001.
[30] Y. Shi, N. Siddharth, B. Paige, and P. H. Torr. Variational mixture-of-experts autoencoders for multi-modal deep generative models. Advances in Neural Information Processing Systems, 2019.
[31] Song, P. Kalluri, A. Grover, S. Zhao, and Ermon, Learning controllable fair representations. international conference on artificial intelligence and statistics, 2018
[32] N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15 156) 1958,, 2014. http::t:: org/ wwtpgg/apprs//eee http:/pppaa//11////esse hwtp://astsaaaaaaaaaa../ html
[33] M. Sutter, I. Daunhawer, and J. Vogt. Multimodal generative learning utilizing jensen-
shannon-divergence. Advances in Neural Information Processing Systems, 2020.
12

---

[34] T. M. Sutter, I. Daunhawer, and J. E. Vogt. Generalized multimodal elbo. International Conference on Learning Representations, 2021.
[35] Y. Tian, X. Peng, L. Zhao, Zhang, and D. N. Metaxas. Cr-gan: Learning complete rep- resentations for multi-view generation. In Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI-18, pages 942-948, International Joint Con- ferences on Artificial Intelligence Organization. 7 2018. doi: 10.24963/ijcai 2018/133.. URL
htttps://oi.. org/10. 20e2224993jcce 1 2018/131
[36] H. K. Ting. On the amount of information. Theory of Probability & Its Applications, 7(4): 439 447, 1962.
[37] N. Tishby, F. C. Pereira, and W. Bialek, The information bottleneck method. arXiv preprint physics/0004057, 2000.
[38] G. Ver Steeg and A. Galstyan. Discovering structure in high-dimensional data through correla- tion explanation. In Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems, volume 27, pages 577-585, Cur- ran Associates, Inc., 2014, URL https: // http://pooceeeeeeeeeeeeee neurips wwtpppppp//////e/e/eee 1ffffo13366476666633222222222222 Paper. pdf
[39] G. Ver Steeg and A. Galstyan. Maximally informative hierarchical representations of high- dimensional data. In Artificial Intelligence and Statistics, pages 1004-1012. PMLR, 2015.
[40] C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie. The caltech-ucsd birds-200-2011 dataset. 2011.
[41] W. Wang, R. Arora, K. Livescu, and Bilmes. On deep multi-view representation learning. In International conference muchine learning, pages 1083-1092. PMLR, 2015.
[42] S. Watanabe. Information theoretical analysis of multivariate correlation. IBM Journal of research and development, 4(1):66-88 82, 1960.
[43] K. Q. Weinberger and L. K. Saul. Distance metric learning for large margin nearest neighbor classification. Journal of machine learning research, 10(2), 2009.
[44] M. White, Yu, X. Zhang, and D. Schuurmans. Convex multi-view subspace learning. In Nips, pages 1690. Lake Tahoe, Nevada, 2012.
[45] J. Wu and J. M. Rehg. Centrist: A visual descriptor for scene categorization. IEEE transactions on pattern analysis and machine intelligence, 1391 2010.
[46] M. Wu and N. Goodman. Multimodal generative models for scalable weakly-supervised learning. Advances in Neural Information Processing Systems, 2018
[47] Y. Yuan, G. Xun, K. Jia, and A. Zhang. A multi-view deep learning framework for eeg seizure detection. IEEE journal of biomedical and health informatics, 23(1):83-944 2018.
[48] C. Zhang, H. Fu, Q. Hu, P. Zhu, and X. Cao. Flexible multi-view dimensionality co-reduction. IEEE Transactions on Image Processing, 26(2):648 (6088866666600 2016
[49] C. Zhang, E. Adeli, T. Zhou, X. Chen, and D. Shen. Multi-layer nuet classification for alzheimer's disease diagnosis. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
[50] C. Zhang, Z. Han, y. cui, H. Fu, J. T. Zhou, and Q. Hu. Cpm-nets: Cross partial multi- view networks. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors. Advances in Neural Information Processing Systems, volume 32, pages 559-569. Curran Associates, Inc., 2019. URL https: / http://rrroeedngn..eeee..ppppppp www.inuuriic.om cctpppprrccppp// http://////./2222222112222220010 13tp/1/-112dddPPPPPr pdf
[51] H. Zhang, T. S. Huang, N. M. Nasrabadi, and Y. Zhang. Heterogeneous multi-metric learning for multi-sensor fusion. In / /4th International Conference on Information Fusion, pages 1-8. IEEE, 2011.
13

---

[52] R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang. The unreasonable effectiveness of deep features perceptual metric. In CVPR, 2018.
[53] H. Zhao, Z. Ding, and Y. Fu, Multi-view clustering via deep matrix factorization, In Proceedings of the AAAI Conference Artificial Intelligence, volume 31, 2017.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes] Please read abstract and introduction
(b) Did you describe the limitations of your work? [Yes] Please read conclusion
(c) Did you discuss any potential negative societal impacts of your work? [Yes] Due to the space limit, we discuss the supplementary material.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes] See supplementary.
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material as URL)? [Yes] we will submit code as the supplementary file.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] specify them in the supplementary material.
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes] We ran 5~100 runs with different random seeds. Related information can be found in the supplementary file.
(d) Did you include the total amount compute and the type of resources used (e.g.. type of GPUs, internal cluster, or cloud provider)? [Yes] included the supplementary material.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] In appropriate places
(b) Did you mention the license of the assets? [N/A] Not applicable.
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you'l 're using/curating? [N/A] All the datasets we used are public.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A] None of datasets contain identifiable content.
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A] Not applicable.
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A] Not applicable.
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A] Not applicable.
14


Reviews:
[{"review_id": "npGxXJt_n8b", "reviewer": "Reviewer_LwhH", "summary": "This paper proposes to measure the quality of a multi-view generative model by the amount of total correlation among observations that is explained by the latent complete representation. While directly maximizing the reduction in total correlation is intractable, the authors derived a variational lower bound as a surrogate objective which encourages the model to learn a minimal sufficient representation and also regularizes the latent representation to be inferable from any view. The proposed formulation works well when some views are missing during training or testing. Hence it's suitable for partial multi-view representation learning. It also naturally calibrates the view-specific encoders to prevent the model heavily relying on only the most informative views. Empirical evaluation shows that the proposed method performs noticeably better than its baselines on most multi-view classification problems and multi-view translation tasks while achieves competitive results on a few others.", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: the idea of using the reduction of total correlation as the objective of learning auto-encoder-like models has been studied in the literature. The authors have also cited related work in the submission. The originality of this paper comes from applying total correlation to multi-view representation learning and deriving a novel variational lower bound that is suitable for learning with partial observations.\n\nQuality: the derivation of the proposed objective looks solid. But i haven't carefully checked the details in the supplementary material. Empirical study on the proposed method is pretty thorough. The paper can be strengthened if the authors can discuss about the computational complexity of their new method and the baselines. The choice of dimension of latent space is not covered in the main text either. It is not clear to me how much tuning effort the proposed method requires.\n\nClarity: this paper is well written in general but can be certainly polished further. On L136-137, the definition of encoder and decoder is confusing to me. The encoder, $r_{\\psi}^v (z | o_v) = N(\\mu_v, \\sigma_v^2 I)$, is a Gaussian distribution that does not involve $o_v$, which feels unintuitive. The same for the decoder $q_{\\psi}^v(o_v | z)$.\n\nSignificance: the novel variational lower bound and its application to partial multi-view representation learning appear to be a solid contribution to this domain. I am not very familiar with the state-of-the-art results on the related tasks. But, based on the experimental results, the proposed method performs well in practice and advances the best results further, though the amount of improvement varies by task.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have mentioned potential societal impact of this work in the supplementary material.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8"}, "extra_text": {}}, {"review_id": "go0mFhJT2Xb", "reviewer": "Reviewer_5CYt", "summary": "The paper proposes the variational-based generative approach to learn the multi-view (multi-modal) representation. The whole framework is motivated and derived from maximizing the total correlation  reduced by the representation, it focuses on learning a shared latent representation that is informative as well as succinct to capture the correlations among the given views (domains). Specifically, the framework is based on VAE framework with product-of-expert posterior encoder, and is trained via extended ELBO with additional regularization KL between joint encoder and view-specific encoder. The authors verify their proposed model on multi-view classification and translation. They also conduct experiments on partial multi-view representation learning where several views are missing in the training, and need to be inferred during testing.  ", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "For originality: \n\n (+) the idea presented makes sense and interesting, though I feel the whole model is actually built on the MVAE with extra conditional VIB term as in eqn. 7. The paper also describes the difference with the existing relevant works. \n\nFor quality: \n\n (+) the authors consider various tasks (e.g., complete-views/partial views, translation/classification) for verification of their model. \n\n (-) however, it's better to also include some \"standard\" dataset for direct comparisons. For example, the existing baseline MMVAE use pair data from SVHN-MNIST for multi-domain translation/prediction (they also include MVAE performance), their models are well-tuned on such data for better performance and could be more illustrative.  Since the main difference is the training objective, if authors could use similar net structures to obtain results on such benchmark, that could easily give us direct and convincing comparisons. \n\n (-) perhaps the word \"multi-view\" is slightly misleading.  There are tons of multi-view representation learning papers in the literature that are mainly focused on face images under multiple viewpoints . This paper seems instead focused on multi-modal(domain) learning. Does the proposed model capable of handling face images under different view-points? Does the paper flexible enough to deal with raw pixels instead of extracted features on Multi-PIE data (for example)? It would be interesting to show the multi-view translation results on face (such as given one view point, use the learned network/representation to generate the face under novel view point). \n\nFor clarity: \n\n (-) the paper can be not so easy to follow. Such difficulty is not coming from the math derivation, but from the presentation of the paper. Quite a few symbols/notations are used without properly definition and descriptions beforehand. For instance: line 83, $r(z) \\approx p_\\theta(z)$, what is this $p_\\theta(z)$? is it empirically marginal distribution based on observation or is it simply prior distribution of your decoder? If the latter, use $\\theta$ indicates you have learnable prior? Another example would be $r_\\phi$ in eqn. 7, when author write down eqn. 7, it didn't clearly show us what is the relationship between $p_\\theta(z|o)$ and $r_\\phi$. So have to go back and forth to check some notations in the previous sections. Changing the presentation would greatly improve the paper and make it more accessible. \n\nFor significance: \n\n (+) the new conditional VIB term might have some interesting properties and may be applicable for follow-up development. \n\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The limitation of the work could be described more clearly. ", "review_text": "", "extra_scores": {"ethical_concerns": "No such concern.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "f7h2crCAJ93", "reviewer": "Reviewer_yxVq", "summary": "This paper proposes a novel Multi-View Representation Learning method to learn a shared representation of multiple observations. The objective is to learn a complete representation through maximizing the reduction of total correlation. The method is based on the VAE framework and is capable of dealing with missing views. ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "*originality*\n\nThe paper proposes a novel objective function for Multi-View Representation Learning (MVRL). The objective optimizes a lower bound of the total correlation between the views and the latent representation. \n\n*quality*\n- In Eq. (6), you mention \"the latent representation is regularized to be inferrable from any view and thus cannot be dominated by a few one\". But the MI terms could still be high for some views and low for some other views. You are taking the average in the end anyway. I don't think this solves the imbalance issue.\n- In Eq. (7), it seems that both $p(z|o)$ and $r(z|o_v)$ are learnable posteriors. Would this cause any training instability?\n- PoE seems to be an aggregation of all the individual posteriors and definitely has a larger support than individual ones. This could also bring some numerical concerns. How did you address this issue?\n- When you say \"VIB in Eq. (5) favors the minimal sufficient encoding\", I'm ok with the minimal due to the minus. But I don't think this term really encourages \"sufficient\". This term could potentially cause \"insufficiency\".\n- For the KL terms, did you use analytical form or sample estimation?\n- In Fig. 1, MMVAE's performance cannot be improved with more views. What might cause this?\n- In Fig. 3, there seems to be no significant gap between MVTCVAE and other unsupervised methods (mmJSD, MoPoE). Besides, as the missing rate increases, the performance drop trends are quite similar. Then I wonder if carefully tuning other methods could achieve the same goal.\n\n*clarity*\n- Authors should really put Fig. 4 to the main text rather than the appendix. \n- When you bring up the limitations \"Missing views\", as a reader, I thought you would immediately explain how you address this. But I have to wait till the end of section 2.4. I would suggest you explaining section 2.4 first. \n\n*significance*\nIt is novel to propose such an idea for MVRL. But maybe authors can demonstrate more scenarios besides the accuracies.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I think the writeup could be improved, especially the order of different modules. Besides, Fig. 4 should be moved to the main text with more descriptions. The whole pipeline would be more clear.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "Uqtk7KjVrSr", "reviewer": "Reviewer_XHEY", "summary": "The paper motivates a variational objective (eq. (7)) for multi-view representation learning. The proposed objective is an alternative to the standard multi-view VAE objective, and should be better suited for extracting the correct representation when views are missing at random and when a subset of the views is overwhelmingly informative. They then propose optimising a convex combination of their proposed objective and the standard multi-view VAE objective.", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The problem tackled by the author seems well motivated, since multi-view representation learning in a partial setting (with missing views) is an important problem.\nThe paper is generally well structured and easy to read.\n\nThe definition of “complete representation” plays a crucial role in the paper, but is never stated formally. Adding a dedicated “Definition” paragraph would be useful for the reader. For example, as done in sec. 2.1 of:\n\nZhang, Changqing, et al. \"CPM-Nets: Cross partial multi-view networks.\" (2019).\n\nLines 128-129 are very puzzling. After motivating and deriving objective (7), the authors say they “sometimes observed that the model is prone to overfitting”, and suggest instead optimising a convex combination of their proposed objective and the usual multi-view variational ELBO, favouring a minimally sufficient encoding. \nThe authors allude to a tradeoff between minimal sufficiency and completeness, but this is not extensively commented upon in the paper, despite apparently constituting a fundamental point for how it informs their final objective. The feeling is that something is missing here, both in theory (characterising this tradeoff, relating it to properties of the ground truth data generating process, etc.) and in experiments (e.g. the authors say “we observed that…”, which seems to refer to some unreported experiments).\n\nI need to say, I am not an expert on the related literature (e.g. references [34] and [35] cited in the paper), so my confidence level in assessing its quality and novelty within that context is not extremely high. Overall I am under the impression that the paper, while being well motivated and written, is not very convincing in some parts, and that some key aspects would require more elaboration and possibly a more rigorous treatment, as detailed above.\n\nTypos and minor comments:\n\nReferences [12] and [13] are the same?\n\nLine 191: “variable.Deep” — space missing after dot\n\nThe captions for figures 1, 2 and 3 could be made more informative.\n\n**Update** I will increase my evaluation to 6 based on the authors' answer.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors address this in the supplementary material.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a total correlation objective within VAE framework for multi-view representation learning, and the proposed method can handle situations with missing views. 

The main concern of the reviewers is with the presentation and clarity. 

The overall assessment is that this is an incrementally novel contribution with incrementally improved experimental results. 
Decision: Accept (Poster)