Multi-View Representation Learning via Total Correlation Objective

1 Introduction . Multi-View Representation Learning ( MVRL ) aims to learn a shared representation of multiple observations from different types of views . In MVRL , it is important to encourage the shared representation to be complete enough to capture the correlation across views without losing viewspecific information so that the learned representation can be readily applied to rich set of downstream tasks . For example , in sensor fusion for the multi-sensor system [ 51 ] and in clinical diagnosis based on patients ’ various types of medical records [ 47 , 49 ] , one would like to aggregate all the information from various observations in order to uncover the true underlying factors under the correlation . Although using views as many as possible seems to be always beneficial for the task of learning a good representation , it can make the problem itself harder depending on the complexity of correlations across all views and the scalability of handling a number of views . MVRL becomes even more challenging when the model does not always have access to complete observations from all views for every data instance during training , which we call Partial Multi-View Representation Learning ( PMVRL ) . PMVRL is closer to the practical setting since it is unrealistic to expect that observations from all different kinds of views are always available . For example , it is unlikely for all the sensors in a sensor-based system to have the same frequency to update their measurements or for all kinds of medical records to be available for any patient . Considering those difficulties , any desirable MVRL methods are encouraged to satisfy three desiderata . The first one is scalability to the number of input views . The method should handle multiple observations in a way computationally scalable to arbitrary many views in both training and testing time . In addition , the method needs to be robust to partial observability . The method should be able to combine any combination of available observations in the representation space in test time . This is important for PMVRL because we have to handle observations arbitrarily missing not only in testing data but also in training data . Lastly , the method should discover cross-view association , 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . which can be learned by identifying both shared and view-specific factors of variation of each view in the representation space . Associating views correctly in the representation space allows the method to utilize any additional observations from different views to improve identification of the shared factors without determining any unobserved ones . Combining multiple of Variational Auto-Encoder ( VAE ) [ 22 ] for each view1 , several generative models [ 30 , 33 , 34 , 46 ] have been recently proposed to address MVRL . Since naive optimization of the evidence lower bound ( ELBO ) on the joint likelihood of multiple views does not address any of the desiderata , those methods impose different structural bias on the joint representation encoders whose strengths and weaknesses are complementary ( see Section 2.4 ) . Although they showed encouraging performance on multiple tasks such as predicting common attributes and inferring missing views , they often fail to simultaneously capture both the shared and view-specific factors of variation and turn out to poorly associate views in our experiments . In this paper , we address the problem of MVRL with a principled approach grounded in information theory . Specifically , we formulate the representation learning task as maximizing the reduction in Total Correlation [ 12 , 38 , 39 ] . Based on our formulation , we derive a novel objective function that not only offers tractable optimization but also introduces multiple types of variational information bottlenecks which successfully associate views . We then show that our method naturally extends to the PMVRL setting via inverse variance weighting , a classical approach used in sensor fusion . We demonstrate the validity and effectiveness of our method in the multi-view translation and classification downstream tasks . Our contributions are three-fold : 1 . Measuring the informativeness of the multi-view representation by Total Correlation , we propose a general information-theoretic framework to learn a complete representation , which encompasses existing multi-view generative models . 2 . Under the proposed framework , we identify drawbacks of optimizing ELBO and derive a novel objective function that resolves them . Specifically , our method yields a representation that correctly associates views by capturing not only the common factors of variation but also the view-specific ones . 3 . We conducted extensive evaluation with comparing methods and ours in translation and classification tasks in both MVRL and PMVRL settings , showing that our method is the most reliable method to obtain the latent representation agnostic to the downstream tasks . 2 Approach . Let ~o = { ov } Vv=1 be the observation of a data instance composed of V different views , which is sampled from an unknown joint distribution pD ( ~o ) , where we emphasize that this is a data distribution using the subscript D. Given these observations , MVRL is the task of learning a complete representation z across the views ~o . Following [ 50 ] , we define a complete representation as follows . Definition 1 ( Completeness for Multi-View Representation [ 50 ] ) A multi-view representation z is complete if each observation , i.e. , ov from ~o , can be reconstructed from a mapping fv ( · ) , i.e. , ov = fv ( z ) . The definition directly indicates that a complete representation describes all factors of variations in ~o , since every view can be reconstructed solely from the complete representation . While MVRL considers complete observations for training data , PMVRL assumes otherwise , i.e . some views being missing in the training data . This poses a unique challenge of ( 1 ) learning to produce the complete representation with partial views ( 2 ) whose availability varies per instance in training and testing . We first relate informativeness of the representation measured in terms of Total Correlation ( TC ) and the goal of MVRL ( Section 2.1 ) . We then show that the TC-based MVRL objective function encompasses existing multimodal generative models and analyze its limits based on a straightforward variational lower bound ( Section 2.2 ) . To resolve those limits , we derive an alternative variational lower bound which suits better for MVRL ( Section 2.3 ) . Lastly , we finalize our formulation by proposing the representation aggregation model that naturally extends to PMVRL ( Section 2.4 ) . 1We follow the conventional terminology in the related literature [ 11 , 14 , 24 , 41 , 44 , 47 , 48 , 50 , 53 ] , but any choice among views , modalities [ 30 , 33 , 34 , 46 ] , and domains [ 19 ] can be suitable for our paper and our baseline methods , as long as there is a common latent representation that explains different views / modalities / domains . 2.1 Multi-View Representation Learning with Total Correlation . A complete representation Z should be informative enough to explain the correlation among V different views . Total correlation ( TC ) [ 42 ] , defined as the Kullback-Leibler divergence of the joint distribution from the factored marginals , measures the amount of information shared among a finite set of random variables . In our MVRL context , TC is defined as TC ( ~O ) , DKL [ pD ( ~o ) ‖ V∏ v=1 pD ( ov ) ] . ( 1 ) We aim to find the encoder pθ ( z|~o ) such that the knowledge of z would reduce TC as much as possible . This can be formulated by maximizing the objective TCθ ( ~O ; Z ) , TC ( ~O ) − TCθ ( ~O | Z ) , ( 2 ) where the conditional TC in the last term is given by TCθ ( ~O|Z ) , Epθ ( z ) [ DKL [ pθ ( ~o|z ) ‖ V∏ v=1 pθ ( ov|z ) ] ] , ( 3 ) which is the expected Kullback-Leibler divergence of the joint conditional from the factored conditionals . The parameterized distributions in the above formula involve the encoder pθ ( z|~o ) as follows : pθ ( z ) = ∫ pθ ( z|~o ) pD ( ~o ) d~o , pθ ( ~o|z ) = pθ ( z|~o ) pD ( ~o ) /pθ ( z ) , and pθ ( ov|z ) = ∫ pθ ( ~o|z ) d~o\v . Intuitively , minimization of Eq . ( 3 ) ( i.e. , maximization of Eq . ( 2 ) ) suits well for MVRL , since ( 1 ) any complete representation Z would minimize Eq . ( 3 ) ( e.g. , Z = ~o ) , and ( 2 ) it accords with the theoretical result that complete representation z should factorize the generative distribution [ 44 , 50 ] . Further decomposition of ( 2 ) reveals that it encourages Z to encode the correlation across views [ 12 ] , a desirable property for MVRL : TCθ ( ~O ; Z ) = V∑ v=1 Iθ ( Ov ; Z ) − Iθ ( ~O ; Z ) . ( 4 ) The first term in Eq . ( 4 ) , Mutual Information ( MI ) between each observation and the representation , enforces the representation to be informative for every observation . On the other hand , the second term takes the role of Information Bottleneck ( IB ) , which encourages the encoder to learn minimal sufficient representation [ 37 ] . Consequently , simultaneous optimization of both terms naturally allows the representation Z to capture correlations among views while being minimally sufficient . Note that when V =2 , Eq . ( 4 ) coincides with Interaction Information [ 4 , 19 , 25 , 36 ] , which quantifies the amount of information shared among two views and their joint representation ( see Section A.4 ) . 2.2 Limitations of VAE models for multi-view data . Unfortunately , a direct optimization of MIs in Eq . ( 4 ) is intractable [ 2 , 12 ] . A straightforward approach would be employing approximate distributions qvφ ( ov|z ) ≈ pθ ( ov|z ) and r ( z ) ≈ pθ ( z ) , and optimize a variational lower bound of Eq . ( 4 ) as follows ( see Section A.1 in the supplementary material for full derivation ) : TCθ ( ~O ; Z ) ≥ V∑ v=1 [ H ( Ov ) + Epθ ( z|~o ) pD ( ~o ) [ ln qvφ ( ov|z ) ] ] − EpD ( ~o ) [ DKL [ pθ ( z|~o ) ‖r ( z ) ] ] ︸ ︷︷ ︸ VIB , ( 5 ) where the entropy terms can be dropped from optimization since they are determined by the true data distribution pD ( ~o ) . Without the entropy terms , we note that Eq . ( 5 ) is essentially identical to evidence lower bound ( ELBO ) of the variational auto-encoder ( VAE ) models for multi-view data [ 30 , 33 , 34 , 46 ] by switching the notations , for p for encoder and q for decoder . This notation switch follows the convention in [ 2 , 12 ] . Although this lower bound contains the variational information bottleneck term that encourages the representation to be minimally sufficient for generalizing well even with small training data [ 2 ] , it has the following fundamental limitations : 1 . Unbalanced representation : If a subset of views is overwhelmingly informative enough to reconstruct the others , the encoder pθ ( z|~o ) may learn to rely on those views while ignoring the rest , yielding a degenerate solution of Eq . ( 4 ) that fails cross-view association . This is problematic in MVRL since such views may not be available at test time . 2 . Missing views : The encoder pθ ( z|~o ) requires complete observations ~o not only in training but also in testing phases , while MVRL requires the model to encode incomplete observations õ ⊆ ~o in test time . Furthermore , PMVRL requires to handle incomplete observations even in training . In order to overcome these challenges , prior methods impose special structures ( i.e . inductive hypotheses ) for pθ ( z|~o ) , such as Product of Experts ( PoE ) [ 17 , 46 ] , Mixture of Experts ( MoE ) [ 30 ] , or Mixture of Product of Experts ( MoPoE ) [ 34 ] . In our work , we present a more principled approach to this problem by deriving an alternative lower bound , described in the next section .

Summary: This paper proposes a novel Multi-View Representation Learning method to learn a shared representation of multiple observations. The objective is to learn a complete representation through maximizing the reduction of total correlation. The method is based on the VAE framework and is capable of dealing with missing views.