Multi-View Disentangled Representation

1 INTRODUCTION . Multi-view representation learning ( MRL ) involves learning representations by effectively leveraging information from different perspectives . The representations produced by MRL are effective when correlations across different views are accurately modeled and thus properly exploited for downstream tasks . One representative algorithm , Canonical Components Analysis ( CCA ) ( Hotelling , 1992 ) , aims to maximize linear correlations between two views under the assumption that factors from different views are highly correlated . Under a similar assumption , the extended versions of CCA , including kernelized CCA ( Akaho , 2006 ) and Deep CCA ( Andrew et al. , 2013 ) , explore more general correlations . There are also several methods ( Cao et al. , 2015 ; Sublime et al. , 2017 ) that maximize the independence between different views to enhance the complementarity . Going beyond the simple assumptions above , the latent representation encodes different views with a degradation process implicitly exploiting both consistency and complementarity ( Zhang et al. , 2019 ) . These existing MRL algorithms are effective , however , the assumed correlations between different views are usually simple thus can not accurately model or explicitly disentangle complex real-world correlations , which hinders the further improvement and interpretability . Although there are a few heuristic algorithms ( Tsai et al. , 2019 ; Hu et al. , 2017 ) that explicitly decompose the multiview representation into shared and view-specific parts , they are especially designed for supervised learning tasks without any disentangled representation guarantee and fall short in formally defining the relationships between different parts . To address this issue , we propose to unsupervisedly disentangle the original data from different views into shared representation across different views and exclusive ( private ) part within each view , which explicitly depicts the correlations and thus not only enhances the performance of existing tasks but could also inspire potential applications . Specifically , we firstly provide a definition for the multi-view disentangled representation by introducing the sufficient and necessary conditions for guaranteeing the disentanglement of different views . According to these conditions , an information-theory-based algorithm is proposed to accurately disentangle different views . To summarize , the main contributions of our work are as follows : • To the best of our knowledge , this is the first work to formally study multi-view disentangled representation with strict conditions , which might serve as the foundations of the future research on this problem . • Based on our definition , we propose a multi-view disentangling model , in which information- theory-based multi-view disentangling can accurately decompose the information into shared representation across different views and exclusive representation within each view . The explicit decomposition enhances the performance of multi-view analysis tasks and could also inspire new potential applications . • Different from the single-view unsupervised disentangled representation learning ( Locatello et al. , 2019 ) , we provide a new paradigm for unsupervised disentangled representation learning from a fresh perspective - disentangling factors between different views instead of each single view . • Extensive experiments on a range of applications verify that the proposed information-theory- based multi-view disentangling algorithm can accurately disentangle data from multiple views into expected shared and exclusive representations . 2 MULTI-VIEW DISENTANGLED REPRESENTATION Existing multi-view representation learning methods ( Wu & Goodman , 2018 ; Zhang et al. , 2019 ) can obtain a common representation for multi-view data , however , the correlations between different views are not explicitly expressed . The supervised algorithms ( Hu et al. , 2017 ; Tan et al. , 2019 ) can decompose multiple views into a common part and private parts , but there is no disentangling guarantee . Therefore , we propose a multi-view disentanglement algorithm that can explicitly separate the shared and exclusive information in unsupervised settings . Formally , we first propose a definition of a multi-view disentangled representation by introducing four criteria , which are considered as sufficient and necessary conditions of disentangling multiple views . The definition is as follows : Definition 2.1 ( Multi-View Disentangled Representation ) Given a sample with two views , i.e. , X = { xi } 2i=1 , the representation Sdis = { si , ei } 2i=1 is a multi-view disentangled representation if the following conditions are satisfied : • Completeness : 1 The shared representation si and exclusive representation ei should jointly contain all information of the original representation xi ; • Exclusivity : 2 There is no shared information between common representation si and exclusive representation ei , which ensures the exclusivity within each view ( intra-View ) . 3 There is no shared information between ei and ej , which ensures the exclusivity between private information of different views ( inter-View ) . • Commonality : 4 The common representations si and sj should contain the same informa- tion . Equipped with the exclusivity constraints , the common representations are guaranteed to not only be the same but also contain maximized common information . The necessity for each criterion is illustrated in Fig . 1 ( satisfaction of all the four conditions produces exact disentanglement , and violation of any condition may result in an unexpected disentangled representation ) . Note that , existing ( single-view ) unsupervised disentanglement focuses on learning a representation to identify explanatory factors of variation , which has been proved fundamentally impossible ( Locatello et al. , 2019 ) . The goal of the proposed multi-view disentanglement is to disentangle multiple views into the shared and exclusive parts which can be well guaranteed as illustrated in definition 2.1 and Fig . 1 . Mutual information has been widely used in representation learning ( Hjelm et al. , 2019 ; Belghazi et al. , 2018 ) . In probability theory and information theory , the mutual information of two random variables quantifies the “ amount of information ” obtained about one random variable when observing the other one , which is well-suited for measuring the amount of shared information between two different views . To approach the disentangling goal , according to conditions 1 ∼ 4 , the general form of the object function is naturally induced as : max 2∑ i , j=1 [ I ( xi ; ei , si ) ︸ ︷︷ ︸ 1 −I ( ei ; si ) ︸ ︷︷ ︸ 2 ] − ∑ i6=j I ( ei ; ej ) ︸ ︷︷ ︸ 3 + ∑ i 6=j I ( si ; sj ) ︸ ︷︷ ︸ 4 , ( 1 ) where I ( · ; · ) denotes the mutual information . We provide an implementation in Fig . 2 and , in the following subsections , we will describe this implementation in detail . 2.1 CONDITION ¬ : INFORMATION PRESERVATION FOR THE SHARED AND EXCLUSIVE REPRESENTATIONS . • How to maximize I ( x ; e , s ) ? For simplicity , x , s , e and xi , si , ei are denoted with the same meanings and used alternately , where the former and latter are used for intra-view and inter-view cases , respectively . To preserve the information from the original data in the shared and exclusive representations , the mutual information I ( x ; e , s ) should be maximized . There are different ways to implement the maximization of I ( x ; e , s ) based on the following assumptions . Assumption 2.1 The shared representation s and exclusive representation e are simultaneously independent and conditionally independent : p ( s , e ) = p ( s ) p ( e ) , p ( s , e|x ) = p ( s|x ) p ( e|x ) . ( 2 ) Firstly , we expand I ( x ; e , s ) to obtain the following equation ( more details are shown in supplement C ) : I ( x ; e , s ) = ∫ ∫ ∫ p ( x ) p ( e , s|x ) log p ( e , s|x ) p ( e , s ) dedsdx . Then , under Assumption 2.1 , the following equation is derived ( more details are shown in supplement C ) : I ( x ; e , s ) = I ( x ; e ) + I ( x ; s ) . ( 3 ) According to the above equation , it seems that we can maximize I ( x ; e ) + I ( x ; s ) to maximize I ( x ; e , s ) , which involves making s and e contain as much information from x as possible ( ideally , it will produce e and s to meet I ( x ; e ) = I ( x ; s ) = H ( x ) , where H ( x ) is the entropy of x ) . This actually leads to a strong correlation between s and e , which is in conflict with the independence Assumption 2.1 about s and e. In other words , it is difficult to balance the completeness ( condition 1 ) and intra-view exclusivity ( condition 2 ) ( see experimental results in supplement B.4 ) . Fortunately , there is an alternative strategy which avoids the difficulty in balancing the completeness and intra-view exclusivity . Specifically , we introduce a latent representation r generated by two independent distributions with respect to s and e under a mild assumption : Assumption 2.2 ( Relationship between s , e and r ) : p ( s , e , x ) = p ( r , x ) . ( 4 ) In our formulation , we define r = f ( s , e ) , where r is derived from s and ewith the underlying function f ( · ) and satisfies p ( r , x ) = p ( s , e , x ) . Eq . 4 is a mild assumption , for example the invertibility of mapping r = f ( s , e ) ensuring a sufficient condition which can be easily verified . Note that r = [ s , e ] is one special case and will be discussed later . Based on Assumption 2.2 , we can get ( more details are shown in supplement C ) : p ( r ) = p ( s , e ) , p ( r|x ) = p ( s , e|x ) . ( 5 ) Then , we can induce the following result ( more details are shown in supplement C ) : I ( x ; e , s ) = I ( x ; r ) . ( 6 ) This result indicates that the maximization of I ( x ; e , s ) can be achieved by maximizing the mutual information of agency r and x . In this way , the independence of e and s is well preserved and the previous conflict is dispelled . Next , we will explain how to encode the information of x into independent representations s and e by introducing the agency r. • How to obtain independent representations e and s by maximizing I ( x ; r ) ? First , we consider encoding the observed data x into a latent representation r by maximizing the mutual information between x and r. Considering robustness and effectiveness ( Alemi et al. , 2018 ) , we can maximize the mutual information between r and x through Variational Autoencoders ( VAEs ) ( Kingma & Welling , 2014 ) . Accordingly , we have the following objective function : min qr , d Ex∼p ( x ) [ − Er∼qr ( r|x ) [ log ( d ( x|r ) ) ] + Er∼qr ( r|x ) log qr ( r|x ) p ( r ) ] , ( 7 ) where d ( x|r ) ( the “ decoder ” ) is a variational approximation to p ( x|r ) , and qr ( r|x ) ( the “ encoder ” ) is a variational approximation to p ( r|x ) , which converts the observed data x into the latent representation r. Second , we consider how to obtain independent representations e and s by modeling qr ( r|x ) . For this goal , the relationships between s , e and r should be jointly modeled . As shown in Eq . 5 , we obtain p ( r|x ) = p ( s , e|x ) . Under Assumption 2.1 , Eq . 5 can be rewritten as p ( r|x ) = p ( s|x ) p ( e|x ) , which implies that qr ( r|x ) can be considered as the product of p ( s|x ) and p ( e|x ) . Furthermore , we introduce PoE ( product-of-experts ) ( Hinton , 2002 ; Wu & Goodman , 2018 ) to model the product of qs ( s|x ) and qe ( e|x ) , where the variational networks qs ( s|x ) and qe ( e|x ) are designed to approximate p ( s|x ) and p ( e|x ) . It is worth noting that the key difference from MVAE ( Multimodal Variational Autoencoder ) ( Wu & Goodman , 2018 ) is that our model obtains the latent representation r from two independent components within each single view , while MVAE achieves the unified representation of all views by assuming independence of representations of different views . Under the assumption that the true posteriors for individual factors p ( s|x ) and p ( e|x ) are contained in the family of their variational counterparts qs ( s|x ) and qe ( e|x ) , we have qr ( r|x ) = qs ( s|x ) qe ( e|x ) . With Gaussian distributions , we can obtain the closed-form solution for the product of two distributions : µr = µsσ 2 s+µeσ 2 e σ2s+σ 2 e , σ2r = σ2sσ 2 e σ2s+σ 2 e . Therefore , the independent representations between e and s are well preserved by modeling qr ( r|x ) . Accordingly , with the results qr ( r|x ) = qs ( s|x ) qe ( e|x ) and p ( r ) = p ( s ) p ( e ) , the objective in Eq . 7 is rewritten as : min qs , qe , d Ex∼p ( x ) [ − Er∼qs ( s|x ) qe ( e|x ) [ log ( d ( x|r ) ) ] + Es∼qs ( s|x ) log qs ( s|x ) p ( s ) + Ee∼qe ( e|x ) log qe ( e|x ) p ( e ) ] , where p ( s ) and p ( e ) are set to Gaussian distributions , which in turn forces qs ( s|x ) and qe ( e|x ) to be closer to a Gaussian distribution , allowing us to find the product of the two distributions . The above objective is actually the ELBO ( evidence lower bound ) ( Kingma & Welling , 2014 ) with the first term being the reconstruction loss , and the second and third terms being the KL divergence . The proposed variant of VAE inherits two advantages from VAE and PoE , respectively . The first is that we can obtain approximate distributions of s and e given x to preserve the independence . The second is that the proposed model still works even when there is a missing case for e or s in the testing . This means that we can use only s or e as input to the decoder to reconstruct x ( shown in the experimental section ) , which is quite different from the concatenation of e and s or other forms that require e and s simultaneously to obtain r. In addition , the way of concatenating s and e does not well exploit the independent prior of s and e .

Summary: The goal of this paper is to define multi-view disentanglement in an unsupervised manner. The authors list four principal rules for representation disentanglement, including completeness of combining shared and specific representations, the exclusivity of two specific representations and between specific and shared representation, and commonality of two shared representation. The authors follow the above rules to design a VAE-based model and demonstrate favorable results on image clustering and classification tasks.