Disentanglement Analysis with Partial Information Decomposition

1 INTRODUCTION . Disentanglement is a guiding principle for designing a learned representation separable into parts that individually capture the underlying factors of variation . The concept is originally concerned as an inductive bias towards obtaining representations aligned with the underlying factors of variation in data ( Bengio et al. , 2013 ) and has been applied to controlling otherwise unstructured representations of data from several domains , e.g. , images ( Karras et al. , 2019 ; Esser et al. , 2019 ) , text ( Hu et al. , 2017 ) , and audio ( Hsu et al. , 2019 ) to name just a few . While the concept is appealing , defining disentanglement is not clear . After Higgins et al . ( 2017 ) , generative learning methods with regularized total correlation have been proposed ( Kim & Mnih , 2018 ; Chen et al. , 2018 ) ; however , it is still not clear if independence of latent variables is essential for better disentanglement ( Higgins et al. , 2018 ) . Furthermore , it is not obvious to measure disentanglement given true generative factors . Towards understanding disentanglement , it is crucial to define disentanglement metrics , for which several attempts have been made ( Higgins et al. , 2017 ; Kim & Mnih , 2018 ; Chen et al. , 2018 ; Eastwood & Williams , 2018 ; Do & Tran , 2020 ; Zaidi et al. , 2020 ) ; however , there are still problems to be solved . Current disentanglement metrics may fail to detect entanglement involving more than two variables . In these metrics , one first measures how each variable explains one generative factor and then compares or contrasts them among variables . With such a procedure , we may overlook multiple variables conveying information of one generative factor . For example , let z = ( z1 , z2 ) be a representation consisting of two vectors , where each variable in z1 disentangles a distinct generative factor and z2 is a rotation of z1 not axis-aligned with the factors . Since any dimension of z2 alone may convey little information of one generative factor , these metrics do not detect that multiple variables encode one generative factor redundantly . Although this is a simple example , this kind of information sharing may arise in learned representations as well , if not the variables are linearly correlated . In this work , we present a disentanglement analysis framework aware of interactions among multiple variables . Our key idea is to decompose the information of representation into entangled and disentangled components using Partial Information Decomposition ( PID ) , which is a framework in modern information theory to analyze information sharing among multiple random variables ( Williams & Beer , 2010 ) . As illustrated in Figure 1 , the mutual information I ( u ; v1 , v2 ) = E [ log p ( u , v1 , v2 ) p ( u ) p ( v1 , v2 ) ] between a random variable u and a pair of random variables ( v1 , v2 ) is decomposed into four nonnegative terms : unique information U ( u ; v1 \ v2 ) and U ( u ; v2 \ v1 ) 1 , redundant information R ( u ; v1 , v2 ) , and complementary ( or synergistic ) information C ( u ; v1 , v2 ) . While these partial information terms have no agreed-upon concrete definitions yet ( Bertschinger et al. , 2014 ; Finn & Lizier , 2018 ; Lizier et al. , 2018 ; Finn & Lizier , 2020 ; Sigtermans , 2020 ) , we can derive universal lower and upper bounds of the partial information terms only with well-defined mutual information terms . We apply the PID framework to representations learned from data by letting u be a generative factor and v1 , v2 be one and the remaining latent variables , respectively . The unique information of a latent variable intuitively corresponds to the amount of disentangled information , while the redundant and complementary information correspond to different types of entangled information . We can quantify disentanglement and multiple types of entanglement through the framework , which enriches our understanding on disentangled representations . Our contributions are summarized as follows . • PID-based disentanglement analysis framework : We propose a disentanglement analysis framework that captures interactions among multiple variables with PID . With this framework , one can distinguish two different types of entanglement , namely redundancy and synergy , which provide insights on how a representation entangles generative factors . • Tractable bounds of partial information terms : We derive lower and upper bounds of partial information terms . We formulate a disentanglement metric , called UNIBOUND , using the lower bound of unique information . We design entanglement attacks , which inject entanglement to a given disentangled representation , and confirm through experiments using them that UNIBOUND effectively captures entanglement involving multiple variables . • Detailed analyses of learned representations : We analyze representations obtained by variational autoencoders ( VAEs ) . We observe that UNIBOUND sometimes disagrees with other metrics , which indicates multi-variable interactions may dominate learned representations . We also observe that different types of entanglement arise in models learned with different methods . This observation provides us an insight that we may require distinct approaches to remove them for disentangled representation learning . PROBLEM FORMULATION AND NOTATIONS Let x be a random variable representing a data point , drawn uniformly from a dataset D. Assume that the true generative factors y = ( y1 , . . . , yK ) > are available for each data point ; in other words , we can access the subset D ( y ) ⊂ D of the data points with any fixed generative factors y . Let z = ( z1 , . . . , zL ) > be a latent representation consisting of L random variables . An inference model is provided as the conditional distribution p ( z|x ) . Our goal is to evaluate how well the latent variables z disentangle each generative factor in y . The inference model can integrate out the input as p ( ·|y ) = Ep ( x|y ) [ p ( ·|x ) ] = 1|D ( y ) | ∑ x∈D ( y ) p ( ·|x ) ; therefore , we only use y and z in most of our discussions . 1Note that this \ is not a set difference operator . It is just a common notation used in the PID literature to emphasize the unique information is not symmetric and resembles the set difference as depicted in Fig.1 . We denote the mutual information between random variables u and v by I ( u ; v ) = E [ log p ( u , v ) p ( u ) p ( v ) ] , the entropy of u by H ( u ) = −E [ log p ( u ) ] , and the conditional entropy of u given v by H ( u|v ) = −E [ log p ( u|v ) ] . We denote a vector of zeros by 0 , a vector of ones by 1 , and an identity matrix by I . We denote by N ( µ , Σ ) the Gaussian distribution with mean µ and covariance Σ and denote its density function by N ( · ; µ , Σ ) . 2 RELATED WORK . The importance of representing data with multiple variables conveying distinct information has been recognized at least since the ’ 80s ( Barlow , 1989 ; Barlow et al. , 1989 ; Schmidhuber , 1992 ) . The minimum entropy coding principle ( Watanabe , 1981 ) , which aims at representing data by random variables z with the sum of minimum marginal entropies ∑ ` H ( z ` ) , is found to be useful for unsupervised learning to remove the inherent redundancy in the sensory stimuli . The resulting representation minimizes the total correlation and is called factorial coding . Recent advancements in disentangled representation learning based on VAEs ( Kingma & Welling , 2014 ) are guided by the same principle as minimum entropy coding ( Kim & Mnih , 2018 ; Chen et al. , 2018 ; Gao et al. , 2019 ) . Understanding better representations , which is tackled from the coding side as above , is also approached from the generative perspective . It is often expected that data are generated from generative factors through a process that entangles them into high dimensional sensory space ( DiCarlo & Cox , 2007 ) . As generative factors are useful as the basis of downstream learning tasks , obtaining disentangled representations from data is a hot topic of representation learning ( Bengio et al. , 2013 ) . Towards learning disentangled representations , it is arguably important to quantitatively measure disentanglement . In that regard , Higgins et al . ( 2017 ) established a standard evaluation procedure using controlled datasets with balanced and fully-annotated ground-truth factors . A variety of metrics have then been proposed on the basis of the procedure . Among them , Higgins et al . ( 2017 ) and Kim & Mnih ( 2018 ) propose metrics based on the deviation of each latent variable conditioned by a generative factor . In contrast , Mutual Information Gap ( MIG ) ( Chen et al. , 2018 ) and its variants ( Do & Tran , 2020 ; Zaidi et al. , 2020 ) are based on mutual information between a latent variable and a generative factor . We extend the latter direction , considering multi-variable interactions . Barlow ( 1989 ) discussed redundancy by comparing the population and the individual variables by their entropies , i.e. , total correlation . It is though less trivial to measure redundancy as an information quantity . The PID framework ( Williams & Beer , 2010 ) provides an approach to understanding redundancy among multiple random variables as a constituent of mutual information . The framework provides some desirable relationships between decomposed information terms , while it leaves some degrees of freedom to determine all of them , for which several definitions have been proposed ( Williams & Beer , 2010 ; Bertschinger et al. , 2014 ; Finn & Lizier , 2018 ; 2020 ; Sigtermans , 2020 ) . The PID framework has been applied to machine learning models . For example , Tax et al . ( 2017 ) measured the PID terms for restricted Boltzmann machines using the definition of Williams & Beer ( 2010 ) . Yu et al . ( 2021 ) took an alternative route , similar to our approach , where they measured linear combinations of PID terms by corresponding linear combinations of mutual information terms . These work aims at analyzing the learning dynamics of models in supervised settings . In contrast , we use the PID framework for analyzing disentanglement in unsupervised representation learning . 3 PARTIAL INFORMATION DECOMPOSITION FOR DISENTANGLEMENT . In this section , we analyze the current metrics and introduce our framework . In Section 3.1 , we introduce PID of the system we concern . In Section 3.2 , we investigate the current metrics in terms of multi-variable interactions . In Section 3.3 and 3.4 , we construct our disentanglement metric with bounds for PID terms . We provide a method of computing the bounds in Section 3.5 . 3.1 PARTIAL INFORMATION DECOMPOSITION . We tackle the problem of evaluating disentanglement of a latent representation z relative to the true generative factors y from an information-theoretic perspective . Let us consider evaluating how one generative factor yk is captured by the latent representation z . The information of yk captured by z is measured using mutual information I ( yk ; z ) = H ( z ) −H ( z|yk ) . In a desirably disentangled representation , we expect one of the latent variables z ` to exclusively capture the information of the factor yk . To evaluate a given representation , we are interested in understanding how the information is distributed between a latent variable z ` and the remaining representation z\ ` = ( z ` ′ ) ` ′ 6= ` . This is best described by the PID framework , where the mutual information is decomposed into the following four terms . I ( yk ; z ) = R ( yk ; z ` , z\ ` ) + U ( yk ; z ` \ z\ ` ) + U ( yk ; z\ ` \ z ` ) + C ( yk ; z ` , z\ ` ) . ( 1 ) Here , the decomposed terms represent the following non-negative quantities . • Redundant informationR ( yk ; z ` , z\ ` ) is the information of yk held by both z ` and z\ ` . • Unique information U ( yk ; z ` \ z\ ` ) is the information of yk held by z ` and not held by z\ ` . The opposite term U ( yk ; z\ ` \ z ` ) is also defined by exchanging the roles of z ` and z\ ` . • Complementary information ( or synergistic information ) C ( yk ; z ` , z\ ` ) is the information of yk held by z = ( z ` , z\ ` ) that is not held by either z ` or z\ ` alone . The following identities , combined with Eq.1 , partially characterize each term . I ( yk ; z ` ) = R ( yk ; z ` , z\ ` ) + U ( yk ; z ` \ z\ ` ) , I ( yk ; z\ ` ) = R ( yk ; z ` , z\ ` ) + U ( yk ; z\ ` \ z ` ) . ( 2 ) The decomposition of this system is illustrated in Figure 2a . We expect disentangled representations to concentrate the information of yk to a single latent variable z ` , and to let the other variables z\ ` not convey the information in either unique , redundant , or synergistic ways . This is understood in terms of PID as maximizing the unique information U ( yk ; z ` \ z\ ` ) while minimizing the other parts of the decomposition . The above formulation is incomplete as one degree of freedom remains to determine the four terms with the three equalities . Instead of stepping into searching for suitable definitions of these terms , we build discussions applicable to any such definitions that fulfill the above incomplete requirements2 .

Summary: The paper proposes an information theoretic approach to analyze the disentanglement. Using Partial Information Decomposition (PID) framework, the authors attempt to characterize the interaction among latent factors, correlation/disentanglement, for more than two latent variables. They generalize the existing mutual information gap (MIG) metric for multiple latent variables, by decomposing mutual information term into unique, redundant, and complementary information. The authors discussed that the pairwise correlation analysis endorses a portion of the redundant information as a measure of disentanglement, which should have been considered as entanglement. Additionally, they derive bounds for the partial information terms and demonstrate the limitations of existing disentanglement measures, i.e., BetaVAE, FactorVAE, and MIG in quantifying multivariate correlation in the latent space. The authors also designed two entanglement attacks to assess the disentanglement measures under injected correlation to the latent representation. The authors used two datasets, dSprites and 3dshapes to empirically substantiate their findings.