Disentanglement Analysis with Partial Information Decomposition

1 INTRODUCTION . Disentanglement is a guiding principle for designing a learned representation separable into parts that individually capture the underlying factors of variation . The concept is originally concerned as an inductive bias towards obtaining representations aligned with the underlying factors of variation in data ( Bengio et al. , 2013 ) and has been applied to controlling otherwise unstructured representations of data from several domains , e.g. , images ( Karras et al. , 2019 ; Esser et al. , 2019 ) , text ( Hu et al. , 2017 ) , and audio ( Hsu et al. , 2019 ) to name just a few . While the concept is appealing , defining disentanglement is not clear . After Higgins et al . ( 2017 ) , generative learning methods with regularized total correlation have been proposed ( Kim & Mnih , 2018 ; Chen et al. , 2018 ) ; however , it is still not clear if independence of latent variables is essential for better disentanglement ( Higgins et al. , 2018 ) . Furthermore , it is not obvious to measure disentanglement given true generative factors . Towards understanding disentanglement , it is crucial to define disentanglement metrics , for which several attempts have been made ( Higgins et al. , 2017 ; Kim & Mnih , 2018 ; Chen et al. , 2018 ; Eastwood & Williams , 2018 ; Do & Tran , 2020 ; Zaidi et al. , 2020 ) ; however , there are still problems to be solved . Current disentanglement metrics may fail to detect entanglement involving more than two variables . In these metrics , one first measures how each variable explains one generative factor and then compares or contrasts them among variables . With such a procedure , we may overlook multiple variables conveying information of one generative factor . For example , let z = ( z1 , z2 ) be a representation consisting of two vectors , where each variable in z1 disentangles a distinct generative factor and z2 is a rotation of z1 not axis-aligned with the factors . Since any dimension of z2 alone may convey little information of one generative factor , these metrics do not detect that multiple variables encode one generative factor redundantly . Although this is a simple example , this kind of information sharing may arise in learned representations as well , if not the variables are linearly correlated . In this work , we present a disentanglement analysis framework aware of interactions among multiple variables . Our key idea is to decompose the information of representation into entangled and disentangled components using Partial Information Decomposition ( PID ) , which is a framework in modern information theory to analyze information sharing among multiple random variables ( Williams & Beer , 2010 ) . As illustrated in Figure 1 , the mutual information I ( u ; v1 , v2 ) = E [ log p ( u , v1 , v2 ) p ( u ) p ( v1 , v2 ) ] between a random variable u and a pair of random variables ( v1 , v2 ) is decomposed into four nonnegative terms : unique information U ( u ; v1 \ v2 ) and U ( u ; v2 \ v1 ) 1 , redundant information R ( u ; v1 , v2 ) , and complementary ( or synergistic ) information C ( u ; v1 , v2 ) . While these partial information terms have no agreed-upon concrete definitions yet ( Bertschinger et al. , 2014 ; Finn & Lizier , 2018 ; Lizier et al. , 2018 ; Finn & Lizier , 2020 ; Sigtermans , 2020 ) , we can derive universal lower and upper bounds of the partial information terms only with well-defined mutual information terms . We apply the PID framework to representations learned from data by letting u be a generative factor and v1 , v2 be one and the remaining latent variables , respectively . The unique information of a latent variable intuitively corresponds to the amount of disentangled information , while the redundant and complementary information correspond to different types of entangled information . We can quantify disentanglement and multiple types of entanglement through the framework , which enriches our understanding on disentangled representations . Our contributions are summarized as follows . • PID-based disentanglement analysis framework : We propose a disentanglement analysis framework that captures interactions among multiple variables with PID . With this framework , one can distinguish two different types of entanglement , namely redundancy and synergy , which provide insights on how a representation entangles generative factors . • Tractable bounds of partial information terms : We derive lower and upper bounds of partial information terms . We formulate a disentanglement metric , called UNIBOUND , using the lower bound of unique information . We design entanglement attacks , which inject entanglement to a given disentangled representation , and confirm through experiments using them that UNIBOUND effectively captures entanglement involving multiple variables . • Detailed analyses of learned representations : We analyze representations obtained by variational autoencoders ( VAEs ) . We observe that UNIBOUND sometimes disagrees with other metrics , which indicates multi-variable interactions may dominate learned representations . We also observe that different types of entanglement arise in models learned with different methods . This observation provides us an insight that we may require distinct approaches to remove them for disentangled representation learning . PROBLEM FORMULATION AND NOTATIONS Let x be a random variable representing a data point , drawn uniformly from a dataset D. Assume that the true generative factors y = ( y1 , . . . , yK ) > are available for each data point ; in other words , we can access the subset D ( y ) ⊂ D of the data points with any fixed generative factors y . Let z = ( z1 , . . . , zL ) > be a latent representation consisting of L random variables . An inference model is provided as the conditional distribution p ( z|x ) . Our goal is to evaluate how well the latent variables z disentangle each generative factor in y . The inference model can integrate out the input as p ( ·|y ) = Ep ( x|y ) [ p ( ·|x ) ] = 1|D ( y ) | ∑ x∈D ( y ) p ( ·|x ) ; therefore , we only use y and z in most of our discussions . 1Note that this \ is not a set difference operator . It is just a common notation used in the PID literature to emphasize the unique information is not symmetric and resembles the set difference as depicted in Fig.1 . We denote the mutual information between random variables u and v by I ( u ; v ) = E [ log p ( u , v ) p ( u ) p ( v ) ] , the entropy of u by H ( u ) = −E [ log p ( u ) ] , and the conditional entropy of u given v by H ( u|v ) = −E [ log p ( u|v ) ] . We denote a vector of zeros by 0 , a vector of ones by 1 , and an identity matrix by I . We denote by N ( µ , Σ ) the Gaussian distribution with mean µ and covariance Σ and denote its density function by N ( · ; µ , Σ ) . 2 RELATED WORK . The importance of representing data with multiple variables conveying distinct information has been recognized at least since the ’ 80s ( Barlow , 1989 ; Barlow et al. , 1989 ; Schmidhuber , 1992 ) . The minimum entropy coding principle ( Watanabe , 1981 ) , which aims at representing data by random variables z with the sum of minimum marginal entropies ∑ ` H ( z ` ) , is found to be useful for unsupervised learning to remove the inherent redundancy in the sensory stimuli . The resulting representation minimizes the total correlation and is called factorial coding . Recent advancements in disentangled representation learning based on VAEs ( Kingma & Welling , 2014 ) are guided by the same principle as minimum entropy coding ( Kim & Mnih , 2018 ; Chen et al. , 2018 ; Gao et al. , 2019 ) . Understanding better representations , which is tackled from the coding side as above , is also approached from the generative perspective . It is often expected that data are generated from generative factors through a process that entangles them into high dimensional sensory space ( DiCarlo & Cox , 2007 ) . As generative factors are useful as the basis of downstream learning tasks , obtaining disentangled representations from data is a hot topic of representation learning ( Bengio et al. , 2013 ) . Towards learning disentangled representations , it is arguably important to quantitatively measure disentanglement . In that regard , Higgins et al . ( 2017 ) established a standard evaluation procedure using controlled datasets with balanced and fully-annotated ground-truth factors . A variety of metrics have then been proposed on the basis of the procedure . Among them , Higgins et al . ( 2017 ) and Kim & Mnih ( 2018 ) propose metrics based on the deviation of each latent variable conditioned by a generative factor . In contrast , Mutual Information Gap ( MIG ) ( Chen et al. , 2018 ) and its variants ( Do & Tran , 2020 ; Zaidi et al. , 2020 ) are based on mutual information between a latent variable and a generative factor . We extend the latter direction , considering multi-variable interactions . Barlow ( 1989 ) discussed redundancy by comparing the population and the individual variables by their entropies , i.e. , total correlation . It is though less trivial to measure redundancy as an information quantity . The PID framework ( Williams & Beer , 2010 ) provides an approach to understanding redundancy among multiple random variables as a constituent of mutual information . The framework provides some desirable relationships between decomposed information terms , while it leaves some degrees of freedom to determine all of them , for which several definitions have been proposed ( Williams & Beer , 2010 ; Bertschinger et al. , 2014 ; Finn & Lizier , 2018 ; 2020 ; Sigtermans , 2020 ) . The PID framework has been applied to machine learning models . For example , Tax et al . ( 2017 ) measured the PID terms for restricted Boltzmann machines using the definition of Williams & Beer ( 2010 ) . Yu et al . ( 2021 ) took an alternative route , similar to our approach , where they measured linear combinations of PID terms by corresponding linear combinations of mutual information terms . These work aims at analyzing the learning dynamics of models in supervised settings . In contrast , we use the PID framework for analyzing disentanglement in unsupervised representation learning . 3 PARTIAL INFORMATION DECOMPOSITION FOR DISENTANGLEMENT . In this section , we analyze the current metrics and introduce our framework . In Section 3.1 , we introduce PID of the system we concern . In Section 3.2 , we investigate the current metrics in terms of multi-variable interactions . In Section 3.3 and 3.4 , we construct our disentanglement metric with bounds for PID terms . We provide a method of computing the bounds in Section 3.5 . 3.1 PARTIAL INFORMATION DECOMPOSITION . We tackle the problem of evaluating disentanglement of a latent representation z relative to the true generative factors y from an information-theoretic perspective . Let us consider evaluating how one generative factor yk is captured by the latent representation z . The information of yk captured by z is measured using mutual information I ( yk ; z ) = H ( z ) −H ( z|yk ) . In a desirably disentangled representation , we expect one of the latent variables z ` to exclusively capture the information of the factor yk . To evaluate a given representation , we are interested in understanding how the information is distributed between a latent variable z ` and the remaining representation z\ ` = ( z ` ′ ) ` ′ 6= ` . This is best described by the PID framework , where the mutual information is decomposed into the following four terms . I ( yk ; z ) = R ( yk ; z ` , z\ ` ) + U ( yk ; z ` \ z\ ` ) + U ( yk ; z\ ` \ z ` ) + C ( yk ; z ` , z\ ` ) . ( 1 ) Here , the decomposed terms represent the following non-negative quantities . • Redundant informationR ( yk ; z ` , z\ ` ) is the information of yk held by both z ` and z\ ` . • Unique information U ( yk ; z ` \ z\ ` ) is the information of yk held by z ` and not held by z\ ` . The opposite term U ( yk ; z\ ` \ z ` ) is also defined by exchanging the roles of z ` and z\ ` . • Complementary information ( or synergistic information ) C ( yk ; z ` , z\ ` ) is the information of yk held by z = ( z ` , z\ ` ) that is not held by either z ` or z\ ` alone . The following identities , combined with Eq.1 , partially characterize each term . I ( yk ; z ` ) = R ( yk ; z ` , z\ ` ) + U ( yk ; z ` \ z\ ` ) , I ( yk ; z\ ` ) = R ( yk ; z ` , z\ ` ) + U ( yk ; z\ ` \ z ` ) . ( 2 ) The decomposition of this system is illustrated in Figure 2a . We expect disentangled representations to concentrate the information of yk to a single latent variable z ` , and to let the other variables z\ ` not convey the information in either unique , redundant , or synergistic ways . This is understood in terms of PID as maximizing the unique information U ( yk ; z ` \ z\ ` ) while minimizing the other parts of the decomposition . The above formulation is incomplete as one degree of freedom remains to determine the four terms with the three equalities . Instead of stepping into searching for suitable definitions of these terms , we build discussions applicable to any such definitions that fulfill the above incomplete requirements2 .

Summary: The authors leveraged partial information decomposition (PID) for analyzing multi-variable interactions in latent representations. The PID framework shows that (i) the mutual information between a latent variable and a generative factor can be divided into unique, redundant, and synergistic information terms, and (ii) the uniqueness term corresponds to the degree of disentanglement. The authors also introduced a disentanglement metric by modifying MIG and conducted experiments with VAE-based models on two datasets.