Year: 2022

Abstract:
We propose a framework to analyze how multivariate representations disentangle ground-truth generative factors. A quantitative analysis of disentanglement has been based on metrics designed to compare how one variable explains each generative factor. Current metrics, however, may fail to detect entanglement that involves more than two variables, e.g., representations that duplicate and rotate generative factors in high dimensional spaces. In this work, we establish a framework to analyze information sharing in a multivariate representation with Partial Information Decomposition and propose a new disentanglement metric. This framework enables us to understand disentanglement in terms of uniqueness, redundancy, and synergy. We develop an experimental protocol to assess how increasingly entangled representations are evaluated with each metric and confirm that the proposed metric correctly responds to entanglement. Through experiments on variational autoencoders, we find that models with similar disentanglement scores have a variety of characteristics in entanglement, for each of which a distinct strategy may be required to obtain a disentangled representation.

heading: 1 INTRODUCTION
text: Disentanglement is a guiding principle for designing a learned representation separable into parts that individually capture the underlying factors of variation. The concept is originally concerned as an inductive bias towards obtaining representations aligned with the underlying factors of variation in data (Bengio et al., 2013) and has been applied to controlling otherwise unstructured representations of data from several domains, e.g., images (Karras et al., 2019; Esser et al., 2019), text (Hu et al., 2017), and audio (Hsu et al., 2019) to name just a few.
While the concept is appealing, defining disentanglement is not clear. After Higgins et al. (2017), generative learning methods with regularized total correlation have been proposed (Kim & Mnih, 2018; Chen et al., 2018); however, it is still not clear if independence of latent variables is essential for better disentanglement (Higgins et al., 2018). Furthermore, it is not obvious to measure disentanglement given true generative factors. Towards understanding disentanglement, it is crucial to define disentanglement metrics, for which several attempts have been made (Higgins et al., 2017; Kim & Mnih, 2018; Chen et al., 2018; Eastwood & Williams, 2018; Do & Tran, 2020; Zaidi et al., 2020); however, there are still problems to be solved.
Current disentanglement metrics may fail to detect entanglement involving more than two variables. In these metrics, one first measures how each variable explains one generative factor and then compares or contrasts them among variables. With such a procedure, we may overlook multiple variables conveying information of one generative factor. For example, let z = (z1, z2) be a representation consisting of two vectors, where each variable in z1 disentangles a distinct generative factor and z2 is a rotation of z1 not axis-aligned with the factors. Since any dimension of z2 alone may convey little information of one generative factor, these metrics do not detect that multiple variables encode one generative factor redundantly. Although this is a simple example, this kind of information sharing may arise in learned representations as well, if not the variables are linearly correlated.
In this work, we present a disentanglement analysis framework aware of interactions among multiple variables. Our key idea is to decompose the information of representation into entangled and disentangled components using Partial Information Decomposition (PID), which is a framework in modern information theory to analyze information sharing among multiple random variables (Williams
& Beer, 2010). As illustrated in Figure 1, the mutual information I(u; v1, v2) = E [ log p(u,v1,v2)p(u)p(v1,v2) ] between a random variable u and a pair of random variables (v1, v2) is decomposed into four nonnegative terms: unique information U(u; v1 \ v2) and U(u; v2 \ v1)1, redundant information R(u; v1, v2), and complementary (or synergistic) information C(u; v1, v2). While these partial information terms have no agreed-upon concrete definitions yet (Bertschinger et al., 2014; Finn & Lizier, 2018; Lizier et al., 2018; Finn & Lizier, 2020; Sigtermans, 2020), we can derive universal lower and upper bounds of the partial information terms only with well-defined mutual information terms. We apply the PID framework to representations learned from data by letting u be a generative factor and v1, v2 be one and the remaining latent variables, respectively. The unique information of a latent variable intuitively corresponds to the amount of disentangled information, while the redundant and complementary information correspond to different types of entangled information. We can quantify disentanglement and multiple types of entanglement through the framework, which enriches our understanding on disentangled representations.
Our contributions are summarized as follows.
• PID-based disentanglement analysis framework: We propose a disentanglement analysis framework that captures interactions among multiple variables with PID. With this framework, one can distinguish two different types of entanglement, namely redundancy and synergy, which provide insights on how a representation entangles generative factors.
• Tractable bounds of partial information terms: We derive lower and upper bounds of partial information terms. We formulate a disentanglement metric, called UNIBOUND, using the lower bound of unique information. We design entanglement attacks, which inject entanglement to a given disentangled representation, and confirm through experiments using them that UNIBOUND effectively captures entanglement involving multiple variables.
• Detailed analyses of learned representations: We analyze representations obtained by variational autoencoders (VAEs). We observe that UNIBOUND sometimes disagrees with other metrics, which indicates multi-variable interactions may dominate learned representations. We also observe that different types of entanglement arise in models learned with different methods. This observation provides us an insight that we may require distinct approaches to remove them for disentangled representation learning.
PROBLEM FORMULATION AND NOTATIONS
Let x be a random variable representing a data point, drawn uniformly from a dataset D. Assume that the true generative factors y = (y1, . . . , yK)> are available for each data point; in other words, we can access the subset D(y) ⊂ D of the data points with any fixed generative factors y. Let z = (z1, . . . , zL)
> be a latent representation consisting of L random variables. An inference model is provided as the conditional distribution p(z|x). Our goal is to evaluate how well the latent variables z disentangle each generative factor in y. The inference model can integrate out the input as p(·|y) = Ep(x|y)[p(·|x)] = 1|D(y)| ∑ x∈D(y) p(·|x); therefore, we only use y and z in most of our discussions.
1Note that this \ is not a set difference operator. It is just a common notation used in the PID literature to emphasize the unique information is not symmetric and resembles the set difference as depicted in Fig.1.
We denote the mutual information between random variables u and v by I(u; v) = E [ log p(u,v)p(u)p(v) ] , the entropy of u by H(u) = −E[log p(u)], and the conditional entropy of u given v by H(u|v) = −E[log p(u|v)]. We denote a vector of zeros by 0, a vector of ones by 1, and an identity matrix by I. We denote by N (µ,Σ) the Gaussian distribution with mean µ and covariance Σ and denote its density function by N (·;µ,Σ).

heading: 2 RELATED WORK
text: The importance of representing data with multiple variables conveying distinct information has been recognized at least since the ’80s (Barlow, 1989; Barlow et al., 1989; Schmidhuber, 1992). The minimum entropy coding principle (Watanabe, 1981), which aims at representing data by random variables z with the sum of minimum marginal entropies ∑ `H(z`), is found to be useful for unsupervised learning to remove the inherent redundancy in the sensory stimuli. The resulting representation minimizes the total correlation and is called factorial coding. Recent advancements in disentangled representation learning based on VAEs (Kingma & Welling, 2014) are guided by the same principle as minimum entropy coding (Kim & Mnih, 2018; Chen et al., 2018; Gao et al., 2019).
Understanding better representations, which is tackled from the coding side as above, is also approached from the generative perspective. It is often expected that data are generated from generative factors through a process that entangles them into high dimensional sensory space (DiCarlo & Cox, 2007). As generative factors are useful as the basis of downstream learning tasks, obtaining disentangled representations from data is a hot topic of representation learning (Bengio et al., 2013). Towards learning disentangled representations, it is arguably important to quantitatively measure disentanglement. In that regard, Higgins et al. (2017) established a standard evaluation procedure using controlled datasets with balanced and fully-annotated ground-truth factors. A variety of metrics have then been proposed on the basis of the procedure. Among them, Higgins et al. (2017) and Kim & Mnih (2018) propose metrics based on the deviation of each latent variable conditioned by a generative factor. In contrast, Mutual Information Gap (MIG) (Chen et al., 2018) and its variants (Do & Tran, 2020; Zaidi et al., 2020) are based on mutual information between a latent variable and a generative factor. We extend the latter direction, considering multi-variable interactions.
Barlow (1989) discussed redundancy by comparing the population and the individual variables by their entropies, i.e., total correlation. It is though less trivial to measure redundancy as an information quantity. The PID framework (Williams & Beer, 2010) provides an approach to understanding redundancy among multiple random variables as a constituent of mutual information. The framework provides some desirable relationships between decomposed information terms, while it leaves some degrees of freedom to determine all of them, for which several definitions have been proposed (Williams & Beer, 2010; Bertschinger et al., 2014; Finn & Lizier, 2018; 2020; Sigtermans, 2020).
The PID framework has been applied to machine learning models. For example, Tax et al. (2017) measured the PID terms for restricted Boltzmann machines using the definition of Williams & Beer (2010). Yu et al. (2021) took an alternative route, similar to our approach, where they measured linear combinations of PID terms by corresponding linear combinations of mutual information terms. These work aims at analyzing the learning dynamics of models in supervised settings. In contrast, we use the PID framework for analyzing disentanglement in unsupervised representation learning.

heading: 3 PARTIAL INFORMATION DECOMPOSITION FOR DISENTANGLEMENT
text: In this section, we analyze the current metrics and introduce our framework. In Section 3.1, we introduce PID of the system we concern. In Section 3.2, we investigate the current metrics in terms of multi-variable interactions. In Section 3.3 and 3.4, we construct our disentanglement metric with bounds for PID terms. We provide a method of computing the bounds in Section 3.5.

heading: 3.1 PARTIAL INFORMATION DECOMPOSITION
text: We tackle the problem of evaluating disentanglement of a latent representation z relative to the true generative factors y from an information-theoretic perspective. Let us consider evaluating how one generative factor yk is captured by the latent representation z. The information of yk captured by z is measured using mutual information I(yk; z) = H(z)−H(z|yk).
In a desirably disentangled representation, we expect one of the latent variables z` to exclusively capture the information of the factor yk. To evaluate a given representation, we are interested in
understanding how the information is distributed between a latent variable z` and the remaining representation z\` = (z`′)`′ 6=`. This is best described by the PID framework, where the mutual information is decomposed into the following four terms.
I(yk; z) = R(yk; z`, z\`) + U(yk; z` \ z\`) + U(yk; z\` \ z`) + C(yk; z`, z\`). (1) Here, the decomposed terms represent the following non-negative quantities.
• Redundant informationR(yk; z`, z\`) is the information of yk held by both z` and z\`.
• Unique information U(yk; z` \ z\`) is the information of yk held by z` and not held by z\`. The opposite term U(yk; z\` \ z`) is also defined by exchanging the roles of z` and z\`.
• Complementary information (or synergistic information) C(yk; z`, z\`) is the information of yk held by z = (z`, z\`) that is not held by either z` or z\` alone.
The following identities, combined with Eq.1, partially characterize each term. I(yk; z`) = R(yk; z`, z\`) + U(yk; z` \ z\`), I(yk; z\`) = R(yk; z`, z\`) + U(yk; z\` \ z`). (2)
The decomposition of this system is illustrated in Figure 2a.
We expect disentangled representations to concentrate the information of yk to a single latent variable z`, and to let the other variables z\` not convey the information in either unique, redundant, or synergistic ways. This is understood in terms of PID as maximizing the unique information U(yk; z` \ z\`) while minimizing the other parts of the decomposition. The above formulation is incomplete as one degree of freedom remains to determine the four terms with the three equalities. Instead of stepping into searching for suitable definitions of these terms, we build discussions applicable to any such definitions that fulfill the above incomplete requirements2.

heading: 3.2 UNDERSTANDING DISENTANGLEMENT METRICS FROM INTERACTION PERSPECTIVE
text: Current disentanglement metrics are typically designed to measure how each latent variable captures a factor and compare it among latent variables, i.e., output a high score when only one latent variable captures the factor well.
For example, the BetaVAE metric (Higgins et al., 2017) is computed by estimating the mean absolute difference (MAD) of two i.i.d. variables following p(z`|yk) = Ep(x|yk)p(z`|x) for each ` by Monte-Carlo sampling and training a linear classifier that predicts k from the noisy estimations of the differences. The FactorVAE metric (Kim & Mnih, 2018) is computed similarly, except that the MAD is replaced with variance (following normalization by population), and a majority-vote classifier is used to eliminate a failure mode of ignoring one of the factors and to avoid depending on hyperparameters. These metrics have the same goal of finding a mapping between ` and k by comparing the deviation of z` when fixing yk. Since the deviation is computed for each z` separately, these metrics do not count how each latent variable z` interacts with the other variables z\`.
Another example is MIG (Chen et al., 2018), which compares mutual information I(yk; z`) for all ` and uses the gap between the maximum and the second maximum among them. More precisely, MIG is defined by the following formula.
MIG = 1
K K∑ k=1 1 H(yk) max ` min `′ 6=` (I(yk; z`)− I(yk; z`′)). (3)
Here, dividing each summand by H(yk) balances the contribution of each factor when they are discrete. The difference in mutual information is rewritten as a difference in unique information as
I(yk; z`)− I(yk; z`′) = U(yk; z` \ z`′)− U(yk; z`′ \ z`) ≤ U(yk; z` \ z`′). (4) In that sense, MIG effectively captures the pairwise interactions between latent variables. This metric still ignores interplays between more than two variables. Figure 2b reveals that some of the redundant information R(yk; z`, z\`) is positively evaluated in MIG, which should have been considered as a signal of entanglement. Note that there are several extensions to MIG (Do & Tran, 2020; Li et al., 2020; Zaidi et al., 2020); see Appendix B for detailed discussions on them.
2Most studies on PID only deal with discrete systems, while deep representations often include continuous variables. There have been attempts to define and analyze PID for continuous systems (Barrett, 2015; SchickPoland et al., 2021; Pakman et al., 2021). Note that the domain of variables, which our framework depends on, is not limited with the PID framework.

heading: 3.3 UNIBOUND: NOVEL DISENTANGLEMENT METRIC
text: We can lower bound the unique information in any possible PID definitions by computable components, as we did in Eq.4. To bound U(yk; z` \z\`) instead of U(yk; z` \z`′), we replace z`′ with z\`, obtaining
U(yk; z` \ z\`) ≥ [ U(yk; z` \ z\`)− U(yk; z\` \ z`) ] + = [ I(yk; z`)− I(yk; z\`) ] +
(5)
where we use [·]+ = max{0, ·} as the difference in mutual information is not guaranteed to be nonnegative. The decomposed terms evaluated by the bound is illustrated in the lower part of Figure 2b. It effectively excludes, from the positive term, the effect of interaction between z` and any other latent variables. In a similar way to MIG, we summarize this bound over all the generative factors to obtain the metric we call UNIBOUND.
UNIBOUND := 1
K K∑ k=1 1 H(yk) max ` [ I(yk; z`)− I(yk; z\`) ] + . (6)
Dividing each summand by the entropy H(yk) has the same role as in MIG; it makes the evaluation fair between factors and eases the comparison as the metric is normalized when yk is discrete.

heading: 3.4 OTHER BOUNDS FOR PARTIAL INFORMATION TERMS
text: The UNIBOUND metric is a handy quantity to compare representations by a single scalar, while PID itself may provide more ideas on how a given representation entangles or disentangles the factors. To fully leverage the potential, we derive bounds for all the terms of interest, including redundancy and synergy terms, from both lower and upper sides.
Let II(yk; z`; z\`) = I(yk; z`) + I(yk; z\`) − I(yk; z) be the interaction information of a triple (yk, z`, z\`). Using nonnegativity of PID terms, we can derive the following bounds from Eq.1-2.[
I(yk; z`)− I(yk; z\`) ] + ≤ U(yk; z` \ z\`) ≤ I(yk; z`)− [ II(yk; z`; z\`) ] + ,[
II(yk; z`; z\`) ] + ≤ R(yk; z`, z\`) ≤ min{I(yk; z`), I(yk; z\`)},[
− II(yk; z`; z\`) ] + ≤ C(yk; z`, z\`) ≤ min{I(yk; z`), I(yk; z\`)} − II(yk; z`; z\`). (7)
Note that all six bounds are computed by arithmetics on I(yk; z`), I(yk; z\`), and I(yk; z). We can summarize each lower bound in a similar way as we did in Eq.6 and summarize the corresponding upper bound using the same ` for each k as the lower bound. While these bounds only determine the terms as intervals, they provide us enough insight into the type of entanglement dominant in the representation (redundancy or synergy).

heading: 3.5 ESTIMATING BOUNDS BY EXACT LOG-MARGINAL DENSITIES
text: When the dataset is fully annotated with discrete generative factors and the inference distribution p(z|x) and its marginals p(z`|x), p(z\`|x) are all tractable (e.g., mean field variational models), we can compute the bounds in a similar way as is done by Chen et al. (2018) for MIG. Let zS be either of z`, z\` or z. We denote by D(yk) the subset of the dataset D with a specific value of yk. The mutual information I(yk; zS) = −H(zS |yk) +H(zS) can then be computed by the following formula.
I(yk; zS) = Ep(yk,zS) log 1 |D(yk)| ∑ x∈D(yk) p(zS |x) − Ep(yk,zS) [ log 1 |D| ∑ x∈D p(zS |x) ] , (8)
Assuming that each generative factor yk is discrete and uniform, we employ stratified sampling over p(yk, zS). We approximate the expectation over p(zS |yk) by sampling x from the subset D(yk) and then sampling zS from p(z|x) to avoid quadratic computational cost. Following Chen et al. (2018), we used the sample size of 10000 in experiments. We use log-sum-exp function to compute log ∑ p(zS |x) = log ∑ exp(log p(zS |x)) for numerical stability. The PID bounds and the UNIBOUND metric are computed by combining these estimations.
When the inference distribution p(z|x) is factorized, its log marginal log p(zS |x) is computed by just adding up the log marginal of each variable as ∑ `∈S log p(z`|x). Otherwise, we need to explicitly derive the marginal distribution and compute the log density. For example, when p(z|x) is a Gaussian distribution with mean µ and non-diagonal covariance Σ, p(zS |x) is a Gaussian distribution with mean (µi)i∈S and covariance (Σij)i,j∈S . Such a case arises for the attacked model we will describe in the next section.
Let M be the sample size for the expectations, and N = |D| be the size of the dataset. Then, the computation of Eq.8 requires O(MN) evaluations of the conditional density p(zS |x).

heading: 4 ENTANGLEMENT ATTACKS
text: To confirm that the proposed framework effectively captures interactions among multiple latent variables, we apply it to adversarial representations that entangle any generative factors. Instead of making a single artificial model, we modify a given model that disentangles factors well into a noised version that entangles the original variables. We call this process an entanglement attack.
Let z ∈ RL be the representation defined by a given model. Our goal is to design a transform from z to an attacked representation z̃ so that disentanglement metrics fail to capture entanglement unless they correctly handle multi-variable interactions.
As we mentioned in Section 3, metrics that do not take into account the interaction among multiple latent variables may underestimate redundant information. To crystallize such a situation, we first design an entanglement attack to inject redundant information into multiple variables. For completeness, we also design a similar attack to inject synergistic information as well.

heading: 4.1 REDUNDANCY ATTACK
text: Let U be an L × L orthonormal matrix and ∈ RL be a random vector following the standard normal distribution N (0, I). Using a hyperparameter α ≥ 0, we define the redundancy attack by
z̃red =
( z
αUz +
) . (9)
The coefficient α adjusts the degree of entanglement. When α = 0, the new representation just appends noise elements to a given representation, which does not affect the disentanglement. Increasing α makes the additional dimensions less noisy, resulting in z̃red redundantly encoding the information of factors conveyed by the original representation. The mixing matrix U chooses how the information of individual variables in z is distributed to the additional dimensions. If we choose
U = I, the dimensions are not mixed; thus considering one-to-one interaction between pairs of variables is enough to capture the redundancy. To mix the dimensions, we can use U = I− 2L11 > instead, which is an orthonormal matrix that mixes each variable with the others.
To evaluate mutual information terms for MIG and UNIBOUND metrics after the attack, we need an explicit formula for the inference distribution p(z̃red|x). When the original model has a Gaussian inference distribution p(z|x) = N (z;µ(x),Σ(x)), the attacked model is a summation of linear transformation of z and a standard normal noise, which results in a Gaussian distribution
p(z̃red|x) = N ( z̃red; ( µ(x)
αUµ(x)
) , ( Σ(x) αΣ(x)U>
αUΣ(x) I + α2UΣ(x)U>
)) .

heading: 4.2 SYNERGY ATTACK
text: For completeness, we also define an attack that entangles representation by increasing synergistic information. Using the same setting with an L × L orthonormal matrix U and a noise vector ∼ N (0, I), we construct the synergy attack by
z̃syn =
( αU + z ) . (10)
Here again, the coefficient α adjusts the degree of entanglement. When α = 0, the attacked version just extends the original representation with independent noise elements. By increasing α, the noise are mixed with the parts conveying the information of factors. The attacked vector z̃syn fully conveys the information of factors in z regardless of α, as we can recover the original representation by z = z̃syn1:L−αUz̃ syn L+1:2L. Note that most existing metrics correctly react to this attack since the information of individual variables is destroyed by the noise. The upper bound of the unique information is one of the quantities that positively evaluate synergistic information and is expected to ignore the attack.

heading: 5 EVALUATION
text: We evaluated the metrics in a toy model in Section 5.1 and in VAEs trained on datasets with the true generative factors in Section 5.2. We also performed detailed analyses of VAEs by plotting the PID terms of each factor, which is deferred to Appendix E due to the limited space.

heading: 5.1 EXACT ANALYSIS WITH TOY MODEL
text: We consider a toy model with attacks defined in Section 4 as a sanity check. Suppose that data are generated by factors y ∼ N (0, I), and we have a latent representation that disentangles them up to noise as z|y ∼ N (y, σ2I), where σ > 0. With this simple setting, we can analytically compute MIG and UNIBOUND. For example, when we set σ = 0.1 and K = 5, we can derive the scores after the redundancy attack as MIG = 1c log ( 101× 1+0.65α 2 1+1.01α2 ) and UNIBOUND = 1c log ( 101× 1+0.01α 2 1+1.01α2 ) ,
where c = log(2πe). As a function of α, UNIBOUND decreases faster than MIG. The difference comes from the information distributed among the added dimensions of the attacked vector; while UNIBOUND counts all the entangled information, MIG only deals with one of the added dimensions. Indeed, the amount of the untracked entanglement remains in MIG after taking the limit of α→∞ as MIG→ 1c log 65, while UNIBOUND → 0. We can also compute the scores after the synergy attack as MIG = UNIBOUND = 1 c log ( 1 + 1α2+0.01 ) , where both metrics correctly capture the injected synergy. Refer to Appendix C for the derivations of the exact scores for arbitrary parameter choice.

heading: 5.2 EMPIRICAL ANALYSIS WITH ANNOTATED DATASETS
text: We used the DSPRITES and 3DSHAPES datasets for our analysis. The DSPRITES dataset consists of 737, 280 binary images generated from five generative factors (shape, size, rotation, and x/y coordinates). The 3DSHAPES dataset consists of 480, 000 color images generated from six generative factors (floor/wall/object hues, scale, shape, and orientation). All the images have 64 × 64 pixels. We used all the factors for y, encoded as a set of discrete (categorical) random variables.
We used variants of VAEs as methods for disentangled representation learning: β-VAE (Higgins et al., 2017), FactorVAE (Kim & Mnih, 2018), β-TCVAE (Chen et al., 2018), and JointVAE (Dupont, 2018). We trained all the models with six latent variables, one of which in JointVAE is
a three-way categorical variable for DSPRITES and a four-way categorical variable for 3DSHAPES. We trained each model eight times with different random seeds and chose the best half of them for each metric to avoid cluttered results due to training instability. We optimized network weights with Adam (Kingma & Ba, 2015). We used the standard convolutional networks used in the literature for the encoder and the decoder. See Appendix D for the details of architectures and hyperparameters.
For disentanglement metrics, we compare BetaVAE metric (Higgins et al., 2017), FactorVAE metric (Kim & Mnih, 2018), MIG (Chen et al., 2018), and the proposed UNIBOUND metric.
We first compared the models with each metric as shown in Figure 3a. The trend is basically similar between UNIBOUND and the other metrics, while they disagree in some cases. For example, FactorVAE achieves a higher MIG score for DSPRITES than β-VAE, while its UNIBOUND score is low. As we saw in Figure 2b, such a case occurs when a part of the redundancyR(yk; z`, z\`)−R(yk; z`, z`′) is large. This observation indicates that FactorVAE effectively forces each variable to encode information of distinct factors (i.e., one-vs-one redundancy is small), while it fails to avoid entangling the information over multiple variables (i.e., one-vs-all redundancy is large).
We can confirm that the redundancy is indeed large in FactorVAE by computing the PID bounds. In Figure 3b, we plot the aggregated bounds of U(yk; z` \ z\`),R(yk; z`, z\`), and C(yk; z`, z\`). The plot reveals that FactorVAE tends to encode the factors redundantly into multiple latent variables.
To understand the tasks and models more deeply, we evaluated the PID bounds for each factor in Figure 4. We can see that the factors are captured by the models in various ways. For example, βTCVAE succeeds to disentangle the position and scale factors in DSPRITES, while it encodes orientation synergistically. It may reflect the inherent difficulty of disentangling this factor in DSPRITES, as the image of each shape corresponding to 0◦ orientation is chosen arbitrarily. These observations help us to choose what kind of inductive biases to introduce into the models.
FactorVAE, on the other hand, tends to encode factors redundantly. This indicates FactorVAE succeeds to make individual variables encode each factor, while it fails to prevent other variables from encoding the same information. JointVAE also encodes factors redundantly. While it fails to disentangle all the factors, this is the only model that encodes the shape and orientation to single latent variables. This can be viewed as the effect of introducing a discrete variable into the representation.
We can understand large redundancy as the models failing to make variables independent enough. The lower bound of redundancy in Eq.7 is related to independence as
[II(yk; z`; z\`)]+ = [I(zk; z\`)− I(z`; z\`|yk)]+ ≤ I(zk; z\`). (11)
Therefore, a high redundancy lower bound indicates large mutual information I(zk; z\`); i.e., the latent variables are highly dependent. We conjecture that FactorVAE, which approximates the total correlation DKL(p(z)‖ ∏ ` p(z`)) by a critic, fails to make the critic capture the dependency in z enough in our experiments. Since the MIG score is relatively high, the critic succeeds to capture pairwise dependency, while it fails to capture higher dimensional dependency. The high redundancy in JointVAE can also be explained by the lack of independence; see Appendix F for details.
As in Figure 3a, some metrics have large deviations from the median. This is caused by the randomness in training rather than in evaluation; see Appendix I for detailed analyses.
As we did for the toy model, we performed entanglement attacks in Figure 3c to assess its effect on each metric in learned representations. We selected the best training trials of β-VAE and β-TCVAE. Here, we only plot the results with the redundancy attack, as each metric already behaves well against the synergy attack. The plot reveals that BetaVAE and FactorVAE metrics do not detect the redundancy injected by the attack. MIG slightly decreases with the attack, which is not significant against the score variation between learning methods as we observed in Figure 3a. UNIBOUND strongly reacts against the attack, indicating that it effectively detects the injected redundancy.

heading: 6 CONCLUSION
text: We established a framework of disentanglement analysis using Partial Information Decomposition. We formulated a new disentanglement metric, UNIBOUND, using the unique information bounds, and confirmed with entanglement attacks that the metric correctly responses to entanglement caused by multi-variable interactions which are not captured by other metrics. UNIBOUND sometimes disagrees with other metrics on VAEs trained with controlled datasets, which indicates that multivariable interactions arise not only in artifitial settings but in learned representations. We found that VAEs trained with different methods induce representations with a variety of ratios between PID terms, even if their disentanglement scores are close. It is a major future work to develop learning methods of disentangled representations on the basis of these observations.

heading: ACKNOWLEDGMENTS
text: We thank members of Issei Sato Laboratory and researchers in Preferred Networks for fruitful discussions, and thank the reviewers for helpful comments to improve the work.

heading: A ADDITIONAL DIAGRAMS

heading: B RELATIONSHIPS TO OTHER INFORMATION-THEORETIC METRICS
text: There have been several metrics proposed in the literature for information-theoretic measurement of disentanglement.
For example, Do & Tran (2020) proposed four metrics, namely WSEPIN, WINDIN, RMIG, and JEMMIG. Among them, WSEPIN is computed based on the information gap I(x; z`|z\`) = I(x; z) − I(x; z`). In the PID perspective, this quantity upper bounds the unique information of x held by z`, i.e., I(x; z`|z\`) ≥ U(x; z` \ z\`). It is similar to the upper bound of unique information that we derived in Eq.7, as
I(yk; z`)− [II(yk; z`; z\`)]+ ≤ I(yk; z`)− II(yk; z`; z\`) = I(yk; z)− I(yk; z\`). (12)
In that sense, WSEPIN is an upper bound of unique information, where yk is replaced with x and the nonnegativity of redundant information is ignored. As WSEPIN does not handle generative factors separately, it is useful when the generative factors are unknown, while our approach provides more detailed information of (dis)entanglement when the generative factors are available.
RMIG is an extension to MIG, where the conditioning between yk and x is inverted so that one can compute the quantity with uncontrolled dataset. This approach may be extended for our framework as well, in a similar way to applying the inversion to MIG. We did not use this extension as we only deal with controlled datasets in this paper to contrast our approach with a wider variety of metrics.
JEMMIG is also an extention to MIG, which provides a measurement of how each variable captures only one generative factor. This aspect is also studied by Eastwood & Williams (2018) and Li et al. (2020). As we used a set of simple latent variables each of which is either a one-dimensional real variable or a categorical variable with a small number of arms (three or four depending on the dataset), we expect that each variable does not capture much information of multiple generative factors. Indeed, we did not observe any latent variable selected for more than two generative factors, and when a latent variable is selected for two generative factors, the overall disentanglement score is low so that the effect of duplicated selection is ignorable. We still consider it an interesting future work to extend our approach for analyzing entanglement of multiple generative factors into a single latent variable.

heading: C DERIVATIONS FOR THE EXACT ANALYSIS WITH THE TOY MODEL
text: In this section, we provide a sketch of analytically computing the metrics for the toy model we used in Section 5.1. We use the following formulae.
• Let u be a Gaussian vector with covariance Σ. Then, its entropy is given by H(u) = 1 2 log(2πe|Σ|), where |·| is the matrix determinant. When we remove the `-th element from u, the remaining vector u\` follows a Gaussian distribution whose covariance matrix Σ\` is obtained by removing the `-th row and column from Σ. The determinant of Σ\` is the cofactor of Σ at the `-th diagonal element, i.e.,
∣∣Σ\`∣∣ = |Σ|(Σ−1)``. With this formula, we can derive the entropy of u\` as H(u\`) = H(u) + 12 log(Σ −1)``.
• Suppose that an invertible matrix Σ is written by blocks as Σ = (
A B C D
) . If A and its
Schur complement S := D−CA−1B are invertible, the determinant and inverse of Σ are
Provided that z is a Gaussian vector with covariance Σ, we can derive the entropies of the attacked vectors. Recall that the redundancy and synergy attacks are defined as follows.
z̃red = ( I 0 αU I )( z ) , z̃syn = ( I αU 0 I )( z ) where U is an orthonormal matrix, α > 0, and ∼ N (0, I). Here, 0 is the zero matrix. These again follow Gaussian distributions, whose covariance matrices are computed as follows.
Cov(z̃red) = ( I 0 αU I )( Σ 0 0 I )( I αU> 0 I ) = ( Σ αΣU> αUΣ I + α2UΣU> ) ,
Cov(z̃syn) = ( I αU 0 I )( Σ 0 0 I )( I 0 αU> I ) = ( α2I + Σ αU αU> I ) .
Using Eq.13 and Eq.14, we obtain their determinants and inverses.∣∣Cov(z̃red)∣∣ = |Σ|, Cov(z̃red)−1 = (α2I + Σ−1 −αU>−αU I ) ,
|Cov(z̃syn)| = |Σ|, Cov(z̃syn)−1 = (
Σ−1 −αΣ−1U −αU>Σ−1 I + α2U>Σ−1U
) .
Therefore, we obtain the following entropies for each ` ∈ {1, . . . , L}.
H(z̃red` ) = 1
2 log(2πeΣ``), H(z̃
red \` ) =
1 2 log(2πe|Σ|(α2 + (Σ−1)``)),
H(z̃redL+`) = 1
2 log(2πe(1 + α2u>` Σu`)), H(z̃ red \L+`) =
1 2 log(2πe|Σ|),
H(z̃syn` ) = 1
2 log(2πe(α2 + Σ``)), H(z̃ syn \` ) =
1 2 log(2πe|Σ|(Σ−1)``),
H(z̃synL+`) = 1 2 log(2πe), H(z̃syn\L+`) = 1 2 log(2πe|Σ|(1 + α2v>` Σ−1v`)),
H(z̃red) = H(z̃syn) = 1
2 log(2πe|Σ|).
(15) Here, u` and v` are the `-th columns of U> and U, respectively. When we use U = I − 2K11
>, these are written as u` = v` = e` − 2K1, where (e1, . . . , eL) is the standard basis of R L.
Recall that, in the toy model, the representation z is drawn from N (y, σ2I), where the generative factors y follow the standard Gaussian distribution. Therefore, the marginal distribution of the representation is p(z) = N (z; 0, (1 + σ2)I). When the representation is conditioned by a single factor yk, it follows p(z|yk) = N (z; ykek, (1 + σ2)I − eke>k ). By substituting the covariance matrices of these distributions to Σ in Eq.15 and computing their differences, we obtain the mutual information terms as follows.
I(yk; z̃ red ` ) =
{ 1 2 log 1+σ2 σ2 (k = `),
0 (k 6= `), I(yk; z̃
red L+`) =  1 2 log
1+α2(1+σ2)
1+α2(1+σ2−(1− 2K )2) (k = `),
1 2 log
1+α2(1+σ2)
1+α2(1+σ2− 4 K2
) (k 6= `),
I(yk; z̃ red \` ) =
{ 1 2 log 1+α2(1+σ2) 1+α2σ2 (k = `),
1 2 log
1+σ2 σ2 (k 6= `), I(yk; z̃
red \L+`) =
1 2 log
1 + σ2
σ2 ,
I(yk; z̃ syn ` ) =
{ 1 2 log 1+σ2+α2 σ2+α2 (k = `),
0 (k 6= `), I(yk; z̃
syn L+`) = 0, (16)
I(yk; z̃ syn \` ) = { 0 (k = `), 1 2 log 1+σ2 σ2 (k 6= `), I(yk; z̃ syn \L+`) =  1 2 log
(1+σ2)(1+σ2+α2)
σ2(1+σ2+α2)+α2(1− 2K )2 (k = `),
1 2 log
(1+σ2)(1+σ2+α2)
σ2(1+σ2+α2)+α2 4 K2
(k 6= `)
I(yk; z̃ red) = I(yk; z̃ syn) = 1
2 log
1 + σ2
σ2 .
The metrics in Table 1 are computed from these quantities. In addition, we can compute other partial information terms as follows.
R(yk; z̃redk , z̃red\k) = 1
2 log
1 + α2(1 + σ2)
1 + α2σ2 , C(yk; z̃redk , z̃red\k) = 0,
R(yk; z̃synk , z̃ syn \k ) = 0, C(yk; z̃ syn k , z̃ syn \k ) =
1 2 log
(1 + σ2)(σ2 + α2)
σ2(1 + σ2 + α2) .
We can observe that the redundancy and synergy terms effectively increase with the corresponding attacks.

heading: D MODEL ARCHITECTURES AND TRAINING HYPERPARAMETERS
text: The architectures of the encoder and decoder used in the experiments are listed in Table 2. We used ReLU nonlinearity at each convolutional layer except for the final output of each network. In addition, we used for the critic in FactorVAE a feedforward network with five hidden layers each of which consists of 1,000 leaky ReLU units with the slope coefficient of 0.2.
The hyperparameters used in training are listed in Table 3 and Table 4. For hyperparameters not listed in the tables, we used the values suggested in the original papers.

heading: E FULL RESULTS FOR FACTOR-WISE PID ANALYSES
text: We illustrate the estimated bounds of PID terms for each factor in Figure 6, whose small version appeared in Figure 4. We summarize these results in Table 5. This table is made by observing and categorizing the plots in Figure 6 into some patterns as follows. Note that we ignore the error bars here.
1. If the lower bound of the unique information is larger than the upper bounds of the redundant and complementary information, mark the plot as disentangled. In this case, the model is determined as successfully disentangling the factor regardless of the concrete definition of PID. Note that the model does not necessarily learn the factor completely; see the figure for how much the information of the factor is uniquely captured by a latent variable.
2. If the upper and lower bounds of the redundant information are larger than those of the complementary information by a margin, mark the plot as redundant. In this case, the model entangles the factor in a redundant way. As we analyzed in Section 5.2, this indicates that the latent variables are highly dependent.
3. If the upper and lower bounds of the complementary information are larger than those of the redundant information by a margin, mark the plot as synergistic. In this case, the model entangles the factor in a synergistic way. This may occur even if the latent variables are independent. We may require additional inductive biases to help the model disentangle the factor.
4. Otherwise, mark the plot as flat.

heading: F POSSIBLE EXPLANATIONS FOR HIGH REDUNDANCY IN JOINTVAE
text: We observed in Table 5 and Figure 6 that JointVAE suffers from high redundancy in all the factors. To explain this phenomenon, we review the training scheme of JointVAE (Dupont, 2018). The
JointVAE model consists of one categorical variable z1 andL−1 Gaussian variables z2:L. LetU(z1) be the uniform categorical distribution and pd(x|z) be the decoder to be learned simultaneously. Then, the objective function of JointVAE is
L = Ep(z|x)[log pd(x|z)]−γ|DKL(p(z1|x)‖U(z1))− c1|−γ|DKL(p(z2:L|x)‖N (z2:L; 0, I))− c2|, which involves three hyperparameters: the regularization coefficient γ, the capacity of the discrete variable c1, and the capacity of continuous variables c2. Throughout training, γ is kept constant, while the capacities c1 and c2 are gradually increased and saturated at the predefined maximum values (Cc and Cz , respectively) in the middle of training. As the capacities are positive at the end of training, this indicates that the KL terms, which include the total correlation of the latent variables (Kim & Mnih, 2018; Chen et al., 2018), are large in the trained model. Therefore, as we discussed in Section 5.2, this model does not add pressure on the representation to be less redundant, which may explain the high redundancy3. See Appendix G for the results with varying Cz , whose results also support the above hypothesis as low capacities induce lower redundancy. This training scheme is designed to align the amount of information captured by discrete and continuous variables, which seem to be successful as we observed, i.e., it effectively captures the shape factor in both datasets compared to the other methods.
G INFLUENCE OF REGULARIZATION HYPERPARAMETERS ON DISENTANGLEMENT
We trained each model with varying hyperparameters (β for β-VAE and β-TCVAE, γ for FactorVAE, and the capcity of continuous variables Cz for JointVAE) and investigated how the metrics as well as PID terms are affected by the regularization strengths4.
3To confirm that large capacity actually causes large dependency, we measured the KL terms and the total correlation of the trained JointVAE models. We summarize the results in Table 6. These values indicate that the KL terms are actually close to the capacity hyperparameters (see Table 3 and Table 4), and more than a half of them are occupied by the total correlation.
4For JointVAE, we chose the final capacity of continuous variables instead of the regularization coefficient as we expect the capacity to be more relevant to disentanglement. The former controls the KL divergence
The results for DSPRITES and 3DSHAPES are illustrated in Fig.7 and Fig.8, respectively. From the left panels, we can observe that the UNIBOUND metric is positively correlated with the regularization strength5. It indicates that the regularization method introduced by each model positively contributes to disentanglement in the PID perspective. We also estimated PID bounds in the right panels. They show some interesting effects of regularization against the types of entanglement. For example, in JointVAE, the representation has low redundancy when the capacity of continuous variables is small, and the redundancy grows significantly when we increase the capacity. It indicates that a large capacity makes the model enforce each latent variable to capture details of the input images, ignoring how the information is redundantly captured by other variables.

heading: H LARGE FIGURES OF EXPERIMENTAL RESULTS
text: We put large versions of Figure 3 and Figure 4 in Figure 9 and Figure 6, respectively, for finer rendering.

heading: I TRAINING AND EVALUATION STABILITY
text: We illustrate the disentanglement scores of models trained with eight training seeds in Figure 10 and Figure 11. As each disentanglement metric involves sampling during evaluation, the evaluated score has some randomness even if we fix the training random seed. This plot reveals that the deviation caused by randomness in evaluating disentanglement metrics is much smaller than the deviation caused by randomness in training each model.
between the aggregated posterior of continuous variables and their prior at the end of training, while the latter controls the strength of enforcing the KL divergence to be close to the capacity. See the original paper (Dupont, 2018) for more details.
5In JointVAE, the hyperparameter controls the final KL term; hence, a smaller hyperparameter should induce more disentangled representation.

Review summaries:
The paper proposes a novel disentanglement metric 'Unibound'. To construct the metric the method uses partial information decomposition, which decomposes the the mutual information between the latent variables and the labels into a sum of other information terms with specific roles: - Redundant information - Unique information - Complementary information  Each of these terms are precisely defined and an intuitive explanation os given. The proposed UnibBund metric is defined similarly to MIG based on the above terms. The main difference is that UnibBund uses a better composition than MIG, that: - better captures our intuitive notion of disentanglement - handles adversarial representation attacks better

The authors leveraged partial information decomposition (PID) for analyzing multi-variable interactions in latent representations. The PID framework shows that (i) the mutual information between a latent variable and a generative factor can be divided into unique, redundant, and synergistic information terms, and (ii) the uniqueness term corresponds to the degree of disentanglement. The authors also introduced a disentanglement metric by modifying MIG and conducted experiments with VAE-based models on two datasets.

The paper proposes an information theoretic approach to analyze the disentanglement. Using Partial Information Decomposition (PID) framework, the authors attempt to characterize the interaction among latent factors, correlation/disentanglement, for more than two latent variables. They generalize the existing mutual information gap (MIG) metric for multiple latent variables, by decomposing mutual information term into unique, redundant, and complementary information. The authors discussed that the pairwise correlation analysis endorses a portion of the redundant information as a measure of disentanglement, which should have been considered as entanglement. Additionally, they derive bounds for the partial information terms and demonstrate the limitations of existing disentanglement measures, i.e., BetaVAE, FactorVAE, and MIG in quantifying multivariate correlation in the latent space. The authors also designed two entanglement attacks to assess the disentanglement measures under injected correlation to the latent representation. The authors used two datasets, dSprites and 3dshapes to empirically substantiate their findings.