SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Gács-Körner Common Information Variational Autoencoder
Michael Kleinman $^{1}$  Alessandro Achille $^{2,3*}$  Stefano Soatto $^{1}$  Jonathan C. Kao $^{1}$
<sup>1</sup>University of California, Los Angeles <sup>2</sup>AWS AI Labs <sup>3</sup>Caltech
michael.kleinman@ucla.edu aachille@amazon.com
soatto@cs.ucla.edu kao@seas.ucla.edu
We propose a notion of common information that allows one to quantify and separate the information that is shared between two random variables from the information that is unique to each. Our notion of common information is defined by an optimization problem over a family of functions and recovers the Gács-Körner common information as a special case. Importantly, our notion can be approximated empirically using samples from the underlying data distribution. We then provide a method to partition and quantify the common and unique information using a simple modification of a traditional variational auto-encoder. Empirically, we demonstrate that our formulation allows us to learn semantically meaningful common and unique factors of variation even on high-dimensional data such as images and videos. Moreover, on datasets where ground-truth latent factors are known, we show that we can accurately quantify the common information between the random variables.2
Data coming from different sensors often capture information related to common latent factors. For example, many animals have two eyes that capture different but highly-correlated views of the same objects in the scene. Similarly, sensors of different modalities, such as eyes and ears, capture correlated information about the underlying scene, as do videos and other time series, where the sensors are separated in time rather than in modality. Learning how information of one sensor maps to information of another provides a self-supervised signal to disentangle the variability that is intrinsic in a sensor from the latent causes (e.g., objects) that are shared between multiple sensors. Indeed, there is evidence that infants spend a long time during development purposefully experiencing objects through different senses at the same time [1].
Motivated by this, we propose to learn meaningful representations of multi-view data by quantifying and exploiting such structure in a self-supervised fashion by using an information theoretic notion of common information as the guiding signal to disentangle common shared information present in high dimensional sensory data (Fig. 1).
However, defining a notion of common information is itself not trivial. The most natural and typical way to quantify the "common part" between random variables would be by quantifying their mutual information. But mutual information has no clear interpretation in terms of a decomposition of random variables in unique and common components. In particular, [2] note that there is generally no way to write two variables  $X$  and  $Y$  using a three part code  $(A, B, C)$  such that  $X = f(A, C)$ ,  $Y = g(B, C)$  and where  $C$  encodes all and only the mutual information  $I(X; Y)$ . Discovering the
*Work performed as external collaboration not related to Amazon
$^{2}$ Code available at: https://github.com/mjkleinman/common-vae
Figure 1: High level schematic of approach. Red denotes shared latent factors (size, shape, floor, background and object color) and black denotes unique latent (viewpoint). The aim is to extract  $z_{c}$ , which is a random variable that is a function of both inputs  $x_{i}$ . We also allow for unique latent variables  $z_{u}$  to capture information that unique to each view. The latent representations are used to reconstruct the inputs.
largest common factor  $C$ , which encodes what is known as the Gács-Körner common information, from high dimensional data is then a distinct problem on its own [3, 4].
To the best of our knowledge, there are currently no approaches to compute or approximate the Gács-Körner common information from high-dimensional samples [4]. In this work, we seek to learn common representations that satisfy the constraint that they are (approximately) a function of each input. A contribution in this paper is that we generalize the constraint that the representation needs to be a deterministic function and allow it to be a stochastic map. As we later show, this is helpful for quantifying and interpreting the latent representation, and allows us to parameterize the optimization with deep neural networks.
We show that our objective can be optimized using a multi-view Variational Auto-Encoder (VAE). Since in general each view can contain individual factors of variation that are not shared between the views, we augment our model with a set of unique latent variables that can capture unexplained latent factors of variation, and show that the common and unique component can be efficiently inferred from data through standard training. While training the multi-view VAE, we simultaneously develop a scalable approximation for the Gács-Körner common information, as we describe in Section 3.
To empirically evaluate the ability to separate the common and unique latent factors we introduce two new datasets, which extend commonly used datasets for evaluating disentangled representations learning: dSprites [5] and 3dShapes [6]. For each dataset, we generate a set of paired views  $(x_{1}, x_{2})$  such that they share a set of common factors. This allows us to quantitatively evaluate the ability to separate the common and unique factors. We also compare our method to multi-view contrastive learning [7] and show that thanks to our definition we avoid learning degenerate representations when the views share little information.
We hypothesize that a key reason precluding the identification of latent generating factors from observed data is that receiving a single sample of a scene is quite limiting. Indeed, classical neuroscience experiments has shown that the ability to interact with an environment, as opposed to passively observing sensory inputs, is critical for learning meaningful representations of the environment [8].
The entropy  $H(x)$  of a random variable  $x$  is  $\mathbb{E}_{p(x)}[\log \frac{1}{p(x)}]$ . The mutual information  $I(x;z) = H(z) - H(z|x)$ . Another useful identity for mutual information that we use is  $I(x;z) = \mathbb{E}_{p(x)}[KL(p(z|x)||p(z))]$  where KL denotes the Kullback-Leibler divergence.
Gács-Körner Common Information. The Gács-Körner common information [2] is defined as
$$
C _ {G K} \left(x _ {1}; x _ {2}\right) := \max  _ {z} H (z) \quad \text {s . t .} z = f \left(x _ {1}\right) = g \left(x _ {2}\right), \tag {1}
$$
where  $f$  and  $g$  are deterministic functions. The Gács-Körner common information is thus defined through a random variable  $z$  that is a deterministic function of both inputs  $x_{1}$  and  $x_{2}$ . Among all
such random variables,  $z$  is the random variable with maximum entropy. This has also been referred to as the "zero error information" in applications to cryptography [3]. It is an attempt to formalize and operationalize the idea of the common part between sources, which mutual information lacks. It is also a lower bound to the mutual information [2, 3]. To the best of our knowledge, there are no efficient techniques for computing the GK common information for high-dimensional  $x_{1}, x_{2}$ . We elaborate on the difference between GK common information and mutual information in App. E.
Variational Autoencoders Variational Autoencoders (VAEs) [9] are latent variable generative models that are trained to maximize the likelihood of the data by maximizing the evidence lower bound, or minimizing the loss:
$$
\mathcal {L} _ {\mathrm {V A E}} = \mathbb {E} _ {p (x)} \left[ \mathbb {E} _ {q _ {\phi} (z | x)} [ - \log p _ {\theta} (x | z) ] + K L \left(q _ {\phi} (z | x) \mid \mid p (z)\right) \right]. \tag {2}
$$
The VAE loss can be motivated in an information theoretic manner as optimizing an Information Bottleneck [10], where the reconstruction term encourages a sufficient representation and the KL regularization term encourages a minimal representation [11, 12]. The addition of a parameter  $\beta$  to modify how the KL regularization is penalized leads to the following loss (and corresponds to the traditional VAE loss when  $\beta = 1$ ):
$$
\mathcal {L} _ {\beta - \mathrm {V A E}} = \mathbb {E} _ {p (x)} [ \mathbb {E} _ {q _ {\phi} (z | x)} [ - \log p _ {\theta} (x | z) ] + \beta K L (q _ {\phi} (z | x) \mid | p (z)) ]. \tag {3}
$$
For larger values of  $\beta$  the representations become more disentangled, shown analytically in [13] and empirically in [14], although reconstructions become worse.
Disentangled representations A guiding assumption for representation learning is that the observed data  $x$  (i.e an image) can be generated from a (simpler) set of latent generating factors  $z$ . Assuming the latent factors are independent, the idea of learning disentangled representations involves learning these latent factors of variation in an unsupervised manner [15]. However, despite apparent empirical progress in learning disentangled representations [14, 16, 17], there remains inherent issues in both learning and defining disentangled representations [18]. In many cases, different independent latent factors may lead to equivalent observed data, and without an inductive bias, disentanglement remains ill-defined. For example, color can be decomposed into an RGB decomposition, or an equivalent HSV decomposition.
In [18, Theorem 1] it is shown that without any inductive bias, one cannot uniquely identify the underlying independent latent factors in a purely unsupervised manner from observed data. Empirically, they also found that there was no clear correlation between training statistics and disentanglement scores without supervision. Later, and related to our work, the authors examined the setting where there is paired data and no explicit supervision (weak supervision), and found that such a setup was helpful for learning disentangled representations [19]. The authors examined the setting in which the set of shared latent factors changed for each example, which was necessary for their identifiability proof. This also required using the same encoder for each view, and thus is a restricted setting that does not easily scale to multi-modal data.
Here, we study the scenario where the set of generating factors is the same across examples, as in the case of a pair of fixed sensors receiving correlated data. Additionally, our objective is motivated in an information theoretic way and our method generalizes to the case where we have different sensory modalities, which is relevant to neuroscience and multi-modal learning. Finally, our variational objective is flexible and allows estimation of the common information in a principled way.
Approximating Mutual Information Estimating mutual information from samples is challenging for high-dimensional random vectors [20]. The primary difficulty in estimating mutual information is constructing high-dimensional probability distribution from samples, as the number of samples required scales exponentially with dimensionality. This is impractical for realistic deep learning tasks where the representations are high dimensional. To estimate mutual information, [21] used a binning approach, discretizing the activations into a finite number of bins. While this approximation is exact in the limit of infinitesimally small bins, in practice, the size of the bin affects the estimator [22, 23]. In contrast to binning, other approaches to estimate mutual information include entropic-based estimators (e.g., [23]) and a nearest neighbours approach [24]. Although mutual information is difficult to estimate, it is an appealing quantity to summarily characterize neural network behavior because of its invariance to smooth and invertible transformations. In this work, rather than estimate the mutual information directly, we study the "usable information" in the network [25, 26], which corresponds to a variational approximation of the mutual information [27, 28].
Contrastive and Multi-View Approaches While (multi-view) contrastive learning aims to learn a representation of only the common information between views [7, 29, 30], we aim to learn a decomposition of the information in the views into common and unique components. Our work naturally extends to multi-sensor data that have different amounts of common/unique information (e.g., touch and vision). Moreover, contrastive approaches assume that the unique information is nuisance variability, and discard this information. Similarly, [31] also seeks to identify common information in both views, but also does not provide an objective to retain the unique information. While the multi-view literature is broad, we are not aware of previous attempts to quantify the common and unique information. Related to our approach, [32] aim to find shared and private representations using VAEs, but it differs in how the alignment of shared information is specified and the resulting objective, and they do not provide a way to quantify the information content of the private and shared components. We discuss additional related work in Appendix C.
3 Method: Gács-Körner Variational Auto-Encoder
Our formulation involves generalizing the Gács-Körner common information in eq. (1) to the case where  $f$  and  $g$  are stochastic functions so that the optimization problem becomes:
$$
\tilde {C} _ {G K} \left(x _ {1}; x _ {2}\right) := \max  _ {z} I \left(x _ {i}; z\right) \tag {4}
$$
$$
\text {s . t .} z = f _ {s} \left(x _ {1}\right) = g _ {s} \left(x _ {2}\right), \tag {5}
$$
where  $f_{s}$  and  $g_{s}$  are stochastic functions. By the equality in eq. (5), we mean that  $p(z|x_1) = p(z|x_2)$  for all  $(x_{1},x_{2})\sim p(x_{1},x_{2})$ . Note that when  $f$  and  $g$  are deterministic functions<sup>3</sup> (which are a subset of stochastic functions), then  $H(z|x_{i}) = 0$  and we recover the original definition since
$$
I \left(x _ {i}; z\right) = H (z) - H \left(z \mid x _ {i}\right) = H (z). \tag {6}
$$
Our latter generalization (eq. 4-5) is more amenable to optimization and interpretable, as we will later demonstrate. In eq. (4), we used  $x_{i}$  as a placeholder since when  $p(z|x_1) = p(z|x_2)$  for all  $(x_{1},x_{2})\sim p(x_{1},x_{2},z)$  then  $I(z;x_1) = I(z;x_2)$  since
$$
\begin{array}{l} I (z; x _ {1}) = \mathbb {E} _ {x _ {1} \sim p (x _ {1})} [ K L (p (z | x _ {1}) | | p (z)) ] \\ = \mathbb {E} _ {\left(x _ {1}, x _ {2}\right) \sim p \left(x _ {1}, x _ {2}\right)} [ K L (p (z | x _ {1}) | | p (z)) ] \\ = \mathbb {E} _ {\left(x _ {1}, x _ {2}\right) \sim p \left(x _ {1}, x _ {2}\right)} [ K L (p (z | x _ {2}) | | p (z)) ] \\ = \mathbb {E} _ {x _ {2} \sim p (x _ {2})} [ K L (p (z | x _ {2}) | | p (z)) ] \\ = I (z; x _ {2}). \\ \end{array}
$$
This means that another equivalent formulation to maximize is  $\max_z\frac{1}{2}\sum_iI(x_i;z) = \max_zI(x_i;z)$ , for any  $i$ . To optimize the objective in eq. 4-5, we need to learn a set of latent factors  $z$  that maximize  $I(x_{i};z)$ , while satisfying the constraint in eq. 5. We propose an optimization reminiscent of the VAE objective. Define  $x = (x_{1},x_{2})$  as the concatenation of both views, and  $z = (z_u^1,z_c,z_u^2)$  as a decomposition of the representation into common and unique components, and  $z_{i} = (z_{u}^{i},z_{c})$ . In particular, we seek to learn latent encodings through an encoder  $q_{\phi}(z|x)$ , which maps  $x$  to  $z$ . To optimize the objective, the representation  $z$  should maximize  $I(x_{i};z)$ , and so we should also learn a decoder  $p_{\theta}(x|z)$  that minimizes  $H(x_{i}|z)$ . This corresponds to the reconstruction term in a traditional VAE, though note here we reconstruct both views.
$$
\mathcal {L} _ {\mathrm {C V A E}} ^ {1} = \mathbb {E} _ {p (x)} [ \mathbb {E} _ {q _ {\phi} (z | x)} [ - \log p _ {\theta} (x | z) ] ] \tag {7}
$$
Without any constraints, this could be achieved trivially by using an identity mapping. To ensure that the latents encode only common information between the different views, we decompose the encodings to ensure the following constraint corresponding to eq. (5):
$$
D \left(q _ {\phi_ {c _ {1}}}, q _ {\phi_ {c _ {2}}}\right) = K L \left(q _ {\phi_ {c _ {1}}} \left(z _ {c} | x _ {1}\right) \mid \mid q _ {\phi_ {c _ {2}}} \left(z _ {c} | x _ {2}\right)\right) = 0. \tag {8}
$$
Here  $q_{\phi_{c_i}}(z_c|x_i)$  maps  $x_i$  to  $z_{c_i}$ . Rather than enforcing a hard constraint, in practice it is easier to optimize the corresponding Lagrangian relaxation:
$$
\mathcal {L} _ {\mathrm {C V A E}} ^ {2} = \mathbb {E} _ {p (x)} [ \mathbb {E} _ {q _ {\phi} (z | x)} [ - \log p _ {\theta} (x | z) ] + \lambda_ {c} D \left(q _ {\phi c _ {1}}, q _ {\phi c _ {2}}\right) ]. \tag {9}
$$
<sup>3</sup>If the constraint in Eq. 5 is satisfied, the map to the mean of the posterior  $p(z|x_1)$  (and  $p(z|x_2)$ ) for observations  $(x_{1},x_{2})$  is a deterministic function and satisfies the constraint of Gács-Körner common information.
After optimizing this objective, for a sufficiently large  $\lambda$  so that  $D(q_{1},q_{2})\approx 0$ , the common information would be:
$$
\tilde {C} _ {G K} \left(x _ {1}; x _ {2}\right) = \mathbb {E} _ {p (x)} [ K L \left(q _ {\phi_ {c _ {i}}} \left(z _ {c} \mid x _ {i}\right) \mid \mid q ^ {*} (z)\right) ], \tag {10}
$$
where  $q^{*}(z)$  is the marginal distribution induced by the encoder. However, estimating the true marginal  $q^{*}(z)$  is difficult for high-dimensional problems. In practice, we follow [9] and learn an approximate prior  $p(z) \approx q^{*}(z)$ , where both  $q_{\phi}(z|x)$  and  $p(z)$  are taken from a given family of distributions (such as multivariate Gaussians with diagonal covariance matrix). This will additionally enable us to sample from the distribution, and interpret the latent factors. To learn  $p(z)$  we also add the following regularization to our training objective:
$$
\mathbb {E} _ {p (x)} [ K L \left(q _ {\phi} (z | x) \mid \mid p (z)\right) ]. \tag {11}
$$
Alternatively, we can also exploit the degree of freedom in learning  $q_{\phi}(z|x)$  and fix  $p(z)$  to be  $\mathcal{N}(0,I)$ . In both cases, our overall objective becomes:
$$
\mathcal {L} _ {\mathrm {C V A E}} = \mathbb {E} _ {p (x)} [ \mathbb {E} _ {q _ {\phi} (z | x)} [ - \log p _ {\theta} (x | z) ] + \lambda_ {c} D \left(q _ {\phi c _ {1}}, q _ {\phi c _ {2}}\right) + \beta K L \left(q _ {\phi} (z | x) \mid p (z)\right) ]. \tag {12}
$$
Optimizing this objective alone could lead to unexplained components of information, for example the unique components. Alternatively, unique information present in the individual views may be encoded in the "common" latent variable if the reconstruction benefits outweighed the cost of the divergence between the posteriors of the encoders (the term corresponding to the  $\beta$ ).
In addition to these common latent components, we can learn unique latent components by optimizing a traditional VAE objective (i.e. with  $\lambda_c = 0$ ) for a subset of the latent variables. Importantly we also need to ensure that the KL penalty for the unique component subset is greater than for the common subset (so that it is beneficial to encode common information in the common latent components). Our final objective becomes
$$
\begin{array}{l} \mathcal {L} _ {\mathrm {C V A E}} = \mathbb {E} _ {p (x)} [ \mathbb {E} _ {q _ {\phi} (z | x)} [ - \log p _ {\theta} (x | z) ] + \lambda_ {c} D \left(q _ {\phi_ {c _ {1}}}, q _ {\phi_ {c _ {2}}}\right) \\ + \sum_ {i = 1} ^ {2} \beta_ {c} K L \left(q _ {\phi_ {c _ {i}}} \left(z _ {c} \mid x _ {i}\right) | | p \left(z _ {c}\right)\right) + \beta_ {u} K L \left(q _ {\phi_ {u _ {i}}} \left(z _ {u} \mid x _ {i}\right) | | p \left(z _ {u}\right)\right) ], \tag {13} \\ \end{array}
$$
where  $\beta_{c}$  and  $\beta_{u}$  correspond to a multiplier enforcing the cost of encoding common and unique information respectively. Importantly  $\beta_{u} > \beta_{c} > 0$ , resulting in a larger penalty on the unique latent variables (otherwise all the information would be encoded in the "unique" components). The summation in the bottom part of Eq. 13 corresponds to a summation over both encoders.  $p(z_{u})$  and  $p(z_{c})$  are both sampled from  $\mathcal{N}(0, I)$  of appropriate dimensionality.
We now show that, if the network architecture used for the VAE implements a generic enough class of encoder/decoders our method will recover the GK common information.
Theorem 3.1 (GK VAE recovers the common information). Suppose our observations  $(x_{1}, x_{2})$  have GK common information defined through the random variable  $z_{c}$  satisfying eq. 4-5 and that our parametric function classes  $q(z|x)$  and  $p(x|z)$  optimized over can express any function. Then, our optimization (with  $\beta_{c} = 0$ ,  $\beta_{u} < 1$  and decoder  $p(x|z) = \sum_{i=1}^{2} p_{i}(x_{i}|z_{i})$ ) will recover latents  $\hat{z} = (\hat{z}_{u}^{1}, \hat{z}_{u}^{2}, \hat{z}_{c})$  where  $\hat{z}_{c}$  is the common random variable that maximizes the "stochastic" GK common information in eq. 4-5, while  $\hat{z}_{u}^{i}$  is the unique information of the  $i$ -th view, which maximizes  $I(x_{i}; z_{u}^{i}, \hat{z}_{c})$ .
We provide the proof in Appendix A. Note that while the previous theorem guarantees that we will be able to separate the common and unique factors at the block level, we might not be able to disentangle the individual common factors.
3.1 Quantifying the common information
Suppose  $D(q_{\phi_{c_1}}, q_{\phi_{c_2}}) = 0$ . The term corresponding to the rate  $R_c$  of the VAE
$$
R _ {c} = I _ {q} \left(z _ {c}; x\right) = \mathbb {E} _ {p (x)} K L \left(q _ {\phi_ {c}} \left(z _ {c} \mid x\right) \mid p \left(z _ {c}\right)\right) \tag {14}
$$
is neither an upper nor lower bound on the true common information. It represents an upper bound to the information encoded in the representation specified by  $q_{\phi_c}(z_c|x)$ , but does not bound the true common information in the data, since  $q_{\phi_c}(z_c|x)$  itself is a variational approximation.
Figure 2: Latent traversals and DCI plots show optimization results in separation of common and unique information. (Left) 3dshapes: The top 3 rows shows the unique latents, the middle 3 the common (and the bottom 3 are the unique latents for the second view). The ground truth unique generative factors are  $(0,1,2)$  corresponding to floor color, wall color, and object color. Our model correctly recovers that those factors are unique (first three rows in the figure), and that the other factors are common (middle three rows). (Right) dsprites: The top 2 rows shows the unique latent variables, the middle 3 the common (and the bottom 2 are the unique latent variables for the second view). The ground truth unique generative factors are  $(3,4)$  corresponding to x-position, y-position respectively. Our model correctly recovers that those factors are unique (first two rows in the figure), and that the other factors are common (middle three rows).
To find a lower bound on the common information encoded in the dataset, we can use any mutual information estimator  $\hat{I}$  that is a lower bound (see [28] for several). The approximate common information can then be quantified by  $\hat{I}(z_q, x)$ , where  $z_q \sim q_{\phi_c}(z_c|x)$ . We report both the rate  $R_c$  and  $\hat{I}$  in the paper. We emphasize that  $\hat{I}$  can be any mutual information estimator. When the data generating distribution is known, as in our synthetic examples, we employ the "Usable Information" estimator, described in Sect. 4.2, which is a variational approximation [27].
3.2 Identifiability of the common and unique components
We now show the conditions under which our optimization will identify the common and unique latent components. Usually we do not directly observe the latent factors  $z$ , but rather an observation generated from them. We may then ask whether the common latent factors can still be reconstructed from this observation. The following proposition shows that this is indeed the case, as long as the function generating  $f$  the observation is invertible, i.e., we can recover the latent factors from the observation itself.
Proposition 3.1. ([3], Ex. 1): Define