SlowGuess/NeurIPS_2023_OCR huggingface dataset:
To quantify the information contained in the representation, we calculate the usable information (in bits), which is a lower bound to the information contained in the representation [25, 26]. To train our decoder, we used the GradientBoostingClassifier from sklearn with default parameters. We trained on 8000 samples and tested on 2000. We evaluated the information on a held-out test set, and hence the negative values correspond to overfitting on the training set. In Table 1, the numbers in parentheses correspond to the number of ground truth factors. We used the same setup for the rotated Mnist experiments (Fig. 3). For the rotation angle, we discretized the angle of rotation  $(-45^{\circ}, 45^{\circ})$  into 10 bins of equal size, and predicted the discrete bin. We predicted the rotation angle applied to the first view. When comparing with a constractive learning approach (Fig. 3, right), we used the same encoder backbone as our GK-VAE<sup>6</sup>. We pre-trained with a batch size of 256 for 60 epochs with a learning rate of 0.001, and then trained the linear classifier for 10 epochs with an initial learning rate of 0.03. We used a latent dimension of 20.
<sup>6</sup>We used the code from: https://github.com/HobbitLong/CMC
Figure 5: Latent traversals and DCI plots show optimization results in separation of common and unique information when using partitioned decoder. Same conventions as Fig. 2. (Left) 3dshapes: The top 3 rows shows the unique latents, the middle 3 the common (and the bottom 3 are the unique latents for the second view). The ground truth unique generative factors are  $(0,1,2)$  corresponding to floor color, wall color, and object color. Our model correctly recovers that those factors are unique (first three rows in the figure), and that the other factors are common (middle three rows). (Right) dsprites: The top 2 rows shows the unique latent variables, the middle 3 the common (and the bottom 2 are the unique latent variables for the second view). The ground truth unique generative factors are  $(3,4)$  corresponding to x-position, y-position respectively. Our model correctly recovers that those factors are unique (first two rows in the figure), and that the other factors are common (middle three rows).
B.1 DCI Plots and Latent Traversals:
Let  $d$  be the dimension of the representation  $z$  and let  $t$  be the true generating factors. The idea is to train a regressor  $f_{j}(z):\mathbb{R}^{d}\to \mathbb{R}$  to predict the ground truth factors  $t_j$  for each  $j$  from the representation  $z$ . This results in a matrix of coefficients that describe the importance of each component of the representation for predicting each ground truth factors. This matrix  $R$  is the importance matrix that we visualize in the paper, where  $R_{ij}$  reflects the relative importance of  $z_{i}$  for predicting  $t_j$  (where  $z_{i}$  refers to the  $i^{th}$  component of the representation). We used the GradientBoostingClassifier from sklearn with default parameters, similar to [18] to predict the ground truth factors.
We visualize the latent components by traversing one latent variable at a time, while keeping the others fixed. We plot traversals of the prior  $p(z)$  and the posterior  $q(z|x)$  in different figures (traversals of the prior are in Fig. 2, Fig. 3, and Fig. 5, while the others are posterior traversals). We obtain the posterior traversals by encoding an observation through  $q(z|x)$  and traversing each component of the representation.
C Other Related Work
A similar formulation has been used for multi-view learning [41], with two separate autoencoders, with an constraint that each latent representation be similar, with the similarity measured by the canonical correlation of the latent representations. They did not motivate it from an information-theoretic perspective; and rather empirically found that such an optimization lead to good representations in the multi-view setting.
Also related to our work is [4]. [4] defined the approximate Gács-Körner information in the following manner:
$$
\begin{array}{l} \max _ {z} I (x _ {1}; z) \\ \text {s . t .} \quad H (z | x _ {2}) <   \delta \tag {19} \\ z \leftrightarrow x _ {1} \leftrightarrow x _ {2} \\ \end{array}
$$
By showing that they could perform the optimization over deterministic functions  $f$  such that  $z = f(x_{1})$ , they formed a Lagrangian corresponding to:
$$
\max  _ {f} H \left(f \left(x _ {1}\right)\right) - \lambda H \left(f \left(x _ {1}\right) \mid x _ {2}\right) \tag {20}
$$
They noted that the above optimization is difficult to perform and that future work should look into avenues for computing this quantity; indeed it looks difficult to learn the function  $f$  from the above optimization problem. They also suggested that this approximate form of the Gács-Körner common information had potential applications in terms of compression, since the (approximate) common information only needs to be represented once.
C.1 Relationship to redundant information in the Partial Information Decomposition
Our approach also relates to approaches that aim understanding how the information that a set of sources contain about a target variable is distributed among the sources. In particular, [42] proposed the Partial Information Decomposition (PID), which decomposes the information that two sources  $X_{1}$  and  $X_{2}$  contain about a target variable  $Y$  into a set of components that are unique, redundant, and synergistic. A central quantity in this decomposition, the redundant information, reflects the shared information about a target variable. The Gacs-Korner common information is equivalent to existing definitions redundant information if the target is reconstructing the sources (i.e.,  $Y = (X_{1},X_{2})$ ) [43]. We note that computing the redundant information from high dimensional samples has been challenging. Recently [44] proposed an approach that could be applied on high dimensional sources but where the target was low dimensional. Here, our approximation of the common information reflects a further step which can be applied on high dimensional samples (and targets).
C.2 Other multi-view representation learning approaches for partitioning common/unique information
Our work relates to a growing body of multi-view representation learning approaches aiming to disentangle common and unique information from grouped data. [45] examine the setting of extracting a common content and variable style from a set of grouped images based on content. [46, 47, 48] are also similar in spirit, aiming to disentangle the shared and unique component between paired data.
These existing methods appear to focus on qualitatively partitioning the information through the use of different objectives, however these approaches lack a well-defined notion of common (and unique) information, making it difficult to prove theoretical guarantees for the recovery of the common/unique component, or enable quantification of the information contained in the representation. In contrast, our setup allows us to formalize the decomposition of information using a well-defined information theoretic notion of common/unique information and we show both formally and empirically that we can optimize the objective with a simple modification to a traditional VAE setup and that we can quantify the amount of information contained in the representation.
D Limitations of our approach
To validate our approach, we focused on the simpler setting where we have paired data, however, we could extend our formulation to find common information between  $n > 2$  sources, as well as finding common information between subsets of sources. While our approach can be naturally extended to find the common information between  $n$  sources, future work could investigate a scalable approach to identify common and unique information between arbitrary subsets of the sources. Additionally, to validate our approach empirically, we focused on using a convolutional encoder on relatively small images and video frames, but our formulation is general and the encoder could be interchanged depending on the complexity and inductive biases of the task and data.
E Differences between GK Common Information and Mutual Information
The following examples highlight the difference between GK information and mutual information, and suggest why using GK information may be desirable in practical settings.
Case 1: Let  $X_{1} = C + N_{1}$  and  $X_{2} = C + N_{2}$ , where  $C$  and  $N_{i}$  are independent variables, but the noise  $N_{1}$  and  $N_{2}$  are correlated. Suppose  $C$  is discrete - for example, a 0 or 1 feature - and the variance of the noise  $N_{i}$  is relatively small ( $\| \Sigma_1\| \ll 1$ ). In this case, since  $C$  can be recovered deterministically from  $C + N_{i}$  (for example by thresholding its value) the GK common information between  $X_{1}$  and  $X_{2}$  is 1 bit. On the other hand, the mutual information between  $X_{1}$  and  $X_{2}$  - which is given by  $I(X_{1},X_{2}) = 1 + 0.5\log (\frac{\operatorname*{det}\Sigma_{1}\operatorname*{det}\Sigma_{2}}{\operatorname*{det}\Sigma}) -$  can be made arbitrarily large by increasing the correlation between  $N_{1}$  and  $N_{2}$ , i.e., making  $\operatorname*{det}\Sigma \to 0$ . Hence, the the value of MI is unrelated to the actual semantic information shared between the two variables. In contrast, the GK common information only identifies the common component  $C$ . In a sense, the GK is able to recover the underlying "discrete" or "symbolic" information that is common between two different continuous sources. This behavior is valuable in settings like neuroscience, where one wants to measure what is robustly encoded by both representations and not just how much they are correlated (which could be due to any amount of spurious factors). Additionally, with our relaxation of the problem we can measure how much "almost" discrete information is encoded in both, which makes the method more applicable to real-world cases.
Case 2: Let  $X_{1} = f_{1}(C;U_{1})$  and  $X_{2} = f_{2}(C;U_{2})$  where  $C$  and  $U_{i}$  are independent but  $U_{1}$  and  $U_{2}$  are correlated. Suppose  $f_{1}$  is an invertible data generating function mapping latent factors to observations (such as pixels). By maximizing mutual information between views, one implicitly measures those correlations (based on a similar computation to above in Case 1), whereas in the GK sense one cares about only the fully shared components between views. These different perspectives can offer different utility depending on the use case.
We show in Fig. 6 and Fig. 7 that we can recover the partitioning of common and unique factors in practice from the generative model in Case 2.
In Table 2, we compute the information encoded in the common and unique latent components for the common-dsprites experiment described in the main text, with corresponding DCI matrix and latent traversals in Fig. 2. We also report additional runs for the common-dsprites and common-3dshapes experiments in Fig. 8 and Fig. 9 respectively. These additional runs are consistent with what was reported in the text, separating the common and unique factors.
To introduce correlations between the unique components (Case 2 in App. E) we modified the generative model from Eq. 15 where the generative model for the data  $(x_{1},x_{2})$  was
$$
x _ {1} = f \left(z _ {u} ^ {1}, z _ {c}\right), \quad x _ {2} = f \left(z _ {u} ^ {2}, z _ {c}\right), \tag {21}
$$
where  $z_{c}$  is shared between the views and  $z_{u}^{i}$  is the unique information encoded in the  $i^{th}$  view and  $f$  corresponds to a rendering function. To allow the unique components to be dependent (but not the same as the common latent component  $z_{c}$ ), we modified the generative model so that  $z_{u}^{2}$  was conditionally dependent on the value of  $z_{u}^{1}$ . We implemented this constraint component-wise for each component of  $z_{u}^{i}$  by reducing the space of possible latent values of the corresponding components of  $z_{u}^{2}$  (to half, floored if odd) depending on the component-wise values of  $z_{u}^{1}$ . We show in Fig. 6 and Fig. 7 that we can recover the partitioning of common and unique factors in practice from this generative model.
In addition to the experiments described in the main text, we report variants of common-dsprites and common-3dshapes. In particular, we change the set of ground-truth common and latent factors.
For the common-3dshapes, we specified that the viewpoint was the unique latent variable  $z_{u}$ , whereas the other latent variables (background color, floor color, object color, shape, size) were common to both views. We show the DCI matrix and the traversals in Fig 10. For the common-dSprites variant, we set the unique components to be the size, scale, and orientation, and the common latent factors to be the x and y position. We show the DCI matrix and latent traversals in Fig. 11. The common and unique latent variables from our optimization separated these ground truth factors.
Figure 6: Decoder from all latents partitions common and unique information even when unique generating factors are dependent. We modified the generative model from Eq. 15 so that the common and unique generating variables were independent but there were correlations between the unique components. In particular, the latent factors for the second view were conditionally dependent on the unique latent factors. We still observed a partitioning of the learned common and unique latent variables, as in the paper. The top 3 rows show the unique factors, the middle 3 the common (and the bottom 3 are the unique factors for the second view). The ground truth unique generative factors are  $(0,1,2)$  corresponding to floor color, wall color, and object color. Our model correctly recovers that those factors are unique (first three rows in the figure), and that the other factors are common (middle three rows).
Figure 7: Decoder from partitioned latents also partitions common and unique information, similar to when decoding from all latents in Fig. 6. The top 3 rows show the unique factors, the middle 3 the common (and the bottom 3 are the unique factors for the second view). The ground truth unique generative factors are  $(0,1,2)$  corresponding to floor color, wall color, and object color. Our model correctly recovers that those factors are unique (first three rows in the figure), and that the other factors are common (middle three rows).
Figure 8: Additional runs for the same experiment as Fig. 2 (right) for common-dsprites, with the same conventions as Fig 2. These runs also show the same blockwise partitioning of common and unique information, as in the paper.
SHAPE (3)	SCALE (6)	ANGLE (40)	X-POS (32)	Y-POS(32)	KL TOTAL
COMMON	1.54	2.45	2.88	-0.33	-0.29	9.57
UNIQUE	0.08	0.03	-0.5	3.58	3.63	9
TOTAL	1.54	2.45	2.76	3.68	3.69	18.57
Table 2: Usable Information (in bits) in representation for a `django` experiment. The common information is separated from the unique information. The ground truth unique factors (x-position and y position) were almost perfectly encoded in the corresponding latents latents, and the other factors (shape, scale, and angle) were correctly encoded in the common latent variables. The numbers in parenthesis represent the number of discrete factors for each latent variable.
Figure 9: Additional runs for the same experiment as Fig. 2 (left) for common-3dshapes, with the same conventions. These runs also show the same blockwise partitioning of common and unique information, as in the paper.
Figure 10: DCI matrix of common-3dshapes variant for different random seeds. The top row shows the unique factor, the middle 5 the common (and the bottom row is the unique factors for the second view). The ground truth unique generative factors are (5) corresponding to the viewpoint. Our model generally correctly recovers that the factor is unique (first row in the figure), and that the other factors are common (middle five rows).
Figure 11: DCI Matrix of `dsprites` experiment variant for different random seeds. The top 3 rows show the unique latent variables, the middle 2 the common (and the bottom 3 are the unique latent variables for the second view). The ground truth unique generative factors are  $(0,1,2)$  corresponding to shape, scale, and viewpoint angle. Our model correctly recovers that those factors are unique (first three rows in the figure), and that the other factors (x and y position) are common (middle two rows).
Figure 12: Effect of  $\beta_{u}$ . Additional runs for Fig. 2 (right) for common-3dshapes with 5 unique and 5 common latents per view. For large  $\beta_{u}$  (e.g.  $\beta_{u} = 100$ , right), as in the paper, we see a similar blockwise separation of the common and unique latents, even when the number of latents does not match the ground-truth number of latents.