SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
z _ {1} = \left(z _ {c}, z _ {u} ^ {1}\right), \quad z _ {2} = \left(z _ {c}, z _ {u} ^ {2}\right)
$$
where  $z_{c}, z_{u}^{1}$ , and  $z_{u}^{2}$  are mutually independent. Then for any invertible transformation  $t_{i}$  the random variable  $z_{c}$  encodes all the common information:
$$
z _ {c} = \arg \max  _ {\hat {z}} C _ {G K} (t _ {1} (z _ {1}), t _ {2} (z _ {2}))
$$
We provide the proof in Appendix A. The above proposition shows that when a set of factors is shared between views and when the unique factors are sampled independently, then the GK common random variable corresponds to shared latent factors. In particular, if the observations  $x_{i}$  are generated through an invertible function  $x_{i} = f(z_{c},z_{u}^{i})$  where  $z_{c}\sim p(z_{c})$  corresponds to the shared factors, the proposition shows that such factors can be recovered from the observations by maximizing the GK common information. In our GK VAE optimization, we optimize the "stochastic" GK common information and we also find in our experiments that we can (approximately) recover the latent factors from observations  $x_{i}$  generated from this process.
FLOOR HUE (10)	WALL HUE (10)	OBJ. HUE (10)	SCALE (8)	SHAPE (4)	ANGLE (15)	KL TOTAL
COMMON	-0.01	-0.02	-0.03	2.73	1.98	3.83	15.0
UNIQUE	3.31	3.31	3.31	0.19	0.37	0.19	12.7
TOTAL	3.31	3.31	3.31	2.69	1.98	3.82	27.7
Table 1: Usable Information (in bits) in representation for  $3dShapes$ . The ground truth unique generative factors are floor color, wall color, and object color, and the common generative factors are scale, shape and angle. The common information is separated from the unique information. The ground truth factors were almost perfectly encoded in the latents. The numbers in parenthesis represent the number of discrete factors for each latent variable.
We train our GK-VAE models with Adam using a learning rate of 0.001, unless otherwise stated. When the number of ground truth latent factors is known, we set the size of the latent vector of the VAE equal to the number of ground truth factors. This choice was not necessary, and we obtain analogous results when the size of the latent vector of the VAE was larger than the number of ground-truth factors (Fig. 12). To improve optimization, we use the idea of free bits [33] and we set  $\lambda_{\mathrm{free-bits}} = 0.1$ . This was easier than using  $\beta$  scheduling [16], since it only involved tuning one parameter. We set  $\beta_{u}$  to be  $10$ ,  $\beta_{c}$  to be  $0.1$  and  $\lambda_{c} = 0.1$ . We trained networks for 70 epochs, except for the MNIST experiments, where we trained for 50 (details in the Appendix).
To ensure that the latents are shared to both encoders, during training we randomly sample  $z$  from either encoder  $q_{\phi_i}(z_c|x_i)$  with  $p = 0.5$ . We opted to randomly sample the latents from each encoder, as opposed to performing averaging, to ensure that the latent will always be a function of an individual view  $x_i$ . This is in addition to the soft constraint governed by  $\lambda_c$  in the loss.
4.1 Evaluation Datasets
We primarily focus on the setting where the ground truth latent factors and generative model are known, in order to quantitatively benchmark our approach. To do so, we constructed datasets with ground truth latent factors so that some of the latent factors are shared between each views. That is, the generative model for the data  $(x_{1}, x_{2})$  is
$$
x _ {1} = f \left(z _ {u} ^ {1}, z _ {c}\right), \quad x _ {2} = f \left(z _ {u} ^ {2}, z _ {c}\right), \tag {15}
$$
where  $z_{c}$  is shared between the views and  $z_{u}^{i}$  is the unique information encoded in the  $i^{th}$  view and  $f$  corresponds to a rendering function.
To construct such datasets, we modified the 3dshapes [6] and the dsprites dataset [5]. We select a subset of the latent factors to be shared between the views, while the remaining factors are sampled independently for each view. The 3dshapes dataset [6] contains six independent generating factors: floor color, background color, shape color, size, shape, and viewpoint. Each latent factor can only take one of a discrete number of values. The dsprites dataset [5] contains five independent generating factors: shape, scale, rotation, x and y position. Each latent factor can only take one of a discrete number of values. When we generate multi-view data following the generative model in eq. (15), we refer to these datasets as Common-3dshapes and Common-dsprites respectively. We consider additional variants in the Appendix.
We also examine the Rotated Mnist Dataset, where the two views are two random digits of the same class to which a random rotation is applied. In particular, the class of the digit is common information between the views whereas the rotation is unique. We also examine the synthetic video dataset Sprites (not to be confused with `dsprites`) described in [34] and evaluate the common information in frames separated  $t$  frames apart.
DCI Disentanglement Matrix [35]. Let  $d$  be the dimension of the latent space and let  $t$  be the true generating factors. The idea is to train a regressor  $f_{j}(z):\mathbb{R}^{d}\to \mathbb{R}$  to predict the ground truth factors
4Our experiments can be reproduced in approximately 3 days on a single GPU (g4dn instance).
Figure 3: Left. Traversals for Rotated Mnist. The unique components of the latent (rows 1,2) appear to encode the "thickness" and rotation of the digit, whereas the common components appear to represent the overall digit (rows 3-6); and also the output of view 1 does not depend on the latents in rows 7,8 (these correspond to the unique components for view 2.) Middle. Corresponding DCI matrix, where factor 0 corresponds to the label, while factor 1 corresponds to the rotation (discretized into 10 bins). Right. Comparison against contrastive implementation from [7], where the contrastive approach does not encode any usable information about the unique factor (the rotation).
$t_j$  for each  $j$ . This results in a matrix of coefficients that describe the importance of each latent for predicting each ground truth factors. This can then be visualized as a matrix where the size of the square reflects the coefficient. We use this metric to assess the partitioning of the learned common and unique representations. We used the random forest regressor, similar to [18] to predict a discrete number of latent classes.
Usable Information [25, 36, 26]. We use this to approximate the mutual information when  $H(x)$  is known, as it is in the datasets previously described. It is a lower bound to mutual information. We use this to lower bound the information contained in the representation  $z$  in the next section.
Separation of Common and Unique Latent Variables. We first examine whether our formulation can correctly separate the common and unique latent factors. After optimizing a network on our Common-3dshapes dataset we examined how much information about the ground-truth latent factors were encoded in the common latents  $z_{c}$  and the unique latents  $z_{u}$  (Table 1).
Given the encoded representation specified by  $q_{\phi}(z|x)$ , we evaluated the usable information for the two latent components  $(z_{c}$  and  $z_{u}$ ), as well as by using the complete latent variable  $z$ . As done in previous work [18], we directly use the mean of  $q_{\phi}(z|x)$  as our representation  $z$  rather than sampling. In Table 1, we see that the common and unique information was perfectly separated. Note, that information values reported are a lower bound to the true information, as our variational approximation is a lower bound to  $I_{q}(z;x)$  (which is itself a variational approximation). Our method accurately encodes all common information between views (ground truth: 3.32 bits for floor, wall, and background hue; 3 bits for scale; 2 bits for shape; 3.91 bits for orientation).
We also performed these analyses on the Common-dsprites dataset and found similar results (Table 2, Appendix). In particular, the unique latent factors corresponding to position are encoded in the unique components of the latent representation, while the other factors are encoded in the common latent representation. We emphasize that the generative model was not used at all during training, and was only used for quantitative evaluation after training.
In Fig. 2, we show the DCI matrix [35] which visually reaffirm that the common and unique factors are properly identified at the block-level for both the Common-3dshapes and Common-dsprites datasets. We also include traversals of the prior shown in Fig. 2 to show qualitatively that the learned factors of variation are meaningful and can be interpreted. Additional runs are in Appendix F.
Rotated Mnist and Comparison with Contrastive Learning. As described before, we generate a dataset of paired views of digits of the same class, each rotated by an independent random amount. In this manner, the unique information is about the rotation, whereas the common information is about the class. In Fig. 3 we see that the unique components of the latent (rows 1, 2) appear to encode the
Figure 4:Sprites [34] video experiment. (Left) Example views separated 2 frames apart. (Right) Common information as a function of delay between frames. In general common information is decreasing as the delay gets longer.
rotation and "thickness" of the digit, whereas the common components seem to represent the class of the digit (rows 3-6). Also, as expected, the output of view 1 does not depend on the latents in rows 7, 8 which by construction correspond to the unique components of view 2.
This setup is reminiscent of contrastive learning, where the goal is to learn a representation which is invariant to a random data augmentation of the input (such as a random rotation). By construction, contrastive learning aims to encode the common information before and after data augmentation, but may not encode any other information. This can lead to degraded performance on downstream tasks, as the discarded unique information may still be important for the task [29, 37]. On the other hand, our GK-VAE separates the unique and common information without discarding information.
To highlight this difference between approaches, we trained using a contrastive objective [7], and found that indeed while we can decode the shared class label, we cannot decode the unique rotation angle of view 1 (discretized into 10 bins; Fig. 3, right). On the other hand, using our method we recover the common and unique information.
Common information across time in sequences from videos. The existence of common information though time is another important learning signal. To study it, we perform an experiment on theSprites dataset described in [34]. This dataset consists of synthetic sequences all with 8 frames. We optimized using the same architecture and hyperparameters except we set  $\lambda_{c} = 0.5$ . We examine the common information between frames  $t$  frames apart, approximated using the KL divergence term. In particular, the two views are two frames  $(x_{1}, x_{t+1})$ , where each pair belongs to a different video sequences. In Fig. 4 we see that in general, as  $t$  increases the common information between the frames decreases evidencing the fact that, due to the random temporal evolution of the video, common information is lost as time progresses. We also note that the common information appears to increase in the last frame; this could be that in many of the sequences the sprite returns close to the initial state (see Fig. 3 in [34]).
We show formally and empirically that we can partition the latent representation of multi-view data into a common and unique component, and also provide a tractable approximation for the Gács-Körner common information between high dimensional random variables, which has been a difficult problem [4]. In many practical scenarios where high dimensional data comes from multiple sensors, such as neuroscience and robotics, it is desirable to understand and quantify what is common and what is unique between the observations. Motivated by the definition of common information proposed by Gács and Körner [2], we propose a variational relaxation and show that it can be efficiently learned from data by training a slightly modified VAE. Empirically, we demonstrate that our formulation allows us to learn semantically meaningful common and unique factors of variation. Our formulation is also a generative multi-view model that allows sampling and manipulation of the common and unique factors.
We used the code from: https://github.com/HobbitLong/CMC (BSD 2-Clause License)
As the common information was motivated by an information theoretic coding problem [2], our work naturally relates to compression schemes. Indeed, approximate forms of the common information, discussed further in Appendix C, are scenarios for distributed compression, since the common information needs to only be transmitted once [38, 4]. It may be interesting to combine our approach with recent advances in practical compression algorithms that leverage VAEs [39].
MK was supported by the National Sciences and Engineering Research Council (NSERC) and a fellowship from the Science Hub for Humanity and Artificial Intelligence (UCLA/Amazon). JCK acknowledges financial support from NSF CAREER 1943467. SS acknowledges financial support from ONR N00014-22-1-2252.
[1] Linda Smith and Michael Gasser. The development of embodied cognition: Six lessons from babies. Artificial life, 11(1-2):13-29, 2005. 1
[2] Peter Gács and János Körner. Common information is far less than mutual information. Problems of Control and Information Theory, 2(2):149-162, 1973. 1, 2, 3, 9, 10
[3] Stefan Wolf and J Wultschleger. Zero-error information and applications in cryptography. In Information Theory Workshop, pages 1-6. IEEE, 2004. 2, 3, 6, 15
[4] S. Salamatian, A. Cohen, and M. Médard. Approximate gács-körner common information. In 2020 IEEE International Symposium on Information Theory (ISIT), pages 2234–2239, 2020. 2, 9, 10, 16
[5] Loic Matthew, Irina Higgins, Demis Hassabis, and Alexander Lerchner. dsprites: Disentangle-ment testing sprites dataset. https://github.com/deepmind/dSprites-dataset/, 2017. 2, 7
[6] Chris Burgess and Hyunjik Kim. 3d shapes dataset. https://github.com/deepmind/3dshapes-dataset/, 2018. 2, 7
[7] Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. In European conference on computer vision, pages 776-794. Springer, 2020. 2, 4, 8, 9
[8] Richard Held and Alan Hein. Movement-produced stimulation in the development of visually guided behavior. Journal of comparative and physiological psychology, 56(5):872, 1963. 2
[9] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013. 3, 5
[10] Naftali Tishby, Fernando C. Pereira, and William Bialek. The information bottleneck method. In Proc. of the 37-th Annual Allerton Conference on Communication, Control and Computing, pages 368-377, 1999. 3
[11] Alessandro Achille and Stefano Soatto. Emergence of invariance and disentanglement in deep representations. Journal of Machine Learning Research, 19(50):1-34, 2018. 3, 14
[12] Alexander Alemi, Ben Poole, Ian Fischer, Joshua Dillon, Rif A Saurous, and Kevin Murphy. Fixing a broken elbo. In International Conference on Machine Learning, pages 159-168. PMLR, 2018. 3, 14
[13] Alessandro Achille and Stefano Soatto. Information dropout: Learning optimal representations through noisy computation. IEEE transactions on pattern analysis and machine intelligence, 40(12):2897-2905, 2018. 3
[14] Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with a constrained variational framework. International Conference on Learning Representations, 2017. 3, 14
[47] Matthew J Vowels, Necati Cihan Camgoz, and Richard Bowden. Nestedvae: Isolating common factors via weak supervision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9202-9212, 2020. 17
[48] Eduardo Hugo Sanchez, Mathieu Serrurier, and Mathias Ortner. Learning disentangled representations via mutual information estimation. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, Computer Vision – ECCV 2020, pages 205–221, Cham, 2020. Springer International Publishing. 17
Theorem A.1 (GK VAE recovers the common information). Suppose our observations  $(x_{1}, x_{2})$  have GK common information defined through the random variable  $z_{c}$  satisfying eq. 4-5 and that our parametric function classes  $q(z|x)$  and  $p(x|z)$  optimized over can express any function. Then, our optimization (with  $\beta_{c} = 0$ ,  $\beta_{u} < 1$  and decoder  $p(x|z) = \sum_{i=1}^{2} p_{i}(x_{i}|z_{i})$ ) will recover latents  $\hat{z} = (\hat{z}_{u}^{1}, \hat{z}_{u}^{2}, \hat{z}_{c})$  where  $\hat{z}_{c}$  is the common random variable that maximizes the "stochastic" GK common information in eq. 4-5, while  $\hat{z}_{u}^{i}$  is the unique information of the  $i$ -th view, which maximizes  $I(x_{i}; z_{u}^{i}, \hat{z}_{c})$ .
Proof. We consider the hard-constrained optimization problem with an infinitely expressive function class (i.e. so that the cross-entropy loss corresponds to the conditional entropy). Our VAE objective corresponds to the optimization problem
$$
\min  _ {z _ {c}, z _ {u} ^ {1}, z _ {u} ^ {2}} H (x _ {1} | z _ {u} ^ {1}, z _ {c}) + \beta_ {u} I (z _ {u} ^ {1}; x) + \beta_ {c} I (z _ {c}; x) +
$$
$$
H \left(x _ {2} \mid z _ {u} ^ {2}, z _ {c}\right) + \beta_ {u} I \left(z _ {u} ^ {2}; x\right) + \beta_ {c} I \left(z _ {c}; x\right)
$$
s.t.  $D(q_{\phi_1},q_{\phi_2}) = 0$
We consider a sequential optimization of finding  $\hat{z}_c$  and  $\hat{z}_u^i$ , and then show that this solution minimizes the joint objective above. We first consider the hard-constrained version of eq. (12).
$$
\min  _ {z _ {c}} H (x _ {1} | z _ {c}) + \beta_ {c} I (z _ {c}; x _ {1}) +
$$
$$
H \left(x _ {2} \mid z _ {c}\right) + \beta_ {c} I \left(z _ {c}; x _ {2}\right)
$$
Note that  $H(x_{i}) = H(x_{i}|z_{c}) + I(z_{c};x_{i})$ . We can rewrite the loss as:
$$
\begin{array}{l} \mathcal {L} = H \left(x _ {1} \mid \hat {z} _ {c}\right) + H \left(x _ {2} \mid \hat {z} _ {c}\right) + \beta_ {c} \left(I \left(\hat {z} _ {c}, x _ {1}\right) + I \left(\hat {z} _ {c}, x _ {2}\right)\right) \\ = H \left(x _ {1}\right) + H \left(x _ {2}\right) + \left(\beta_ {c} - 1\right) \left(I \left(\hat {z} _ {c}, x _ {1}\right) + I \left(\hat {z} _ {c}, x _ {2}\right)\right) \\ \end{array}
$$
This tells us that the optimal  $z_{c}$  maximizes  $I(\hat{z}_{c}, x_{1}) + I(\hat{z}_{c}, x_{2})$ . This is exactly the definition that we give of "stochastic" GK common information. Note we have previously shown  $I(z_{c}; x_{1}) = I(z_{c}; x_{2})$ . Given  $\hat{z}_{c}$  found above, the remaining objective (eq. (13)) becomes:
$$
\min _ {z _ {u} ^ {1}, z _ {u} ^ {2}} H (x _ {1} | z _ {u} ^ {1}, \hat {z} _ {c}) + \beta_ {u} I (z _ {u} ^ {1}; x _ {1}) +
$$
$$
H \left(x _ {2} \mid z _ {u} ^ {2}, \hat {z} _ {c}\right) + \beta_ {u} I \left(z _ {u} ^ {2}; x _ {2}\right).
$$
For  $\beta_{u} < 1$ , the objective maximizes  $I(x_{1};z_{u}^{1},\hat{z}_{c})$ , which was the definition of the unique information. (For  $\beta_{u} > 1$ , this corresponds to a  $\beta$ -VAE, and will have the corresponding trade-off between rate and reconstruction [14, 11, 12].)
Finally, suppose  $\tilde{z}_c$  did not contain all the common information as  $\hat{z}_c$ ; i.e.  $I(\tilde{z}_c; x_i) < I(\hat{z}_c; x_i)$ . Write the final equation as a maximization by noting that
$$
H \left(x _ {i} \mid z _ {u} ^ {i}, \hat {z} _ {c}\right) = - I \left(x _ {i}; z _ {u} ^ {i}, \hat {z} _ {c}\right) + H \left(x _ {i}\right)
$$
Then the final optimization (for any  $i$ ) is equivalent to
$$
\begin{array}{l} \max  _ {z _ {u} ^ {i}} I \left(x _ {i}; z _ {u} ^ {i}, \hat {z} _ {c}\right) - \beta_ {u} I \left(z _ {u} ^ {i}; x _ {i}\right) - H \left(x _ {i}\right) = \max  _ {z _ {u} ^ {i}} I \left(x _ {i}; \hat {z} _ {c}\right) + I \left(x _ {i}; z _ {u} ^ {i} \mid \hat {z} _ {c}\right) - \beta_ {u} I \left(z _ {u} ^ {i}; x _ {i}\right) - H \left(x _ {i}\right) (16) \\ > \max  _ {z _ {u} ^ {i}} I \left(x _ {i}; \tilde {z} _ {c}\right) + I \left(x _ {i}; z _ {u} ^ {i} \mid \tilde {z} _ {c}\right) - \beta_ {u} I \left(z _ {u} ^ {i}; x _ {i}\right) - H \left(x _ {i}\right) (17) \\ = \max  _ {z _ {u} ^ {i}} I \left(x _ {i}; z _ {u} ^ {i}, \tilde {z} _ {c}\right) - \beta_ {u} I \left(z _ {u} ^ {i}; x _ {i}\right) - H \left(x _ {i}\right) (18) \\ \end{array}
$$
For any  $z_{u}^{i}$ , Eq. 16 is maximized with  $\hat{z}_{c}$  that encodes all the common information. Thus the GK VAE optimization is minimized with  $\hat{z} = (\hat{z}_{u}^{1},\hat{z}_{u}^{2},\hat{z}_{c})$ .
Proposition A.1. ([3], Ex. 1): Define
where  $z_{c}, z_{u}^{1}$ , and  $z_{u}^{2}$  are mutually independent. Then for any invertible transformation  $t_{i}$  the random variable  $z^{*}$  that satisfies
$$
\arg \max  _ {\hat {z}} C _ {G K} \left(t _ {1} \left(z _ {1}\right), t _ {2} \left(z _ {2}\right)\right)
$$
is  $z_{c}$
Proof. Note that if  $t$  is the identity transformation  $t(z) = z$ , then
$$
\arg \max  _ {\hat {z}} C _ {G K} (z _ {1}, z _ {2})
$$
is  $z_{c}$ . In general, if  $t_{i}$  are invertible transformations, suppose that  $f_{1}$  and  $f_{2}$  are the functions satisfying  $\hat{z} = f_{1}(z_{1}) = f_{2}(z_{2})$  corresponding to  $C_{GK}(z_1,z_2)$ . Then the functions corresponding to  $C_{GK}(t_1(z_1),t_2(z_2))$  will be  $\hat{z} = f_{1}\circ t_{1}^{-1}(t_{1}(z_{1})) = f_{2}\circ t_{2}^{-1}(t_{2}(z_{2}))$  and the random variable  $\hat{z}$  is equivalent.
We trained networks with Adam with a learning rate of 0.001, unless otherwise stated. When the number of ground truth latent factors is known, we set the number of latents equal to the number of ground truth factors. To improve optimization, we use the idea of free bits [33] and we set  $\lambda_{free-bits} = 0.1$ . This was easier than using  $\beta$  scheduling, since it only involved one parameter  $\lambda_{free-bits}$ . We set  $\beta_u$  to be 10 and  $\beta_c$  to be 0.1. We trained networks for 70 epochs, except for the Mnist experiments, where we trained for 50 epochs. We used a batch size of 128 and we set  $\lambda_c = 0.1$ . For all our experiments we used the same encoders and decoders as [16], which has been also used in recent work [18]. We used Gaussian encoders and Bernoulli decoders, basing our implementation off [40]. Our architecture is schematized in Fig. 1. Note that we optimized encoders and decoders separately between the views (i.e. weights were not shared).
In our implementation, unless otherwise noted, we used two separate view-dependent decoders that each processed all the latents  $z$  to reconstruct the observations. We also verified that view specific decoders that processed only view-specific latents  $z_{i}$ , as in Thm. 3.1, also led to a partitioning of the common and unique information Fig. 5. This was the case both when the unique factors were pairwise independent, as in generative model described in Sect. 4.1 used for the majority of our experiments, and also if there were correlations between the unique components (Fig. 7) and (Fig. 6), with the experiments described in Sect. F. Empirically, we found the separation of the common and unique information was robust to the partitioning of the latents fed to the decoder of our GK-VAE. We provide both decoder implementations in our repository available at https://github.com/mjkleinman/common-vae.