Gacs-Korner Common Information Variational Autoencoder
Authors: Michael Kleinman
Alessandro Achille
Stefano Soatto
Jonathan Kao
Venue: NeurIPS 2023 poster (2023)
Keywords: Common Information
Gacs-Korner
Variational Autoencoder

Abstract: We propose a notion of common information that allows one to quantify and separate the information that is shared between two random variables from the information that is unique to each. Our notion of common information is defined by an optimization problem over a family of functions and recovers the G\'acs-K\"orner common information as a special case. Importantly, our notion can be approximated empirically using samples from the underlying data distribution. We then provide a method to partition and quantify the common and unique information using a simple modification of a traditional variational auto-encoder. Empirically, we demonstrate that our formulation allows us to learn semantically meaningful common and unique factors of variation even on high-dimensional data such as images and videos. Moreover, on datasets where ground-truth latent factors are known, we show that we can accurately quantify the common information between the random variables.

Gács-Körner Common Information Variational Autoencoder
Michael Kleinman' Alessandro Achille2.3+
Stefano Soatto Jonathan C. Kao! AWS Al Labs *Caltech
University of Californa,, Los Angeles michael w.chhee..keiimaan www.iimmnuuaaaaccmmmmmm... edu soattolcs ucla. edu
aachille@amazon. com
kaoλseas, www.uclalee... edu
Abstract
We propose a notion of common information that allows one quantify and separate the information that is shared between two random variables from the information that is unique each. Our notion of common information is defined by an optimization problem over a family of functions and recovers the Gács- Körner common information as a special case. Importantly, our notion can be approximated empirically using samples from the underlying data distribution. We then provide a method to partition and quantify the common and unique information using simple modification of traditional variational auto-encoder. Empirically, we demonstrate that our formulation allows us to learn semantically meaningful common and unique factors of variation even on high-dimensional data such as images and videos. Moreover, on datasets where ground ground-truth latent factors are known, we show that can accurately quantify the common information between the random variables.
Introduction
Data coming from different sensors often capture information related common latent factors. For example, many animals have two eyes that capture different but highly-correlated views of the same objects in the scene. Similarly, sensors of different modalities, such eyes and ears, capture correlated information about the underlying scene, as do videos and other time series, where the sensors are separated in time rather than modality. Learning how information of one sensor maps to information of another provides self-supervised signal to disentangle the variability that is intrinsic in sensor from the latent causes (e-g., objects) that are shared between multiple sensors. Indeed, there is evidence that infants spend long time during development purposefully experiencing objects through different senses at the same time [1].
Motivated by this, we propose to learn meaningful representations of multi-view data by quantifying and exploiting such structure in a self-supervised fashion by using an information theoretic notion of common information as the guiding signal to disentangle common shared information present in high dimensional sensory data (Fig. 1).
However, defining notion of common information is itself not trivial. The most natural and typical way to quantify the "common part" between random variables would be by quantifying their mutual information. But mutual information has no clear interpretation in terms of decomposition of random variables in unique and common components. In particular, [2] note that there is generally no way to write two variables and Y using a three part code (A, B,C) such that X = f(A, C),, (B, C') and where C encodes all and only the mutual information 1 (X; YY. Discovering the
* "Work performed as external collaboration not related to Amazon "Code available at: https:// thub. wwlinmann/mmmmmmmmmmmmmmmmmmmmam common-vas vae
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

X
2i
2
12
X2
22
Figure 1: High level schematic of approach. Red denotes shared latent factors (size, shape, floor, background and object color) and black denotes unique latent (viewpoint). The aim is to extract Fe. which is random variable that is function of both inputs 24. We also allow for unique latent variables &u to capture information that unique each view. The latent representations are used to reconstruct the inputs.
largest common factor C, which encodes what is known as the Gåcs-Körner common information, from high dimensional data is then a distinct problem on its own [3, 4].
To the best of our knowledge, there are currently approaches compute approximate the Bács-Körner common information from high sigh-dimensional samples [4]. this work, we seek to learn common representations that satisfy the constraint that they (approximately) function of each input. A contribution in this paper that we generalize the constraint that the representation needs to be a deterministic function and allow it to be stochastic map. As later show, this is helpful for quantifying and interpreting the latent representation, and allows us to parameterize the optimization with deep neural networks.
We show that our objective can optimized using multi-view Variational Auto-Encoder (VAE). Since in general each view contain individual factors of variation that not shared between the views, we augment our model with a set of unique latent variables that can capture unexplained latent factors of variation, and show that the common and unique component can be efficiently inferred from data through standard training. While training the multi-view VAE, we simultaneously develop describe in Section
scalable approximation for the Gács-Körna common information,
To empirically evaluate the ability to separate the common and unique latent factors we introduce two new datasets, which extend commonly used datasets for evaluating disentangled representations learning: dSprites [5] and 3dShapes [6]. For each dataset, we generate a set of paired views (æ1 X22 such that they share set of common factors. This allows to quantitatively evaluate the ability to separate the common and unique factors. We also compare our method to multi-vion viet contrastive learning [7] and show that thanks to definition we avoid learning degenerate representations when the views share little information.
We hypothesize that a key reason precluding the identification of latent generating factors from observed data that receiving a single sample of scene is quite limiting. Indeed, classical neuroscience experiments has shown that the ability to interact with an environment, as opposed to passively observing sensory inputs, is critical for learning meaningful representations of environment [8].
2 Preliminaries and Related Work
The entropy (x)) of random variable x is Ep(a) [log |. The mutual information [2x22 (=)) (z|zx) Another useful identity for mutual information that use is (x;;2) Ep(z) [K](p(]|=))l((]) where KL denotes the Kullback-Leibler divergence.
Gács-Körner Common Information.
The Gáes-Körner common information [2] defined as
CGk (ox(x1)))) max (2)) s.t. f(11) g(x2),
where f and y are deterministic functions. The Gács-Körner common information thus defined through random variable that is deterministic function of both inputs ø and z2. Among all

---

such random variables, is the random variable with maximum entropy. This has also been referred to as the "zero error information" in upplications to cryptography [3]. It is an attempt to formalize and operationalize the idea of the common part between sources, which mutual information lacks. It also a lower bound the mutual information [2,3), 3). To the best of our knowledge, there are no efficient techniques for computing the GK common information for high-dimensional æ1; .2- We elaborate on the difference between GK common information and mutual information in App. E.
Variational Autoencoders Variational Autoencdoers (VAEs) [9] are latent variable generative models that are trained to maximize the likelihood of the data by maximizing the evidence lower bound. or minimizing the loss:
vAV = Ep(x)] (=)t)= (ogooo + (= II [[z)]]]
The VAE loss can be motivated in an information theoretic manner optimizing an Information Bottleneck [10], where the reconstruction term encourages a sufficiens representation and the KL regularization term encourages a minimal representation [11, 12]. The addition of a parameter / to modify how the KL regularization is penalized leads to the following loss (and corresponds to the
traditional VAE loss when 3 = 1):
CR-
Ep(= Es (=|m)\ log (=] + [KL(())) ll p(z))]
(3)
For larger values of 33 the representations become more disentangled, shown analytically in [13] and empirically in [14], although reconstructions become worse.
Disentangled representations A guiding assumption for representation learning that the observed data (i.e an image) can be generated from a (simpler) set of latent generating factors z. Assuming the latent factors are independent, the idea of learning disentangled representations involves learning these latent factors of variation in an unsupervised manner [15]. However, despite apparent empirical progress in learning disentangled representations [14, 16, 17], there remains inherent issues in both learning and defining disentangled representations [18]. In many cases, different independent latent factors may lead to equivalent observed data, and without an inductive bias, disentanglement remains ill-defined. For example, color can be decomposed into an RGB decomposition, or equivalent HSV decomposition.
In [18. Theorem 1] it is shown that without any inductive bias, one cannot uniquely identify the un- derlying independent latent factors in a purely unsupervised manner from observed data. Empirically, they also found that there was no clear correlation between training statistics and scores without supervision. Later, and related to our work, the authors examined the setting where there is paired data and no explicit supervision (weak supervision), and found that such setup was helpful for learning disentangled representations [19]. The authors examined the setting in which the set of shared latent factors changed for each example, which was necessary for their identifiability proof. This also required using the same encoder for each view, and thus is a restricted setting that does not easily scale to multi-modal data.
Here, we study the scenario where the set of generating factors is the same across examples, as in the case of a pair of fixed sensors receiving correlated data. Additionally, our objective is motivated in an information theoretic way and our method generalizes to the case where we have different sensory modalities, which is relevant to neuroscience and multi-modal learning. Finally, our variational objective is flexible and allows estimation of the common information in a principled way.
Approximating Mutual Information Estimating mutual information from samples is challenging for high-dimensional random vectors [20]. The primary difficulty in estimating mutual information is constructing nigh-dimensional probability distribution from samples, as the number of samples required scales exponentially with dimensionality. This is impractical for realistic deep learning tasks where the representations are high dimensional. To estimate mutual information, [21] used a binning approach, discretizing the activations into a finite number of bins. While this approximation is exact in the limit of infinitesimally small bins, in practice, the size of the bin affects the estimator [22, 23]. In contrast to binning, other approaches to estimate mutual information include entropic- based estimators (e.g., [23]) and a nearest neighbours approach [24]. Although mutual information is difficult to estimate, is an appealing quantity to summarily characterize neural network behavior because of its invariance to smooth and invertible transformations. In this work, rather than estimate the mutual information directly, we study the "usable information" in the network [25, 26), which corresponds to a variational approximation of the mutual information [27, 28].
3

---

Contrastive and Multi- View Approaches While (multi-view) contrastive learning aims to learn a representation of only the common information between views [7. 29, 30], we aim to learn decomposition of the information in the views into common and unique components. Our work naturally extends to multi-sensor data that have different amounts of common/unique information (e.g., touch and vision). Moreover, contrastive approaches assume that the unique information is nuisance variability, and discard this information. Similarly, [31] also seeks to identify common information in both views, but also does not provide an objective to retain the unique information. While the multi-view literature is broad, we are not aware of previous attempts to quantify the common and unique information. Related representations using VAEs, but differs in how the alignment of shared information is specified and the resulting objective, and they do not provide way quantify the information content of the private and shared components. We discuss additional related work in Appendix C.
approach, [32] aim to find shared and private
Method: Gács-Körner Variational Auto-Encoder
Our formulation involves generalizing the KirseKKönner common information in eq. (1) to the case where f and g are stochastic functions so that the optimization problem becomes:
ČGK(F1; max (x
(4)
2 s.t. 2 = f (x2),
(5)
where fs and ga are stochastic functions. By the equality in eq. (5), we mean that p(z (2))))) ppaz))pppalaa (x2) for all (2102)) p(x1, 22). Note that when f and g are deterministic functions (which a subset of stochastic functions), then (2( = and we recover the original definition since
I I(x;; 2) ((2) H=ll;;) H(z).
(6)
Our latter generalization (eq. 4-5) is more amenable optimization and interpretable, will later demonstrate. In eq. (4), we used x; as a placeholder since when p(z(z1) = p(# (2) for all
(x1, ~ T2, then [[2; m1) = [[2; 22) since
1(2;21) Emr~n(a) LLL(
[(m
K [<<p(e|<))]l](]]]
Engen(az) [L(p(2)
This means that another equivalent formulation to maximize is max, 2 E; [xx;; 2) = max, [[x; 2). for any i. To optimize the objective in eq. 4-5, we need to learn set of latent factors z that maximize I 12,22) while satisying the constraint in eq. 5. We propose an optimization reminiscent of the VAE objective. Define x (I1, as the concatenation of both views, and (2 Zey = as decomposition of the representation into common and unique components, and = (2 2c). In particular, we seek learn latent encodings through an encoder qs (z|z), which maps x to 2. To optimize the objective, the representation should maximize 7 [ =), and so we should also learn decoder po (z|z) that minimizes H (ara 1²). This corresponds to the reconstruction term in a traditional VAF, though note here we reconstruct both views.
L F p(=)) ( (=))) log (x][=)]
Without any constraints, this could be achieved trivially by using an identity mapping. To ensure that the latents encode only common information between the different views, we decompose the encodings to ensure the following constraint corresponding to eq. (5):
D(yoe Udez) = K L(ge,, (ze|=1) || Dtte (ze(x2)) = 0.
(8)
Here qo., (zc(=% maps x; to 2e, Rather than enforeing hard constraint, in practice is easier to optimize the corresponding Lagrangian relaxation:
Co = Ep(=)] (g .)) log po DeD - A0nz
'If the constraint in Eq. 5 is satistied, the map to the mean of the posterior p(z(æ1) (and p(2[za)) for observations (x1, 222 is deterministic function and satisfies the constraint of Gáes Körner common information.

---

After optimizing this objective, for a sufficiently large À so that D(q 92) = 0, the common information would be:
(ax(21;22) = Ep(z)l K (z)) l1 (=))
(10)
where (=)) ) is the marginal distribution induced by the encoder. However, estimating the true marginal q" (2) is difficult for high-dimensional problems. In practice, we follow [9] and learn an approximate prior p(z) = (z), where both 4s (z(zz) and p(z) are taken from a given family of distributions (such as multivariate Gaussians with diagonal covariance matrix). This will additionally enable us sample from the distribution, and interpret the latent factors. To learn p(z) we also add
following regularization to our training objective:
Ep(x) K (gg((())) p(=))
(11)
Alternatively, we can also exploit the degree of freedom in learning 90 and fix p(z) to be N(0, l). In both cases, our overall objective becomes:
LeVAE = Ep(z) [= Po
XeDio,, ( 3KL( [l
(12)
Optimizing this objective alone could lead unexplained components of information, for example the unique components. Afternatively, unique information present in the individual views may be encoded in the "common" latent variable if the reconstruction benefits outweighed the cost of the divergence between the posteriors of the encoders (the term corresponding to the (3).
In addition to these common latent components, we can learn unique latent components by optimizing a traditional VAE objective (i.e. with A = 0) for subset of the latent variables. Importantly we also need to ensure that the KL penalty for the unique component subset is greater than for the common subset (so that it is beneficial to encode common information in the common latent components). Our final objective becomes
LoVAE Ep(x)] (gg()))) [x]1)( + (eDe, (dez)
B.KL (2cl04) p(zc)) + B.KILgg (2ul2z) ((pz)()))))))))))))))))) (13)
where 3e and Bu correspond to a multiplier enforcing the cost of encoding common and unique information respectively. Importantly B. > 3. > O, resulting in a larger penalty on the unique latent variables (otherwise all the information would be encoded in the "unique" components). The summation in the bottom part of Eq. 13 corresponds to summation over both encoders. p(2.) and p(z.) are both sampled from (O, /) of appropriate dimensionality.
We now show that, the network architecture used for the VAE implements a generic enough class of encoder/decoders our method will recover the GK common information.
Theorem 3.1 (GK VAE recovers the common information). Suppose our observations (x2) have GK common information defined through the random variable 2e satisfying eq. 4-5 and that our parametric function classes q( and p(z[z) optimized over can express any function. Then, our optimization (with S. 0, Bu < 1 and decoder p(z|z) Σ px(z;/2;)) will recover latents i = (El, $2. 22.2.) 2e) where é. is the common random variable that maximizes the "stochastic" GK common information in eq. 4-5, while şt is the unique information of the i-th view, which maximizes T(m;; = Ez).
"u
We provide the proof in Appendix A. Note that while the previous theorem guarantees that will be able to separate the common and unique factors at the block level, we might not be able to disentangle the individual common factors.
3.1 Quantifying the common information
Suppose D(age, (ooz) = The term corresponding to the rate R. of the VAE
(14) is neither upper nor lower bound on the true common information. It represents upper bound the information encoded in the representation specified by go, go (=ela)) but does not bound the true common information in the data, since 4o. (z)(z) itself is variational approximation.
R. = T,(==; .r) Fp(x) K (L(gs, || p(==))
5

---

lalent
latent
factor
factor
Figure 2: Latent traversals and DCI plots show optimization results in separation common and unique information. (Left) 3dshapes: The top 3 rows shows the unique latents, the middle 3 the common (and the bottom? are the unique latents for the second view). The ground truth unique generative factors are (0, 2) corresponding correctly recovers that those factors are unique (first three rows in the figure), and that the other factors are common (middle three rows). (Right) dsprites: The top rows shows the unique latent variables, the middle the common (and the bottom 2 are the unique latent variables for the second view). The ground truth unique generative factors (3, 4) corresponding x-position, y-position respectively. Our model correctly recovers that those factors are unique (first two rows in the figure), and that the other factors common (middle three rows).
floor color, wall color, and object color. Our model
To find lower bound on the common information encoded in the dataset, we can use any mutual information estimator that is a lower bound (see [28] for several). The approximate common information can then be quantified by [(2y, x²), where 2g Qá. (Zelzz). We report both the rate R. and / in the paper. We emphasize that can be any mutual information estimator. When the data generating distribution is known, as in our synthetic examples, we employ the "Usable Information" estimator, described in Sect. 4.2, which is variational approximation [27].
3.2 Identifiability of the common and unique components
We now show the conditions under which our optimization will identify the common and unique latent components. Usually we not directly observe the latent factors 2, but rather an observation generated from them. We may then ask whether the common latent factors can still be reconstructed from this observation. The following proposition shows that this is indeed case, as long as function generating f the observation is invertible, i.e., can recover the latent factors from the observation itself.
Proposition 3.1. ([3), Ex. 1): Define
21 (Zey 2
22
(Zes 2
where ze, 2/. and 2 are mutually independent. Then for any invertible transformation t, the random variable ze encodes all the common information:
Ze arg max (cx(t
We provide the proof in Appendix A. The above proposition shows that when set of factors is shared between views and when the unique factors are sampled independently, then the GK common random variable corresponds shared latent factors. In particular, if the observations I; are generated through an invertible function 2; f(ze. 2 where 2e p(ze) corresponds to the shared factors, proposition shows that such factors can be recovered from the observations by maximizing the GK common information. In our GK VAE optimization, we optimize the "stochastic" GK common information and we also find our experiments that we can (approximately) recover the latent factors from observations I; generated from this process.

---

FLOOR HUE (10)
WALL HUE (10)
OBI. HUB (10)
SCALE (8)
SHAPE (4)
ANGLE (15)
KL TOTAL
COMMON -0.01
-0.02
-0.03
2.73
1.98
UNIQUE 3.31
3.31
3.31
0.19
0.37
0.19
12.7
TOTAL 3.31
3.31
3.31
2.69
1.98
3.82
27.7
Table 1: Usable Information (in bits) in representation for 3dShapes. The ground truth unique generative factors are floor color, wall color, and object color, and the common generative factors are scale, shupe and angle. The common information separated from the unique information. The ground truth factors were almost perfectly encoded in the latents. The numbers in parenthesis represents the number of discrete factors for each latent variable.
Experiments
We train EKEV models with Adam using learning rate of 0.001, unless otherwise stated. When the number of ground truth latent factors is known, we set the size of the latent vector of the VAE equal to the number of ground truth factors. This choice was not necessary, and obtain analogous results when the size of the latent vector of the VAE was larger than the number of ground-truth factors (Fig. 12). To improve optimization, we use the idea of free bits [33] and set Atres-bits = 0.1. This was easier than using 3 scheduling [16], since it only involved tuning one parameter. We set 8. be 10, Be to be 0.1 and A 0.1. We trained networks for 70 epochs, except for the MNIST experiments, where we trained for 50 (details in the Appendix).
To ensure that the latents are shared to both encoders, during training randomly sample from either encoder 96, (z)( with p 0.5. We opted to randomly sample the latents from each encoder, opposed to performing averaging, to ensure that the latent will always be a function of individual view Z₁. This is in addition to the soft constraint governed by A in the loss.*
4.1 Evaluation Datasets
We primarily focus the setting where the ground truth latent factors and generative model are known, in order to quantitatively benchmark our approach. To do so, we constructed datasets with ground truth latent factors so that some of the latent factors are shared between each views. That is, the generative model for the data (x1, 222) is
I = f( (c),
x2 =2=
zo);
(15)
where 2e shared between the views and 2 is the unique information encoded in the zth view and f corresponds to rendering function.
To construct such datasets, we modified the 3dshapes [6] and the dsprites dataset [5]. We select subset of the latent factors to be shared between the views, while the remaining factors are sampled independently for each view. The 3dshapes dataset [6] contains six independent generating factors: floor color, background color, shape color, size, shape, and viewpoint. Each latent factor can only take one of a discrete number of values. The dsprites dataset [5] contains five independent generating factors: shape, scale, rotation, and position. Each latent factor can only take one of discrete number of values. When we generate multi-view data following the generative model eq. (15), we refer to these datasets as Common-3dshapes and Common-dspriter respectively. We consider additional variants in the Appendix.
We also examine the Rotated Mnist Dataset. where the two views are two random digits of the same class which random rotation is applied. In particular, the class of the digit is common information between the views whereas the rotation is unique. We also examine the synthetic video dataset Sprites (not be confused with dsprites) described in [34] and evaluate the common information frames separated t frames apart.
4.2 Metrics
DCT Disentanglement Matrix [35]. Let d be the dimension of the latent space and let be the true generating factors. The idea is to train regressor fz(2): Rd -> R to predict the ground truth factors
*Our experiments can be reproduced in approximately 3 days on single GPU (g4dn instance).
7

---

8
8
8
8
8
8
8
8
Info. in latent (bits)
8
9
7
Contrastive
3.285
0.014
2
2
8
8
O
O
latent
6
6
8
5 8
GKVAE 3.248
0.798
8
8
8
8
8
8
label
rot.
8
8
8 8 8
8
8
8
8
factor
Figure 3: Left. Traversals for Rotated Mnist. The unique components of the latent (rows 1,2) appear to encode the "thickness" and rotation of the digit, whereas the common components appear to represent the overall digit (rows 3-6); and also the output of view does not depend the latents in rows 7,8 (these correspond to the unique components for view 2.) Middle. Corresponding DCI matrix, where factor 0 corresponds the lahel, while factor corresponds the rotation (discretized into 10 bins). Right. Comparison against contrastive implementation from [7], where the contrastive approach does not encode any usable information about the unique factor (the rotation).
t, for each j. This results in matrix of coefficients that describe the importance of each latent for predicting each ground truth factors. This can then visualized as a matrix where the size of the square reflects the coefficient. We use this metric assess the partitioning of the learned common and unique representations. We used the random forest regressor, similar to [18] 8] to predict a discrete number of latent classes
Usable Information [25, 36, 26). We this to approximate the mutual information when H (ax) known, as it is in the datasets previously described. It a lower bound to mutual information. We use this to lower bound the information contained the representation in the next section.
4.3 Results
Separation of Common and Unique Latent Variables. We first examine whether our formulation can correctly separate the common and unique latent factors. After optimizing network on our Nommon-3dshappe dataset we examined how much information about the ground-truth latent factors were encoded in the common latents and the unique latents (Table 1).
Given the encoded representation specified by qo(=(æ)), two latent components (z, and 2,), as well as by using the complete latent variable 2. As done in previous work [18], we directly use the mean of (=(=)xr) our representation = rather than sampling. In Table 1, we see that the common and unique information was perfectly separated. Note, that information values reported are lower bound the true information, as our variational approximation is lower bound to I, (2; æ) (which is itself a variational approximation). Our method accurately encodes all common information between views (ground truth: 3.32 bits for floor, wall, and background hue; 3 bits for scale; 2 bits for shape; 3.91 bits for orientation).
evaluated the usable information for the
We also performed these analyses the Common-dsprice dataset and found similar results (Table 2, Appendix). In particular, the unique latent factors corresponding to position are encoded in the unique components of the latent representation, while the other factors encoded in the common latent representation. We emphasize that the generative model was not used at all during training, and was only used for quantitative evaluation after training.
In Fig. 2, we show the DCI matrix [35] which visually reaffirm that the common and unique factors properly identified at the block-level for both the Sommon-3dshaper and Common- datasets. We also include traversals of the prior shown in Fig. 2 to show qualitatively that the learned factors of variation meaningful and can be interpreted. Additional runs are in Appendix F.
Rotated Mnist and Comparison with Contrastive Learning. As described before, we generate a dataset of paired views of digits of the same class, each rotated by independent random amount. In this manner, the unique information about the rotation, whereas the common information about the class. In Fig. we that the unique components of the latent (rows 1, 2) appear encode the

---

Cormon Information Rate (bits)
View 1
View 2
2
Delay beoween
Figure 4: Sprites [34] video experiment. (Left) Example views separated frames apart. (Right) Common information function of delay between frames. In general common information is decreasing as the delay gets longer.
rotation and "thickness" of the digit, whereas the common components seem to represent the class of the digit (rows 3-6). Also, as expected, the output of view does not depend on the latents in rows 8 which by construction correspond to the unique components of view 2.
This setup reminiscent of contrastive learning, where the goal is to learn a representation which is invariant to random data augmentation of the input (such as random rotation). By construction, contrastive learning aims to encode the common information before and after data augmentation, but may not encode any other information. This can lead to degraded performance on downstream tasks, as the discarded unique information may still important for the task [29, 37J. the other hand, our GK-VA -KK separates the unique and common information without discarding information.
To highlight this difference between approaches, we trained using a contrastive objective* [7]. and found that indeed while we can decode the shared class label, we cannot decode the unique rotation angle of view (discretized into 10 bins; Fig. 3. right). On the other hand, using our method recover the common and unique information.
Common information across time in sequences from videos. The existence of common information though time another important learning signal. To study it, we perform an experiment on the Sprites dataset described in [34]. This dataset consists of synthetic sequences all with 8 frames. We optimized using the same architecture and hyperparameters except we set A = 0.5. We examine the common information between frames /. frames apart, approximated using the KI divergence term. particular, the two views are two frames (#1, (r+1), where each pair belongs to a different video sequences. In Fig. 4 see that in general, as increases the common information between the frames decreases evidencing the fact that, due to the random temporal evolution of the vidco, common information lost time progresses. We also note that the common information appears to increase in the last frame; this could be that in many of the sequences the sprite returns close to the initial state (see Fig. in [34]).
Discussion
We show formally and empirically that can partition the latent representation of multi-view data into a common and unique component, and also provide a tractable approximation for the Gács- Körner common information between high dimensional random variables, which has been difficult problem [4]. In many practical scenarios where high dimensional data comes from multiple sensors, such as neuroscience and robotics, it is desirable to understand and quantify what common and what is unique between the observations. Motivated by the definition of common information proposed by Gáes and Körner [2], we propose a variational relaxation and show that it can be efficiently learned from data by training slighly modified VAE. Empirically. we demonstrate that our formulation allows us learn semantically meaningful common and unique factors of variation. Our formulation is also generative multi-view model that allows sampling and manipulation of the common and unique factors.
We used the code from: http::/ggthabbcom///oobbboooooog (BSD 2- 2-Clause License)

---

As the common information was motivated by an information theoretic coding problem [2], our work naturally relates compression schemes. Indeed, approximate forms of the common information, discussed further in Appendix C, are scenarios for distributed compression, since the common information needs only transmitted once [38, 4]. It may be interesting to combine our approach with recent advances in practical compression algorithms that leverage VAEs [39].
Acknowledgments
MK was supported by the National Sciences and Engineering Research Council (NSERC) and a fellowship from the Science Hub for Humanity and Artificial Intelligence (UCLA/Amazon). JCK acknowledges financial support from NSF CAREER 1943467. SS acknowledges financial support from ONR N00014-22-1-2252.
References
[1] Linda Smith and Michael Gasser. The development of embodied cognition: Six lessons from babies. Artificial life, 11(1-2):13-2) 2005.
[2] Peter Gács and János Körner. Common information far less than mutual information. Problems of Control and Information Theory, 2(2):149-162, 1973. 1, 2, 3, 9, 10
[3] Stefan Wolf and J Wultschleger. Zero-error information and applications in cryptography. In Information Theory Workshop, pages 1-6. IEEE, 2004. 2. 3, 6, 15
[4] S. Salamatian, A. Cohen, and M. Médard. Approximate gács-körner common information. In 2020 IEEE International Symposium on Information Theory (ISIT), pages 2234-2239, 2020. 10, 16
[5] Loic Matthey, Irina Higgins, Demis Hassabis, and Alexander Lerchner. dsprites: Disentangle- 2017 2,7
ment testing sprites dataset.
[6] Chris Burgess and Hyunjik Kim. 3d shapes dataset. https//ggthbbbooommepppm///////s dataset/, 2018. 2.
[7] Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. European conference on computer vision, pages 776-794. Springer, 2020. 2, 4, 8,9
[8] Richard Held and Alan Hein. wevemmnnt-roduueed stimulation in the development of visually guided behavior. Journal of comparative and physiological psychology. 56(5):872, 1963. 2
[9] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint 131iv:1312.61 14, 2013. 3,5
[10] Naftali Tishby, Fernando C. Pereira, and William Bialek. The information bottleneck method. In Proc. of the 37-th Annual Allerton Conference on Communication, Control and Computing, pages 368-37 377, 1999.
[11] Alessandro Achille and Stefano Soatto. Emergence of invariance and disentanglement in deep representations. Journal of Machine Learning Research, 19(50)1-34, 2018. 3, 14
[12] Alexander Alemi, Ben Poole, Ian Fischer, Joshua Dillon, Rif A Saurous, and Kevin Murphy. Fixing a broken elbo. In International Conference on Machine Learning, pages 159-168. PMLR, 2018. 3, 14
[13] Alessandro Achille and Stefano Soatto. Information dropout: Learning optimal representations through noisy computation. IEEE transactions on pattern analysis and machine intelligence, 40(12):2877-2055 2018. 3
[14] Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with constrained variational framework. International Conference on Learning Representations,
2017. 3, 14
10

---

[15] Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence, 35(8):1798-
1828, 2013. 3
[16] Christopher Burgess, Irina Higgins, Arka Pal, Loic Matthey, Nick Watters, Guillaume Des- jardins, and Alexander Lerehner. Understanding disentangling in f-vac. arXiv preprint 180. 2018. 3, 15
[17] Ricky Q. Chen, Xuechen Li, Roger Grosse, and David Duvenaud. Isolating sources of disentanglement in variational autoencoders. Advances in Neural Information Processing Systems, 2018.
[18] Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Raetsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentangled representations. In international conference machine learning, pages 41 14-
4124. PMLR, 2019. 3, 8, 15, 16
[19] Francesco Locatello. Ben Poole, Gunnar Rätsch, Bernhard Schölkopf, Olivier Bachem, and Michael Tschannen. Weakly "eakkl-suupervised disentanglement without compromises. In International Conference on Machine Learning, pages 6348 6348-6359. PMLR, 2020. 3
[20] Liam Paninski. Estimation of entropy and mutual information.
Neural Comput.,
156):119 June 2003. 3
[21] Ravid Shwartz-Ziv and Naftali Tishby. Opening the black box of deep neural networks via information. CoRR, 0Ds/1703.00810 2017. 3
[22] Andrew Michael Saxe, Yamini Bansal, Joel Dapello, Madhu Advani, Artemy Kolchinsky, Brendan Daniel Tracey, and David Daniel Cox. On the information bottleneck theory of deep learning. In International Conference on Learning Representations, 2018.
[23] Ziv Goldfeld, Ewout Van Den Berg, Kristjan Greenewald, Igor Melnyk, Nam Nguyen, Brian Kingsbury, and Yury Polyanskiy. Estimating information flow deep neural networks. In Pro- ceedings the 36th International Conference Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 2299-2300 PMLR, 09-15 Jun 2019.
[24] Alexander Kraskov, Harald Stögbauer, and Peter Grassberger. Estimating mutual information. Physical review E, 69666)066138, 2004. 3
[25] Yilun Xu, Shengjia Zhao, Jiaming Song, Russell Stewart, and Stefano Ermon. theory of usable information under computational constraints. In 8th International Conference Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
3, 8, 15
[26] Michael Kleinman, Alessandro Achille, Daksh Idnani, and Jonathan Kao. Usable information and evolution of optimal representations during training. In International Conference on Learning Representations, 2021. 3, 8, 15
[27] David Barber and Felix Agakov. maximization. In Proceedings of the 16th International Conference on Neural Information Processing Systems, NIPS'03, page 201-208, Cambridge, MA, USA, 2003. MIT Press. 3, 6
The im algorithm: A variational approach to information
[28) Ben Poole, Sherjil Ozair, Aaron Van Den Oord, Alex Alemi, and George Tucker. On variational bounds of mutual information. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Pre- ceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 5171-5180, Long Beach, California, USA, 09-15 Jun 2019. PMLR. 3, 6
[29] Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning? Advances Neural Information Processing 33:6ems, 33:6827-6839, 2020. 4,

---

[30] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597-1607. PMLR, 2020. 4
[31] Marco Federici, Anjan Dutta, Patrick Forré, Nate Kushman, and Zeynep Akata. Learning robust representations via multi-view information bottleneck. In International Conference on Learning Representations, 2020. 4
[32] Weiran Wang, Xinchen Yan, Honglak Lee, and Karen Livescu. Deep variational canonical correlation analysis. arXiv preprint arXiv:1610.03454, 2016. 4
[33] Durk P Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, Ilya Sutskever, and Max Welling. Improved variational inference with inverse autoregressive flow. Advances in neural information processing systems, 29:4743-4755 2016. 7, 15
[34] Li Yingzhen and Stephan Mandt. Disentangled sequential autoencoder. In Jennifer Dy and An- dreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning. volume 80 of Proceedings of Machine Learning Research, pages 5670-5679, PMLR, 10-15 Jul 2018. 7.9
[35] Cian Eastwood and Christopher K. Williams. A framework for the quantitative evaluation of disentangled representations. In International Conference on Learning Representations, 2015 8.
[36] Yann Dubois, Douwe Kiela, David J Schwab, and Ramakrishna Vedantam. Learning optimal representations with the decodable information bottleneck. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 18674-18690. Curran Associates, Inc., 2020. 8
[37] Haoqing Wang, Xun Guo, Zhi-Hong Hong Deng, and Yan Lu. Rethinking minimal sufficient repre- sentation in contrastive learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16041-160515 2022.
[38] Salman Salamatian, Asaf Cohen, and Muriel Médard. Maximum entropy functions: Ap- proximate gacs-korner for distributed compression. arXiv preprini 1604.::600.33777,, 2016.
10
[39] James Townsend, Thomas Bird, and David Barber. Practical lossless compression with latent variables using bits back coding, In International Conference on Learning Representations 2019. 10
[40] Yann Dubois, Alexandros Kastanos, Dave Lines, and Bart Melman. Disentangling vae. http: //gi thub. cwm/ Yomm/YannDub////// / /iieentnggninnnggggn/a//n vae/, march 2019. 15
[41] Weiran Wang. Raman Arora, Karen Livescu, and Jeff Bilmes. On deep multi-view representation learning. In Proceedings of the 32nd International Conference on International Conference on Machine Learning Volume 37, ICML 15, page 1083-1092. JMLR.org, 2015. 16
[42] Paul L Williams and Randall D Beer. Nonnegative decomposition of multivariate information. arXiv preprint arXiv: 2010. 17
[43] Artemy Kolchinsky. A novel approach to the partial information decomposition. Entropy. 24(3):403, 2022. 17
[44] Michael Kleinman, Alessandro Achille, Stefano Soatto, and Jonathan C. Kao. Redundant information neural estimation. Entropy, 23(7), 2021. 17
[45] Diane Bouchacourt, Ryota Tomioka, and Sebastian Nowozin. Multi-level variational autoen- coder: Learning disentangled representations from grouped observations. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018. 17
[46] Ananya Harsh Jha, Saket Anand, Maneesh Singh, and VS Rao Veeravasarapu. Disentangling factors of variation with cycle-consistent variational auto-encoders. In Proceedings of the European Conference on Computer Vision (ECCV), pages 805-820, 2018. 17
12

---

[47] Matthew J Vowels, Necati Cihan Camgoz, and Richard Bowden. Nestedvae: Isolating common factors via weak supervision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9202-9212, 2020. 17
[48] Eduardo Hugo Sanchez, Mathieu Serrurier, and Mathias Ortner. Learning disentangled repre-
sentations via mutual information estimation. Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, Computer Viston ECCV 2020, pages 205-221, Cham, 2020. Springer International Publishing. 17
13

---

A Proofs
Theorem A.1 (GK VAE recovers the common information). Suppose our observations (I1, T2) have GK common information defined through the random variable ie satisfying eq. 4-5 and that our parametric function classes q( (z)) and p(z|z) optimized over can express any function. Then, our optimization (with & = 0, Bu < 1 and decoder p(z|2) E pi (=,(2,)) will recover latents = (zy, 22, 26) where & is the common random variable that maximizes the "stochastic" GK common information in eq. 4-5, while 2 is the unique information of the i-th view, which maximizes [x zu, 22).
Proof. We consider the hard-constrained optimization problem with infinitely expressive function class (i.e. so that the cross-entropy loss corresponds to the conditional entropy). Our VAE objective corresponds to the optimization problem
min Res PM
H ( 2 ze) (2 + (el(
(12)222) ze) +e)++ B.d (zu: x) (ol(ze)))
s.t. D(gg, 96x)= =0
We consider sequential optimization of finding & and a and then show that this solution minimizes joint objective above. We first consider the hard-constrained version of eq. (12).
min Re
H (2) Bed(zei)
H(z2 + (2e)
s.t.
D(9 (oo.)622))
Note that H(x;) = (xx + 1 (Ze5 a.). We can rewrite the loss as:
L = H (z, (2)) + H 222 (2e) + (.( + I(2n, x2)) H + H (x22) + (Be () 21) + I(ãc; x22)
This tells us that the optimal ze maximizes [(,n, 21) + (é 22). This is exactly the definition that we give of "stochastic" GK common information. Note we have previously shown ((2,; [(ze: 22).
Given & found above, the remaining objective (eq. (13)) becomes:
min 2
H 2 2c) + (22)
H 2 2c) B. x2).
For Bgg < the objective maximizes [[x 2/, 2c), which was the definition of the unique information. (For 3, > 1, this corresponds to a 8- VAE,, and will have the corresponding trade-off between rate and reconstruction [14, 11, 12].)
Finally, suppose & did not contain all the common information as dn i.e. T(Zo;2;) < T(2))))) Write the final equation maximization by noting that
Ha 2c) - (as 2.) + H(:))
Then the final optimization (for any i) is equivalent to max I(x;; " 2c) - B. I (2) H(x;) = max I(x + I I(xx: (2)) Bu (2)( - H(x))
(16)
max (xx + [ (x Bu (x11
(xx)
#
(17)
max 1(2 2yg Ec) T (z:) - H(xx))
(18)
For any = Eq. 16 is maximized with &, that encodes all the common information. Thus the GK VAE optimization is minimized with 2 = (2), 22. 2.).
14

---

Proposition A.1. ([3), Ex. I): Define
21 =1 = (20)
22 =2
- (zez =
where Se: z/. and z mutually independent. Then for any invertible transformation i, the random variable 2" that satisfies
arg max Cox (t(((t)))))
is Ze-
Proof. Note that if t is the identity transformation t(2) 2, then arg max Cak (21, 22)
&
is Ze. In general, if t are invertible transformations, suppose that fi and fa are the functions satisfying f1 = f2(22) corresponding to CGK (31, 22). Then the functions corresponding to CC (21), to (32) will be fi ti (t1()1) = f2 o to (t2())) and the random variable is equivalent.
B Experimental Details
We trained networks with Adam with learning rate of 0,001, unless otherwise stated. When the number of ground truth latent factors is known, we set the number of latents equal to the number of ground truth factors. To improve optimization, we use the idea of free bits [33] and we set bitec-bits 0.1. This was easier than using / scheduling, since it only involved one parameter bite We set 3,, to 10 and S. be 0.1. We trained networks for 70 epochs, except for the Mnist experiments, where we trained for 50 epochs. We used a batch size of 128 and we set A = 0.1. For all our experiments we used the same encoders and decoders as [16], which has been also used in recent work [18]. We used Gaussian encoders and Bernoulli decoders, basing our implementation off [40]. Our architecture schematized in Fig. 1. Note that we optimized encoders and decoders separately between the views (i.e weights were not shared).
To ensure that the latents are shared to both encoders, during training we randomly sample from either encoder qo. (2:( with p 0.5. We opted to randomly sample the latents from each encoder, as opposed to performing averaging, to ensure that the latent will always be a function of individual view z4. This is in addition to the soft constraint governed by X in the loss.
In our implementation, unless otherwise noted, used separate fiew-dependen decoders that each processed all the latents z to reconstruct the observations. We also verified that view specific decoders that processed only view-specific latents 2,, in Thm. 3.1, also led to partitioning of the common and unique information Fig. 5. This was the case both when the unique factors generative model described in Sect. 4.1 used for the majority of our experiments, and also if there were correlations between the unique components (Fig. 7) and (Fig. 6), with the experiments described in Sect. Empirically, we found the separation of the common and unique information was robust to the partitioning of the latents fed to the decoder of our GK- KAEE. We provide both decoder implementations in our repository available at https:// thub. com/mj kllijiiiimmi eilman////////m http///oooooo/////
were pairwise independent,
To quantify the information contained in the representation, we calculate the usable information (in bits), which is a lower bound to the information contained in the representation [25, 26). To train our decoder, we used the madieentBosstincl wetiiinnnssiiiiee from sklearn with default parameters. We trained on 8000 samples and tested on 2000. We evaluated the information on a held-out test set, and hence the negative values correspond parentheses correspond to the number of ground truth factors. We used the same setup for the rotated Mnist experiments (Fig. 3). For the rotation angle, we discretized the angle of rotation (-45 45%) into 10 bins of equal size, and predicted the discrete bin. We predicted the rotation angle applied to the first view. When comparing with constrastive learning approach (Fig. 3, right), we used the same encoder backbone as our GK-S We pre-trained with a batch size of 256 for 60 epochs with a learning rate of 0.001, and then trained the linear classifier for epochs with an initial learning rate of 0.03. We used latent dimension of 20.
overfitting on the training set. In Table 1, the numbers in
We used the code from: http://gitabb.om///Hobbitmmggng
15

---

lalent
latent
factor
Factor
Figure 5: Latent traversals and DCI plots show optimization results separation common and unique information when using partitioned decoder. Same conventions as Fig. 2. (Left) 3dshapes: The top rows shows the unique latents, the middle 3 the common (and the bottom the unique latents for the second view). The ground truth unique generative factors are (0, 1,2) corresponding to floor color, wall color, and object color. Our model correctly recovers that those factors unique (first three rows in the figure), and that the other factors common (middle three rows). (Right) dsprites: The top 2 rows shows the unique latent variables, the middle the common (and the bottom are the unique latent variables for the second view). The ground truth unique generative factors are (3, 4) corresponding to x-position, y-position respectively. Our model correctly recovers that those factors are unique (first rows in the figure), and that the other factors are common (middle three rows).
B.1 DCI Plots and Latent Traversals:
Let d be the dimension of the representation and let é be the true generating factors. The idea to train a regressor fz(2) : Rd + to predict the ground truth factors i, for each j from the representation 2. This results in matrix of coefficients that describe the importance of each component the representation for predicting each ground truth factors. This matrix R is the importance matrix that visualize in the paper, where R. reflects the relative importance of of for predicting tj (where 2; refer to the ith component of the representation). We used the BradientBood wnnnnaasssiiii from sklearn with default parameters, similar to [18] to predict the ground truth factors.
visualize the latent components by traversing one latent variable at time, while keeping the others fixed. We plot traversals of the prior p(z) and the posterior g(=[ær) in different figures (traversals of the prior are Fig. 2, Fig. 3, and Fig. 5, while the others are posterior traversals). We obtain the posterior traversals by encoding observation through (=](z) and traversing each component of the representation.
C Other Related Work
similar formulation has been used for multi-view learning [41]. with two separate autoencoders. with an constraint that each latent representation be similar, with the similarity measured by the canon- ical correlation of the latent representations. They did motivate from an information-theoretic perspective; and rather empirically found that such an optimization lead to good representations in the multi-view setting.
Also related to our work is [4]. [4] defined the approximate Gåcs-Kömer information in the following manner:
max 1(x122)
H(2)22) <8 s.t.
(19)
2 2 σ + 02
16

---

By showing that they could perform the optimization over deterministic functions f such that f(x1), they formed a Lagrangian corresponding to:
max f
- ((oG))))H((((1)))))
(20)
They noted that the above optimization is difficult to perform and that future work should look into avenues for computing this quantity; indeed it looks difficult to learn the function f from the above optimization problem. They also suggested that this approximate form of the Gárs-Körner common information had potential applications in terms of compression, since the (approximate) common information only needs to be represented once.
C.1 Relationship to redundant information in the Partial Information Decomposition
Our approach also relates approaches that aim understand how the information that a set of sources contain about a target variable is distributed among the sources. In particular, [42] proposed the Partial Information Decomposition (PID), which decomposes the information that two sources Xi and X2 contain about a target variable Y into a the components that are unique, redundant, and synergistic. A central quantity this decomposition, the redundant information, reflects the shared information about a largel variable. The Gors-Korner common information is equivalent to existing definitions redundant information if the target is reconstructing the sources (i.e, Y = (2), [43]. We note that computing the redundant information from high dimensional samples has been challenging. Recently [44] proposed an approach that could be applied on high dimensional sources but where the targer was low dimensional. Here, approximation of the common information reflects a further step which can be applied on high dimensional samples (and targets).
C.2 Other multi-view representation learning approaches for partitioning common/unique information
Our work relates to growing body of multi-view representation learning approaches aiming to disentangle common and unique information from grouped data. [45] examine the setting of extracting common content and variable style from set of grouped images based on content. [46, 47, 48] are also similar spirit, aiming to disentangle the shared and unique component between paired data.
These existing methods appear to focus on qualitatively partitioning the information through the use of different objectives, however these approaches lack well-defined notion of common (and unique) in- formation, making it difficult to prove theorectical guarantees for the recovery of the common/unique component, or enable quantification of the information contained in the representation. our setup allows us to formalize the decomposition of information using well-defined information theoretic notion of common/unique information and we show both formally and empirically that we can optimize the objective with simple modification traditional VAE setup and that we can quantify the amount of information contained in the representation.
contrast,
D Limitations of our approach
To validate our approach, we focused on the simpler setting where we have paired data, however, we could extend our formulation find common information between common information between subsets of sources. While our approach can be naturally extended to find the common information between sources, future work could investigate scalable approach to identify common and unique information between arbitrary subsets of the sources. Additionally, to validate our approach empirically, we focused using convolutional encoder on relatively small images and video frames, but our formulation is general and the encoder could be interchanged depending on the complexity and inductive biases of the task and data.
sources, well finding
E Differences between GK Common Information and Mutual Information
The following examples highlight the difference between GK information and mutual information, and suggest why using GK information may be desirable in practical settings.
17

---

Case 1: Let X1 = N1 and X2 = C N2, where C' and N; are independent variables, but the noise N1 and N₂ are correlated. Suppose C is discrete for example, 0 or feature and the variance of the noise N₂ is relatively small [|) || < 1). In this case, since C' can be recovered deterministically from C' N; (for example by thresholding its value) the GK common information between X1 and X2 is bit. On the other hand, the mutual information between X₁ and X2 which 2 can be made arbitrarily large by increasing the
is given by , = 1 + 0.5 Log( des correlation between N1 and N2. L.e. making det Σ - O. Hence, the the value of MI is unrelated to the actual semantic information shared between the two variables. contrast, the GK common information only identifies the common component C. In sense, the GK is able to recover the underlying "discrete" or "symbolic" information that is common between two different continuous sources. This behavior is valuable in settings like neuroscience, where one wants to measure what is robustly encoded by both representations and not just how much they are correlated (which could be due to any amount of spurious factors). Additionally, with our relaxation of the problem can measure how much "almost" discrete information encoded in both, which makes the method more applicable to real-world cases.
det
Case 2: Let X, = fi Ui) and X2 = f2CC; U2) where C' and U, are independent but U1 and U2 are correlated. Suppose fi is an invertible data generating function mapping latent factors to observations (such as pixels). By maximizing mutual information between views, one implicitly measures those correlations (based sense one cares about only the fully shared components between views. These different perspectives can offer different utility depending on the case.
similar computation to above in Case 1), whereas in the GK
We show in Fig. 6 and Fig. 7 that we can recover the partitioning of common and unique factors in practice from the generative model in Case 2.
F Additional Experiments
In Table 2, we compute the information encoded in the common and unique latent components for the common-dsprites experiment described in the main text, with corresponding DCI matrix and latent traversals in Fig. 2. We also report additional runs for the common -ssriiss and common-3dshqge experiments in Fig. 8 and Fig. 9 respectively. These additional runs consistent with what was reported in the text, separating the common and unique factors.
To introduce correlations between the unique components (Case 2 in App. E) we modified the generative model from Eq. 15 where the generative model for the data (x1, was
I = f(2h 2x); x2 = f(z zn),
(21)
where ze is shared between the views and 2 is the unique information encoded in the jth view and corresponds rendering function. To allow the unique components be dependent (but not the same as the common latent component ze), we modified the generative model so that z was conditionally dependent on the value of 2/) We implemented this constraint component-wir component of 2 by reducing the space of possible latent values of the corresponding components of 2 (to half, floored if odd) depending on the component- wise values of 2 We show in Fig. and Fig. 7 that we can recover the partitioning of common and unique factors in practice from this generative model.
for each
In addition to the experiments described in the main text, we report variants of common-dsprites and common-3dshapee particular, we change the set of ground-truth common and latent factors.
For the common-3dshapes we specified that the viewpoint was the unique latent variable z,, whereas the other latent variables (backgroud color, floor color, object color, shape, size) were common to both views. We show the DCI matrix and the traversals in Fig 10. For the common-dspriter variant, we set the unique components to be the size, scale, and orientation, and the common latent factors to be the x and y position. We show the DCI matrix and latent traversals in Fig. 11. The common and unique latent variables from optimization separated these ground truth factors.
18

---

latent
latent
latent
factor
factor
factor
Figure 6: Decoder from all latents partitions common and unique information when unique generating factors are dependent. We modified the generative model from Eq. 15 so that the common and unique generating variables were independent but there were correlations between the unique components. In particular, the latent factors for the second view were conditionally dependent the unique latent factors. We still observed partitioning of the learned common and unique latent variables, as the paper. The top 3 rows shows the unique factors, the middle 3 the common (and the bottom 3 the unique factors for the second view). The ground truth unique generative factors are (0, 1, 2) corresponding to floor color, wall color, and object color. Our model correctly recovers that those factors are unique (first three in the figure), and that the other factors are common (middle three rows).
19

---

latent
latent
latent
factor
factor
factor
Decoder from partitioned latents also partitions common and unique information, similar to when decoding from all latents in Fig. 6. The top 3 rows shows the unique factors, the middle 3 the common (and the bottom 3 the unique factors for the second view). The ground truth unique generative factors are (0, 2) corresponding to floor color, wall color, and object color. Our model correctly recovers that those factors are unique (first three rows in the figure), and that the other factors are common (middle three rows).
Figure 7:
20

---

latent
latent
latent
2
factor
factor
factor
Figure 8: Additional runs for the same experiment as Fig. 2 (right) for common-dsprites with the same conventions Fig 2. These runs also show the same blockwise partitioning of common and unique information, as in the paper.
SHAPE (3)
SCALE (6)
ANGLE (40)
X-Pos (32)
Y-Pos(32)
KL. TOTAL
COMMON
1.54
2.45
2.88
-0.33
-0.29
9.57
UNIQUE 0.08
0.03
-0.5
3.58
3.63
9
TOTAL 1.54
2.45
2.76
3.68
3.69
18.57
Table 2: Usable Information (in bits) in representation for dsprite experiment. The common information is separated from the unique information. The ground truth unique factors (x-position and y position) were almost perfectly encoded in the corresponding latents latents, and the other factors (shape, scale, and angle) were correctly encoded the common latent variables. The numbers in parenthesis represents the number of discrete factors for each latent variable.
21

---

latent
latent
latent
factor
factor
factor
Figure 9: Additional runs for the same experiment as Fig. 2 (left) for common- -dsmon-ddshape, with the same conventions. These runs also show the same blockwise partitioning of common and unique information, in the paper.
22

---

latent
latent
latent
factor
factor
factor
Figure 10: DCI matrix of common-3 -dshhpps variant for different random seeds. The top row shows the unique factor, the middle 5 the common (and the bottom row is the unique factors for the second view). The ground truth unique generative factors are (5) corresponding the viewpoint. Our model generally correctly recovers that the factor is unique (first row in the figure), and that the other factors are common (middle five rows).
23

---

latent
latent
latent
factor
factor
factor
Figure 11: DCI Matrix of dsprites experiment variant for different random seeds. The top 3 rows shows the unique latent variables, the middle 2 the common (and the bottom are the unique latent variables for the second view). The ground truth unique generative factors are (0, 1, 2) corresponding to shape, scale, and viewpoint angle. Our model correctly recovers that those factors are unique (first three rows in the figure), and that the other factors (x and position) are common (middle two rows).
Bu 1
Bu 100
1 = 2
latent
latent
10
10
14
14
2
2
factor
factor
Figure 12: Effect of 8., Additional runs for Fig. 2 (right) for common-3dshapes with unique and 5 common latents per view. For large 8n, (e.g- 8n 100, right), in the paper, we see similar blockwise separation of the common and unique latents, even when the number of latents does not match the ground- truth number of latents.


Reviews:
[{"review_id": "ec2oZBr7yl", "reviewer": "Reviewer_uZkW", "summary": "This paper proposes a new way to partition common and unique information in multi-view data, by leveraging and optimizing the objective of common information defined by Gacs and Korner. Authors extended the deterministic setup to stochastic scenario and used a variational autoencoder to realize the objective. Authors also carefully designed experiments in both static and time series data to validate the effectiveness of their architecture.\n", "questions": "Please refer to weaknesses above.", "limitations": "Authors did not discuss potential limitations and negative societal impacts. ", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The introduction of the Gacs-Korner common information to partition common and unique information in realistic multi-view data is novel.\n2. A straightforward application of Gacs-Korner common information is hard. The stochastic relaxation made by authors makes sense to me.\nIt is also good that authors constructed different datasets to validate the effectiveness of their method.", "weaknesses": " I found the biggest issue of this manuscript is that some points are very unclear or need more explanations. Below I listed a few of them:\n1) Can you elaborate more on the difference between common information and mutual information. \nI can understand that mutual information has no clear interpretation in terms of information decomposition. \nIt would be much better to describe more differences. \nFor example, in Eq.~(15), what would be the mutual information between $x_1$ and $x_2$; and why mutual information should be much larger than common information $z_c$?\nAre there more illustrative examples?\n2) In terms of implementation, I appreciate that authors relax the constraint of $Z=f(X_1)=g(X_2)$ with a conditional divergence minimization term $D(p(z|x_1);p(z|x_2))$. \nHow to implement this term in your VAE objective?\n3) Also in implementation, it requires that $\\beta_u > \\beta_c$. How to balance the trade-off between $\\lambda_c$ and $\\beta$?\nIn Eq.~(13), there should be two $\\beta_u$ corresponding to two views, i.e., $\\beta_{u1} KL(q_{\\phi_{u1}}||...)+ \\beta_{u2} KL(q_{\\phi_{u2}}||...)$?\n4) Proposition 3.1 requires an invertible mapping from $z$ to $x$. How is this condition reflected in your implementation? ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback and address their concerns below.\n\nQ1: We elaborate on the difference between common information and mutual information in our response to [Reviewer iAUJ](https://openreview.net/forum?id=e4XidX6AHd&noteId=HqfjeyTDaR) (see Case 1 and Case 2 in particular). \n\nQ2: This was achieved through a combination of using $\\lambda_c$ and also sampling independently from both encoders with $p=0.5$ (Line 226). As per the request of reviewer iAUJ, we ran the ablation with $\\lambda_c = 0$ and found that the implicit bias due to the random swapping suggested by the reviewer is enough to have a weak qualitative separation between common and unique information (Fig. 1 of rebuttal pdf). However, in Table 1 of the rebuttal we observe that in this same setting the divergence between the distribution and was exceedingly large (7.5 * 10^5) which would lead to severe mis-estimation of the GK information in the common component. Increasing the value of solves the problem (Table 1 of rebuttal material). Higher-values of will increasingly converge to the exact GK information, but may destabilize training. Empirically, we found that $\\lambda_c = 0.1$ provided a good trade-off.\n\nWe parametrized the encoders $p(z|x_1)$ and $p(z|x_2)$ using neural networks that outputted the mean and (diagonal) covariance of a Gaussian distributuion. In this case, the KL divergence (which we denote by D) has a close form, which we minimize during training.\n\nQ3: Related to the above, the purpose of $\\lambda_c$ was to ensure that the information was common. We found that we could set $\\lambda_c$ to be a large value provided we started it off at a small value at the beginning of training (Fig. 2 of rebuttal).\nIn this way, the effect of the parameter $\\lambda_c$ appears to depend more strongly on it's value during training rather than the values of $\\beta_u$ and $\\beta_c$. \n\nIndeed there should be two $\\beta_u$ in Eq.~3 corresponding to the two views (in our experiments they were equal). We will clarify the text.\n\nQ4: This assumption refers to condition on the data generating model so that the latents can be recovered: it’s saying that we need to be able to recover the latents from the data in order for our optimization to recover the ground-truth latents. In the datasets we examine, this appears to hold, as indeed we can decode the common and unique latents from the common and unique factors. Note that even if this assumption does not hold in practice, our method will still recover the common information and unique information present in the observed data.\n\nWe discussed limitations in Appendix D, and we do not foresee any negative societal impacts.\n\n"}, {"review_id": "iOl1ZMfizp", "reviewer": "Reviewer_hfAm", "summary": "The paper proposes the Gracs-Korner Common information to measure common information between two random variables and propose a variational relaxation to compute the practical loss. The paper then derives an objective that eases disentanglement requirement.", "questions": "Please see weakness above. ", "limitations": "Yes. ", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The paper proposes the Gracs-Korner Common information to measure common information between two random variables and propose a variational relaxation to compute the practical loss. To this point, the paper is novel to me. Empirical evidence demonstrates the advantages of the proposed method. ", "weaknesses": "1. The proposed method is only compared with basic VAE where as the advantage of the proposed method is not significant. \n\n2. The proposed method seems to be loosely connected with Gacs-Korner Common Information. The derivation is a minor variation of a vanilla VAE. In this regard, it seems the impact of the work is limited, where the modification introduces additional encoder architecture but only brings limited empirical advantage. The final loss further decompose the distance between prior and posterior inference into “common components-prior distances” and the “unique components-prior distances\". The driven force that separates the DNN to learn the decomposition remains unclear. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their comments and address their concerns below.\n\n> The proposed method is only compared with basic VAE where as the advantage of the proposed method is not significant. \n\nThe purpose of our study was to show formally and empirically that we can partition the latent representation of multi-view data into a common and unique component, and also provide a tractable approximation for the Gacs-Korner common information between high dimensional random variables. In comparison against a (Beta)-VAE, in addition to our empirically observed higher disentanglement scores, the (Beta)-VAE cannot provide insight into which factors are shared and which are unique, whereas our optimization directly provides this separation. Additionally, even theoretically it has been shown that VAE cannot disentangle components without additional signals (like, in our case, the common information between different sensors) (Locatello et al., 2019).\n\n> The proposed method seems to be loosely connected with Gacs-Korner Common Information. The derivation is a minor variation of a vanilla VAE. In this regard, it seems the impact of the work is limited, where the modification introduces additional encoder architecture but only brings limited empirical advantage. The final loss further decompose the distance between prior and posterior inference into “common components-prior distances” and the “unique components-prior distances\". The driven force that separates the DNN to learn the decomposition remains unclear.\n\nIn Theorem 3.1, we prove that our notion of common information is a direct generalization of the original Gacs-Korner common information. Moreover, we show that our variational relaxation of the problem provides a tractable way to compute the common information between high-dimensional random variables, which is a major obstacle with the original proposal of GK-Common Information. We show that our approach recovers the correct value of the GK common information in all benchmarks. Moreover, we introduce for the first time a series of benchmarks with high-dimensional random variables (which previous algorithms could not address) and we show that even here we can obtain the correct theoretical value of the common information. \n\nMost results in structured representation learning with VAE involve loss functions modified with additional constraints. We believe that the fact that the changes need to decompose the representation in common and unique factors are relatively simple to implement, theoretically grounded, and widely applicable to any existing VAE architecture are an advantage of our method, not a downside. \n\nThe key parameters enabling this optimization are the relative values of $\\beta_u$ and $\\beta_c$, with $\\beta_u > \\beta_c$, which encourages common information to be encoded in the common latents by paying a smaller cost. $\\lambda_c$ encourages the distributions of the encoders to be similar so that the factor encodes common information (we additionally bias the information to be common through random sampling of the common latents from either encoder during training).\n\n---\n\nLocatello et al. Challenging common assumptions in the unsupervised learning of disentangled representations. ICML, 2019\n"}, {"review_id": "llAQUo2RO1", "reviewer": "Reviewer_iAUJ", "summary": "The basis of the proposed work is a practical means to decompose the information contained in two random variables into common and unique components.  With a couple tweaks to a vanilla VAE setup, data that has been paired such that certain factors of variation have a constant value within the pair can be used to train latent spaces that separate the components of the information.  The common component of the information -- the Gacs-Korner common information -- is the primary goal here, and it’s generally difficult to acquire because it requires finding a random variable that is simultaneously a deterministic function of both input variables.  Except for highly constrained joint distributions, such a variable generally does not exist (beyond the vacuous solution).  The current work proposes a relaxation where the common component can be a stochastic function of the inputs, and this facilitates optimization as well as makes the method applicable to cases without cleanly shared information (at least in theory; it’s not shown here).\n\nAs a route to Gacs-Korner common information, the paper offers a principled and pragmatic methodology that (as far as I can tell) offers something original and of value.  The example scenarios and applied metrics, however, paint the work more as a route to partially disentangled latent spaces given grouped data, and in that regard it’s lacking novelty and missing relevant comparisons.  The following papers leverage the same weak supervision (paired/grouped data with a subset of factors held constant) to learn latent spaces that separate common from unique information:\n\n- “Multi-Level Variational Autoencoder: Learning Disentangled Representations from Grouped Observations”, Bouchacourt 2018\n- “Disentangling Factors of Variation with Cycle-Consistent Variational Auto-Encoders”, Jha 2018\n- “Unsupervised Robust Disentangling of Latent Characteristics for Image Synthesis”, Esser 2019\n- “NestedVAE: Isolating Common Factors via Weak Supervision” Vowels 2020\n\nAnd one that does so without a generative model:\n- “Learning Disentangled Representations via Mutual Information Estimation”, Sanchez 2020\n\nIn summary, the method is an interesting contribution but the experimental results are not very effective.\n", "questions": "- As common information is a lower bound to mutual information (L73), how can a lower bound to MI be used as a lower bound for common information (L194)?  \n- Was $\\lambda_c$ shown to be necessary given the random swapping of the common representation (L225)?  ", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The principled route to extracting the common information in a pragmatic methodology is great.  Though it is not demonstrated in this work, the authors motivate the work in terms of multi-modality data, which could be a rich area of application of the method.", "weaknesses": "See the summary for the primary weakness.  I can imagine a couple routes to make the experimental results more strongly support the method.  First, the current experiments could include direct comparisons to some of the attached methods with a demonstration of what GKVAE brings.  However, if GKVAE is not more effective in the scenarios currently in the manuscript, qualitatively different experiments could help show its merit.  Paired data streams from different modalities could highlight the strengths of GKVAE, or perhaps an example where the content of the common information allows for insight about the relationship between the variables (ie, through inspection of the learned latent variable).  Or some example where GK information is desirable.\n\nOn a much more minor scale: I doubt the majority of readers will have a better intuition for “usable information” than for standard mutual information; it seems a weakness to me to rely on that rather than any of a number of mutual information estimators or bounds that should work fine on the low-entropy examples of this work.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their comments and address their concerns below.\n\nTo understand the **difference between GK information and mutual information and why GK information is desirable**, consider the following settings.\n\n**Case 1:**\nLet $X_1 = C + N_1$ and $X_2 = C + N_2$, where $C$ and $N_i$ are independent variables, but the noise $N_1$ and $N_2$ are correlated. \nSuppose $C$ is discrete (e.g. a 0 or 1 feature), and have $N_i$ be a relatively small amount of gaussian noise. Now $C$ can still be recovered deterministically from $C + N_i$ (just threshold) but the mutual information between $C + N_1$ and $C + N_2$ is $1 + 0.5 \\log (\\frac{\\det(\\Sigma_1) \\det(\\Sigma_2) } {\\det(\\Sigma)} )$ (the first bit comes from the MI of $C$, the log comes from the MI between gaussian variables). Then the MI can be made arbitrarily large by increasing the correlation between $N_1$ and $N_2$ (making $\\det(\\Sigma) \\to 0$).\n \nIn contrast, the GK common information only identifies the common component C. In a sense, the GK is able to recover the underlying \"discrete\" or \"symbolic\" information that is common between two different continuous sources. This is a valuable property to have in settings like neuroscience where one wants to measure what is actually encoded by both representations and not just how much they are correlated (which could be due to any amount of spurious factors). Additionally, with our relaxation of the problem we can measure how much is almost encoded in both, which makes the method more applicable to real-world cases where it is unlikely the same exact thing is encoded in both.\n\n**Case 2:**\nLet  $X_1 = f_1(C; U_1)$ and $X_2 = f_2(C; U_2)$ where $C$ and $U_i$ are independent but $U_1$ and $U_2$ are correlated. Suppose $f_i$ is an invertible data generating function mapping latent factors to observations (such as pixels). By maximizing mutual information between views, one is implicitly caring about those correlations (based on a similar computation to above in Case 1), whereas in the GK sense one cares about only the fully shared components between views. These different perspectives can offer different utility depending on the use case. \n\nWe will clarify this distinction in the main text, which we hope is helpful for readers. In addition to the benefits of interpreting a well-defined common core between variables, the GK common information can be helpful for compression schemes, as we mention in the discussion.\n\nWe appreciate the reviewer noting that our contribution is interesting in bridging the efforts in information theory to define a well-defined notion of common information and recent progress in machine learning to exploit such knowledge coming from multiple data streams. We agree that there is a growing literature on multi-view representation learning that aims to partition the information between views (which we discuss in part in our paper) and we will update our citations with the references you mention which share a similar goal -- thank you for suggesting them! The existing methods appear to focus on qualitatively partitioning the information through the use of different objectives, however, the setups appear to be difficult to prove theorectical guarantees (as they lack a well-defined definition) or enable quantification of the information contained in the representation. In contrast, our setup allows us to formalize the decomposition of information using a well-defined information theoretic notion of common/unique information and we show both formally and empirically that we can optimize the objective with a simple modification to a traditional VAE setup, and also enables quantification of such information. We compare our method against contrastive training, finding that contrastive approaches only identify the common information (Fig. 3). \n\nWe also agree with the reviewer that the ultimate goal is to leverage our framework for application on real-world high dimensional multi-modal data, or for interpreting high dimensional scientific data (such as neural recordings from different brain areas) and are excited about the insights this can yield. We believe showing that a well-defined partitioning of information exists and can be approximated from data using our framework is an important step to this goal. For example, accurately modeling for example high-resolution RGB + Depth data requires task specific architectures, and we leave it to future work, and we believe our work is an important step towards this ultimate goal. \n\nQ1: We are looking at mutual information contained in the representation that satisfies the constraint that it is the common information (approximately). We then look at the usable/mutual information that the representation contains about the ground-truth latent variable.\n\nQ2: In terms of the necessity of $\\lambda_c$, the random sampling of the latent may not be enough to enforce that only common information is encoded in $z_c$, as in principle $z_c$ could encode unique information about a view that is then discarded by the view-specific decoder. The constraint on $\\lambda_c$ guarantees that that will not be the case.  In Fig. 1 in the rebuttal material, we run an ablation experiment setting $\\lambda_c=0$. We observe that the implicit bias due to the random swapping suggested by the reviewer is enough to have a qualitative separation between common and unique information in the experiment (Fig. 1, left). However, in Table 1 of the rebuttal we observe that in this same setting the divergence between the distribution $q_1(z_c|x_1)$ and $q_2(z_c|x_2)$ was exceedingly large (7.5 * 10^5) which would lead to severe mis-estimation of the GK common information. Increasing the value of $\\lambda_c$ solves the problem (Table 1 of rebuttal material). Higher-values of $\\lambda_c$ will increasingly satisfy the constraint of the GK information, but may destabilize training. Empirically, we found that $\\lambda_c=0.1$ provided a good trade-off in our experiments.\n"}, {"review_id": "TSTiEri0vB", "reviewer": "Reviewer_cgd1", "summary": "The authors present a concept of common information that assesses and distinguishes shared information from unique information between two random variables. This notion, which is a variant of the Gacs-Korner common knowledge, is more optimizable and experimentally approximable when employing sample data. \n\n\n\n", "questions": "- Could you describe how the multi-view VAE model's unique latent variables capture the individual factors that cause variation, and how this contributes to the efficiency of inferring common and unique components from data?\n", "limitations": "The authors address the limitation adequately. ", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": " I believe this paper made a good contributions by proposing to relax the requirement that the representation must be a deterministic function and instead allows it to be a stochastic mapping. Based on the results, it has been shown that such an approach is more  beneficial for quantifying and interpreting the latent representation.\n\n\n\n", "weaknesses": "\n-  The authors experimentally demonstrating that adding unsupervised viewpoints improves disentanglement; which is likely due to limitations in single-sample analysis; active interaction, not passive observation, fosters better learning of environment representations. but the observation has not been sufficiently explained or explored in the current study.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their comments and address their questions below.\n\n> The authors experimentally demonstrating that adding unsupervised viewpoints improves disentanglement; which is likely due to limitations in single-sample analysis; active interaction, not passive observation, fosters better learning of environment representations. but the observation has not been sufficiently explained or explored in the current study.\n\nWe agree with the reviewer’s observation and intuition; and we will happily elaborate on this in the text. Indeed, as we briefly mentioned in the text, a source of inspiration for our work was an old experiment where neuroscientists showed that the ability for animals to act within an environment, rather than passively observe it strongly impacts their representations of it and ability to identify relevant environmental variables (Held & Hein, 1963 ). One explanation for these findings is that the correlations between the actions and the environmental stimuli are meaningful and important for the animal to leverage.  Here, in a similar spirit, we indeed found that the addition of an additional viewpoint can provide a guiding signal for finding the common information between the views, which can identify and reveal interesting structure in the data.\n\n> Could you describe how the multi-view VAE model's unique latent variables capture the individual factors that cause variation, and how this contributes to the efficiency of inferring common and unique components from data?\n\nIn our optimization, adding the unique latent variables (with the constraint that $\\beta_u > \\beta_c$) was important so that unique information would not be encoded in the common latent factors. While this is penalized by $\\lambda_c$, as well as by randomly sampling $z$ from $q_1(z_c|x_1)$ and $q_2(z_c|x_2)$ with $p = 0.5$, without the addition of the unique components, the unique information could leak into the common latents if the unique information helped for reconstructing the output more than the cost of not satisfying the (approximate) equality constraint. By adding in the unique latent variables (with $\\beta_u > \\beta_c$) the common and unique information will be partitioned, as we show in Thm 3.1 and our experiments.\n\n--- \nRichard Held and Alan Hein. Movement-produced stimulation in the development of visually guided behavior. Journal of comparative and physiological psychology, 56(5):872, 1963."}]

Author rebuttal:
We thank all the reviewers for their thoughtful comments and suggestions, which we believe will lead to a clearer and improved manuscript. 

We have responded to each of the reviewer's questions individually. Please see the attached pdf for figures and a table that we refer to in our responses to specific reviewers.

Meta-review: This paper proposes a new way to partition common and unique information in multi-view data, by leveraging and optimizing the objective of common information defined by Gacs and Korner. The paper extends the deterministic setup to stochastic scenario and uses a variational auto-encoder to achieve this objective. Carefully designed experiments in both static and time series data validate the effectiveness of the method.


I agree with the reviewers that the contributions are sound and novel and would be of interest to the community.
Decision: Accept (poster)