Multi-View Disentangled Representation
Authors: Zongbo Han
Changqing Zhang
Huazhu Fu
Qinghua Hu
Joey Tianyi Zhou
Venue:  (2021)

Abstract: Learning effective representations for data with multiple views is crucial in machine learning and pattern recognition. Recently great efforts have focused on learning unified or latent representations to integrate information from different views for specific tasks. These approaches generally assume simple or implicit relationships between different views and as a result are not able to flexibly and explicitly depict the correlations among these views. To address this, we firstly propose the definition and conditions for multi-view disentanglement providing general instructions for disentangling representations between different views. Furthermore, a novel objective function is derived to explicitly disentangle the multi-view data into a shared part across different views and a (private) exclusive part within each view. Experiments on a variety of multi-modal datasets demonstrate that our objective can effectively disentangle information from different views while satisfying the disentangling conditions.

Under review as a conference paper at ICLR 2021
MULTI- VIEW DISENTANGLED REPRESENTATION
Anonymous authors
Paper under double-blind review
ABSTRACT
Learning effective representations for data with multiple views is crucial in machine learning and pattern recognition. Recent great efforts have focused on learning unified or latent representations to integrate information from different views for specific tasks. These approaches generally assume simple or implicit relationships between different views and a result are not able to accurately and explicitly depict the correlations among these views. To address this, we firstly propose the definition and conditions for unsupervised multi-view disentanglement providing general instructions for disentangling representations between different views. Furthermore, a novel objective function is derived to explicitly disentangle the multi- view data into shared part across different views and a (private) exclusive part within each view. The explicit guaranteed disentanglement is of great potential for downstream tasks. Experiments that our objective can effectively disentangle information from different views while satisfying the disentangling conditions.
variety of multi-modal datasets demonstrate
INTRODUCTION
Multi-view representation learning (MRL) involves learning representations by effectively leveraging information from different perspectives. The representations produced by MRL are effective when correlations across different views are accurately modeled and thus properly exploited for downstream tasks. One representative algorithm, Canonical Components Analysis (CCA) (Hotelling. (992). aims to maximize linear correlations between two views under the assumption that factors from different views are highly correlated. Under a similar assumption, the extended versions of CCA, including kernelized CCA (Akaho 2006) and Deep CCA (Andrew et al.) 2013), explore more general correlations. There are also several methods (Cao et al., 2015 Sublime et al.) 2017) that maximize the independence between different views to enhance the complementarity. Going beyond the simple assumptions above, the latent representation encodes different views with a degradation process
implicitly exploiting both consistency and complementarity (Zhang et al. 2019).
These existing MRL algorithms are effective, however, the assumed correlations between different views usually simple thus cannot accurately model explicitly disentangle complex real-word world correlations, which hinders the further improvement and interpretability. Although there are a few heuristic algorithms (T'sai et al., 2019, Hu et al., 2017) that explicitly decompose the multi- view representation into shared and view-specific parts, they are especially designed for supervised learning tasks without any disentangled representation guarantee and fall short in formally defining the relationships between different parts. To address this issue, we propose to unsupervisedly disentangle the original data from different views into shared representation across different views and exclusive (private) part within each view, which explicitly depicts the correlations and thus not only enhances the performance of existing tasks but could also inspire potential applications. Specifically, we firstly provide a definition for the multi-view disentangled representation by introducing the sufficient and necessary conditions for guaranteeing the disentanglement of different views. According to these conditions, information whhoortttthhhaaaaee -bsed algorithm is proposed to accurately disentangle different views. To summarize, the main contributions of our work are as follows:
To the best of our knowledge, this is the first work to formally study multi-view disentangled representation with strict conditions, which might serve as the foundations of the future research on this problem.
Based on our definition, we propose multi- view disentangling model, in which information- theory- based multi-view disentangling can accurately decompose the information into shared

---

Under review as a conference paper at ICLR 2021
(a) Original data (b) 2 (C)
(d)
Figure 1: Illustration of multi-view disentangled representation. (a): The red and white graphics indicate the shared information between different views, and the (private) exclusive information within each view, respectively. (b): The exact disentangled representation can be achieved the shared (gray area) and exclusive (white area) components are separated. when the four conditions in definition 2. are satisfied. (c))))e)(f) The exact disentangled representation cannot be guaranteed when any condition is violated. Intuitively, the proposed four conditions are necessary and sufficient conditions since any change of (b) will violate the definition.
representation across different views and exclusive representation within each view. The explicit decomposition enhances the performance of multi-view analysis tasks and could also inspire new potential applications.
Different from the single-view unsupervised disentangled representation learning (Locatello et al. 2019). we provide a new paradigm for unsupervised disentangled representation learning from a fresh perspective disentangling factors between different views instead of each single view.
Extensive experiments on range of applications verify that the proposed information cimmmatioo-thooyy- theory- based multi-view disentangling algorithm can accurately disentangle data from multiple views into expected shared and exclusive representations.
2 MULTI- DISENTANGLED REPRESENTATION
Existing multi-view representation learning methods (u & Good- man 20188] Zhang et al.] 2019) can obtain common representation for multi-view data, however, the correlations between different views are not explicitly expressed. The supervised algorithms (Hu et al. 2017 Tan et al. 2019) can decompose multiple views into common part and private parts, but there is no disentangling guaran- tee. Therefore, we propose multi-view disentanglement algorithm that can explicitly separate the shared and exclusive information in unsupervised settings. Formally, we first propose definition of a multi-view disentangled representation by introducing four criteria, which are considered sufficient and necessary conditions of disentangling multiple views. The definition is as follows:
Illustration of our model, which corresponds to the objective in Eq. and the conditions in definition 2. Re- fer the text for PoE (Product of Expert) in 2.1
Figure 2:
Definition 2.1 (Multi- View Disentangled Representation) Given a sample with two views, i.e., X' {>'} the representation Sais {s', e² e²)? li=1 is view disentangled representation if
the following conditions are satisfied:
Completeness: ! The shared representation s² and exclusive representation e should jointly contain all information of the original representation x²;
Exclusivity: ② There is no shared information between common representation sf and exclusive representation et, which ensures the exclusivity within each view (intra-Vien View). A There is no shared information between et and e², which ensures the exclusivity between private information of different views (inter- View).
Commonality: ④ The common representations s² and s' should contain the same informa- tion. Equipped with the exclusivity constraints, the common representations are guaranteed only be the same but also contain maximized common information.
The necessity for each criterion is illustrated in Fig. (satisfaction of all the four conditions produces exact disentanglement, and violation of any condition may result in an unexpeeted disentangled representation). Note that, existing (single-view) unsupervised disentanglement focuses learning
2

---

Under review as a conference paper at ICLR 2021
representation identify explanatory factors of variation, which has been proved fundamentally impossible (Locatello et al.] 2019). The goal of the proposed multi-view disentanglement to disentangle mulriple views into the shared and exclusive parts which be well guaranteed as illustrated in definition 2.7 and Fig. 7
Mutual information has been widely used in representation learning (Hjelm al. 2019 Belghazi et al. 2018). In probability theory and information theory, the mutual information two random variables quantifies the "amount of information" obtained about one random variable when observing the other one, which is well-suited for measuring the amount of shared information between two different views. To approach the disentangling goal, according conditions D~C, the general form of the object function is naturally induced as:
I(x'ie'ss') I(e'ss)] - EI(elee) (I(stt) iF
EI(ss:')
max
V==
itj
where I(:;) :) denotes the mutual information. We provide an implementation in Fig. and, in the following subsections, we will describe this implementation in detail.
2.1 CONDITION @: INFORMATION PRESERVATION FOR THE SHARED AND EXCLUSIVE REPRESENTATIONS
How maximize e(z;e,
For simplicity, 2, and æ², s", e¹ are denoted with the same meanings and used alternately, where the former and latter are used for intra-view and inter-view cases, respectively. To preserve the information from the original data in the shared and exclusive representations, the mutual information I(a;e, 6,8) 8) should be maximized. There are different ways implement the maximization of I(x; e, 8) based on the following assumptions.
Assumption 2.1 The shared representation s and exclusive representation e are simultaneously independent and conditionally independent:
p(s,e) p(s)p(e), p(s, elx)) p(s)z)p(e|x).
(2)
Firstly, we expand [(x; e, 8) to obtain the following equation (more details are shown in supple- ment
J p(a)p(e, s|x) log p(e,s)
p(e, s|x) dedsdr.
I(x; e, 8)
Then, under Assumption 2.1 the following equation is derived (more details are shown in supple- ment
I(x; I(x; e) I(a;s).
According to the above equation, it seems that we can maximize I(x;e) + I(x; 8) to maximize e)); which involves making s and e contain as much information from as possible (ideally, it will produce and to meet I(z; e) [(2; s) H (z). where H(x) is the entropy of æ). This actually leads to a strong correlation between s and e, which in conflict with the independence Assumption 2. about and In other words, it is difffcullt to balance the completeness (condition
①) and intra-view exclusivity (condition ( (see experimental results in supplement B.4..
Fortunately, there is an alternative strategy which avoids the difficulty in balancing the completeness and intra-view exclusivity. Specifically, we introduce a latent representation r generated by two independent distributions with respect and under mild assumption:
Assumption 2.2 (Relationship between s,e and r):
p(s,e,x) = =
(4)
In our formulation, we define f(s, e). where r is derived from and with the underlying function f(-) and satisfies p(r, x = p(s, e). Eq. 4 a mild assumption, for example the invertibility of mapping r = f(8, e) ensuring sufficient condition which be easily verified. Note that {s, e)
3

---

Under review as a conference paper at ICLR 2021
is one special case and will be discussed later. Based on Assumption 2.2 are shown in supplement C
get (more details
p(r)= p(,, p(rlz) p(s, e|z).
(5)
Then, we can induce the following result (more details are shown in supplement C
e(); I(z;)).
(6)
This result indicates that the maximization of I[æ; s) can be achieved by maximizing the mutual information of agency r and z. In this way, the independence
e and & is well preserved and the previous conflict is dispelled. Next, we will explain how to encode the information of into independent representations s and e by introducing the agency r.
How to obtain independent representations and by maximizing [(x;))
First, we consider encoding the observed data æ into a latent representation r by maximizing the mutual information between and r. Considering robustness and effectiveness (Alemi et al. 2018),
we can maximize the mutual information between r and æ through Variational Autoencoders (VAEs)
(Kingma & Welling 2014). Accordingly, we have the following objective function:
gr (r(r(2))
min E pp) (r(x) [Log(d(z]r))] + B, Br (rlar) log 4..d
(7)
p(r)
where d(x/|r) (the "decoder") is a variational approximation to p(z(r), and qr (r(zx) (the "encoder") is a variational approximation to p(rlær) which converts the observed data into the latent representation r.
Second, we consider how obtain independent representations and by modeling qr(rlaz). For this goal, the relationships between and r should be jointly modeled. As shown in Eq, 5 we obtain p(r(z) = p(s, elæ:). Under Assumption 2.1 Eq. 5 can be rewritten as p(r|z) = p(s (()ss)))u ())(e)))) (r). which implies that gr (r(æz)
(z) can be considered as the product of p(s and p(e[z). Furthermore, we introduce PoE (product-of-experts) (Hinton 2002; Wu & Goodman 2018) to model the product 9s x)) and (e(e).z). where the variational networks q, (s|z) and gr(e (e x are designed to approximate p(s and p(elxr). worth noting that the key difference from MVAE (Multimodal Variational Autoencoder) (Wu & Goodman 2018) is that our model obtains the latent representation from two independent components within each single view, while MVAE achieves the unified representation of all views by assuming independence of representations of different views. Under the assumption that the true posteriors for individual factors p(s[z) and p(elz) are contained in the family of their variational counterparts qs (s(x)) and qe (c(rr), we have (r(r))) 9s qs(s)2 (c(x)) With Gaussian distributions, we can obtain the closed-form solution for the product of two distributions: Pr = p.of +ugo
of+ož
of afo2 -z modeling (-(r)x).
Therefore, the independent representations between e and s are well preserved by
Accordingly, with the results qr (r(zz) = qs(x)))) and p(r) = p(s)p(e), the objective in Eq. is rewritten
min E Ex-p(1) Rx
E Eypoq (s12)a [log(d(x\r))]
(s(x)) (elg)(el>) log (e|x)
+ E Bawg (s|m)
log
p(s)
p(e)
where p(s) and p(e) are set to Gaussian distributions, which in turn forces (s|xz) and qe (e)xz) to be closer to a Gaussian distribution, allowing us to find the product of the two distributions. The above objective is actually the BLBO (evidence lower bound) (Kingma & Welling 2014) with the first term being the reconstruction loss, and the second and third terms being the KI. divergence.
The proposed variant of VAE inherits two advantages from VAE and PoE, respectively. The first is that we can obtain approximate distributions of and given æ to preserve the independence, The second is that the proposed model still works even when there is a missing case for e or in the testing. This means that we can use only or e as input to the decoder to reconstruct æ (shown in the experimental section), which is quite different from the concatenation of and or other forms that require e and s simultaneously to obtain r. In addition, the way of concatenating s and e does not well exploit the independent prior of s and e.
4

---

Under review as a conference paper at ICLR 2021
2.2 CONDITIONS 2-:: EXCLUSIVITY
To fulfill conditions 2 and 3, we minimize the mutual information between two variables by enhancing the independence. There are different strategies promote the independence between variables, which are endowed with different properties. Specifically, the straightforward way is to promote the independence by minimizing the linear correlation. Accordingly, we have the following loss function:
c'es²
min 2.
8
||
for condition ④ and similar objective could be induced for condition 3 Although simple and effective, the linearity property may not be powerful enough to handle complex real-world complex correlations. Therefore, we also propose an alternative strategy for general correlation cases in the supplementary material A.1
2.3 CONDITION 4: ALIGNMENT OF THE SHARED REPRESENTATION FROM DIFFERENT VIEWS
For condition ④ we ensure the commonality between st and ş by maximizing the mutual information follows:
I(s / p(s' st) log p(st,s)) ds'ds'
difficult calculate the mutual information, since the distribution is usually unknown. Based on the scalable and flexible MINE (Belghazi et al., (018), we introduce two different strategies for maximizing the mutual information between the shared representations s² [x")) and s ~ (s" (x3) from different perspectives.
MINE can estimate mutual information of two variables by training a classifier distinguish whether the samples come from the joint distribution J or the product of marginals M. MINE actually aims to optimize the tractable lower bound estimate mutual information based on the Donsker- Varadhan representation (Donsker & Varadhan 1983) of the KL-divergence, with the following form:
I(s 5²) 2 E To('', s²)] - -log
Ex
where To is a discriminator function modeled by a neural network with parameters 8. J and M are the joint and product of marginals, respectively. We can maximize the mutual information of s' and sf by maximizing the lower bound.
Although the KL--based MI is effective for tasks, it tends overemphasize the similarity between samples and thus cannot thoroughly explore the underlying similarity between different distributions. To address this issue, could replace the KL divergence with JS divergence (Hjelm ct al. 2019), which can focus on the similarity in terms of different distributions instead of samples. Accordingly, y, we maximize the mutual information of s² and s by the following form:
E, [- sp (-To(s', (TT(ssss)) ('))s Ers ss (To(s ss"))]
d...T.
where s², s² corresponds to one sample with two views, i.e., the ith and jth views, respectively. s corresponds to another sample from the jth view, and sp(=) = log(1 + e*) the softplus function. Specifically, the inner product is employed in the classifier, i..., To b) aTbb. We have discussed these two methods in the supplementary material A.2
RELATED WORK
The disentanglement of representations aims to depict an object through independent factors order to provide a more reliable and interpretable representation (Bengio et al. 2013), Most unsupervised disentangled representation learning methods (Chen et al. 2018 Esmaeili et al. 2019 Kumar et al.) 2018) are based on Variational Autoencoders (VAEs) (Kingma & Welling 2014). Existing VAE- based methods basically increase the independence between different factors of the representation. On the basis of VAES, 3-VAE (Higgins et al. 2017) implicitly obtains promising disentanglement
5

---

Under review as a conference paper at ICLR 2021
performance by increasing 3 in ELBO to constrain the capacity of the latent space. Different from 3-VAE, several methods (Chen et al. 2018 Esmaeili et al. 2019 increase the independence between different factors by minimizing a total correlation (TC) loss. DIP (Disentangled Inferred Prior) (Kumar et al., 2018) encourages disentanglement by introducing a disentangled prior to constrain the disentangled representation. However, there are theoretical problems in unsupervised disentanglement learning (Locatello et al., 22119) Thus there are also several remi-supervised methods (Kingma et al. 2014; Narayanaswamy et al. 2017 Bouchacourt et al., 2018) for disentanglement representation
learning which have access to partial real factors of the data. There are also various real-world
applications using disentangled representations (Gonzalez Garcia et al., 2018 Liu et al. 2018).
Miew representation learning aims to jointly utilize information from multiple views for better performance. To jointly learn a unified representation between multiple views, CCA-based (Hotelling 1992 Akaho, 2006; Andrew et al.) 2013) Wang et al., 2015) algorithms maximize the correlation between different views to extract shared information. KCCA (Akaho 2006) and DCCA (Andrew al. 2013) extend the traditional CCA using kernel and deep neural networks, respectively. DCCAE (Wang et al., 2013) jointly considers the reconstruction of each single view and the correlation across different views. To jointly encode the shared and view-specific information, some latent- representation--asee models (Zhang et al.,,2011 (2019) have been proposed. There are also models (Wul et al. 2019: Suzuki al., 2016; Vedantam al., 2018) that employ VAE learn unified multi-modal representation.
4 EXPERIMENTS
Experimental Settings. We conduct quite comprehensive experiments to evaluate the disentangled representation. Specifically, we investigate the disentanglement quantitively by conducting clustering and classification (section D., and provide visualization results to intuitively evaluate the disentan- glement (section 4.2 and section B.5). Furthermore, we conducted ablation experiments (section B.3) and present application experiments based on the disentangled representation (section B.6 Due to the space limitation, some experiments are detailed in the supplementary material.
Datasets: Similar to the work (Gonzalez- Garcia al. 2018), construct the dataset MNIST- CBCD, comprising MNIST-CB (MNIST with Colored Background) and MNIST-CD (MNIST with Colored Digit) as two views, by randomly modifying the color of digits and background of digit images from the dataset MNIST. Intuitively, the shape of a digit corresponds to the shared information, while the colors of background and digit correspond to the private information within each view. The same strategy is applied to FashionMNIST. We also conduct experiments on the face image dataset CelebA (Liu al. 2015), which is large-scale face-attributes dataset with more than 200K celebrity images, each of which is with 40 attribute annotations. The image and attribute domains are considered as two different views. We select the 18 most significant attributes as the attribute vector (Perarnau et al.) 2016). For these two views, the shared information are atribbute-elattdd information (e.g., viewpoint, hair color, w/ or w/o glasses etc), and the exclusive representation is non-attribute information.
To verify the disentanglement, we compare our algorithm with: (1) Raw-data (Raw), which reshapes
images directly into vectors representations; (2) Variational autoencoders (VAE (Kingma &
Welling 2014)), which uses VAE to extract features from the data of each view; (3) CCs-
methods (CCA (Hotelling 1992). KCCA (Akaho 2006). DCCA (Andrew et al.) 2013) and DCCAE (Wang et al., 2013)). which obtain the common representation by maximizing the correlation between different views. (4) Multimodal variational autoencoders (MVAE (Wu al., (019)), which can learn a common representation of two views. For Raw-data and VAF, we report the clustering results using the representations obtained by view-1, view-2, and representation by concatenating view- I and view-2. In our method, we use s!, s² and the concatenated representation for clusteringccassifccction
4.1 QUANTITATIVE ANALYSIS
To evaluate the disentanglement of our algorithm, we conduct clustering and classification based on the representations, respectively. For simplicity, we employ k-means as the clustering algorithm, since k-means is based on the Euclidean distance, which makes it more objective in measuring the quality of representations. KNN (K-Nearest Neighbour) and linear SVM are employed to conduct

---

Under review as a conference paper at ICLR 2021
Raw
VAE
CCA-basee
Ours
MVAE
viewl
view2
Concat
view1
view2
Concat
KCCA
DCCA
DCCAE
view
view2
Concat
ACC 11.
26.8
12.2
35,4
38.5 56.2
46,9
48.2
40.2
47.4
50,4
62.0
62.1 62.4
NMr 0.30
18.2
0.50
25.0
30.6 50.2
54.2
52.9
51.3
51.9
41.9
54.8 56.4 57.0
ACC 15.1
36.8
28.7
42.5
45.6
52.0
48.6
46.2
42.5
46.4
51.8
56.1"
56.3
F
NMI 3.90
32.3
24.5
41.1
45.1
55.1
55.1
56.7
52.8
54.0
54.1
58.5" 59.4"
Table 1: Comparison on the clustering task. 'M' and 'F' indicate MNIST-CBCD and FMNIST-CBCD, respectively. The top three results are in bold and marked with superseript.
Raw
VAE
Ours
MVAE
viewl
view2
Concat
view
view2
Concat
KCCA
DCCA
DCCAF
view
view2
Concat
LSVM KNN
M 77.8 90.6*
77.9
78.7
89.3
79.2
83.0
55.5
52.9
59.5
87.2
88.1
39.9991.2
70.7 75.2 67.8
74.8 78.7 74.9
64.1
53.3
52.8
56.8
73.2
74.2
78.1 77.53
M 82.3
90.4
91.5
85.6
91.0
91.1
83.5
61.8
51.6
64.9
92.4
89.2
90.5
92.2
F 59.7
78.1
79.6
77.9
81.3
81.43
65.1
57.4
47.1
59.0
79.8
79.4
81.9 81.91
Table 2: Comparison on the classification task. 'KNN' and 'LSVM' indicate the K-Nearest Neighbor and linear SVM respectively. 'M' and 'F' indicate MNIST-CBCD and FMNIST-CBCD, respectively
MNIST-CBCD
FMNIST-CBCD
MVAE
Ours
View-1 (e))
8.48
70.80++5.99
top
64.12
95.15
Raw View-1
10.39+10.79
57.65++6.94
top
63.24
94.57
View-2 (c?)
53.30
58.12++4.63
top 10
62.85
94.16
Raw View-2
53.19
51.92:44.80
top 100
61.53
91.96
Table 3: Clustering with exclusive representation.
Table 4: Cross-modal retrieval.
classification experiments based on the shared representations. All experiments run 20 times and the means are reported in terms of accuracy (refer to the supplement for standard deviations).
From the quantitative results in Tables and the following observations are drawn: (1) directly using the raw features for clustering/Classiiaatton is not promising, as the digital and color information mixed. Moreover, since the background region is much larger than the area of the digit, the accuracy of using view-1 is relatively low on MNIST; (2) compared with the raw features, the shared information extracted by our model is competitive due the clear semantic information; (3) by extracting the shared (digit) information explicitly, our model obtains much better results.
Furthermore, we evaluate the exclusive representations clustering. For the MNIST, the colors of background (MNIST-CB: MNIST with Colored Background) or digits (MNIST-CD: MNIST with Colored Digit) are considered as class labels. According Table 3 our algorithm obtains more promising clustering performance with the exclusive representation compared with the raw data, while existing algorithms cannot obtain exclusive representation explicitly. The performance improvement of view-2 (on MNIST-CBCD) is not substantial. The possible reason is that exclusive information (the color of digit) from the images not so significant due to small area ratio of digits, which increases the difficulty of disentanglement.
We verify our disentangled representation with cross-modal retrieval on CelebA (Liu ct al. 2015). Specifically, after training the disentangling networks, we can obtain the shared representations from the image and attribute views, respectively. Therefore, the attribute vector can be used to retrieve the related face images (attribute- speciffc cross-modal retrieval). The quantitative results reported in Table 4 and examples are in Fig. ([a) (in the supplement). Given the specific attributes represented as vector by we can obtain attribute vector 4nk for the 4th most similar retrieved image, which is associated with D attributes (the value of each one 0 or 1). Accordingly, for the top K
retrieved images, we have accuracy- = DN Σ E. 6 6(tars, where ò(a, b) = when a = b, Nx x2
otherwise 5 b) = 0. According to the results in Table the performances of our model are much higher than those of MVAE due the promising disentanglement.
7

---

Under review as a conference paper at ICLR 2021
KV
(a)
(Viow-1) Original
(b) (Viex-1) Shared
(c) (View-1) Exclasive
(d) (View-1) SALE
(View-2) Shared
4
4
S B
7
4
3
4
T
3
3 1
8
7 4 /
3
4
0
7
2
6
0
?
d 5
7 1
9
6
7
/
7 7
6
0
7
3
3
2
6
#
9
3
2
6
6
4
5
3
5
6
4 5 9
5
3
4
4
1
4/ 4
5
5
5
7
5
3
8
9
7
3
4
0
8
9
3
3
4
0
4
&
7
4
7
5
( (View-2) Original
(g) (View-2) Shared
(h) (Vieu-2) Exclosive
(i) (View-2) SAF
(Viow-1) Shared
Figure 3: Visualization of reconstruction with shared and exclusive representations. The top and bottom rows correspond to the reconstruction results from the decoders of view-1 and view-2, respectively. 'Shared', 'Exclusive' and 'S&E' indicate shared, exclusive, and the combination of shared and exclusive representations, respectively. 'View-1' and 'View-2' in the parentheses indicate the view where these representations come from. Note that, the images in (e) (()) are reconstructed results using decoder of view-1 (view-2) using the share representation from view-2 (view-1).
4.2 QUALITATIVE ANALYSIS
We further intuitively demonstrate that our model can well fulfill the four conditions definition 2. with visualization. On MINIST-CBCD, we train the model using the training set and randomly select 64 images in the test set for visual analysis. The disentangled shared and exclusive representations used input decoders corresponding different views reconstruct the original data with different combinations. For example, we can input the shared representation extracted from view-1 into the decoder corresponding view-2 to obtain the reconstructed images from view-2. The visualization of reconstruction results are shown in Fig 3
From Fig. 3 have the following observations, which are consistent with the definition of multi- view disentanglement: (1) By combining the shared and exclusive information, the original image can be fully reconstructed ((d) and (i)), satisfying condition ! (completeness): (2) The shared and exclusive representations contain different information. With the shared representation, we can reconstruct images ((b) and (g)) with clear digit shapes rather than color information as in the original images. In contrast, with the exclusive representations, we can reconstruct the color information ((c) and (h)) of the original images rather than the digit shapes. This verifies that the condition ② (intra- view exclusivity) is satisfied. (3) The exclusive representations from different views contain different information. Specifically, the exclusive representation (c) from view-1 contains the information of background color, while the exclusive representation (h) from view-2 contains information of the digit color. This verifies that our model satisfies condition 3 (inter-view exclusivity). (4) The shared representations ((b). (g). (e) and (j) from different views contain (almost) the same information, i.e., condition ④ (commonality). We verify this by reconstructing digit shapes in view-2 using the shared representations from view-1 and vice versa. Similar experiments are done on CelebA (section B.5
CONCLUSION
In this work, we proposed a formal definition for disentangling multi-view data, and based on this developed a principled algorithm which focuses on automatically disentangling multi-view data into shared and exclusive representations without supervision. Extensive experiments validate that the proposed algorithm can promote subsequent analysis tasks (e.g., clusterin/llassfiiattoonnee We consistently validated that the proposed algorithm can provide promising disentanglement and thus is quite effective and flexible in analyzing and manipulating multi-view data. We will focus on the semi-supervised setting improve the discriminative ability in the future.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Shotaro Akaho. A kernel method for canonical correlation analysis. arXiv preprint cs/0609071, 2006.
Alexander Alemi, Ben Poole, Ian Fischer, Joshua Dillon, Rif A Saurous, and Kevin Murphy. Fixing broken elbo. In International Conference on Machine Learning, pp. 159-168, 2018.
Galen Andrew, Raman Arora, Jeff Bilmes, and Karen Livescu, Deep canonical correlation analysis. In International Conference on Machine Learning, pp. 1247-1255, 2013.
Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeshwar, Sherjil Ozair, Yoshua Bengio, Devon Hjelm, and Aaron Courville. Mutual information neural estimation. In International Conference on Machine Learning, pp. 530-539, 2018.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(8):1798-1828 2013.
Diane Bouchacourt, Ryota Tomioka, and Sebastian Nowozin. Multi-level variational autoencoder: Learning disentangled representations from grouped observations. In Thirty-Second AAAI Confer- ence on Artificial Intelligence, 2018.
Xiaochun Cao, Changqing Zhang, Huazhu Fu, Si Liu, and Hua Zhang. Diversity-induced multi-view subspace clustering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 586-594, 2015.
Tian Qi Chen, Xuechen Li, Roger B Grosse, and David K Duvenaud. Isolating sources of disentan- glement in variational autoencoders. In Advances in Neural Information Processing Systems, pp- 2610-2620, 2018.
Yunjey Choi. Minje Choi, Munyoung Kim, Jung- Woo Ha, Sunghun Kim, and Jaegul Choo. Star- gan: Unified generative adversarial networks for multi-domain image-to-image translation. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 8789-8797, 2018.
Jung-Woo
Monroe D Donsker and SR Srinivasa Varadhan. Asymptotic evaluation of certain markov process expectations for large time. iv. Communications on Pure and Applied Mathematics, 36(2):183-212. 1983.
Babak Esmaeili, Hao Wu, Sarthak Jain, Alican Bozkurt, N Siddharth, Brooks Paige, Dana H Brooks, Jennifer Dy, and Jan Willem Meent. Structured disentangled representations. In The 22nd International Conference Artificial Intelligence and Statistics, pp.
2019.
Abel Gonzalez-Garcia, Joost de Weijer, and Yoshua Bengio. Image-to-image translation for cross- domain disentanglement. In Advances in Neural Information Processing Systems, pp. 1287-1298, 2018.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with constrained variational framework. In International Conference on Learning Representations, 2017.
Geoffrey E Hinton. Training products of experts by minimizing contrastive divergence. Neural computation, 14(8):1771-1800, 2002.
R Devon Hjelm, Alex Fedorov, Samuel Lavoie- Marchildon, Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. In International Conference on Learning Representations, 2019.
Harold Hotelling. Relations between two sets of variates. In Breakthroughs in statistics, pp. 162-190. Springer, 1992.
Junlin Hu, Jiwen Lu, and Yap-Peng Tan. Sharable and individual multi-view metric learning. JEEE Transactions Pattern Analysis and Machine Intelligence, 40(9):2281-2288, 2017.

---

Under review as a conference paper at ICLR 2021
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167, 2015.
Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. In Yoshua Bengio and Yann
LeCun (eds.), International Conference on Learning Representations, 2014.
Durk P Kingma, Shakir Mohamed, Danilo Jimenez Rezende, and Max Welling. Semi-supervised learning with deep generative models. In Advances in Neural Information Processing Systems, pp. 3581-3589, 2014.
Abhishek Kumar, Prasanna Sattigeri, and Avinash Balakrishnan. Variational inference of disen- tangled latent concepts from unlabeled observations. In International Conference on Learning Representations, 2018.
Alexander Liu, Yen-Chemg Liu, Yu- Ying Yeh, and Yu-Chiang Frank Wang. A unified feature disen- tangler for multi-domain image translation and manipulation. In Advances in Neural Information Processing Systems, pp. 2590-2599, 2018.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of the IEEE International Conference on Computer Vision, pp. 3730-3738, 2015.
Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Raetsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning of dis- entangled representations. In International Conference on Machine Learning, pp. 41 14-4144, 2019.
Siddharth Narayanaswamy, Brooks Paige, Jan-Willem Van de Meent, Alban Desmaison, Noah Goodman, Pushmeet Kohli, Frank Wood, and Philip Torr. Learning disentangled representations with sepi-super deep generative models. In Advances in Neural Information Processing Systems, pp. 99925-5935, 2017.
Guim Perarna,, Joost Van De Weijer. Bogdan Raducanu, and Jose M Álvarez. Invertible conditional gans for image editing. arXiv preprint 1x1.vv6611.666 2016.
Prajit Ramachandran, Barret Zoph, and Quoc V Le. Searching for activation functions. arXiv preprint arXiv:1710.05941, 2017.
Jérémie Sublime, Basarab Matei, and Pierre-Alexande Murena. Analysis of the influence of diversity in collaborative and mieti-view clustering. In International Joint Conference on Neural Networks, pp. 4126-4133. IEEE, 2017.
Masahiro Suzuki, Kotaro Nakayama, and Yutaka Matsuo. Joint multimodal learning with deep generative models. arXiv preprint arXiv:1611.0188 1.001191 2016.
Qiaoyu Tan, Guo-Xian Yu, Jun Wang. Carlotta Domeniconi, and Xiangliang Zhang. Individuality- and commonality- baadd multiview multilabel learning. IEEE Transactions on Cybernetics, pp- 1-12, 11 2019.
Y Tsai, P Liang, Zadeh, L Morency, and R Salakhutdinov. Learning factorized multimodal representations. In International Conference on Learning Representations, 2019.
Ramakrishna Vedantam, Ian Fischer, Jonathan Huang, and Kevin Murphy. Generative models of visually grounded imagination. In International Conference on Learning Representations, 2018.
Weiran Wang, Raman Arora, Karen Livescu, and Jeff Bilmes. On deep multi-viev representation learning. In International Conference on Machine Learning, pp. 1092-1092, 2015.
Mike Wu and Noah Goodman. Multimodal generative models for scalable weakly-superviee learning. In Advances in Neural Information Processing Systems, pp. 5575-5585, 2018.
Xuan Wu, Qing-Gue Chen, Yao Hu, Dengbao Wang, Xiaodong Chang, Xiaobo Wang, and Min-Ling Zhang, Multi-view multi-label learning with view-specific information extraction. In Proceedings of the 28th International Joint Conference on Artificial Intelligence, pp. 3884-3890, 2019.
Changqing Zhang, Yeqing Liu, and Huazhu Fu. Ae2-nets: Autoencoder in autoencoder networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2577-2585, 2019.
10

---

Under review as a conference paper at ICLR 2021
Supplemental Materials: Multi-Vies Disentangled Representation
A SUPPLEMENTAL MATERIAL FOR METHODS
A.1 SUPPLEMENTAL MATERIAL FOR CONDITIONS 2--
Inspired by Choi al. 2018 introduce classifier distinguish these (independent) representa- tions generated by the encoders. The loss function of the classification can be defined as:
min BR - p(z) log C(z[R)d2),
where C is a classifier that distinguishes the representation from different sources (independent representations), R is from representation set with different sources, e.g., private representation e and e from different views, and q corresponds to the encoder of different views. is a label which indicates the source of R.
Generally, it is difficult to strictly guarantee the independence; however, the two different strategies can promote the independence between different representations (generated by the encoders) to certain extent. We implement both strategies and similar results are observed in practice.
A.2 DISCUSSION OF CONDITION
Both KL- and JS-based estimators maximize the mutual information. However, due to the different properties of the KI. and IS divergence, the two estimators are suitable for different scenarios. Since the JS divergence is bounded, in theory, it prevents the estimator from overemphasizing the similarity of two representations for the same sample (even if they are exactly the same, it will not obviously reduce the loss). This prevents the encoder from paying too much attention generating the exact s² coordination with s instead of the overall objective function. In contrast, since the estimator based on the KL divergence unbounded, s* and s are forced to as similar as possible. Although this is not appropriate for most tasks, it helps us to observe whether sf and s² intuitively have high mutual information. For example, we can replace: 8 and o with each other to if they can accomplish the same task (which is demonstrated in the experimental part).
B SUPPLEMENTAL EXPERIMENTS
B. NETWORK ARCIITECTURES
For MNIST and FashionMNIST, two convolutional layers and two fully connected layers are used for the encoder, while we employ two fully connected layers and two deconvolution layers for the decoder. For the face image dataset CelebA, we use four convolutional layers and two fully connected layers to build the encoder for handling the image view, while the decoder is built using two fully connected layers and four deconvolutional layers. For the attribute vector view, three fully connected layers are used construct both the encoder and decoder. The batch normalization Joffe & Szegedy (2015) and Swish activation functions Ramachandran et al. (2017) are used between the convolutional layers.
B.2 DETAILED EXPERIMENTAL RESULTS OF QUANTITATIVE EXPERIMENTS
Due to space limitations, we only report the means of clustering and classification experiments in the text, and here we add their standard deviations in Table 5 and
B.3 ABLATION EXPERIMENTS
To verify the necessity of each criterion in definition 2.1 we conduct experiments the MNIST- CBCD dataset. Specifically, we conduct the similar experiments by removing 2, + and 4 in the objective function, and the corresponding results are shown in Fig. 5 and respectively.
11

---

Under review as a conference paper at ICLR 2021
viewi
view2
Raw
Concat
view
view2
VAE
Concut
CCA
KCCA
LLA-basse
DCCA
DCCAE
MVAE
view1
vsew2
Ous
Concal
M
NMI 0
ACC 11,720
26,822
18:222
0.40
122240
25.0433
35.4::2
38.544
304-1
50242
54.24
46993
52.9=1
48.222
51.342
40243
51.923
47,424
41.923
504=7
020023
54.21
56.411
$7.0
ACC 15.111
36.812
28.112
02.533
45613
52013
18.6:3
06211
42513
16.433
51824
56.123
54913
56.3
NMI 3911
32,322
24541
41.142
45.143
55.143
55.143
56.742
52843
54,022
541422
58.222
58.522
59.4-2
Table Comparison between existing multi-modal representation learning methods and ours on the clustering task. 'M' and 'F' indicate the MNIST-CBCD and FMNIST-CBCD datasets respectively.
viewl
view2
Raw
Coscat
view1
view2
Concat
CCA
KOCA
CCA-hased
DCCA
DCCAB
MVAF
view1
view2
Concor
KNN M
7..8..4
17.98
78.7
1931
79.23.6
83,0:.44
55.59 51.11.6 5951111
87.21.8
88.11
95.91.6
91215.
70.741175214
67.84:11
74.82.8
78.71.6
74.9..3
73.21.9
74.4
78.12.9
7..
LSVM
F
M
82.3+.7
59.7 2
9044.3 9154.3
77.9.88
85.66.6
910+7
513455 61.426
91.14.4
65.1=
83.5577
6 47.1 590+12
6184 31.42.8 6494.9
924
79.8
79.41000
59243
101.9
90.++.
922446
8199
Table 6: Comparison between existing multi-modal representation learning methods and ours on the classification task. 'KNN' and 'LSVM' indicate the K-Nearest Neighbor and linear SVM respectively.
As shown in Fig. due to the removal of the intra-view exclusivity (2), there are shared information between s' and which is clearly validated by the reconstructed images (g) and (h). Similarly, as shown in Fig. after removing the inter-view exclusivity (G), the performance of disentangle- ment becomes much worse. As shown in Fig. 6 after removing the condition ④ we can hardly disentangle the information from different views due the significant difference between the shared representation sf and s².
6
6 3
5
(View-1) Original
(b) (View- 1) Shared
(c) (View- 13 Exxelusive
(d) (View-1) SALE
(e)
(View:2) 2) Shared
3
6 0 5 2
/
6 4 5
6
0
/
2
/
3
0
0 $ 5 3 g
3 6
0
/ 5 2
/
5 4
0
/ 5
2 5
7
15015
0
7
7
O
5 >
7 5
0
1
0
8 3
7 5
7
0
5 7
7
5
7
O
7
6
2
9
8
a
7
6
2
9
9
2
4
4 0
1
8
1
2
9
+
8
2
g
8 a
0
2
4
4 0
1
8
Q
9
4
3 y O 6
8
4
3
7
6
&
p
8
5
3
q
0
0
3 9 5
3
9 O
6
8
4
3
/
C
6
9
9
8
9
7
0
5
7
5
6
9
7
4
9
5
7
7
0
5
7
6
6
9
7
9
R
8
2
9
3
4
4
2
8
7
0
7
6 7
2
7
0
7
7
2
7
0
(1) (View Odiginal
(g)
(View-2) Shared
(h) (View-2) Exclnsive
ii) (Vie-2) SAF
(Vizw-1) Shared
Figure 4: Visualization of reconstruction with shared and exclusive representations after removing the condition ② (intra-view exclusivity between s and e²). (Zoom in for best view).
B.4 VERIFICATION MAXIMIZING I(z;e) ((NIZIINGG[(((e))
+ I(x;; s) IS CONFLICT WITH MINIMIZING I(e; s)
we analyze the reason why we do not maximize I(x; s) and [(x; to realize minimizing I[e; s). We provide qualitatively verification the MNIST-CBCD dataset by only changing the way of maximizing [(x; e, s). According to the experimental results in Fig. 7 we can find that both the shared representation (corresponding to Fig. b))) and (e)) and the exclusive representation (corresponding to Fig. ( and (f)) contain almost the same information from the original views. This actually leads to poor disentanglement performance and also empirically validates the conflict discussed in Section 2.1
In Section 2.1
B.5 SUPPLEMENTAL VISUALIZATION RESULTS ON CELEBA
Furthermore, we conduct experiments on the face amage-attribute dataset: CelebA Liu et al. (2015). The results are shown in Fig. 8 The reconstructed images with the shared representations from
12

---

9 Under review as a conference paper at ICLR 2021
9
(a)
(Niew-1) Original
bb) (Niew-1) Shared
(View- Exchusive
(d) (Niew-ll) S&F
(View-2) Shared
5
0 4
7
3
5
0
7
3 3 9
2
8
8
to
3
A
3
f
2
8
Le
3
4 8
3
@ 2
b
2
A
3
2
8
8
Lo
3
A
3
2
2
Lo
3
4
7
1
2
8
4
④
7
V
8
2
4
7 g
2
2
0
7
V
1
8
4
0
7
4
8
2
8
9
6
R
9
6
9
9
2
8
7
3
4
7
7
3
0
P
7
3
4
4
8
7
8
6
0
5
6
0
6
8
0
5
6
7
6
0
3
6
8
7
8
6
0
5
6
8
6
0
5
6
2
4
7
0
4
&
6
0
9
2
4
7
2
6
0
3
8
0
(1) (View Original
(g) (View-2) Shared
(h) (Niew-2)
(i) (View-2)
i (Niew Shared
Figure 5: Visualization of reconstruction with shared and exclusive representations after removing the condition 3 (inter-view exclusivity between e' and e²). (Zoom in for best view).
3
2
2
(a)
(Viow-1) Original
(b) (View-1) Shared
(Nee-11 Exclasive
dd (View-1) SAE
(View-2) Shared
9
4
8 O
z
B 9 X
6
g
3 9
4
3 0 2
9
4
/
8 7
0
z
8
9
8
0
7
0 1/
3
2
g
0
2
3
1
0
0
0
0 /
0
0
8
9
0
0
7
6 9
8
L
7
4
7 g
9
0
3
0
3
0
8
0
2
3
3
0
3
9
0
3
9
3
0
2
0
4 7
9
0
3
3
0
8
9
5
6
6
2
6
4
7
3
0
2
0
4 7
9
3
5
3
/
3
8
8
3
6
g
4
4
5
4
7
8 0
0
7
8
(f) (View Original
gg)
(View-2) Shared
(h) (View-2) Exclusive
(i) (View22) S&E
(i) (View-1) Shared
Figure 6: Visualization of reconstruction with shared and exclusive representations after removing the condition ④ (commonality between s' and s'). (Zoom in for best view).
2
3
5
2
3
3
3
3
8
à
7
0
8
1
8
8 7
8
2
5
/
5
0
8
Q
1
5
/
9
2
3
Y 2
7
4
2
5
1
9 6 2
2 O
2.2
O
4
(a) (View-1)
(b) (View-1) 1)
(c) (View-1) E (d) (View-2) O
(c) (View-2) S
(1) (View-2) E
Figure 7: Visualization of reconstruction with shared and exclusive representations. "O'' 'S' and 'E' indicate original image, shared representation and exclusive representation respectively. 'View-1' and 'View-2' in the parentheses indicate the view where these representations come from. (Zoom in for best view).
different views ((b) and (e)) are relatively similar, and the results of (b) and (e) both reflect the same underlying attributes, including smile, hair color, hairstyle, gender (condition ④ commonality). The exclusive representation from the image view demonstrates that the reconstructed images correlate little to the attributes. For example, none of the reconstructed people have their mouths open and
13

---

Under review as a conference paper at ICLR 2021
their genders cannot be easily identified (condition @: intra-view exclusivity). By combining the shared and exclusive representations, the original images can be accurately reconstructed (condition D: completeness). Our model recovers the most critical information without emphasizing details because the current task is to obtain good representations for clustering/retrieaal By setting the goal to improve image reconstruction, we can use additional techniques (e.g., using more deeper networks only 4 layers in our implementation, or using adversarial strategy). It is worth noting that there is a small difference from MNIST-CBCD; the information of view-2 (attributes) is actually contained in view-1 1 (images). Therefore, it is rather difficult to reconstruct face images using the exclusive representation from view-2 the reconstructed images are almost all the same (condition (): inter-view exclusivity). The visualization experiments on CelebA further verify that our disentangled representation can promisingly satisfy the four conditions in the definition 2.1
(a) Original
(b) Shared
(c) Exclusive
W
+
(d) (View-1) S&E
(e) (View-2) 2) Shared
(f) (View-2) Excl
Figure 8: Visualization of image reconstruction with shared and exclusive representations. We use the decoder of the image view to reconstruct images by inputting the shared and exclusive representations, where 'Original' indicates the original images, 'Shared' and 'Exclusive' indicates the shared and exclusive representations, respectively. 'View-1 S&E' indicates the combination of shared and exclusive representations. Similarly, 'View-2 shared' and 'View-2 exclusive' indicate the shared and exclusive representations from View-2, respectively, which are used as inputs into the decoder. (Zoom in for best view).
B.6 SUPPLEMENTAL RESULTS FOR ATTRIBUTE-SPEES SPECIFIC CROSS-MODAL RETRIEVAL AND EDITING
In this section, validate the potential use of our multi-view disentangled representation in two real applications: attribute-specific retrieval and attribute- -spebfff editing.
First, we verify our disentangled representation the attribute-specffic face retrieval task on the CelebA et al. al. (2013) dataset. The details are as described in the text, and here we show some examples in Fig. a).
Second, we demonstrate the potential use of our model in attribute -peeufe face editing by manip- ulating the shared representations. The shared representation from the image view allows to
14

---

Under review as a conference paper at ICLR 2021
systrowey Smiingg
chanping
Alale Syeglasnes Fandacree Shregh
Smilo apceoring
Pangg growing
(a) Attribute-specific cross-modal retrieval (label- free for the gallery set).
(b) Attribute-specfie editing.
Figure 9: Example results for face retrieval and face editing.
manipulate the specific properties of an image. The magnitude of the change can be controlled by interpolating between two shared representations. Specifically, to modify the visual properties of person in an image, exclusive mmgeg representations for a given image; (2) modifying the values the attribute vector corresponding to the properties to be changed, and extracting the shared representation (sattribute) from the modified vector; (3) replacing the original shared representation (gmage) with seew where gnew is the linear interpolation between sn Sattribute and smmage' (4) using gnew and Emage as the input to reconstruct the intended image. Representative experimental results are shown in Fig. b)).
can perform the following steps: (1) disentangling the shared (s) (mmage) and
We provide more results of attribute-speciee editing for more attributes. The experimental results shown in Fig. 10
Becoming bald
Makoup removing
Chancing g DICK har
Oponing mouth sighty
Channirgg to blond hoir
OO DF to pol 2din
Figure Example results for attribute-specifie face editing.
C
PROOFS
PROOF OF I(z;e,))= 5 [S [[[[[[[[aaaap_ p(x,,(()))) log p(x.elz)
dsdedx pla.e)
In order to obtain Eq. in Section 2.1 first according the chain rule for mutual information,
can get
I(see,)) I(xxe) I(z;
(10)
where
I(x;e) = / / p(a, eog p(ale) dedr p(x)
(11)
= / p(x,, e(zxe) (e - - p(z, e(z,e) log log
15

---

Under review as a conference paper at ICLR 2021
[(x;s|e) = / p(a, s, log
pla|s, dsdedr p(zle)
(12)
f p(z, s, log
p(r pog e / e) log
Then I(x; s) can be formulated as
I(x; e, = l p(x, s, e) pogss eo e)dsdedx
/ p(z, e) log
= p(z, 8, log p()ps dsddde
/ p(a, (,,,8,) logp())))))))) dsdder
(13)
= =J 8, log p(x|s,e) dsdedæ
p(a))
= [ 8,), log p(a, e) dsdeda.
p(8,e)
C.2 PROOF OF e, I(x;e) - I(r;; 8)
Under Assumption 2.1 we can get p(s. e) = p(s)p(e) and p(s,e|z) p(s)x)p(e(z). Substituting this into Eq. 13 yiclds Eq. in Section 2.1
/ p(z,e,8) log s(e, dedsdr
8)
p(e,s)
| / p()) log ple, s/x) dedsdr
s)e,s)
/ p(s)p(e))))((()) log p(elx)p(s(x) dedsda
p(e)p(s)
(14)
= / p(=)p(elz)p((sz))ll log p(elx)p(s(x) dedsdr
p(E)p(s)
p(z)p(e/z) log p(e|z) dedæ + / / I(x,e) 5(r;s).
p(x) ((x) log p(s|z)
dedæ
p(e)
p(s)
C.3 PROOF OF e( = I(:;r)
First, based on Assumption 2.2 we can get
p(r) / ((r; / p(s,e,z)de== p(s,e),
(15)
and
p(r[x)) p(r,,)) p(0,e,x)) = p(
(16)
p(x) p(x)
Accordingly, Eq. 6 Section 2.1 can be derived as follows
[ e(z,e,s) log p(e, dedsdr
p)e,s)
/ / p(z)p(e, s(r) log p(e,slz) dedsdr s)e,s)
(17)
/ p(z)p(r(z) log p(r(#) drdr r() p(r)
16


Reviews:
[{"review_id": "f4ktmyz_ihL", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work aims to achieve disentangled representations for multi-view data. It explicitly summarizes the conditions for disentangled representations and proposed example solutions to achieve each of these criteria.\n\nIn my aspect, the three conditions for disentanglement is intuitional reasonable and can present insight for other works. However, I have the following concerns on this work:\n\n1. the author presented the three conditions to be necessary conditions for disentanglement and argued them to be 'strict conditions'. However, I am afraid sufficient proof is needed to support these conditions to be necessary. In my opinion,  Information theory can be a candidate perspective to perform your proof, e.g. [1].\n\n2. The proposed model achieves disentanglement on s and e with the product of the expert. So, will the design results in high computational cost at this phase? \n\n3.  The compared baseline methods are not up-to-date.  This work specifically targets disentanglement, it is not fair to compare vanilla CCA methods which simply considers the shard information. VCCA-private [2] can be a good candidate to compare here. \n\n4.  the work in [3] also specifically studies the multi-view disentanglement problem. Can you please discuss this method? It is also good to compare this method in the reconstruction visualization part.\n\n\n[1] Gao, Shuyang, et al. \"Auto-encoding total correlation explanation.\" The 22nd International Conference on Artificial Intelligence and Statistics. 2019.\n\n[2].Wang, Weiran, et al. \"Deep variational canonical correlation analysis.\" arXiv preprint arXiv:1610.03454 (2016).\n\n[3]. Gonzalez-Garcia, Abel, Joost Van De Weijer, and Yoshua Bengio. \"Image-to-image translation for cross-domain disentanglement.\" Advances in neural information processing systems. 2018.", "extra_scores": {}, "extra_text": {}}, {"review_id": "eSBijstRWry", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The goal of this paper is to define multi-view disentanglement in an unsupervised manner. The authors list four principal rules for representation disentanglement, including completeness of combining shared and specific representations, the exclusivity of two specific representations and between specific and shared representation, and commonality of two shared representation. The authors follow the above rules to design a VAE-based model and demonstrate favorable results on image clustering and classification tasks.\n\n*Strengths*\n1. Clear definition of representation disentanglement: as the prior works do not clearly define all constraints, which are ought to be valid for representation disentanglement, the prior models might not able to disentangle the representations into disjoint representations. This paper clarifies the four conditions and formally define the problem with four mutual-information terms. This sets the standard objective function for future works.\n\n2. While the proposed model is based on VAE with modifications, the proposed model can perform favorably against existing works as presented in Table 1-4.\n\n\n*Questions*\n1. Extension to supervised representation disentanglement: I am wondering how to extend the proposed model to the supervised setting so that the semantic of the disentangled feature can be learned for further manipulation? \n\n2. As beta-VAE is also designed for unsupervised disentanglement (although it is single-view), how beta-VAE performs in the tasks demonstrated in Table 1-4?\n\n3. Generalization to more complicated data: I am wondering if the model is able to perform consistent improvement on a larger scale and more complicated dataset, as modeling mutual information in a higher dimension might be more difficult.", "extra_scores": {}, "extra_text": {}}, {"review_id": "eIQH-FMqcz-", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes definition and conditions for unsupervised multi-view disentanglement providing general instructions for disentangling representations between different views. The authors also provide a novel objective function to explicitly disentangle the multi-view data into a shared part across different views and a (private) exclusive part within each view. I have the following comments on the paper.\n\nmajor comments\n1. In literature, there are several ways to estimate mutual information, such as lower bound of JS divergence (Hjelm 2019, Federici 2020), InfoNCE etc. Also there are other types of mutual information estimators that maximize similarities between two views. Could you provide any indication which of them work better than the other in you model?\n\n2. I think the proposed model presented in the paper is inspired by the paper by Gonzalez-Garcia in 2018, where the authors didn't use the maximization and minimization of mutual information. Why a similar criteria based on mutual information performs better than the distance metric?\n\n3. Maximizing mutual information is often considered as a difficult task and needs complicated sampling strategy (negative, hard negative etc). How do you handle those issues in your setting? Some details will be helpful for the community.\n\n4. Without a bottleneck, i.e. assuming unbounded capacity, maximizing mutual information can be trivially solved by setting the underlying function to identity (Xu Ji et al. ICCV 2019). Have you tried any bottlenecking in your case? If not, how do you ensure that the mutual information maximization does not results in a degenerated solution. Do you think including a bootlenecking would improve the results?\n\n5. The experimental evaluation only shows experimental comparison with some baseline approaches. I think it is also worth to compare the proposed methods with the prior works (for example, Gonzalez-Garcia et al. NeurIPS 2018).\n\nBased on my current understanding and the above comments, I currently recommend the paper as \"marginally below acceptance threshold\". I would like to hear clarification on the proposed models and if satisfied would be happy to increase my recommendation.\n\nminor comments\n1. In ICLR 2020, there were few works that proposed to learn mutual information from diverse domains. I think it is worth to provide to have a discussion on them.\n(i) M. Federici et al., Learning Robust Representations via Multi-View Information Bottleneck, ICLR, 2020.\n(ii) M. Tschannen et al., On Mutual Information Maximization for Representation Learning, ICLR, 2020.\n2. I think it worth providing some details on the implementation and architectures in the paper. I would also recommend to share the code.", "extra_scores": {}, "extra_text": {}}, {"review_id": "610yQZnEEsY", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "For multi-view feature learning, this paper presents the definition as well as implementation for unsupervised multi-view disentanglement. Both consistency and complementarity are well investigated. Experiments on clustering and classification tasks validate the effectiveness.\n\nPros:\n(1) The definition in 2.1 is quite straightforward, and the objective function in Eq. (1) is easy to understand.\n(2) To measure the consistency and complementarity, this paper gives the solution based on mutual information.\n(3) The presented experimental results look good.\n\nCons:\n(1) I'm afraid that the consistency and complementarity have been investigated in multi-view clustering. [1,2] focus on affinity matrix based multi-view learning, and they already divided the affinity matrix of each view into the shared and the view-specific representations in an unsupervised manner. The difference lies in the format of input data and how to measure these properties. In view of this, the novelty of this paper is reduced.\n(2) Please give more details about the optimization process.\n(3) This paper focuses on multi-view learning. But the experiments only involve two views. How about the extension to more views. For example, you can conduct experiments on datasets with the number of views larger than 2 to validate the performance.\n(4) In experiments, only s^{1], s^{2} and the concatenated representation are used for clustering or classification. Do you mean that the view-specific features are useless for the downstream tasks? Please give more explanation.\n\n[1] Robust Multi-View Spectral Clustering via Low-Rank and Sparse Decomposition. AAAI 2014\n[2] Exclusivity-Consistency Regularized Multi-view Subspace Clustering. CVPR 2017", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper focuses on disentangled representation learning from multi-view data, which is an interesting and hot topic. However, there are several papers published in the last couple of years (especially in NeurIPS2020 and ECCV2020) solving very similar problems with closely related contributions to this paper. The contributions of this paper compared to all recent works in this space is unclear. Contributions and benefits of individual components in the method are not investigated. Although the method is designed for multi-view settings, the authors run experiments on simple settings with only two views. The experiments seem quite limited and do not show the method's capabilities. The rebuttal does not properly address the reviewers' concerns either. 

The paper received four reviews with three recommending below acceptance threshold (rejection) and one above the acceptance threshold (although this one was the least confident scoring). Given all the above shortcomings and reviewer recommendations I do not recommend acceptance of the paper.



Decision: Reject