Understanding Latent Correlation-Based Multiview Learning and Self-Supervision: An Identifiability Perspective
Authors: Qi Lyu
Xiao Fu
Weiran Wang
Songtao Lu
Venue: ICLR 2022 Spotlight (2022)

Abstract: Multiple views of data, both naturally acquired (e.g., image and audio) and artificially produced (e.g., via adding different noise to data samples), have proven useful in enhancing representation learning. Natural views are often handled by multiview analysis tools, e.g., (deep) canonical correlation analysis [(D)CCA], while the artificial ones are frequently used in self-supervised learning (SSL) paradigms, e.g., BYOL and Barlow Twins. Both types of approaches often involve learning neural feature extractors such that the embeddings of data exhibit high cross-view correlations. Although intuitive, the effectiveness of correlation-based neural embedding is mostly empirically validated. 
This work aims to understand latent correlation maximization-based deep multiview learning from a latent component identification viewpoint. An intuitive generative model of multiview data is adopted, where the views are different nonlinear mixtures of shared and private components. Since the shared components are view/distortion-invariant, representing the data using such components is believed to reveal the identity of the samples effectively and robustly. Under this model, latent correlation maximization is shown to guarantee the extraction of the shared components across views (up to certain ambiguities). In addition, it is further shown that the private information in each view can be provably disentangled from the shared using proper regularization design. A finite sample analysis, which has been rare in nonlinear mixture identifiability study, is also presented. The theoretical results and newly designed regularization are tested on a series of tasks. 

Published as a conference paper at ICLR 2022
UNDERSTANDING LATENT CORRELATION-BASED MULTIVIEW LEARNING AND SELF-SUPERVISION: AN IDENTIFIABILITY PERSPECTIVE
Qi
Xíao Fu"
Weiran Wang Google Inc.
Songtao Lu IBM Research
School of EECS Oregon State Univ.
School of EECS
Oregon State Univ.
Mountain View
Yorktown Heights
ABSTRACT
Multiple views of data, both naturally acquired (e.g., image and audio) and arti- ficially produced (e-g. via adding different noise to data samples), have proven useful in enhancing representation learning. Natural views are often handled by multiview analysis tools, c.g., (deep) canonical correlation analysis [(D)CCA), while the artificial ones are frequently used in self self-supervised learning (SSL) paradigms, e.g., BYOL and Barlow Twins, Both types of approaches often involve learning neural feature extractors such that the embeddings of data exhibit high cross-view correlations. Although intuitive, the effectiveness of correlation- based neural embedding is mostly empirically validated. This work aims under- stand latent correlation maximization-basse component identification viewpoint. An intuitive generative model of multiview data is adopted, where the views are different nonlinear mixtures of shared and private components. Since the shared components are new/distortionnivaarie representing the data using such components is believed to reveal the identity of the samples effectively and robustly. Under this model, latent correlation max- imization is shown to guarantee the extraction of the shared components across views (up to certain ambiguities). In addition, it is further shown that the pri- vate information in each view can be provably disentangled from the shared using proper regularization design. A finite sample analysis, which has been rare in non- linear mixture identifiability study, is also presented. The theoretical results and newly designed regularization are tested on a series of tasks.
deep multiview learning from a latent
INTRODUCTION
One pillar of unsupervised representation learning is multiview learning. Extracting shared infor- mation from multiple "views" (e.g., image and audio) of data entities has been considered major means to fend against noise and data scarcity. A key computational tool for multiview learning is canonical correlation analysis (CCA) (Hotelling, 1936). The classic CCA seeks linear transforma- tion matrices such that transformed views are maximally correlated. A number of works studied nonlinear extensions of CCA; see kernel CCA in (Lai Fyfe, 2000) and deep learning- based CCA (DCCA) in (Andrew et al., 2013; Wang al., 2015). DCCA and its variants were shown to largely outperform the classical linear CCA in many tasks.
In recent years, series of self-supervised learning (SSL) paradigms were proposed. These SSL approaches exhibit a lot of similarities with DCCA approaches, except that the "views" are noisy data "augmenented" from the original clean data. To be specific, different views generated by distorting data-e.g. using rotating, cropping, and/or adding noise to data samples (Dosovitskiy al., 2015; Gidaris et al., 2018; Chen al., 2020; Grill et al., 2020). Then, neural encoders employed to map these artificial views to embeddings that are highly correlated across views. This genre-whicl will be referred to as artificial multiview SSL (AM-SSL)-includes some empirically successful frameworks, e.g., BYOL (Grill et al., 2020) and Barlow Twins (Zbontar et al., 2021).
"Contact information: Q. Lyu and X. Fu: y xiao. FuJBoregonstate..du W. Wang: wei rawwang@tticcccc edu. S. Lui songtanli imm.. 1mm. com.

---

Published as a conference paper at ICLR 2022
Notably, many DCCA and AM-SSL approaches involve (explicitly implicitly) searching for highly correlated representations from multiple views, using neural feature extractors (encoders). The empirical success of DCCA and AM-SSL bears an important research question: How to under- stand the role of cross-view correlation in deep multiview learning? ? Furthermore, how to use such understanding to design theory-backed learning criteria to serve various purposes?
Intuitively, it makes sense that many DCCA and AM-SSL paradigms involve latent correlation max- imization in their loss functions, as such loss functions lead to similar/identical representations from different views-which identifies view-invariant essential information that is often identity- revealing. However, beyond intuition, theoretical support of latent correlation based deep multiview learning had been less studied, until recent works started exploring this direction in both nonlinear CCA and AM-SSI (see, e-g., (Lyyu & Fu, 2020; Von Kügelgen et al., 2021; Zimmermann et al., 2021; Tian et al.. 2021; Saunshi et al., 2019; Tosh et al., 2021)), but more insights and theoretical underpinnings remain to be discovered under more realistic and challenging settings. In this work, we offer understanding the role of latent correlation maximization that is seen in number of DCCA and AM-SSL systems from a nonlinear mixture learning viewpoint-and use such under- standing to assist various learning tasks, e.g., clustering, cross-view translation, and cross-sample generation. Our detailed contributions are:
ii) Understanding Latent Correlation Maximization Shared Component Identification. We start with a concept that has been advocated in many multiview learning works. In particular, the views are nonlinear mixtures of shared and private latent components; see, e.g., (Huang et al., 2018; Lee et al., 2018; Wang et al., 2016). The shared components are distortion/view invariant and identity. The private components and view-specific nonlinear mixing processes de- termine the different appearances of the views. By assuming independence between the shared and private components and invertibility of the data generating process, we show that maximizing the correlation of latent representations extracted from different views leads to identification of the ground-truth shared components up to invertible transformations.
(ii) Imposing Additional Constraints Private Component Identification. Using the understand- ing to latent correlation maximization-type loss functions in DCCA and AM-SSL, we take step further. We show that with carefully imposed constraints, the private components in the views can also be identified, under reasonable assumptions. Learning private components can facilitate tasks such cioss-view and cross-sample data generation (Huang al., 2018; Lee et al., 2018).
(iii) Finite-Sample Analysis. Most existing unsupervised nonlinear mixture identification works, c.g., those from the nonlinear independent component analysis (ICA) literature (Hyvarinen & Morioka, 2016; 2017; Hyvarinen et al., 2019; Khemakhem et al., 2020; Locatello et al., 2020; Gre- sele et al., 2020), are based infinite data. This is perhaps because finite sample analysis for unsupervised learning is generally much more challenging relative to supervised cases- and there no existing "universal" analytical tools. In this work, we provide sample complexity analysis for the proposed unsupervised multiview learning criterion. We come up with a success metric for char- acterizing the performance of latent component extraction, and integrate generalization analysis and numerical differentiation to quantify this metric. To our best knowledge, this is the first tinite-sapll analysis of nonlinear mixture model-based multiview unsupervised learning.
(iv) Practical Implementation. Based on the theoretical understanding, we propose a latent based multiview learning criterion for extracting both the shared com- ponents and private components. To realize the criterion, a notable innovation is minimax neural regularizer that serves for extracting the private components. The regularizer shares the same pur- pose of some known independence promoters (e.g.. HHilbert-Schmidi Independence Criterion (HSIC) (Gretton et al., 2007)) but is arguably easier to implement using stochastic gradient algorithms.
cerreation-maximizzee
Notation. The notations used in this work are summarized in the supplementary material.
2 BACKGROUND: LATENT CORRELATION IN DCCA AND AM-SSL
In this section, we briefly review some deep multiview learning paradigms that use latent correlation maximization and its close relatives.
2

---

Published as a conference paper at ICLR 2022
2.1 LATENT CORRELATION MAXIMIZATION IN DCCA
DCCA methods aim at extracting common information from multiple views of data samples. Such information expected to be informative and essential in representing the data.
DCCA. The objective of DCCA can be summarized as follows (Andrew et al., 2013):
maximize Tr (EE) (()) (z!!i) ((0) (xl)2) ) s.t. E
((4) (x)(1) f( (z(10)
) (1)
=
where ( RM4 Dy a data sample from view q for q 1,2, Dg the underlying distribution of the gth view, f(1), RM, RD and f(2) RR,, RD are two neural networks. CCA was found particularly useful in fending against unknown and strong view-specific (private) interference (see theoretical supports (Bach & Jordan, 2005; Ibrahim & Sidiropoulos, 2020)). Such properties were also observed in DCCA research (Wang et al., 2015), while theoretical analysis is mostly elusive.
equivalent representation of (1) as follows
E (() (2) (ar (2)
S.L. E f(9) (xl(0) p(9) (al)) =1, (2)
minimize f(a)
which is expressed from latent component matching perspective. Both the correlation maximization form in (1) and the component matching form in (2) are widely used in the literature. As we will see our proofs, although the former popular in the literature (Andrew et al., 2013; Wang al., 2015; Chen et al., 2020), the latter is handier for theoretical analysis.
Slack Variable-Band Based DCCA. In (Benton et al., 2017) and (Lyu Fu, 2020), a deep multiview learning criterion is used:
minimize f(e
Σ E (())
9 =1
s.t. Elgg. 1, E[g,g,] 0.
The slack variable 9 represents the common latent embedding learned from the two views. Con- ceptually, this is also latent correlation maximization (or latent component matching). To see this, assume that there exists (x(()) = g for all (1)) The criterion amounts to learning (f() ))) [f(2) whichh has the maximally attainable correlation.
2.2 LATENT CORRELATION MAXIMIZATION/COMPPNN
MATCHING IN AM-SSL.
Similar to DCCA, the goal AM-SSL is also to find Seuntit--reeallig embeddings of data samples without using labels. The idea is often realized via intentionally distorting the data create multiple artificial views. Then, the encoders are require to produce highly correlated (or closely matched) embeddings from such views. In AM-SSL, the views (1) and (2) are different augmentations (e.g.. by adding noise, cropping, and rotation) of the sample
Barlow Twins. The most recent development, namely, the Barlow Twins network (Zbon- tar et al., 2021) is appealing since it entails a succinct implementation. Specifically, the Barlow Twins network aims to learn a single encoder f RM - RD for two distorted views. The cost function is follows:
D minimize
D
D
Ci where Cij = E [[
(1- + + 1=1
Elllf( (a El[f(x
j
When the learned embeddings are constrained to have zero mean, i.e., E [f(( = 0, Cj is the cross-correlation between ()) and (z)( Note that the normalized representation of cross- correlation Cjj is equivalent to the objective (I) with the orthogonality constraints.
BYOL. The BYOL method (Grill et al., 2020) uses cross-view matching criterion that can be distilled as follows:
minimize E (x)1)) (2)
(2)(((2))) (x)
(4)
3

---

Published as a conference paper at ICLR 2022
where (.) means that the output of the network is normalized. In BYOL, the networks are constructed in a special way (e.g., part of f(2's weights are moving averages the correspond part of f(''s weights). Nonetheless, the cross-view matching perspective still very similar to that in latent component matching in (2).
SimSiam. The loss function of SimSiam (Chen & He, 2021) has similar structure as that of BYOL, but with Siamese network, which, essentially, is also latent component matching.
UNDERSTANDING LATENT CORRELATION MAXIMIZATION
In this section, we offer understandings to latent correlation maximization (and latent component matching) from unsupervised nonlinear multiview mixture identification viewpoint. We will also show that such understanding can help improve multiview learning criteria to serve different purposes, e.g., cross-view and cross-sample data generation.
3.1 MULTIVIEW AS NONLINEAR MIXTURES OF PRIVATE AND SHARED COMPONENTS
We consider the following multiview generative model:
11) ; = (1) (1)
Zf
x = (2))
(2)
21 (2)
(S)
C;
where x; E Ra the éth sample of the qth view for = 1, 2e E IRP is the shared component across views, and c, (() E RDx represents the private information of the gth view-which are the /th samples of continuous random variables denoted by z E RR,, c RD,,, respectively. In addition, RRMM, is an invertible and smooth nonlinear transformation, which is unknown. Additional notes on (5) and the shared-private component-baséd modeling idea in the literature can be found in the supplementary materials (Appendix H). We will the following assumption:
(-)) RD+D,
Assumption (Group Independence) Under (5), the samples z, and (.) are realizations of con- timuous latent random variables z, c( for q = 1,2, whose joint distributions satisfy the following:
>>>()), cl p(c( z Z, cl C.. c(z,c!), (())) p(=)p(c) )pc(!))
(6)
where Z CRR. JRP.CO Cg C RD are the continuous supports of p(z) and p(c(1)). respectively.
Assumption 1 is considered reasonable under both AM-SSI. and DCCA settings. For AM-SSI, the private information can be understood as random data augmentation noise-induced components, and thus it makes sense to assume that such noise is independent with the shared information (which corresponds to the ddentity-revealing components of the data sample). In DCCA problems, the pri- vate style information can change drastically from view to view (e.g., audio, text, video) without changing the shared content information (e.g., identity of the entity) which also shows indepen- dence between the two parts. In our analysis, we will assume that D and D, are known to facilitate exposition. In practice, these parameters are often selected using a validation set.
Learning Goals. Our interest lies in extracting ze and el (up to certain ambiguities) from the views in an unsupervised manner. particular, we hope to answer under what conditions these latent components can be identified and to what extent. As mentioned, ž, is view/distortion- invariant and thus should be identity The ability of extracting may explain DCCA and AM-SSL's effectiveness. In addition, the identification of (g) and the mixing processes may help generate data different views.
3.2 A LATENT CORRELATION -AAIIIINN----DDDDD LEARNING CRITERION
Given observations from both views (a)()) 2 Te=1 generated from (5), we aim to understand how latent correlation maximization (or latent component matching) helps with our learning goals. To
(2) N
4

---

Published as a conference paper at ICLR 2022
this end, we consider the following problem criterion:
maximize 'Ir
11)
(1)
(2)
(2)
(7a)
f()
subject to f(a)
for g = 1, 2 are invertible,
(7b)
N
N
1
44)
(4)
Js
(9)
(9)
S
(4)
(9) 2 = 0, q
1,2,
(7c)
1=1
E=1
Ss
(1)
F
(1)
11
p(o)
(x)
(0)
1,2;
(7d)
where f(a) RM4 + RDD+Da for 1, are the feature extractors of view q. We use the notations
(1)
(9)
= ((x) (x()) ]
((()
DALD+D," -= 1,2,
L.D
to denote the encoder-extracted shared and private components for each view, respectively. Note that designating the first D) dimensions of the encoder outputs to represent the shared information is without loss of generality, since the permutation ambiguity is intrinsic.
The correlation maximization objective is reminiscent of the criteria of learning paradigms such as DCCA and Barlow Twins. In addition, under the constraints in (7), the objective function also equivalent to shared component matching that is similar to those used by BYOL and SimSi i am, i.e.,
max Tr flu)
Σ N
fs
(1)
(1) a
JS
(2)) zt
min fla)
(1) zi
SS
(2)
(2) 2
To explain the criterion, note that we have a couple of goals that we hope to achieve with f and f First, the objective function aims maximize the latent correlation of the learned shared components, i.e., S( This is similar to those in DCCA and AM-SSL, and is based on the belief that the shared information should be identical across views. Second, in (7d), we ask f(9) and f(() to be statistically independent for q 1. This promotes the disentanglement of the shared and private parts of each encoder, following in Assumption 1. Third, the invertibility constraint in (7b) is to ensure that the latent and the ambient data can be constructed from each other-which is often important in unsupervised learning, for avoiding trivial solutions; see, e.g., (Hyvarinen et al., 2019; Von Kügelgen et al., 2021). The orthogonality and zero-mean constraints in (7c) are used to make the correlation metric meaningful. In particular, if the learned components are not zero-mean, the learned embeddings may capture "co-variations" but dominated by some constant terms.
3.3 THEORETICAL UNDERSTANDING
We have the following theorem in terms of learning the shared components:
Theorem (Shared Component Extraction) Under the generative model in (5) and Assumption I,
consider the population form in (7) (i.e, N = co). Assume that the considered constraints hold over all xl4) E Ny for q = 1,2, where X, = { = g ([z
Vz E Z, (49 E Cy).
Denote f(a) any solution of (7). Also assume that the first-order derivative of f(e) g( exists. Then, we have = f( (xx!4) = (2)) no matter if (7d) is enforced or not, where 7(-) RRD RD is an unknown invertible function..
A remark is that 2 7(2) has all the information of due to the invertibility of 7(-). Theo- rem I clearly indicates that latent correlation maximization/ltent component matching can identify the fiew/distortionninaari tortions. This result may explain the reason why many DCCA and AM-SSL schemes use latent correlation maximization/latent component matching as part of their objectives. Theorem I also in- dicates that one only aims to extract z, the constraint in (7d) is not needed. In the next theorem, we show that our designed constraint in (7d) can help disentangle the shared and private components:
information contained in multiple views under unknown nonlinear dis-
5

---

Published as a conference paper at ICLR 2022
Theorem 2 (Private Component Extraction) Under the same conditions as in Theorem 1, also
assume that (7d) is enforced. Then, we further have ((1) = f( (xx(0) 51 (cl()), where IP is an unknown invertible function.
(1)) RD : +
Note that separating and el may be used for other tasks such as cross-view translation (Huang et al., 2018; Lee et al., 2018) and content/style disentanglement.
The above theorems are based the so-called population case (with N and the X₂ observed). This is similar to the vast majority of provable nonlinear ICA/factor disentanglement literature; see (Hyvarinen & Morioka, 2016; Hyvarinen et al., 2019; Locatello al., 2020; Khemakhem et al., 2020). It is of interest to study the finite sample case. In addition, most of these works assumed that the learning function f(() is a universal function approximator. In practice, considering f(9) E F, where is a certain restricted function class that may have mismatches with g g( function class, meaningful. To proceed, assume D1 D2 and M = M1 = M2 for notation simplicity and:
Assumption Assume the following conditions hold: (a) We have g( G and learn f(a) from F, where the function classes F and 9 are third-order differentiable and bounded.
(b) The Rademacher complexity (Bartlett & Mendelson, 2002) of F = =ffa= RRM - Rffa(x) = [f())las S E FF is bounded by Ry given N samples.
(c) Define g-1 {u: RRM RRD+D (ug(ar) 7(2), us us(())) (u(a:)]I.D) Vo E X2 and any
invertible -(-)- There exists f E F such that sup=Ex, [fs(x) us us)))))l2
(d) Any third-order partial derivative of (h (x))a = (f(() g (x))a resides in [-Ca, Cal for all x E Ny. In addition, [c)*)], E [-Cpp Cpl with 0 < Cp < co for j E (Dy).
Assumption 2 specifies some conditions of the function class F where the learning functions are chosen from. Specifically, (a) and (d) mean that the learning function is sufficiently smooth (i.e., with bounded third-order derivatives); (b) means that the learning function is not overly complex (i.e., with bounded Rademacher complexity); and (c) means that the learning function should be expressive enough to approximate the inverse of the generative function.
Theorem 3 (Sample Complexity) Under the generative model in (5), Assumption I and the suite of conditions in Assumption 2, assume that (ar (l) for = 1,..., N are i.i.d. samples of (x(+), (())), Denote f(a) any solution of (7) with the invertibility constraint satisfied. Then, have the following holds with probubility of at least
E
Σ
Σ
D
(o
- -0( + 1og(1/6//N )
1²))
2/3
for any c(a) E Cg such that -Cp + K2 S c, Cp - Kg for all j [Dy] and all [D], where Cp-Nwy
ド2 = (AC + C /lo((166//2)) + 4/2)
If the metric on the left hand side (8) is zero, then fs (2:()) is disentangled from c( The theorem indicates that with N samples, the metric is bounded by ONN-)/s) In addition, Ry de- creases when increases; e.g., a fully connected neural network with bounded weights satisfies Ry = O -/2) (Shalev-Shwartz & Ben-David, 2014). When Ry increases (e.g., by using more complex neural network), the function mismatch often decreases (since F can be more expressive with a higher y) In other words, Theorem 3 indicates a tradeoff between the expressiveness of the function class FF and the sample complexity. If J comprises neural networks, the expressive- ness is increased (or equivalently. the modeling error is reduced) by increasing the width or depth of networks. But this in turn increases Ry and requires more samples reduce (8). This makes hopes to use a sufficiently expressive learning function, but does not hope to use an excessively expressive one, which is similar to the in supervised learning.

---

Published as a conference paper at ICLR 2022
4 IMPLEMENTATION
Enforcing Group Statistical Independence. A notable challenge is the statistical independence constraint in (7d), whose enforcement is often an Early methods such as (Taleb & Jutten, 1999; Hyvärinen & Oja, 2000) may be costly. The HSIC method in (Gretton et al., 2007) which measures the correlation of two variables in a kernel space can be used in our framework, but kernels some- times induce large memory overheads and are sensitive to parameter (e.g., kernel width) selection.
In this work, we provide simple alternative. Note that if two variables X and are statistically independent, then we have p(X; X, Y)= = (Y)) 3 E(p(X)+(Y) E [+()))E(T(()) for all measurable functions <f:) -> and T():
(Gretton et al., 2005). Hence, enforce group independence between variables g and c we propose exhaust the space of all measurable functions ((9) RRD R and 7(1) RR + R. such that
sup R( sup Φfo), (()
|Corfolo) (s(o)) ) (1 ((blo (MMM))))))) (Mrooa))
minimized, It is not hard show the following (see the proof in the supplementary material):
Proposition 1 In (9), if supato at0, ((1) R( 0 over all measurable functions gl and T(0), then, any [2(0]] and [ for E [D] and j E [D,] are statistically independent.
In practice, we use two neural networks to represent (69) and ( respectively, which blends well with the neural encoders for algorithm design.
Reformulation and Optimization. We use deep neural networks to serve as f()), We introduce
slack variable uy and change the objective to minimizing L. = Σ ue S (o) )13 like in
(3). The slack variable can also make orthogonality and zero-mean constraints easier to enforce. A reconstruction loss V/- = - | (9) (f(a) (ar (4) 1> is employed to promote invertibility of
f f( where (9) is a reconstruction network. Let collect the parameters of f(r) and r(0), and r the parameters of ( -(0), The overall formulation is:
N Σ u =I, N
N
min max L+ 3V + AR, s.t. U
ut = 0.
(10)
N
U. [u...... uul E RDXN, R = E2=, and B, 2 0. We design an algorithm for handling (10) with scalable updates; see the supple- mentary materials for detailed implementation and complexity analysis. We should mention that using reconstruction to encourage invertibility is often seen in multiview learning; see, c.g., (Wang et al., 2015). Nonetheless, this needs not be the only avertibility-encorragi realizations such as flow-based (e.g., (Kingma & Dhariwal, 2018)) and entropy regularization-ban approaches (e.g., (Von Kügelgen al., 2021)) are also viable- see more in Appendix
where L =
method. Other
RELATED WORK- NONLINEAR ICA AND LATENT DISENTANGLEMENT
Other than the DCCA and AM-SSL works, our design for private and shared information separation also draws insights from two topics in unsupervised representation learning, namely, nonlinear ICA (nICA) (Hyvarinen Morioka, 2016: 2017: Hyvarinen al., 2019; Khemakhem al., 2020) and latent factor disentanglement (Higgins et al., 2017: Kim & Mnih, 2018; Chen et al., 2018; Zhao al., 2019; Lopez et al., 2018), which are recently offered a unifying perspective in (Khemakhem et al., 2020). The nICA works aim to separate nonlinearly mixed latent components to an individual component level, which is in general impossible unless additional information associated with each sample (e.g., time frame labels (Hyvarinen & Morioka, 2016) and class labels (Hyvarinen et al., 2019; Khemakhem et al., 2020)) is used. Multiple views are less studied in the context of nICA, with the recent exception in (Locatello et al., 2020) and (Gresele et al., 2020). Nonetheless, their models are different from ours and the approaches cannot extract the private information from views with different nonlinear models. The concurrent work in (Von Kügelgen al., 2021) worked on

---

Published as a conference paper at ICLR 2022
Raw Proposed Clust. Ace= 37.4% Chust. Ace- 97.0%
DCCA Clust. Ace= 97.0%
Barlow Twins Clust. Ace= 98.0%
BYOL
2 2
Clust. Acc= 95.6%
3 3
* Sc
566
7
9
Figure I: t-SNE of the results multiview MNIST from (Wang et al., 2015). Baselines: DCCA (Wang et al., 2015), Barlow Twins (Zbontar et al., 2021) and BYOL (Grill et al., 2020).
content-style disentanglement under data augmented SSL settings and considered a similar genera- tive model where both shared and private components are explicitly used. A key difference that their model uses identical nonlinear generative function across the views (i.c., (1) g( while consider two possibly different g( In addition, our learning criterion is able to extract the private information, while the work in (Von Kügelgen et al., 2021) did not consider this aspect. None of the aforementioned works offered finite sample analysis. Our independence promoter is reminiscent of (Gretton et al., 2005), with the extension to handle group variables.
EXPERIMENTS
Synthetic Data. We first use synthetic data for theory validation; see the supplementary materials.
6.1 VALIDATING THEOREM SHARED COMPONENT LEARNING
In this subsection, we show that latent correlation maximization (or latent component matching) leads shared component extraction under the model in (5>-whihh be used explain the effectiveness of number of DCCA and AM-SSL formulations. This is also the objective of our formulation (7) if the constraint in (7d) not enforced.
Multiview MNIST Data. For prooffof-concept, we adopt a multiview MNIST dataset that was used in (Wang et al., 2015). There, the "augmented" view of MNIST contains randomly rotated digits and the other with additive Gaussian white noise (Wang et al., 2015); see Fig. 1. This is similar the data augmentation ideas in AM-SSL. Using this multiview data, we apply different multiview learning paradigms that match the latent representations (or maximize the correlations of learned representations) across views. The dataset has 70,000 samples that are 28 28 images of handwritten digits. Note that without (7d), our method can be understood as a slight variant of (3), To benchmark our method, we use a number of DCCA and AM-SSL approaches mentioned in Sec. 2, namely, DCCA (Wang al., 2015), Barlow Twins (Zbontar al., 2021) and BYOL (Grill et al., 2020). We set D = 10, D2 = 20 and D2 = 50 through a validation set. The detailed settings of our neural networks can be found in the supplementary material.
Since we do not have ground-truth to evaluate the effectiveness of shared information extraction, follow the evaluation method in (Wang al., 2015) and apply k-means all the embeddings
(1) !! (x) (2) and compute the clustering accuracy on the test set (see the visualization of (2) in the supplementary materials). In Fig. visualizations of (())ss on test set of 10,000 samples together with the clustering accuracy. All results are averaged over 5 random initializations.
show the t-SNE (Van der Maaten & Hinton, 2008)
By Theorem 1, all the methods under test should output identity representations of the data samples. The reason is that the two views share the same identity information of sample. Indeed, from Fig. I, one can see that all latent porclation-maximizattio based DCCA and AM-SSL methods learn informative representations that sufficiently distinguishable. The "shape" of the clusters are different, which can be explained by the existence of the invertible function 7()). These results corroborate analysis in Theorem 1.
CIFAR10 Data. We also observe similar results using the CIFARIO data (Krizhevsky et al., 2009). The results can be found in the supplementary materials, due to page limitations.
8

---

Published as a conference paper at ICLR 2022
(2)
J
(1)
(2)
Figure 2: Evaluation Cars3D; rows in blue boxes w/ R; rows in green boxes w/o R.
6.2 VALIDATING THEOREM 2 SHARED AND PRIVATE COMPONENT DISENTANGLEMENT
We data generation examples support claim in Theorem 2 2-i.e., with designed regu- larizer R in (10), one can provably disentangle the shared and private latent components.
Cars3D Data for Cross-sample Data Generation. We use the Cars3D dataset (Reed et al., 2015) that contains different car CAD models. For each car image, there are three defining aspects (namely, 'type', 'elevation' and 'azimuth').
We create two views as follows. We assume that given the car type that is captured by shared vari- ables z and the azimuths that are captured by c(a), the generation mappings g()) and g produce images with low elevations and high elevations, respectively (so they must be different map- pings). Under our setting, each view has N = 8, 784 car images. We model z with D = 10. For c(a), we set D2 = D2 = 2. More details about our settings are in the supplementary material.
We use the idea of cross-sample data generation to evaluate the effectiveness of method. To be precise, we evaluate the learned fla)'s by combining ε{9) ( and 2 f( (m)")
and generating 2 = ((E{(( (oc()))T) where p(9) the learned reconstruction network [ef Eq. (10)). Under our model, the shared components and private components are truly disentangled ml4) in the latent domain, this generated sample zi.) should exhibit the same 'type' and 'elevation' as those of atg (4) and the 'azimuth' of a/
Fig. shows our experiment results. The proposed method's outputs are as expected. For example, on the left of Fig. 2, the zt² is extracted from the red convertible, and the eft''s are extracted from the top row. One can see that the generated (,(1), s under the proposed method with R. are all red F.j
convertibles with the same elevation, but using the azimuths of the corresponding cars from the top
row. Note that if R is not used, then the learned c/ may still contain the 'type' or "elevation' information; see the example highlighted with yellow background on the right of Fig. 2. More results in the supplementary material.
dSprites Data and MNIST Data for Cros--smmleeCroos-viee Data Generation. We offer two extra sets of examples to validate our claim in Theorem 2. Please see the supplementary materials.
CONCLUSION
In this work, we provided theoretical understandings to the role of latent correlation maximization (latent component matching) that is often used DCCA and AM-SSL methods from an unsuper- vised nonlinear mixture learning viewpoint. In particular, we modeled multiview data as nonlinear mixtures of shared and private components, and showed that latent correlation maximization en- sures to extract the shared components-whic showed that, with a carefully designed constraint (which is approximated by a neural regularizer), one can further disentangle the shared and private information, under reasonable conditions. We also analyzed the sample complexity for extracting the shared information, which has not been addressed nonlinear component analysis works, to our best knowledge. To realize our learning criterion, we proposed a slack variable-assisted latent correlation maximization approach, with novel minimax neural regularizer for promoting group independence. We tested our method over synthetic and real data. The results corroborated our design goals and theoretical analyses.
believed to identity-revealing
addition, we

---

Published as a conference paper at ICLR 2022
Acknowledgement. This work is supported in part by the National Science Foundation (NSF) under Project NSF 1808159, and in part by the Army Research Office (ARO) under Project ARO W911NF-21-1-0227.
Ethics Statement. This paper focuses on theoretical analysis and provable guarantees of learning criteria. It does not involve human subjects other ethics-related concerns.
Reproducibility Statement. The authors strive to make the research in this work reproducible. The supplementary materials contain rich details of the algorithm implementation and experiment settings. The source code of our Pychon-implemennted algorithm and two demos with real data are uploaded supplementary materials. The details of the proofs of our theoretical claims are also included in the supplementary materials.
REFERENCES
Galen Andrew, Raman Arora, Jeff Bilmes, and Karen Livescu. Deep canonical correlation analysis. In International Conference on Machine Learning, pp. 1247-1255, PMLR, 2013.
Raman Arora and Karen Livescu. Multi-vied view cca-based acoustic features for phonetic recognition across speakers and domains. 2013 IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 7135-7139. IEEE, 2013.
Francis R Bach and Michael I Jordan. A probabilistic interpretation of canonical correlation analysis. 2005.
Peter L Bartlett and Shahar Mendelson. Rademacher and Gaussian complexities: Risk bounds and structural results. Journal of Machine Learning Research, 3(Nov):463-483 2002.
Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeshwar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and Devon Hjelm. MINE: Mutual information neural estimation. In International Conference on Machine Learning, volume 80, pp. 531-540. PMLR, 10-15 Jul 2018.
Adrian Benton, Huda Khayrallah, Biman Gujral, Drew Reisinger, Shenmin Zhang, and Raman Arora. Deep generalized canonical correlation analysis. In RepLANLP at ACL., 2017.
J Douglas Carroll. Generalization of canonical correlation analysis to three or more sets of vari- ables. In Proceedings of the 76th annual convention of the American Psychological Association, volume 3, pp. 227-228, 1968.
Ricky TQ Chen, Xuechen Li, Roger B Grosse, and David K Duvenaud. Isolating sources of disen- tanglement in variational autoencoders. In Advances in Neural Information Processing Systems, pp. 2610-2620, 2018
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International Conference Machine Learning, pp. 1597-1607. PMLR, 2020.
Xinlei Chen and Kaiming He. Exploring simple Siamese representation learning. In Proceedings of Conference on Computer Vision and Puttern Recognition, pp. 15750-15758, 2021.
Thomas M Cover. Elements of information theory. John Wiley & Sons, 1999.
Darmois. Analyse des liaisons de probabilité. In Proc. Int. Stat. Conferences 1947, pp. 231, 1951.
Richard A Davis, Keh-Shin Lii, and Dimitris N Politis. Remarks on some nonparametric estimates of a density function. In Selected Works of Murray Rosenblatt, pp. 95-100. Springer, 2011.
Paramveer Dhillon, Jordan Rodu, Dean Foster, and Lyle Ungar. Two Step CCA: A new spec- tral method for estimating vector models of words. In Proceedings of the 29th International Conference on Machine Learning (ICML-12), pp. 1551-1558, New York, NY, USA, July 2012. Omnipress.
Alexey Dosovitskiy, Philipp Fischer, Jost Tobias Springenberg, Martin Riedmiller, and Thomas Brox. Discriminative unsupervised feature learning with exemplar convolutional neural networks. 2015.
IEEE Trans. Pattern Anal. Mach. Intell.
10

---

Published as a conference paper at ICLR 2022
John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research, 12(ul))2121-2159, 2011.
Ronald A Fisher. Frequency distribution of the values of the correlation coefficient in samples from an indefinitely large population. Biometrika, 10(4):507-521, 1915.
Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. In International Conference on Learning Representations, 2018.
Yunchao Gong, Qifa Ke, Michael Isard, and Svetlana Lazebnik, A multi-view embedding space for modeling internet images, tags, and their semantics. International Journal of Computer Vision, 106(2):210-233, 2014.
Luigi Gresele, Paul K Rubenstein, Arash Mehrjou, Francesco Locatello, and Bernhard Schölkopf. The incomplete Rosetta stone problem: Identifiability results for multi-view nonlinear ICA. In Proceedings of UAI 2020, pp. 217-227, 2020.
Arthur Gretton, Alexander Smola, Olivier Bousquet, Ralf Herbrich, Andrei Belitski, Mark Augath, Yusuke Murayama, Jon Pauls, Bernhard Schölkopf, and Nikos K Logothetis. Kernel constrained covariance for dependence measurement. In International Conference on Artificial Intelligence and Statistics, volume 10, pp. 112-119. Citeseer, 2005.
Arthur Gretton, Kenji Fukumizu, Choon Teo, Le Song, Bernhard Schölkopf, and Alex Smola. A kernel statistical test of independence. Advances in Neural Information Processing Systems, 20, 2007.
Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Remi Munos, and Michal Valko. Bootstrap your own latent new approach to self-supervised learning. In Advances in Neural Information Processing Systems, volume 33, pp. 21271 121244, 2020.
Gregory Gundersen, Bianca Dumitrascu, Jordan T Ash, and Barbara E Engelhardt. End-to-end training of deep probabilistic CCA on paired biomedical observations. Uncertainty in Artificial Intelligence, 2019.
Michael Gutmann and Aapo Hyvärinen. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In Proceedings of the 13th International Conference on Artificial Intelligence and Statistics, pp. 297-304. JMLR Workshop and Conference Proceedings, 2010.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerehner. Beta-VAE: Learning basic visual concepts with a constrained variational framework. In International Conference Learning Representations, 2017.
Harold Hotelling. Relations between two sets variates. Biometrika, 28(3/4):2213377, 1936.
Xun Huang, Ming- Yu Liu, Serge Belongie, and Jan Kautz. Multimodal unsupervised image-to- image translation. In Proceedings of the European Conference on Computer Vision, pp. 172-189, 2018.
Aapo Hyvarinen and Hiroshi Morioka. Unsupervised feature extraction by time-contrastive learning and nonlinear ICA. In Advances in Neural Information Processing Systems, volume 29, 2016.
Aapo Hyvarinen and Hiroshi Morioka. Nonlinear ICA of temporally dependent stationary sources. In International Conference on Artificial Intelligence and Statistics, volume 54, pp. 460-469, 20-22 Apr 2017.
Aapo Hyvärinen and Erkki Oja. Independent component analysis: Algorithms and applications. Neural Networks, 13(4-5):41 11:5411-43300 1-430, 2000.
11

---

Published as a conference paper at ICLR 2022
Aapo Hyvarinen, Hiroaki Sasaki, and Richard Turner. Nonlinear ICA using auxiliary variables and generalized contrastive learning. In International Conference on Artificial Intelligence and Statistics, pp. 859-868, 2019.
Mohamed Salah Ibrahim and Nicholas D Sidiropoulos. Reliable detection of unknown cell-edge users via canonical correlation analysis. IEFE Trans. Wireless Commun., 19(6):4170-44) 2020.
Jon R Kettenring. Canonical analysis of several sets of variables. Biometrika, 58(3) 1971.
Ilyes Khemakhem, Diederik Kingma, Ricardo Monti, and Aapo Hyvarinen. Variational autoen- coders and nonlinear ICA: A unifying framework. In International Conference on Artificial In- telligence and Statistics, pp. 2207-22117 PMLR, 2020.
Hyunjik Kim and Andriy Mnih. Disentangling by factorising. In International Conference on Machine Learning, volume 80, pp. 2649-2666600 PMLR, 10-15 Jul 2018.
Diederick P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations, 2015.
Durk P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1x1 convolutions. Advances in Neural Information Processing Systems, 31, 2018.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Pei Ling Lai and Colin Fyfe. Kernel and nonlinear canonical correlation analysis. International Journal of Neural Systems, 10605):36--37,, 2000.
Christian Ledig, Lucas Theis, Ferenc Huszár, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang. et al. Photo-realistic sin- gle image super-resolution using a generative adversarial network. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp. 4681-4690, 2017.
Hsin-Y Ying Lee, Hung- Yu Tseng, Jia-Bin Huang, Maneesh Singh, and Ming- -isun Yang. Diverse image-to-image translation via disentangled representations. In Proceedings of the European Conference on Computer Vision, pp. 35-51, 2018.
Francesco Locatello, Ben Poole, Gunnar Rätsch, Berhhard Schölkopf, Olivier Bachem, and Michael Tschannen. Weakly-superviser disentanglement without compromises. International Confer- ence on Machine Learning, pp. 6348 1348-6359, PMLR, 2020.
Romain Lopez, Jeffrey Regier, Michael Jordan, and Nir Yosef. Information constraints on auto- encoding variational Bayes. Advances in Neural Information Processing Systems. 31:6114-6125, 2018.
Qi Lyu and Xiao Fu. Nonlinear multiview analysis: Identifiability and neural network-assisted implementation. IEEE Trans. Signal Process., 68:2697-2712, 2020.
Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar. Foundations of Machine Learning. MIT press, 2018.
Aaron van den Oord, Yazhe Li. and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv preprint arXiv:1807.03744 2018.
Pushpendre Rastogi, Benjamin Van Durme, and Raman Arora. Multiview LSA: Representation learning via generalized CCA. In Proceedings of the 2015 conference of the North American chapter of the Association for Computational Linguistics: Human Language Technologies, pp- 556-566, 2015.
Scott E Reed, Yi Zhang, Yuting Zhang, and Honglak Lee. Deep visual in Neural Information Processing Systems, 28:1 28:1222-1260 2015.
Advances
12

---

Published as a conference paper at ICLR 2022
Nikunj Saunshi, Orestis Plevrakis, Sanjeev Arora, Mikhail Khodak, and Hrishikesh Khandeparkar. A theoretical analysis of contrastive unsupervised representation learning. In Proceedings of the 36th International Conference Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 5628-5637. PMLR, 09-15 Jun 2019.
Shai Shalev-Shwartz and Shai Ben-David. Understanding machine learning: From theory to algo- rithms. Cambridge university press, 2014.
Richard Socher and Li Fei-Fei. Connecting modalities: Semi-supervised segmentation and anno- tation of images using unaligned text corpora. In 2010 IEEE Computer Society Conference Computer Vision and Pattern Recognition, pp. 966-973. IEEE, 2010.
Anisse Taleb and Christian Jutten. Source separation in post-nonlinear mixtures. IEEE Trans. Signal Process., 47(10):287722220 1999.
Yuandong Tian, Xinlei Chen, and Surya Ganguli. Understanding self-supervised learning dynam- ics without contrastive pairs. In Proceedings of the 38th International Conference on Machine Learning. volume 139 of Proceedings of Machine Learning Research, pp. 18 18-24 Jul 2021.
PMLR,
Christopher Tosh, Akshay Krishnamurthy, and Daniel Hsu. Contrastive learning, multi-view redun- dancy, and linear models. In Algorithmic Learning Theory. pp. 1179-1206. PMLR, 2021.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-SNE. Journal of Machine Learning Research, 9(1 (11), 2008.
Julius Von Kügelgen, Yash Sharma, Luigi Gresele, Wieland Brendel, Bernhard Schölkopf, Michel Besserve, and Francesco Locatello. Self-supervised learning with data augmentations provably isolates content from style. Advances in Neural Information Processing Systems, 34, 2021.
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through align- ment and uniformity the hypersphere. In International Conference Machine Learning, pp- PMLR, 2020.
Weiran Wang, Raman Arora, Karen Livescu, and Jeff Bilmes. On deep multi-view representation learning, International Conference Machine Learning, pp. 1083-1092, 2015.
Weiran Wang, Xinchen Yan, Honglak Lee, and Karen Livescu. Deep variational canonical correla- tion analysis. arXiv preprint 1:X0v:6610.0344 2016.
Ka Yee Yeung and Walter L Ruzzo. Details of the Adjusted Rand Index and clustering algorithms, supplement to the paper an empirical study on Principal Component Analysis for clustering gene expression data. Bioinformatics, 17( 17(9):663-777 2001.
Jure Zhontar, Li Jing, Ishan Misra, Yann Lecun, and Stephane Deny. Barlow Twins: Self-supervised learning via redundancy reduction. In Proceedings of the 38th International Conference on Ma- chine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 12310-12320. PMLR, 18-24 Jul 2021.
Shengjia Zhao, Jiaming Song, and Stefano Ermon. IAF: Balancing learning and inference in variational autoencoders. Proceedings of the AAAI Conference on Artificial Intelligence, 33(01): 5885-5892, Jul. 2019.
Jun- Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial nerworks. In Proceedings of the IEEE International Conference on Computer Vision, pp. 2223-2232, 2017.
Roland S. Zimmermann, Yash Sharma, Steffen Schneider, Matthias Bethge, and Wieland Brendel. Contrastive learning inverts the data generating process. In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 12979-12990. PMLR, 18-24 Jul 2021.
13

---

Published as a conference paper at ICLR 2022
Supplementary Materials
A NOTATION
The notations used in this work are summarized in Table A.I.
Table A.I: Definition of notations.
Notation
Definition
æ. X
scalar, vector, and matrix
ás Zt
both represent the ith element of vector æ
lij
the æth row, jth column of matrix X probability density function of random variable x
p x
II
and are statistically independent, i.e., p(z,y) p(z)p())
II y
T2 yj for all i, transpose of æ. X
X
J
Jocobian matrix of a vector-valued function
identity matrix with a proper size
f 9 detX
function composition operation
determinant of square matrix X
expectation
variance
Co.
covariance
the integer set (1, (1,2..... NJ
B PROOF OF THEOREM
Theorem (Shared Component Extraction) Under the generative model in (5) and As- sumption consider the population form in (7) (i.e., N co). Assume that the con- sidered constraints hold over all (x()) E Xg for q 1,2, where X, {a(*) (9)
g ( ([zT, (c' Vz Z, Vela) E Cy). Denote f(a) any solution of (7). Also assume that the first-order derivative of f(v) g(t) exists. Then, we have 2 f (4) (z()) 7(*) matter if (7d) is enforced or not, where 7()) RD + R² is certain invertible functions.
We consider the formulation in (7) without (7d). When N = co and x(1) Xg for q 1, 2 are all available, the sample average version of the formulation in (7) becomes the following expected value version:
(E (1) (x)(
(
maximize Tr
(B.la)
f(2
subject to (()) for q 1,2 are invertible,
(B.
(4)
x |
(9)
= I,
Ss (
[ 0, = 1,2
(B.1c)
First, note that under the generative model in (5), the maximum the objective function in (7) D, which is obtained when every corresponding components of the leameed solutions, i.e., ((1)
RM, + P and ((1) IRM, - RD, are perfectly correlated, i.c..
(())
(1) = S x (2)
(B.2)
14

---

Published as a conference paper at ICLR 2022
Indeed, one may rewrite (B.la)
arg min
2Tr
E
ss
(2(")
x
(2)
f
arg min f()
- 2Tr
[E [E$$)
+1
arg
min
||55 (1)
(a())
f
(2)
a
(B.3)
where the second equality holds because the constraint in (B.lc). Note that the criterion in (B.3) admits the optimal solution in (B.2) under our generative model.
Note that one solution to attain zero cost of (B.3)
x (9)
2, q 1,2
However, the question lies in "uniqueness", i.c., can enforcing (B.2) always yield f(() (z(*) = = (up to certain ambiguities)? This is central to learning criterion design, as the expressiveness of function approximators like neural networks may attain zero cost (B.3) with undesired solutions.
Assume that a solution that satisfies (B.2) is found. Denote flo) for = 1,2 as the solution. Combine the solution with the generative model in (5). Then, following equality can be obtained:
(
)
hg
(2)
((1) (d
2
ng
(!22)
(B.4)
where we have
hg (0)
(w(l))
f(a) g()) (wl(i) 1:D
f. g( (w(())
in which
wl9) = [x= (c
(1) and hg are functions of only z-i.e,, the functions f(9) for q 1,2 We hope to show that hg only extract the shared information.
(2)
To show that ng is function of only z but not function of ( we consider the first-order partial derivatives of hg w.r.t. z and c respectively. Namely, we hope show that the matrix consisting of all the partial derivatives of h w.r.t. is full rank while any partial derivatives of ( w.r.t. clI) is zero.
(1)
Therefore, we investigate the Jacobian of (1) which fully characterizes all the first-order partial derivatives of the function hg (1) and hp w.r.t. and c!)) Let us denote the outputs of h!))
f()) (w) as follows:
$ = h
((i)))
(B.5)
2(1)
The Jacobian of 1 (1) can be expressed using the following block form
(1)
j(m)
21
22
15

---

Published as a conference paper at ICLR 2022
where J1 RDxD, J defined as follows
E RDxx,, Jll
RDixD and J RDixD, Jacobian matrices
a a (w)))
o (w))
l [W))]
8 (wn))
0c
BED
J1
J12
@ 1gg
(w
Ing (1))
on (W))
8 blng (w (W)))
881
OeD
B0pi
[N (W))))) dc""
2 (w)),
a 10 (w))
1 (w)
a
UeD
22
a hh (
@ 1 (6()
a h (w)
dzn
0c1
0e
What we hope to show is that J/2 is an all-zero matrix while the determinant of J( is non-zero.
We first show that ( O. Note that (B.4) holds over the entire domain. Hence, we consider any fixed z and c!?) Then for all e(3), the following equation holds:
1)
hg
(1)
c(L)
hg
(B.6)
for all c E C1 with any fixed z and c(
Let us define matrices H (1) and H (2) where
h (0) (w())
①
1. ... D. 1.........
ig
0c, (1)
By taking partial derivatives of Eq. (B.6) wrt. Jacobian:
1 for j 1...., D1. we have the following
02
05
(6
80
8En
däp
d6
de
H (1)
J (2)
86
86 (2)
J (0)
Onx 0D2XD,
ODXD.
(,0(1)
(22(
2,0(0)
de
de
(2000))
0ep
8c (1)
0e
(B.7)
where Ag E (DDx(D+)s) is the Jacobian of h (2) (a) is by the chain rules and (b) is because we take derivatives of constants. The equation above holds for any z and e(2) Hence, the same derivation holds for all and c(2), which leads to the conclusion that the learned h (() (w)()) is not
a function of (()) Note that another possibility that could lead to (B.7) is that h (wl()) always outputs constant. However, this is not possible because each (() f(s) g is invertible function, which implies that any dimension of h19) (w)()) cannot be a constant if (()) is not constant. To be more precise, note that x is generated from (()) that has D + D, dimensions. If there are D dimensions (i.e., h (w)) E RD) that are constants (and thus no information) in the learned generative domain, then (4) cannot be reconstructed from that domain, which contradicts invertibility.
16

---

Published as a conference paper at ICLR 2022
Then, the Jacobian of h (!)) can be re-expressed by (B.7)
H 11
J
ODXDa J
11
Next, we show that the determinant of J(l non-zero. By the structure of the Jacobian J(1, one can see that 2 is function of only but not determined by c(1), where we denote as E (=)) Besides, since (1) is invertible, have det = [ett
(1) det.
(1) 22
+
by the property of determinant for block matrix. It further indicates that detJ (1)
# (so does
det (1) 22 ) which implies that y(-) is invertible function. This proves Theorem 1.
C PROOF OF THEOREM 2
Theorem 2. (Private Component Extraction) Under the same conditions as in Theorem 1,
also assume that (7d) is enforced, we further have č( f (x) 5(1) (c!o)), where g(0) (.) RD, RR is certain invertible function.
Following the proof of Theorem 1, we further consider the expected value version with (7d), i.e.
maximize Tr E
(()) (a S (
(x(21)
(C.
f()
subject to f(a) for 1,2 are invertible, (zx(1) (9) (0)(1)
(C.1b)
s
(4)
I, E
(()) (a!(1) = 0, 4= 1,2,
(C.1c)
(4) 1L (4) (z(1))
(z)
1,2,
(C.Id)
Again, under our generative model, any optimal solution f
satisfies
(()) (1) = f/ (x)
We hope to further show that
1) fp (x()) h
is an invertible function-transformee version of c(!), where h (w)
= f(v) g()) (w)()) D+1.D+D, = Foa) ) (w))))
Recall that we have the following Jacobian matrix for function (1)
(
J
21
where the second row corresponding to function (1)
Since we have shown that both (1) (2) # 0 and det. (1)
* 0 in Section B, we only need to show
that (1()) is an all-zero matrix. To show this, we will use the condition (C.1d). First, it is not hard to see that
1 01 (w0)),
(1)
21
11
(w)
(w)))
D
D1
17

---

Published as a conference paper at ICLR 2022
by chain rules where the first matrix the right hand side is the Jacobian of 2( w.I.t. 2. By (C.Id), we have 2(1) IL 2, which means that we can observe fixed (( (c+²) C,, _ for any fixed C2 with any possible 2. Therefore, at any specific point of c! the following always holds
[
0c:
0,
02
since the numerator is a constant. Note that the above holds for any a(1) with different ट for i [Di], which further means that we actually have
2 (1))
(wa))
BED
0 hpp (1)
())
0
hpp
821
for hp (w) and
Therefore, 1 Op 11)
Consequently, have the following block diagonal form for (1) by combing with Theorem 1, i.e.
O D
J
0Dx D1 [(]
0DixD
By invertibility of fla) and ( we have
( = det.
11 dde.
(1)
* #0,
which implies that z 72) and ((1)
5(1) (c(1) with invertible functions 7(-) and 500 (+),
respectively. The same proof technique applies to (.))
D PROOF OF THEOREM
Theorem Under the generative model (5) and Assumptions and 2, assume that (x(") x solution (7) with the invertibility constraint satisfied. Then, have the following holds with probability of at least 1 5:
(2) for == 1,..,, N are i.i.d. samples of (x(1), x²)). Denote f(n) E F as any
E
ΣΣCO
D D2
(B
=0 -O(DSNN + V1oB(1/8//N +2²)²
2/3 +22)2
(D.1)
(9) S SCp - for all j E [Dy] and all i E [D]. where Ki = (A(3)))) + CoV/oR(1/8)/22) 4u2)1/6).
for any clu) E C, such that -Cp 5
D.1 A LEMMA ON RADEMACHER COMPLEXITY
To derive the Rademacher complexity the loss function, we have the following lemma
Lemma Consider the following function class
H = -(((d()) 2(1)) (1)
D Σ
( (z)
1!(2 (2)22)
d=1
where (()) ((() E F' are defined in Assumption Assume that f(l)) (2601) S Cj for all
f/ E F,, where Cj O. Then, the Rademacher complexity of class H. is bounded by Rx RDC
18

---

Published as a conference paper at ICLR 2022
Proof: First, we have the function få (z()) f (x(2)) bounded within [0, 20,] According
to the Lipschitz composition property of Rademacher complexity (Bartlett & Mendelson, 2002), have
R F) S I.gRv(F)
where Lg is the Lipschitz constant of Φ. Here o(x) x² and Lg = 4C
Combining with the linearity property of the Rademacher complexity (Bartlett & Mendelson, 2002),
we have
RN(H) & RDC
(D.2)
which completes the proof of the lemma. Note that (D.2) is derived by treating f(9) for [D] individual functions. However, f/ for d [D] the first D outputs of the same function wi)ch means that many parameters of these D functions are constrained be identical. Nonetheless, this fact does not affect the inequality in (D.2) since adding confining constraints to
the function class H only reduces the Rademacher complexity (Bartlett & Mendelson, 2002).
D.2 PROOF OF THEOREM 3
First, we bound the true risk on the view matching loss. Consider the regression problem given (1) xx{ as samples, we have
minimize f(
Z
(1)
2
2
By Lemma and (Mohri et al., 2018, Theorem 3.3), have the following hold with probability least 1
5 (x())
(2)
(x)))
f (2)
(2)
x
fs
x
2
1og(1/8)
2 (H) 4C9
2N
By Assumption 2(c), the first term on the right hand side can be bounded
(1)
fs
a
(2)
2
N
1
Σ
(2)")
us
(1)
(x(1)
ug
(2)
(x(2))
-f-² (x)²) {
N
t=1
1
Σ
(1+ 4u3.
N
where the first equality is because there exist (1) G-1 and (2) E G-1 such that ud! (x) (2) (x) 7(2) and the second inequality is by the triangle inequality.
Therefore, by plugging in Ry (H) we have:
[
||
5
(a(2))
-18²) (1)²)) 11: Se
with the definition
log(1/0)
e:= 2 + 8DC,AR + +4c//
2.V
log(1/8)
= =00 2DAA +CI
+ 422
2NV
19

---

Published as a conference paper at ICLR 2022
Next, we use the bound of true risk to bound the energy of the entries of the Jacobian matrix on the left hand side of Eq. (D.1). Denote hg
f og((). We define the error for any individual sample
pair (x)") (1) 1t
~ p (xx1), x .x(2) as
ss (
(1) (1)
2 (2) SS
(a, 1/2 E2:
with Else] S. Define another two pairs of samples, such that
1) hg
2
hs
(1)
+ Aej
ze
Ce
2
(2)
2
=EG-
1) hg
2
Ze
Ze
se
Eg.
where > 0 and ej is the unit vector the ( direction. Then by triangle inequality we have
([o))
zt
+ 1)
hg
(2)
(1g)
2f
- _e, 1) s V + Ver
Define
Vi (19))
h
(1)
1x,, (d))
600 .... cb. bb)
(1) which is a scalar function of oj
with fixed # and
e for ks + j.
a (x]")),
Then the element
can be estimated using the central difference formula as
0e.")
1 Ss (1)))]
hg
(1)
zt (1)
)
hg
(1)
De, 1)1,
+ Ae;
c
2 (L1) + (E2)
0
24
12
Ze
1)
(Tep
2t (1)
hg
1/g
+ Aej
Ce
De,
A vg((5))
2A
(1) (1)
(1) and by intermediate value theorem e A). & E (oh3 A, C/3
(1)
where E
c.j. Go.
(o -
A c
(1)
+ +
Since |z;|| S ||/|| x
|| we have
@ fs (2n(1))
5 +
Veg
A
(5))))
a (1)
2A
6
By taking expectation, we have
a fs (x)))
ElVEl + E[vEil
A2
E
+
de (1)
2A
6
2
+
(g)((")),
A
6
where the second inequality is by Jensen's 's inequality
F(Ved
V(ee Ve,
20

---

Published as a conference paper at ICLR 2022
due to the concavity of vz.
We aim to find the smallest upper bound, i.e.,
A Wy (5))
inf o<A<min Cott
+
(D.3)
11
6
Note that the function in (D.3) is convex and smooth. We have the minimizer
(19(()1)) ))
3vē
1/3
.ii {c, + 2 - -p-
E
which gives us the minimum
ve A
3
1/3
1/3 v0
+ k3
inf
(e S min
2 2
3
Kj
where Hi min Cp + c C 0
1/3
N2 2
3vE (<a))")
then we can bound
fs (z())
3
(5')
1/3
E
e//3
dc, (1)
3
With fixed N, one can choose c 4C, 22DAN C.V bound
/og(1/0) 2N
+ +4v2 which gives the following
1 fs (D)
3 Ca
1/3
log(1/5)
1/3
E
4C,
2DRN + C
+ 4v/2
2
3
2N
1/3
1/6
AC, 2DR + Cjv
log(1/8) 2N
+ +42
Considering all i. pairs, have
a fs (z))
3
Ca
1/3
log(1/8)
1/3
DD1
40,
2DR
+CI
+ 4v²
3
2N
j=
Since || l|2 is upper bounded by || I/,, we have
D ΣΣ
Js
(x)
27
9
D2
Ca
2/3
2DRN + +Or
log(1/5)
+ 42²
2/3
de
(I)
3
2N
==
which completes the proof. The same holds for q by role symmetry.
E PROOF OF PROPOSITION
The claim be proved by contradiction. Take q 1 for example. Suppose that there exists two elements 2(1) and (211) that are dependent but (9) Let
e
T
71 6
21

---

Published as a conference paper at ICLR 2022
which are valid choices. Then, we have Cov(or ((())) E(2 (f E[2 " IETe
Note that by definition of independence, we have
[ _ प | + (1) 11 T.
Hence, x and ((1) ojj
being dependent means that
E o - E|E EEC + 0.
Consequently, one can that
sup ( 2 Covl Cor(or(E(), TI (c(2))
=T Cool3, 0j + O.
(1)(11)
61.71
The above is contradiction to our assumption that holds.
On the other hand, if 2 and are independent for all E [D] and j E [Di]. then we have ( [[ = 0,
for E [D] and j E [Di), for any Φ : -> and
R.
F DETAILED ALGORITHM IMPLEMENTATION
Recall that the proposed criterion is
N
maximize Tr
SS
(F.la)
f(a)
1=1
subject to p(a) for q 1, 2 invertible,
(F.Ib)
1
(9)
(4)
(4) 22
JS
= 0, q = 1,2, (F.lc)
1- 1,2,
(F.Id)
We will neural networks parameterize the functions that aim leam. To move forward, first, as we have shown in the proof of Theorem 1, Eq. (F.1) is equivalent to the following:
N
minimize (22
(1)
((2)
(2)
(F.2a)
N
1=1
N
N
subject to
f
at
(n)
(g)
(4)
=
I.
1
=
0, 1,2
(F.2b)
N
N
t=1
8=1
f(9) for 9 = 1,2 are invertible,
(F.2c)
(9)
((()
( (4)
1=99111 1,2,
(F:2d)
(1) and fs Direetly optimizing Note that we have manifold constraints both neural networks f over such manifold constraints may be costly and challenging. To reduce the difficulty of this con- strained problem, we introduce a slack variable us and recast the formulation in (F22 follows:
(2)
1 N
N
(()) (x (7) 112
2
minimize f0r
L
C
(F.3a)
N
N
1=1
t=1
q=1
N
subject to
T = I,
Us
= 0.
(F.3b)
N
N
t=1
((1) for 2, are invertible,
(F.3c)
(()
(1) 1l (()
(x)")
(1)
q == 1,2,
(F.3d)
22

---

Published as a conference paper at ICLR 2022
Ideally, we hope that ue >zz)) Introducing ue makes the f()) and f( subproblems uncon- strained. This is a commonly used reformulation in neural network based multiview matching (see (Benton et al., 2017: Lyu & Fu, 2020)), which is reminiscent of the MAX-VA VAR formulation of CCA (Kettenring, 1971; Carroll, 1968; Rastogi et al., 2015). Such reformulations oftentimes make algorithm design easier, since the constraints are simplified.
The inveritibility and independence constraints in (F33c) and (F.3d) are also not straightforward to enforce. Instead of directly enforcing the invertibility constraint in (FFcc,, we design a regularization term. Specifically, we use the idea of autoencoder that reconstructs the samples from their latent representations f(1) (x) We define regularizer
N
2
=
1 N
Σ
V
1 N
Σ t=19=1
x - - (1000)))))))))
(F44
as the reconstruction loss, where (1)'s are the reconstruction neural networks. Note that the above term being zero does not necessarily indicate that the function f( is invertible, since this term only imposed on limited number of samples. But in practice, this idea is effective learning invertible transformations also see (Wang et al., 2015; Lyu & Fu, 2020).
To promote the statistical independence constraint in (F.dd), we use the designed independence regularizer, i.e.,
Cov ( (z(o)) ((0) (2(1)
sup R(a) sup (()) V (stw)) v (T(0) (ct))
(F.5)
( (()) ((0)
where σ(9) and (19) are again represented by neural networks.
Let collect the parameters of f f()) and (()), and n the parameters of ((1) and 7(1), Putting all the terms together, our working cost function summarized as follows:
N
N V( + AR(O,n)
min max U.0
Σ
U) + +
(F.6a)
N
N
t=1
(=1
1 _
subject to N
LUT =1, U1 =
(F.6b)
N
t=1
where [u.-... uN] E RDxN and 3 and À are nonnegative and = = R(1).
In terms of algorithm design, propose handle U, 0 and n cyclically when the other two fixed, i.e., alternating optimization (AO).
First, we stochastic gradient descent and ascent for the unconstrained 0 and n subproblems. To proceed, we sample a batch of data indexed by B S [N]. Then, 0 and can be updated by any stochastic gradient based optimizers. e.g., the plain-vanilla stochastic gradient descent/ascent,
0 - 0-7
{ {
1
Σ (VOX()) EEB
7 - 5 (1 (AV,R(0,,))
where 7 and 5 are the step sizes for the updates of 0 and n,, respectively. The stochastic gradients of C, V are defined as follows:
VOL : 1
(oL,(
(F.7a)
B
EEB
VoV = B
(O).
(F.7b)
FEB
23

---

Published as a conference paper at ICLR 2022
In addition, the terms R and V n R defined similarly. Taking the latter example. We have V Eim R, and V R()) is estimated by taking gradient w.r.t. n of the following batch-estimated R( (the same holds for VoR):
R.9)
(F.8a)
Ø
91)
2(9)
Σ
(2(0)
(
(
Σ
T
(1)
(c(
B
B
[B]
FEB
BEB
FEB
Σ
(1()
(99)
1
Σ
Φ
(2())
Σ EEB
(
(c(0)
1
Σ T(1)
(c(
[8]
V
[8]
[8]
(EB
IEB
EEB
R g) = V,RR
(F.8b)
VOR (9) VoR. (1)
(F.8c)
It was shown in (Fisher, 1915) that the correlation coefficient is computed using random samples of Gaussian variables, the estimator in (F.8a) for R(4) is asymptotically unbiased. For other distri- butions, the estimation also works well in practice; see, e.g., DCCA based works in (Wang et al., 2015).
Consider more general stochastic optimizers, e.g., Adam (Kingma & Ba, 2015) and Adagrad (Duchi et al., 201 I). Then, the updates can be summarized as follows:
0 + wwt.opppiiieeeeee...eeeeppppeeee (0, VoL BV (oo
(F.9)
7 + SCD.optimizer (n. (,R).
(F.10)
where (F.10) uses the negative stochastic gradient since an ascending step, while stochastic optimizers are by default descending the objective function.
The U subproblem consists (F.3a) and (F.3b.. can be re-expressed as follows by expanding (F.aa):
1
(0) (x)
ue
N
f=
N
2
= Σ A=1
1
Tr
2urus - 2ue (f$) (z) (1)
+ (2)
+5
+
Σ
fs
(4)
(4)
(g))
(x))
(z)
N
9=1
Note that the first term is constant by (F.3b) and the last term does not involve ug. Then, rewrite the Ue subproblem as
maximize Tr (U (F()) -
(F.11)
subject -UUT= =I, N UI= =
(a This is an orthogonal projection onto the set of row zero-mean and orthogonal matrices. is shown (Lyu & Fu, 2020, Lemma 1) that such projec- tion problem, although nonconvex, can be solved to optimality via mean-removed singular value decomposition (SVD) procedure, i.c.,
where (()) = [$ (x)
U V. with SDT = =VD ((F)+ F(2) ():
(F.12)
where W = IN 11 removes the mean of F()) + F(2), ,, D E RxD holds the singular values, Se E DXD and T E NDD are the left and right orthogonal matrices in the SVD, respectively.
To summarize, an alternating optimization algorithm is summarized Algorithm 1. Notice that use two different batch sizes (denoted as B, and B, in line of Algorithm 1) to construct the gradient estimations for VoL, V and VoR, V R, respectively. The reason is that accurately
24

---

Published as a conference paper at ICLR 2022
Algorithm 1: Proposed Algorithm.
Data: (1))) for & = 1, .N and g 1,2
Result: f(a), p(1)
while stopping criterion is reached do
VNST" with SDT = SVD ( ( F(t) w):
while stopping criterion is not reached do draw random batch B1 and B2;
// [Ba] [B|]
Vol 1
181 Lees VoLs
VaV -- BB] Etes, VoV;;
V + Σ V (9).
// using Bg and (ESa), (F8b)
83
FoR Ei= VoRg,
// using B2 and (F8a), (F8c)
0, VoL + VoV ABOR):
0 scD.optimizer
// descent
10
17 + sc0.optimizer
17 -)
// ascent
end
12 end
Table F.l: Computational complexity of the proposed algorithm in each iteration, where de and d, denote the parameter dimensions of the encoder/reconstruction and the independence-promooing networks, respectively.
Complexity (flops)
Line 2
D(ND²)
Line 5
O dg)
Line 6
O do)
Line 7
((B2
Line 8
O do)
Line 9
((de)
Line 10
(dd))
Overall
O (BB) + (B2 Dd (2ld,,)
estimating of R using (F.8a) often requires relatively large batch size, while small batches may suffice for the gradient estimations of L and V, according to our extensive simulations.
Computational Complexity. Tab. F.1 summarizes the computational complexity of each step. Specifically, line requires computing thin SVD, which requires O D²) flops. Note that this is linear in the number of samples N, and D is the dimension of the shared component, which is often relatively small in practice.
Inside the inner loop line 4-10, lines and construct the gradient estimations w.r.t. and m These two steps cost O(BB, (do) flops. Similarly, lines 7 and 8 use O (2 and (((2))d)) flops, respectively. Note that we have used do and d, to denote the parameter dimensions of the en- codler/reconsttuccionn and the iirependence-prommooe [Ba] are small numbers compared to N (c.g., [B|] = 128. |B&] 512 while N could easily exceed 10"). For line 9 and 10, when first-order stochastic optimizer (e.g., Mliin-vnnllla SGD, ADAM, Adagrad) is used, this step has a computational complexity that is linear terms the network size.
networks, respectively. Typically, [B] and
One can see that all the steps scale linearly with size the neural networks or the sample size, which makes the algorithm easy with large-scale data sets and large-size neural feature extractors.
25

---

Published as a conference paper at ICLR 2022
scatter plot
t-SNE of xx))
t-SNE of xx
1.0
100
50
22
axcis-
-50
-0.5
-50
-100
-1.0
-1.0
0.0
1.0
1100
100
axis-1
axis-1
Figure G.I: Left: 2; middle: I-SNE of ()) right: t-SNE of x2)))
G EXPERIMENTS: MORE DETAILS AND ADDITIONAL VALIDATIONS
In this section, we show all the experiment results with greater details, e.g., results under more metrics, more setting details, and more demonstrations. We also include more real data experiments using the CIFARIO (Krizhevsky et al., 2009) and dSprites data (Higgins et al., 2017).
G.1 SYNTHETIC DATA VALIDATING MAIN THEOREMS
In this subsection, we describe the synthetic data experiments. For synthetic data, we generate the shared z E RR² that is uniformly drawn from the unit circle, with noise AO, 0.02²) added to each dimension. The private components are scalars c(1) ~ NN 2.0) and c(2) ~ Laplace (0, 4.0). The soureddto--rrvatee energy ratios for the two views are approximately -6 dB and -18 dB. The sample size is N = 5, 0000 And we use two different one-hidden-layer neural networks with 3 neurons and softplus activation to represent the invertible (())ss. 's The network parameters are drawn from standard normal distribution.
The shared component z and the t-SNE of (1) and xx22) are shown in Fig. G.I. One can see that by incorporating strong noise and nonlinear transformations, the shape of circle is hardly to be identified in both views.
In our simulations, f(() is represented by a three-hidden-layee multi-layer perceptrons (MLPs) with 256 neurons in each layer with ReLU activations. In addition, g(9) and 7(1) represented by two- hidden-layer MLPs with 128 neurons in each layer. We set batch size to be 1000, 3 = 1.0, À = 0.1. We the Adam optimizer (Kingma & Ba, 2015) with initial learning rate 0.001 for all the parame- ters. Besides, we also regularize the network parameters using !! with regularization parameter 0.1. This often helps improve numerical stability when optimizing cost functions involving neural networks. We run lines 4-10 10 of Algorithm for 10 epochs to update O and 17.
For ablation study, we test the methods with different combinations C, R and V, i.c.
(i) the proposed method (C + V R);
(ii) the proposed without independence regularization (C. + V);
(iii) the proposed without reconstruction (L. + R);
(iv) the proposed with only latent correlation maximization (C);
we also test the performance with HSIC (Gretton et al., 2007) as the independence regular-
izer (L. + V HTSC);
All methods stop when the average matching loss L reaches 0.01. The learned components by the proposed method are shown in Fig. G.2. One matched, while the second view exhibits relatively large noise level as expected. For the estimated private components, one can see that both (-) and ((2) are approximately invertible functions.
that the estimated shared components
To evaluate the performance of the synthetic experiment, we compute mutual information (MI) be- tween groups of random variables of interest (measured by the mutual information neural estimation
26

---

Published as a conference paper at ICLR 2022
scatter plot of 2
201 as a function of c(1)
aa)
(6)
a
(1)
1.
22
10)
-5.0
-2.5
0.0
c(1)
2.5
5.0
scatter plot of 2(2)
((2)
as a function of c(2)
(c)
dd)
1
(
(2)
(2)
-20
(2)
20
21
Figure G.2: (a) Scatter plot of g(1; (b) 2(1) function of (1) (c) Scatter plot of 2(2); (d) ((2) as function of c(
(MINE) (Belghazi et al., 2018) and Gaussian kernel density estimation (KDE) (Davis al., 2011) 1)). The results are averaged from 10 random trials.
One that all methods successfully extract the information about in the sense that both g(9) = (2(()) for = 1, 2 have similarly high MIs with z. Besides, all methods output g(0) and ((9) (cl9)) that have small MIs mll-ing that they are not dependent.
Although most loss functions using latent correlation maximization extract the shared z's informa- tion well, the difference is articulated in extracting the private information. The proposed C+ V + R objective has the best performance on that regard. The method ++V+ HSIC also works reasonably well since HS IC serves the same purpose as R does butt with a kernel- based implementation.
Moreover, by looking the last two columns, one can see that the methods with R and V perform the best in removing the information of z from c( This corroborates analysis that both (7b) (invertibility) and (7d) (independence) are vital to achieve private-shared information disentangle- ment.
Tab. G.1 shows the results, with all the entries averaged over 10 random trials. One can see that the results via the Gaussian KDE consistent with those under MINE. That is, the proposed C + V + R exhibits the best performance in terms of extracting and disentangling the shared and private information.
G.2 SYNTHETIC DATA ROBUSTNESS TO STRONG PRIVATE INTERFERENCE
In this subsection, we demonstrate the performance of the proposed method under different levels of private component energy (which are often considered interference) (Ibrahim & Sidiropoulos, 2020; Bach & Jordan, 2005; Lyu & Fu, 2020). First, we define the shared-to-private ratio (SPR) for the g-th view
DN DE
1
SPR = 10
dB.
D,N M
1
For the experiment, we make both views have identical SPRs. We test the performance under SPR=- 10 dB, -20 dB and -30 dB, respectively.
27

---

Published as a conference paper at ICLR 2022
Table G.I: Mutual information (MI) between groups of variables. "?" high score preferred; "U":
low score preferred: "n/a": not applicable; g(0)
f( for q 1,2.
,0 (t)
_(2,22 (1)
"(",,(0)) (4)
_ (1) _ed" (t) 20 (t)
2! (4)
2 ,2 (4)
Metric
MINE-bused
MI Estimation (Belghazi et al., 2018)
L.
2.32+00.06
2.36+00.10
0.01+0.00
0.00++0.00
0.:5+0.13 0.33+0.07
0.39
0.43+00044
C+V
2.37:10.05
2.38:10088
0.01=0.00
0.01
0.55+0000
0.33:10008
0.31
0.43
C+R
2.32++0.05
2.35+0.07
0.02+0001
0.00+0.00
0,90+0042
0.22++0.12
0.09:0000
0.22:+0.12
++V+R
2.43:40.04
2.39
0.01=0001
0.01
1.22+t: 0.33
0.85+1023
0.04=10.02
0.11=00.03
C V+HSIC
2.48:10.09
2.43
0.01=0.00
0.01
0.52:10.14 0.681:.22
0.04:10.01
0.07
Metrie
Gaussian kernel density estimate (KDE)-based MI Estimation
I
2.73+0022
2.82
0.00=0000
0.00
0.33.0.11
0.01=0.02
0.34
0.37:40.06
C+V
2.95:10.21 2.91
0.00=1001
0.00
0,3810.11
0.01+0.02
0.24:0.04
0.3910.07
C+R R
2.83:10.18 2.8610.15
0.00
0.00
0.08:10.07 0.10 0.72:0040
0.28110116
V.R.
3,27+0.07
2.88+0.25
0.00
0.00
0.99+0.30
0,35+0.11
0.03+00.04
0.08+0.03
C V+HSIC
3.35110.16 2.95
0.00
0.00
0.78+0.22
0.06=10.06
0.05=0.02
0.0210.01
Table G.2: Mutual information (MI) between groups of variables under different SPR "!": high
score preferred; "L": low score preferred; 2 = g( for q ===-==a0 1,2.
80) (0)) (t)
_00__00)) ) g()) 2() (1)
20) ") (t) 2 c (t)
200 (1)
20)
SPR
MINE-bbsed MI Estimation (Belghwei al., 2018)
dB
2.41:0.07
2.4320.05
0.01=0.01
0.01=0001
1.72=0.41
0.52=0.10
0.02=0.01
0.19=0.03
-20 dB
181100110
2.15+0.09
0.10=0.06
0.01+00011
1.4100155
1.15=0.07
0.08=0.04
0.06=#0.02
dB
1.16+0.07
1.55+0.10
0.11+0.09
0.07+00.4
1.77+0.42
0.73+0.14
0.03-0.02
0.14+0.11
Table G.3: Mutual information (MI) between groups of variables with different (i.e. reconstruction
term). "f"; high score preferred: "L": low score preferred; 2(0) = f5 for = 1,2.
21 (1)
g (t)
g0) (1) _(2)__()) (1) 2 _04) (t) 2 (t)
20), (L)
((2),x (4)
MINe-bbsed MI Istimation (Helghazi
2018)
8=
2.1310026
2.11=0.33
0.0110001
0.01
0.77+0.03
0,4820.18
0.13=0.02
0.17::0.07
lc
2.16::023
2.12:1021
0.01110.11
0.01+10.00
1.09+0.37
0.70:10.00
0.10=00.66
0.16:10.07
100
2.11:0.13
2.38=0.12
0.01100.00
0.01100.00
1.4810.08
0.9320.17
0.02=0.00
0.10:10.04
= lc
2.34+0.19
2.28+0008
0.04++0.4
0.02+0002
0.86+0.40
0.41+0.21
0.15+0.10
0.52+0.22
8 1
2.32+00.11
1.77-0.07
0.07
0.40:10.05
0.58:00.0
0.21
0.31-0.01
0.69-10.05
Tab. G.2 shows the evaluation results averaged from 10 trials. One can see that as SPR decreases, the MI between the extracted 2 and decreases but only gracefully. The slight decline of performance is because the matching of two views becomes harder when the private components get stronger. However, even if SPR=-30 dB, the extraction and disentanglement of private and shared information are still clearly achieved. Such robustness to strong private interference is considered key feature of linear CCA (Ibrahim & Sidiropoulos. 2020) and post-nonlinear CCA (Lyu Fu, 2020). Our analysis and evaluation in this work show that such resilience is also inherited by the proposed approach.
G.3 SYNTHETIC DATA SENSITIVITY TO HYPERPARAMETERS
In this subsection, we investigate the sensitivity the key hy hyperparameters 3 and X. The results are shown in Tab. G.3 and Tab. G.4, respectively. One can see that for the reconstruction regularization parameter (i.e., 3) does not affect the results too much unless it was set be overly large (i.e., 3 or 100 in our simulation). This makes sense, since reconstruction is for preventing trivial degenerate solutions, and giving this part too much attention may not really help the learning goals (e.g., shared and private information extraction) reflected in the other parts of the loss function. From Tab. G.4, one can see that the choice of À affects the performance slightly more than 3. It makes sense, since À reflects the attention that the algorithm puts on the private component extraction part. We should mention that, for real data analysis with a downstream task (e.g., classification), one can choose these hyperparameters using validation set.
28

---

Published as a conference paper at ICLR 2022
Table G.4: Mutual information (MI) between groups of variables with different À (i.e. independence
regularizer). "I" high score preferred; "4": low score preferred; g = fs for 1.2.
20,, (1)
20,), (t)
((0))e(1)) (L) _(ي)_م( (L) (00)_20 (t) ((21, (t)
200 (C)
20 (4)
8= le"
MiNNE-based MI Estimation (Belghazi 2018)
A le
2.36.10.09
2.34-0055
0.01100.00
0.01110.00
0.5610.18
0,51
0.25-0.07
0,36.10.08
le-1
2.411:0133
2.38=0.12
0.011:0.00
0.01+0.00
1.48:10.88
0.93-0.17
0.02-0.01
0.10:10.04
1ee
2.04+0.12
2.00-0.11
0.04++0.01
0.02++0.00
0.71++0.17
0.37+0.10
0.15+0.06
0.22++0.03
let
1.4940018
1.55=0.11
0.17:10.07
0.17:10.10
0.61:0004
0.17=0.07
0.23=0002
0.16:10.05
A le
0.80++0.23
0.96=0.36
0.46++0.13
0.28++0.07
0.18+0.08
0.22=0006
0.75=0.37
0.57=0.118
Table G.5: Network structures for the MNIST experiment. Encoders
Decoders
input: x0(") R28x28x1
input: (z (9) E RD+D2
4 X 4 Conv, 64 ReLU, stride 2 4 4 Conv, 32 ReLU, stride FC 256, ReLU FC D + Dr
FC ReLU
FC x 7 32, ReLU
4 x 4 .Trans, 64 ReLU, stride 2
4 x _Taans, 1, stride 2
G.4 REAL DATA MORE ON VALIDATING THEOREM I
Multiview MNIST Data. In this subsection, provide more details and evaluation results on the MNIST experiment. The way of generating such two views (as shown in Fig. I of the main text) of MNIST data is similar to the data augmentation ideas used in AM-SSL, e.g., rotation, adding noise, cropping, flipping (Chen et al., 2020; Grill et al., 2020). We aim to match the two views and try learn the shared representations, which should encode the class label information. The dataset has 70,000 samples that are 28 x28 images of handwritten digits. For the latent dimension, we set D 10, D1 20 and D2 50. In particular, since the second view consists of large random noise that are not of interest, we only add the independence regularizer R(1) on the first view (i.e., the rotated digits) to learn the private component of x!!) The learned (l)) was used to generate new samples; see the illustration in our main text.
The network structure used for all methods is shown in Tab. G.5. The network structure for ( and ((4) are MLPs with three hidden layers of 64 neurons. For hyperparameters, we set batch size to be |B,1 = 100 and |By] 1000, 3 1.0, À 100.0. For optimizer, Ba, 2015) with initial learning rate 0.001 for and 1.0 for 77. And we add squared &, regularization for both O and n7. with different regularization parameters that are 0.0001 and 0.1, respectively. We also run the SGD optimizer for 10 epochs to update 0 and n7.
also Adam (Kingma &
In Tab. G.6, we show more evaluation results the learned shared information across views. To be specific, we feed the learned =(()'s 's a classifier and the k-means algorithm, to observe if the leamedd representations improve the performance of supervised and unsupervised learning tasks. For the classification task, we split the data as 0000/10,00/1000
sets. We train a linear support vector machine (SVM) using the training data. The performance is measured by classification error (ERR). For the clustering task, we use the standard k-means to cluster on all the data samples. After clustering, we report the performance on the test set. We use number of metrics to measure performance, namely, clustering accuracy (ACC), normalized mutual information (NMI) and adjusted Rand index (ARI) (Yeung & Ruzzo, 2001). Among these metrics, ARI ranges from -1 to +1, with being the best and -1 the worst and NMI range from 0 to with 1 being the best.
for cniniin/validationn//
The "Baseline" denotes the results of simply applying SVM and k-means onto the raw data of the first view, i.c., x/ for = I,...,N, N. All the results of algorithms that involve stochastic methods are averaged over 5 random initializations. One can see that all methods have comparably good results in terms of learning informative representations across views. The results empirically validate our Theorem that latent correlation maximization is useful criterion to extract the shared information with guarantees. In particular, Barlow Twins performs slightly better in terms of benefiting downstream classification and clustering tasks. Nonetheless, the proposed method can
29

---

Published as a conference paper at ICLR 2022
data
Proposed Clust. Acee 95.7%
DCCA Clust. Acc= 96.0%
Barlow Twins Clust. Acc= 97.1%
BYOL Clust, Acc= 94.5%
Clust. Acce 54.8%
Figure G.3: t-SNE of the results on multiview MNIST of the second view. Baselines: DCCA (Wang
et al., 2015), Ba Barlow Twins (Zbontar et al., 2021) and BYOL (Grill et al., 2020).
Table G.6: The classification error (first row) and clustering results (rows 2 to 4) of the two-view
MNIST dataset. "Aat; high score preferred; "L" low score preferred.
Baseline
++
L+Y
DCCA
DCCAE
Berlow Twins
BYCL
ERR (1)
13.40%
2811:100219%
20.2%
2.87% 16%
2.85%
2.05%
2.67%
ACC
37.35%
97.03% 0.13%
96.80% 40.4%%
97.02%%%
96.95%
98.08% 10.05%
95.56%
NMI (t)
0.337
0.92240.003
0,923:00007
0.922:10.003
0.920=20.003
0.947
0.895=0.0023
ARI
0.216
0.936*0.003
0.931#00009
0.935=00003
0.934=0.002
0.958=0.001
0,904=0.047
Table G.7: Augmentation Used for CIFARIO to Generate Multiple Views.
Transformation
Value
Probability
ColorJitter
brightess=0... contrast-0.8, saturation=0.8, hue=0.2
0.8
GrayScale
0.2
RandomReeiizedCrop
scale=(0.2, 1.0), r0..5= 4/3)
HorizontalFlip
0.5
BaussianBlur
0.1,2
0.5
Solarization
0.4
Normalization
also guarantee extracting private information and facilitate cross-view image generation (see the experiments in the main text), which out the reach of Barlow Twins and BYOL.
In addition to showing the visualization of the learned embedding of the firs view in Fig. I of the main paper, we also plot the I-SNE of the learned representation xx(²) of the second view (i.e., the noisy digits). The results are shown in Fig. G.3. One can see that the visualization and clustering accuracy are similar to those obtained from (1
Augmented CIFAR10 Data for SSL. We also use the CIFARIO dataset (Krizhevsky et al., 2009) to validate Theorem 1. The CIFAR10 dataset contains 50,000 and 10,000 images of size 32 x 32 for training and testing, respectively. There are 10 different classes. We use ResNet18 as the backbone network for learning the representation. Since CIFARIO images are small, we replace the first 7x7 Conv layer of stride 2 with 3x3 Conv layer of stride 1. We also remove the max pooling layer. We follow the evaluation method in (Chen & He, 2021) to stop the algorithms after one hundred epochs. In terms of data augmentation, we use the pipeline of different transformations in Tab. G.7. Note that for the proposed method, we only impose constraint (7c) since our goal here in this task is to extract essential shared information.
We evaluate the proposed method and two AM-SSL baselines as mentioned in the main text, namely, Barlow Twins (Zbontar et al., 2021) and BYOL (Grill et al., 2020) by feeding the learned rep- resentations to a linear classifier. We report both the Top-1 I linear classification accuracy and the KNN (with A = 5) accuracy. The results are shown in Tab. G.8. One can see that different methods achieve comparable results. The proposed method and BYOL attain essentially the same accuracy. The t-SNE (Van der Maaten & Hinton, 2008) visualizations of the test set is plotted in Fig. G.4. One can see that all methods extract iidentityrrveealing information to a certain extent, as in the MNIST case.
30

---

Published as a conference paper at ICLR 2022
Table G.8:
Evaluation using CIFAR10. Barlow Twins
BYOL
Proposed
Classification Acc. (%) KNN (h = 5) Acc. (%)
84.2
82.8 78.7
84.2
80.5
81.0
BYOL
Barlow Twins
Proposed
Figure G.4: t-SNE of learned representations for CIFARIO.
samples of viewl
samples of view2
Figure G.5: Samples of the lower elevations (view 1) and higher elevations (view2).
Remark 1 We should remark that the experiment results suggest that latent correlation maximiza- tion (or latent component matching) used in many AM-SSL and DCCA methods works towards the same ultimate goal under our generative model in (5). However, this does suggest that different SSL and DCCA methods are essentially the same in practice-one should not expect that. fact, there are many factors that affect DCCA and AM-SSL methods' results, e.g., model mismatches, optimization procedure, network construction, and the detailed designs in their loss functions. The difference between the methods normally are more articulated with larger data sets or more complex problems. Nonetheless, our interest lies in theoretical understanding of their common properties, other than the differences practical implementations. From this perspective, the results this section support our theoretical analysis in Theorem 1.
G.5 REAL DATA MORE ON VALIDATING THEOREM 2
Cars3D Data for Cross-sample Generation. In this subsection, we provide more detailed settings and results of the Cars experiment. To create multiview dataset, we assume that given the car type that is captured by shared variables z and the azimuths that are captured by c( the generation mappings (1) and g" produce car images with low elevations and high elevations, respectively (so they must be different mappings).
We split the car images as follows. We treat the same car model (e.g., a red convertible) with lower and higher elevations oz
(1) and a!") respectively. The azimuths are randomly shuffled with different pairs of x/" and (2) This way, if our generative model holds, 2, c( and g(o) are responsible for 'type', 'azimuth' and 'elevation', respectively. Some samples are shown in Fig. G.5.
Under our splitting, each view has 2x 24 = 8784 RGB images that all have a size of 64x64x3. We model the 'type" information using D = 10 latent dimensions since many different factors (e.g., color and shape) together give rise to 'type". On the other hand, we set Dj = D2 = 2 to model the "azimuth" information.
Tab. G.9 shows network structures for the encoders and decoders of our formulation in (F.6). In the table, FC denotes fully connected layer, Conv denotes convolutional layer and Conv Trans denotes 2D transposed convolutional layer. As before, the structures of (3 and ( are ML.s with two-
31

---

Published as a conference paper at ICLR 2022
Table G.9: Network structures for the Cars3D experiment.
Encoders
Decoders
input: 2;,") E Re1x6ix3
input: f(ll (ar ()) E R10++
4 4 Conv, 32 ReLU, stride
256, ReLU
4 4 Conv, 32 ReLU, stride 2
FC4 4 64, ReLU
4 4 Conv, 64 ReLU, stride
4 x
64 ReLU, stride 2
4 4 Conv, 64 ReLU, stride
4 4 Conv.Trans, 32 ReLU, stride 2
FC 256, ReLU FC 10+2
4 x Conv Trans, 32 ReLU, stride 2
4 x Conv Trans, 3, stride 2
E,j
(2)
(2) V
3
L,j
Generated samples by fixing 2n and varying 20) rows in blue boxes are w/ R; rows in green boxes are w/o R.
Figure G.6:
hidden- and 256 neurons for each layer. For hyperparameters, we set batch size to be [B] 100 and [Ba] 800, 3 0.1, = 1.0. For this real data experiment, we also use Adam (Kingma & Ba, 2015) as the optimizer with initial learning rate 0.001 for 0 and 1.0 for n.. We add |n| for regularization with parameter 0.0001. We limit the inner loops for solving the 0 and r subproblems to 10 epochs well.
Figs. G.6 and G.7 show more results under the same setting as in Fig. the main text.
dSprites Data for Cross-sample data Generation. We present the results on additional dataset, ie., dSprites (Higgins et al., 2017). the dSprites dataset, 64 x 64 images are generated based on five factors: 3 shapes (square, ellipse, heart), 6 scales, 40 orientations, and 32 different horizontal and vertical coordinates.
In particular, take subset that contains squares and hearts as the two views of data. In this subset, all the data samples are with the same scale. We assume the generating functions g (1) and (2) are responsible for the shape of square and heart, respectively. We treat the orientation and horizontal positions as the shared information, i.e., z, and the vertical position as the private information c(9), The vertical coordinates are random and not matched between different pairs of
2 and a Overall, we have 40, 960 samples for each view. We set = 2 and Dj = D2 = 1.
We use the same neural network structure as in Tab. G.9. The only differences lie in the input and latent dimensions. The network structure for (44) and (44) are MLPs with two hidden layers of 128 neurons, and we set [B|| = 100. |B2l = 500. 3 = 0.1, and À 100.0. Similar as before, we use the Adam (Kingma & Ba, 2015) optimizer with initial learning rate 0.001 for 0. As before, we add a squared & norm regularization on the network parameters 77, and set the regularization parameter to 0.1. We let the inner loop stochastic optimizers for 10 epochs.
32

---

Published as a conference paper at ICLR 2022
(1)
a (1)
(1)
[,
2 (2)
(2) E
j
(2)
3 E,j
Generated samples by fixing o and varying Yn rows in blue boxes are w/ R; rows in green boxes w/o R.
Figure G.7:
We conduct the same cross-sample data generation experiment as in the Cars3D case. Fig. G.8 shows the results. To be specific, we extract 2(() = (z( that represents the rotation and horizontal coordinate information and 2/a1 f And we combine this information to- gether to generate synthetic samples ((()) with the learned reconstruction network r()), i.e., a(4) p( (I(2) y, (c)
The observations are similar to that in the Cars3D experiments. Ideally, the generated samples should have the rotation and horizontal position of x; (contained in 2/ while the vertical po- sition of 2 (contained in c(a)) One see that without using the independence regularizer R, the generated samples may have rotation change, shape deformation compared to (:(9) simply the vertical position is not exactly replicated from the sample (0) However, with the R regularization, the results are exactly what one expects to see. This again verifies our claim in Theorem 2.
Multiview MNIST Data for Cross-view Generation. Using the multiview MNIST data, we also
show the cross-view generation results in Fig. G.9. Here, extract 2/2) f( (m: from the second view and of ft (x) ()) from the first. Then, we generate (1) Ei, = ([(2( (T)
shown in the blue and green boxes. Ideally, the generated samples should have the digit information
of a (2) (contained in 2(22) while the style information of (2 (contained in 2 Clearly, using R attains the desired results. Note that this dataset is challenging as the noise in view is very high,
making it hard achieve perfect matching and reconstruction- but our result still plausible.
We would like to mention that multiview data and pertinent learning tasks are perva- sive in the real world. For example, acoustic features and articulatory recordings are two views of speech signals, and multiview based representation learning can be used to enhance speech recog- nition (Arora & Livescu. 2013; Wang et al., 2015). Another example is cross-media information retrieval (Gong et al., 2014). There, data entity has text view and an image view, and the task to retrieve one view from another. This task can be efficiently done in the learned shared do- main. In natural language processing, multilingual word embedding can also be formulated as CCA-type shared information learning problem; see (Socher Fei-Fei, 2010; Dhillon et al., 2012). In computer vision, there are number of important tasks such as image style translation (e.g. sketch to picture and picture to cartoon) (Zhu et al., 2017: Huang et al., 2018; Lee et al., 2018) and super-resolutie (Ledig et al., 2017) can be considered multiview learning problems. In particu- lar, image style translation will benefit from our method's 's guaranteed shared (content) and private (style) disentanglement.
Remark 2
33

---

Published as a conference paper at ICLR 2022
(1)
(1)
(1)
C,j
(2)
a ]
2
2
(2)
IB E.j
Figure G.8: Generated samples by fixing 2N (rotation and horizontal position) and varying <00) (vertical position). Top: the square view; bottom: the heart view; rows in blue boxes are w/ R; rows in green boxes are w/o R.
R
O
1
2
3 4 8 8
7
q
U
1 Q
3
4 6 6
7
5
9
O
3
9
6 6
?
4
3
4
5
9
%
4
2
4
4
1
4
2
4
?
3
/ /
;
9 4
4
2
2 2
Z
w/o R
(1) Figure G.9: Cross-view generation from a ar
H ADDITIONAL NOTES ON SHAARED-PRIVATE MODELING
Regarding the generative model in (5), some remarks are as follows. The intuition that multiview data consists of shared and private components widely used; see, e.g., (Huang al., 2018; Lee et al., 2018; Wang et al., 2016; Gundersen et al., 2019). However, explicit generative models were only considered in limited threree works.
34

---

Published as a conference paper at ICLR 2022
The model in (5) can be understood nonlinear generalization of the linear CCA model in (lbrahim & Sidiropoulos, 2020), where the views are modeled as
(0) A |-], (c, (T
for q 1,2. In (Lyu & Fu, 2020), special type of nonlinear model, namely, the post-nonlinear mixture model, was analyzed. There, the model is
g 1 A [Z (c("))))
where (g)) applies nonlinear distortion to each element of y individually. However, post- nonlinear models are much less general compared our model in (5) (h)wreereeee g (1)) nonlinearly distorts all elements of jointly in an unknown way. More recently, under the context of AM-SSL, work in (Von Kügelgen et al., 2021) considered a multiview generative model that is similar to our model, but the views share the same generative nonlinear function, i.e.,
g[ (c("))TT))
f
This assumption restricts the applicability of the model scenarios where the two views are gen- erated using exactly the same nonlinear distortions, which may be less flexible. Our model in (5) subsumes the models (Ibrahim Sidiropoulos, 2020; Lyu Fu, 2020; Von Kügelgen al., 2021) as its special cases.
I AVOIDING RECONSTRUCTION USING ENTROPY REGULARIZATION
1.1 ENTROPY REGULARIZATIO AND SHARED COMPONENT IDENTIFIABILITY
we ignore private information extraction, our formulation for shared information extraction as follows:
minimize E
[|SS) (z(1) 2 )1]
(x(3))
(L.la)
(()
subject to p(a) for = 1,2 are invertible,
(I.Ib)
E
(g))
[z (1)
f (z((1) [
= E fs (x(0) = 0, 1,2,
(LIcc
with the latent variables satisfying:
p(z, (()) c(2)) p(z))(c(")) )
We hope to encourage invertibility of f()) without using decoder reconstruction network. To this end, we generalize the idea in Theorem 4.4 (Von Kügelgen al., 2021). Note that (Von Küggelgen al., 2021) deals with the case where only one is learned (i.e., (1) = (2)).. Here, we show that this idea can be used under our case as well. To see this, let us consider the following formulation:
minimize E [|555) (x())
S (2((2) (2)
H ($$) (x)
(I.2a)
f (2)
subject f!!) M, (0
(I.2b)
where (.) computes the differential entropy of its argument. The formulation still aims match the latent representations of the two views, but at the same time maximizes the entropy of the learned features of the first view. The proof of this case consists of three major steps.
Step 1. It is straightforward to see that the optimal solution of (I.2) is
1 (z) f! (x) & Uniform(0., D)
since the first term has optimal value 0 when two view perfectly matched, and the differential en- tropy of a random variable is maximized when the distribution on (0, uniform (Cover, 1999). Next, following the idea in (Von Kügelgen al., 2021), by the Darmois construction (Darmois,
35

---

Published as a conference paper at ICLR 2022
1951), there exists d(-) 2 - (0, which maps the ground-truth to uniform random variable on (0, 1)?. Thus, one can construct an optimal solution of (1.2) as:
((/10)
S& do
1:D
where the first D dimensions of the output of (g(4)) are fed to d(.).
Step 2. By using our proof technique in Theorem 1, employing the equation = (2)((())) = fs ( (z(2)), it can be shown that only depends on the shared component x but does not depend on either e(1) e!²), which we denote as = y(*). Note that the proof of this part holds since it only uses the latent correlation maximization (or (()) matching). Using the same derivation as in Theorem I, we have the Jacobian of h (2) follows:
J
J
O (I)
22
which indicates that only depends on z but not c(L), The above also holds for the second view. Note that the possibility f being trivial constant solution (and thus making H 0) is ruled out since f entropy maximized.
Step 3. The last step in Theorem is to use (an)(J()) D + D1 to show that Ji RDxD has full rank. There, rank( (J)) D + D₁ is natural since f(1)) is constructed to be invertible using an autoencoder (and thus f g is also invertible). Here, we could not use this argument. However, similar Theorem 4.4 in (Von Kügelgen al., 2021), by applying Proposition of (Zimmermann et al., 2021), one can show that 2 7(2) where - is an invertible function, if p(2) regular density, i.e., < p(z) < xo everywhere. Note that under our generative model, f (x)) = (z) for all ((4) Hence, the above derivations can be repeated for f(2), This concludes the proof.
1.2 REALIZATION AND CONNECTION TO CONTRASTIVE SSL
To implement the formulation (I.2), following the idea in (Von Kügelgen et al., 2021), one can use the idea of InfoNCE (Gutmann & Hyvärinen, 2010; Oord et al., 2018), which it has interesting connections to contrastive SSL (Wang & Isola, 2020). In particular, the formulation of InfoNCE is follows:
K Σ
log exp(sim(2, (0tssim(,,))//1 81/T}
(1) E (m/ m ( (p)(()()))))))) ~p(a())))))))) (2())
(L.3)
j= exp(sim (2.ssimm,,,.00 2/)/T}
where & and 2 are the learned representations of the two corresponding samples 2 and x?? respectively, sim(a, b) computes the similarity of its arguments, and there are K samples of each batch where K - of them are negative.
temperature hyperparameter
Note that in (Von Kügelgen et al., 2021), only one generative function g(-) considered. In their (1) (2) K
implementation, given sample pairs {m zt [=1, the above InfoNCE objective can be rewritten with 7 = 1 and sim(a, b) =-laa bll as
E fa ae
(2) (20,2, 101 (10m01 (3))
[(a(") -f(--(()))) f(x,2))
log
exp {
f(0
(L.4)
The second term is a non non-parametric entropy estimator of the representation as - 00 (Wang & Isola, 2020). The above nicely connects AM-SSL with contrastive learning when (1) (2) and only encoder used, i.e., (1) (2)
36

---

Published as a conference paper at ICLR 2022
However, in our problem the generative functions different in each view. Hence, the formulation above is not directly applicable. Nonetheless, one can use the slack variable based design as in (10). Then, the problem can be reformulated as
(a) + log expl - 1|u - uz162)
E. (a("", (11p)(0()))))))
(I.5)
Note that the entropy regularization is imposed on the slack variable w whch indirectly promotes high entropy of f(a's. This way, one can handle Q views with different generative functions g g(9)s.
37


Reviews:
[{"review_id": "O-NFc_vxmOC", "reviewer": "Reviewer_zhWW", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Strengths\n- The paper presents sound (I only checked the proof of Thm. 1 in detail and could not find any major flaws.) theory proving identifiability of both shared and private components for the assumed multi-view setting.\n- A finite sample analysis and result is provided which is rare in this area and thus appreciated.\n- The paper is overall well-written and well structured. \n- Based on the presented theory, a new learning method is developed and implemented.\n- The experimental evaluation is relatively diverse (multiple datasets and baselines)\n\n\n### Weaknesses\n- The theoretical analysis relies on two relatively strong and restrictive assumptions. \n   - The first is **invertibility of the learnt encoders** which, as stated by the authors, ensures that all information is preserved and corresponds to assuming the reconstruction task is solved. In their proposed algorithm, this is implemented in the form of a reconstruction objective within an auto-encoding framework. However, the (DCCA and) AM-SSL methods cited in the paper do not use such reconstruction objectives, partly owing to the complex nature of the data these methods are often applied to (e.g., natural images where perfect reconstruction is not feasible). Instead, these methods avoid trivial representations, e.g., via redundancy reduction (BarlowTwins) or using moving averages/stop gradients (BYOL/SimSiam). Unfortunately, **this undermines one of the paper's main claims that its theory explains the success of DCCA and AM-SSL methods**. In order to justify this claim, further theoretical analysis would be needed that explains why the aforementioned methods still work *without explicitly enforcing invertibility*. There is existing work that actually addresses learning with non-invertible encoders, see, e.g., [S1, S2], (von Kügelgen et al., 2021); cf. contrastive learning theory literature [S3, S4]. \n   - The second is **mutual independence between shared and private components**. This assumption seems central to the proofs of Thms. 1  and 2, but is only very briefly discussed in 3.2. This assumption may be less restrictive than invertibility (see also Questions below), but it deserves a more extensive discussion and justification. Moreover, since specifying a latent distribution is part of specifying a generative model, I believe this should be moved earlier to the beginning of 3.1 and inform the comparison with other generative models of multi-view data. \n- For *some* of the theoretical results, it is unclear to what extent they are novel or can already be found in similar form in the literature. In particular, there seem to be large parallels between Thm. 1 and the cited work of (von Kügelgen et al., 2021) which also identifies the shared latent component without the above assumptions and uses the same notion of identifiability up to invertible function. However, this is not discussed in the theory or related work sections.\n\n### Questions/comments for the authors\n- For the assumed generative model, starting from a setting where $(z,c^1,c^2)$ are *dependent*, can this equivalently be transformed into a setting with independent latents (and possible different $f^1,f^2$), and, if so, how? In other words, does the independence assumption come without loss of generality when arbitrary invertible $f^1,f^2$ are allowed?\n- It seems that $D,D_1,D_2$ are implicitly assumed to be known. If so, this should be highlighted as an assumption. Also, did you perform any ablation on this? \n- The finite sample analysis at the end of 3.3 is hard to parse for non-experts. It would be nice if you could provide some additional intuition on Assumption 2 and Thm. 3.\n- I did not understand exactly how the clustering experiment in 6.1 directly evaluates Thm. 1, i.e., the existence of an invertible function mapping between ground truth and inferred latents. Could you please clarify? In particular, how are the ground truth shared components used in the evaluation? It may be nice to complement the existing presentation with performance at predicting the ground truth latents, or is this the reported clustering accuracy in Fig. 1? Could you also show t-SNE for $\\hat{z}^2$?\n- What does your analysis have to say/how does it relate to contrastive AM-SSL approaches such as SimCLR? Can this also be phrased as  latent correlation maximisation?\n\n### Other minor comments, suggestions, and typos\n- p.2, 2nd para: The claim that \"theoretical understanding has been lacking\" seems too strong and should be toned down and complemented with references, acknowledging at least some of the many works from previous years that seek to theoretically understand CCA & AM-SSL.\n- p.2, (i): the key assumptions (e.g., invertibility) and type of identifiability (up to invertible function) should be stated here for transparency \n- p.3, end of 2.1: presumably the criterion should hold for all x?\n- p.7, 1st para: is \"HSIC [...] maximizes the correlation\" a typo and should read \"minimizes\"?\n- p.7, Reformulation para: invertibility cannot be \"enforced\" by using an autoencoder (at least in practice), this should be toned down, e.g., to encourage or promote as used in the next sentence\n- p.14, 3rd last para: there seems to be a typo in that the partial derivative of $h_S$ w.r.t. $c^1$ (instead of $z$) should be zero; otherwise this contradicts the first part of the sentence.\n- p.14, 2nd para: $h_P$ has not yet been defined\n\n### References\n[S1] Zimmermann, Roland S., et al. \"Contrastive Learning Inverts the Data Generating Process.\" (2021).\n\n[S2] Tian, Yuandong, Xinlei Chen, and Surya Ganguli. \"Understanding self-supervised learning dynamics without contrastive pairs.\" (2021).\n\n[S3] Arora, Sanjeev, et al. \"A theoretical analysis of contrastive unsupervised representation learning.\" (2019).\n\n[S4] Tosh, Christopher, Akshay Krishnamurthy, and Daniel Hsu. \"Contrastive learning, multi-view redundancy, and linear models.\" (2021).", "summary_of_the_paper": "### Scope & Problem setting\nThe paper studies the problem of representation learning from multi-view data from an identifiability perspective. In particular, the focus is on latent correlation maximisation approaches as commonly used in (nonlinear) canonical correlation analysis (CCA) and artificial multi-view self-supervised learning (AM-SSL). The paper postulates a generative model by which views $(x^1,x^2)$ are generated by invertible functions $f_i$ applied to a mixture of independent shared $z$ and private $c^i$ components, $x^i=f^i(z,c^i)$, for $i=1,2$. \n\n\n### Theory\n It is shown that latent correlation maximisation (with a suitable objective and constraints, as well as with invertible encoders) identifies, or separates, both (i) the shared component $z$ and (ii) the private components $c^1,c^2$ up to arbitrary invertible functions. Further, (iii) a finite sample analysis is provided.\n\n\n### Algorithm & Experiments\nThe paper proposes a learning objective that combines latent correlation maximisation with a regulariser and a reconstruction objective to encourage independence among $(z,c^1,c^2)$ and invertibility of the encoders. Several experiments on synthetic data and image benchmarks are used to validate the theory and compare the proposed method against DCCA, BarlowTwins, and BYOL.", "summary_of_the_review": "Overall, I enjoyed reading this paper. However, I have two main concerns (see Weaknesses above for details): (i) the invertibility assumption used to obtain the theoretical results is at odds with the fact that latent correlation-based multi-view learning typically does not use invertible encoders. This is not made sufficiently clear, and the claims that the presented analysis explains or helps understand the success of current methods is not fully justified and should be toned down. (Such an explanation would also have to explain why these methods work without invertible encoders.) (ii) part of the theory appears weaker than results in (von Kügelgen et al., 2021) which are not compared to in sufficient detail; in particular, I encourage the authors to include a more complete comparison.\n\nI believe the paper also provides valuable novel contributions, in particular, the finite sample analysis (Thm. 3), the new learning method (Sec. 4), and the thorough experimental evaluation (Sec. 6); I actually see these as the main contributions (Thms. 1 and 2 seem to follow quite directly from invertibility and independence) and think they could feature more prominently. Together with addressing (i) and (ii) above, this would make the paper a solid and well-rounded submission, and I remain open to increasing my score depending on the authors' response.\n\n### Post-rebuttal updates: \nThe authors have provided detailed responses to my comments and have made several modifications to the manuscript that satisfactorily address my two main concerns (see the discussion below for details). I have decided to increase my score as a result. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "N9zOjUhppnu", "reviewer": "Reviewer_u537", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Multi-view data (e.g. getting both visual and audio data of the same situation) is often analysed with CCA-style algorithms. Contemporary nonlinear CCA models rely on neural encoders to find suitable representations. This paper develops a theoretical analysis of a particular CCA model. This model differs in subtleties from existing models, but reflect the same intuitions as current state-of-the-art. The paper provides theoretical evidence that suitable representations can be recovered from multi-view data. While I am not an expert in this area, this does seem like valuable insights.\n\nTo me, the largest caveat of the paper are the subtleties which is where the proposed model differs from others. These allow for theoretical insights, but it is unclear to which extend the assumptions are reasonable. For example, \n\n* In Eq. 6b it is stated that $f^{(q)}$ should be invertible. This seem like a rather strict assumption in practice. In particular does it not imply that $f^{(q)}: \\mathbb{R}^D \\rightarrow \\mathbb{R}^D$? Is it reasonable that $f^{(q)}$ is dimensionality preserving?\n\n* Eq. 6d seems to be a rather important assumption about independence, but I find it rather hard to determine if that's a suitable assumption. I get that it comes with mathematical convenience, but is it reasonable?\n\nIn general, I would have liked the paper to have a remark after both Assumption  1 & 2 about the reasonability of these assumptions.\n\nAlgorithmically, the assumed constraints are converted in \"soft constraints\" (regularization) and an appropriate optimization is proposed. While the empirical analysis is limited to fairly simple settings, the evidence does point towards these regularizers working well. It would have been nice with an empirical study of the sensitivity towards the scaling of these regularizers (e.g. $\\lambda$).\n\n### Minor comments ###\n\n* In Sec. 3.1, the paragraph \"Generative Models of Multiview Analysis\" feels more like a comment on previous work and it broke my flow while reading. Perhaps this paragraph could be moved elsewhere?\n\n* In Theorem 2, it is unclear what is meant by \"a certain invertible function\".\n\n* The citation associated with t-SNE in Sec. 6.1 appears to be about SNE. This should be corrected.\n\n* The parenthesis remark in the conclusion (\"which is later realized...\") should perhaps indicate that the realization is approximate.\n\n### Update after rebuttal ###\nI am happy with the given replies and retain my positive score.", "summary_of_the_paper": "The paper offers a theoretical analysis of neural canonical correlation analysis (CCA) type methods. This reveals conditions under which these methods may work. The paper then proposes regularizers that approximately realize these conditions, which result in a new CCA-type algorithm. Empirical results indicate that the approximate algorithm behaves as dictated by the theory.", "summary_of_the_review": "The paper is generally easy to read and tackles an interesting problem with a novel solution (as far as I can tell). The issues I have pointed to can fairly easily be fixed. Hence the positive score (I would have given a 7 if that was an option).", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "DEL4tBA0aD", "reviewer": "Reviewer_hUwt", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper proposed a novel approach for multi-view learning. The pros and cons are listed below:\n\nPros:\n\nThe general motivation of the proposed model is reasonable and logical.\n\nConsider there are a few multi-view learning works provide the theoretical analysis, while this work gives a comprehensive analysis of the model. It is potential to extend to wider multi-view methods for deeply understanding multi-view scenario.\n\nThe experimental results demonstrate the effectiveness of the proposed modules.\n\nCons:\n\nThe experimental results mainly based on relatively small and simple datasets. The potential extension to a more large and sophisticated datasets and sample formats should be discussed.\n\nThe computational cost of the proposed model should be discussed. The potential solution for large-scale applications should be discussed.\n", "summary_of_the_paper": "This paper proposed a general multi-view learning approach and provides a theoretical analysis for the proposed method. Specifically, each view is considered as a common/shared and private/view-specific components. Then, the multi-view learn problem is converted to a identification and disentanglement problems. The theoretical analysis provides the bounds of the proposed model. Experimental results demonstrate the effectiveness for downstream tasks.", "summary_of_the_review": "This paper proposed a novel approach for multi-view learning problem. A theoretical analysis of the proposed model is provided. In experiments, most of the datasets are small scale and simple data format. More discussions and analysis about its potential real-world applications could be discussed.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers agreed that this is a strong paper, that the methodological contributions are both relevant and significant, and that the experimental validation is convincing. I fully share this viewpoint!
Decision: Accept (Spotlight)