Adversarial Representation Learning for Canonical Correlation Analysis
Authors: Feng Bao
Steve J. Altschuler
Lani F. Wu
Venue: Submitted to ICLR 2023 (2023)
Keywords: Representation Learning
Canonical Correlation Analysis
Adversarial Learining

Abstract: Canonical correlation analysis (CCA) provides a framework to map multimodality data into a maximally correlated latent space. The deep version of CCA has replaced linear maps with deep transformations to enable more flexible correlated data representations; however, this approach requires optimization over all samples for each iteration and poorly scales. Here, we present a deep, adversarial approach to CCA, adCCA, that can be efficiently solved by standard mini-batch training. We reformulate CCA under the constraint that the different modalities are embedded with identical latent distributions, derive a tractable deep CCA target, and use an adversarial framework to efficiently learn the canonical representations. A consequence of the new formation is that adCCA learns maximally correlated representations across multimodalities meanwhile preserves structure within individual modalities. Further, adCCA removes the need for feature transformation and normalization and can be directly applied to diverse modalities and feature encodings. Numerical studies show that the performance of adCCA is robust to data transformations, binary encodings, and corruptions. Together, adCCA provides a scalable approach to align data across modalities without compromising structure within each modality.

Under review as a conference paper at ICLR 2023
ADVERSARIAL REPRESENTATION LEARNING FOR CANONICAL CORRELATION ANALYSIS
Anonymous authors
Paper under double-blind review
ABSTRACT
Canonical correlation analysis (CCA) provides framework to map multimodal- ity data into maximally correlated latent space. The deep version of CCA has replaced linear maps with deep transformations to enable more flexible correlated data representations; however, optimization for the CCA target requires calcula- tion on sufficiently large sample batches. Here, we present a deep, adversarial approach to CCA, adCCA, that can be efficiently solved by standard mini-batch training. We reformulate CCA under the assumption that the different modali- ties are embedded with identical latent distributions, derive a tractable deep CCA target. We implement the new target and distribution constraint with an adver- sarial framework to efficiently learn the canonical representations. adCCA learns maximally correlated representations across multimodalities meanwhile preserves class information within individual modalities. Further, adCCA removes the need for feature transformation and normalization and can be directly applied to diverse modalities and feature encodings. Numerical studies show that the performance of adCCA is robust to data transformations, binary encodings, and corruptions. Together, adCCA provides a scalable approach to align data across modalities without compromising sample class information within each modality.
INTRODUCTION
Data samples can be measured with different modalities (e.g., image or text), encoded in different formats, and modeled by different distributions. Integrative analysis of multimodality data provides the opportunity in many machine learning tasks to combine partial information from each modality and achieve better performance than any single modality alone (Ngiam et al. (2011); Srivastaval & Salakhutdinov 2012)). Canonical correlation analysis (CCA) (Thompson (1984) is of the most classical and general approaches for multimodality data integration. It Tearns linear mappings
between data modalities that achieve maximal coos-mmodality correlations.
Replacing the linear mappings in CCA with deep functions can achieve non-linear and flexible transformations and provide better correlated representations. However, learning the deep CCA target function requires optimization over all data samples (Andrew et al. (2013)) or a sufficiently large data batch (Wang al. (2015)) ), which is incompatible with standard batch-based learning strategies widely used in deep learning and limits its power on large-scale datasets. Therefore, many recent deep CCA approaches (Wang et al. (201)): Dutton (2020; Karami & Schuurmans 202T
sidestep the original CCA formulation and rather focus on learning joint representation for the paired modalities using approaches that are compatible with mini-batch training (Appendix A
Here, we propose multimodal adversarial learning framework for deep CCA learning: adCCA. Mathematical analysis provides an optimization target for CCA amenable to mini-bated training un- der the requirement that the different modality distributions are identical in latent space. adCCA formulates this requirement as a penalty function that, during optimization, brings the two latent distributions into alignment. As the latent space representations converge (distribution-wiss),, max- imizing the optimization target leads to highly correlated latent representations (sample-wise) for the two modalities, illustrated with numerical experiments. Thus. adCCA is derived from deep CCA framework that follows the original correlation target of CCA, yet can be directly optimized by mini-batch training.

---

Under review as a conference paper at ICLR 2023
In more detail, adCCA learss representations in multiple steps. First, initial latent representations are provided using -oselity-speciffe autoencoders (AEs). Second, the Wasserstein distance is used to measure overall differences between the two latent distributions, and an inner product is used to measure sample-wise similarities. Then, adCCA uses an adversarial framework to minimize la- tent space distribution differences and maximize sample-wist similarity (Goodfellow et al. (2014). Together, the final output representation from each modality is expected to have similar latent dis- tributions (from the Wasserstein distance terms), high cross-modality sample similarity (from the inner product term) while still maintaining a faithful representation of the original features (from the AE).
The design of adCCA framework has several advantages. One major advantage is flexibility in in- put features. Often multimodal analysis, the total loss composed of loss terms from different modalities (Wan et al. (202T); Andrew et al.] (2033) Ngiam et al. (201T): Jain et al.] (205)). If features from diverse modalities have different distributions or data formats, the joint optimization can be biased towards a certain modality. In the adCCA framework, each modality representation is generated from a modality-specifff encoder, and the two latent representations are only related indirectly through the adversarial network. Another major advantage is the ability to learn corre- lation without losing information in the original features. In particular, deep CCA approaches use deep transformations to learn canonical representations with high correlation; however, in doing so these representations may lose class information contained in the original feature space. This point typically not considered in deep CCA frameworks. In adCCA, the use of AE in learning forces the latent representation to retain a faithful reconstruction of the original data.
We benchmark adCCA against 10 different methods, using both synthetic and real datasets. In the evaluation, we focused on two aspects, Cross-modality consistency: Correlation of sample points in the latent space for both modalities (Appendix Fig D.IB). And, Class presevvation: Ability of the latent representation to retain the original class information (Appendix Fig D.ICC. The ideal method should provide embedding that are both correlated per sample and retain class structure across both modalities (Appendix Fig D.IP). We show representations from adCCA both achieve high cross- modality consistency in the joint latent space and preserve class information from the original feature distributions. We demonstrate that adCCA can maintain stable performances across features of different modality types, feature distributions, and degradation levels.
The major contributions of this work are: (1) A reformulation of CCA with deep transformations into a target that can be optimized through standard mini- batch training. (2) An adversarial learn- ing framework for efficient canonical representation learning and optimization strategy for the (3) An expanded approach for evalu-
stable adversarial training between two latent representations. ating the performance of CCA approaches for both cross-modality consistency and single-modality information preservation. (4) An extensive test and demonstration of adCCA's performance.
ADVERSARIAL REPRESENTATION LEARNING FOR CCA
2.1 PRELIMINARY
Here, we consider a general framework for CCA. (An overview of existing CCA variants is provided in Appendix A) Given pair of feature vectors from two modalities, R and y E R?, CCA tries to learn two mapping functions fi and f2 that maximize the correlation after mapping:
max max 2n) s.t. 2g = fxx), = f2(y),
(1)
where 2n E IR and 2g IR are the corresponding canonical variables. Transformation functions fi and f2 can be linear mappings (in the original CCA), implicit kernel functions (in kernel CCA) or deep transformations (in deep CCA). These functions are learned through optimization over the entire dataset (Andrew et al., Appendix II.. and mini-batch training strategies cannot be leveraged.
Here, aim reformulate the CCA learning target form that enables optimization through general deep-lcarnnng optimizers. To this end, we relax the CCA learning target with an additional identical latent distribution assumption:
2

---

Under review as a conference paper at ICLR 2023
Theorem 1. Assuming latent representations from modality and y follow the same distribution with the mean a and variance o², the optimization target of CCA (Eq. 1 can be rewritten as:
max E(xxy))-Paat [z
s.t. 2 = fx(x), 1 2y f2 2x ~ n (11, 0²), 2, a (11, o²),
(2)
where y) ~ Pdata indicates paired modal features x and drawn from the data.
Proof.
Refer to Appendix B.2
The latent distributions are assumed to be the same with the same fixed Jr and a², and the new target function be expressed simply the average of the product of canonical variables. This formulation avoids noted problems with mini-batch optimization in CCA where μ and o² are not fixed.
We next extend the optimization target to learn k canonical variables simultaneously. Deep CCA (Appendix [B.I] employs Singular Value Decomposition (SVD) to output canonical variables that are orthogonal to each other. This step requires the use of large data batch sizes. To achieve more efficient optimization,
forgo the requirement for orthogonality. Instead, we use two independent autoencoders to generate canonical representations. We add the autoencoders as constraint terms and extend Theorem I to a multi-variable form:
Corollary 1. Multidimensional canonical representations 2, E TR and by E R are learned from two individual autoencoders with encoders fi(x) and fz(y) and decoders gi (Zz) and 92(Zy). Un- der the assumption that each entry of the two latent representations 2!!), Zy follow the same latent distribution with mean p and variance o², the optimization target of CCA can be re-formulated as:
max
[2- zyl
8.1. 2 = f1(x); (1xx),0 by f2(y),
xx - 91 (zm) 0, Ily - 92 92(zy)13
z 2/1² ~ T(1, 0²) for
Proof. Refer to Appendix B.3
2.2 MODEL FORMULATION
Based on Corollary [ the new target function for CCA has three requirements: maximiz- ing the inner product between latent represen- tations of the two modalities; 2) requiring the two latent distributions from two modalities to be the same; and fixing the mean and vari- ance of each latent dimension to be constant.
2,
Modality
x
PMr
Overall distribution
Wassersian
Cross-modal similarity
These requirements can be easily satisfied us- ing existing deep-learning paradigms. For (1),
2. 2,
the inner product is a general term to measure sample similarities and is widely used in deep- learning approaches, such as simCLR (Chen al. (2020)); for (2), distribution differences can be measured by the Wasserstein distance (Arjovsky et al. (2017)) or by other types of di- vergences used in machine learning tasks, such as image generation Makhzani et al. (015 )) for (3), constant mean and variance be achieved through batch normalization.
Modality y
geo
Pbr
Figure 1: adCCA framework and loss function de- sign.
3

---

Under review as a conference paper at ICLR 2023
Next, we describe the adCCA framework (Fig. [ Appendix C Here, consider paired modality
features x and y collected from the same sample. First, we employ an autoencoder (AE) for each modality to learn the initial latent representations n, and Zy:
min Exxp, |1 990=
(2) (x)) x112
(4)
min Ey Eymp, 11 19, (y)] y1
(2) (1)
Pall,
where 90,
rip are the encoder and decoder for modality z with network parameter sets Oz (x) and (a)
and Pr (Resp. 90, and 14g latent representations
(2) for modality y): Ps and Py are data distributions for x and y. The given from the modality-specific encoders by 2z Do. [ (x) and Zy
(1) (y). The reconstruction loss encourages the AE bottleneck layer to faithfully encode information contained in the original feature space.
Second, a Wasserstein distance is used to measure the distance between the latent distributions of the modalities:
W (Paas Pey)
inf Elllp Pry)
Eie,, 2. myyr 11|z - zy
inf yEll(p=.P))
(xx,)) 11 2yll]
(5)
s.l. 2g = 90, (x), By = 90, (y);
(2))
where Pz,: Pz, are distributions for Zg and Zy. and IT (pz, Pz.)) represents the space of joint dis- tributions for 2z and Zy with marginal distributions Pa.. and Pz, The same notation also applies to [1 Py). As previously shown (Arjovsky et al. (2017)), the least upper bound of Wasserstein distance can also be formulated in terms of I-Lipschitz function fwo
W (Pa.. (Pa.)
sup 11
(Ex~pa (f Eyr,, [f (zy)))
(6)
Following the Wasserstein GAN structure (Arjovsky et 2017) ), we minimize this distance in an adversarial paradigm:
max min (Ex-p, (2.z)) E Ey-p, ( (2,)))
to 02.0y
(v) s.t. 2, 90 (x), Zy 90, (y)
(2) and 9o,
(9) from above are reused as generative networks. fo is mod-
Here, the AB encoders 9o. eled by discriminator network with parameter set w. Maximizing the function by optimizing the discriminator parameter w increases the difference between two representations 2, and Zy. while minimizing the function by tuning generator parameters decreases the distance by finding better encoder representations.
Third, we add the inner product to Eq. 7 and formulate the learning target used by adCCA:
max min - -E(x.)) [y zn] + [Ex=pt [Ln 0.,0,
(8)
(v) (z) (x). Zy 90, (y). s.t. 2 = 90_
where À is multiplier used balance the distribution penalty (given by the Wasserstein dis- tance) with the sample pairwise-similarig term (given by the inner product of latent representations). Taken together, the overall adCCA model requires the optimization of Eqs. and

---

Under review as a conference paper at ICLR 2023
We note that the two latent representations are provided from two modality-specific AEs and not linked directly in the model. Thus, there is no requirement to process raw features x and y for balanced learning and the two AEs can be replaced with modality specific structure (e.g.. one AE can be a convolutional network for images and the other an attention model for language).
2.3 INTUITIVE INTERPRETATION OF THE ADCCA LOSS FUNCTION
In this section, we provide intuition for how the formulated loss function can provide correlated embeddings from multimodal data. In Eq. X the first two Wasserstein distance terms force the two latent representations, 2 and Zy. to share similar distribution shapes. However, this overall distribu- tion penalty does not align different modality features from the same sample. The last inner product term explicitly encourages representations generated from paired modality features corresponding to the same samples to maximize their similarities. Together, adCCA encourages similarity across modalities in the latent space representations at both distribution (Wasserstein distance) and sample (feature correlation) scales.
The AEs in Eq. encourage 2y and Zy to preserve information from each modality. This allows adCCA to avoid the adversarial network converging to trivial solutions. For example, 2s and Zy could converge zero vectors, which satisfy both the distribution and similarity requirements, but do not carry any information from the original data. By using both Eqs. 4 and 8 multimodal representations can be learned that both preserve the original information and achieve high cross- modality correlations.
2.4 MODEL OPTIMIZATION
In the standard generative adversar- ial network framework (Goodfellow
Algorithm 1: adCCA optimization
Data: y)}" ~ Pdata; X y Result: 2y))",
et al. 22044; Arjovsky et al. (2017).
R2 :y E R²
we only need to optimize one genera- tor and discriminate its output against real samples. However, in adCCA, need optimize two genera- tive networks and match the distri- butions between both latent represen- tations. During training, these two latent space representations change epoch by epoch, which creates chal- lenge for stable optimization.
Initialize 2r and by with autcencoders for +- 1: epoch (AB) do
Px, Oz min Bxxxx ( Le.e (xx] -x]3:
Py: 0y t min Eynp, [I 190y [y)] y|
(2) (v)
end
Ka (z) (x), k, = 98, (y): Opt immis 2s and by iteratively
(9)
for i +- 1: epoch do
y-step: fix 2, and optimize By
Here, we propose multi-step train- ing process for stable adCCA opti- mization in Algorithm I (illustrated in Appendix Fig. D.2) First, au- toencoders are trained independently
Pv. 0g + min Eyapp (2) go, [y)] - y113:
w + max mxx [f.(2.) By-py If (z
0g f- min -Eyspy,loot, [Ro() X 1ot 2n]:
s-step: fix By and opt imtiieze us (| (a) (a)) (x)] x11
Ps, O2 min Bx-p,
for each modality to provide an initial embedding. This enables adCCA to work with raw input feature data from different modalities and different dis- tributions (see Section 3.1.2). Next, return 2. ((2) (x), By (w)) (y)
w + max Exxp, (2x)) By-py [f bw)
0g min Ezeps (/n(2.)) ( (2 2ny:
end
we fix the representation for modality and optimize the generative network for modality y through the adversarial learning step (y-step)- During this y-step, we optimize the autoencoder, discriminator and generator. The discriminator identifies distributional differences, which are then minimized by "moving" the latent distribution of y towards the latent distribution of æ. Meanwhile, the autoencoder ensures that the latent repre- sentation faithfully reflects information in the original data. Then, fix the representation of y and optimize the generator for x (æ-step). The ststep and y-step are performed iteratively.
3 EXPERIMENTS
In this section, show the results of experiments using simulated and real data. We compared the proposed adCCA with other CCA approaches as well as multimodal learning frameworks that can complete the same task.
5

---

Under review as a conference paper at ICLR 2023
Specifically, we compared adCCA to two different classes of existing approaches. The first class (in- dividual representation learning methods) provides an individual representation for each modality. This class contains 6 approaches: classical CCA, kernel CCA (radial basis function and polyno- mial kernel) (Perry et al. )) CCA with deep transformations (DCCA) (Andrew et al. (2033). simCLR (Chen al. (2020) that employs the same encoder to learn similar representations from multimodal inputs, and contrastive autoencoders (contrast AE) that uses a contrastive loss to con- strain the similarities in the latent space (which can be viewed as adCCA without the distribution penalty). The second class (joint representation learning methods) provided a combined representa- tion for both modalities. This class contained approaches: CCA with the latent distribution con- straint under variational framework (VCCA and VCCA-p with private latents) (Wang et al. (2016) or adversarial framework (ACCA) (Dutton (2200) and deep probabilistic CCA (DPCCA) Karami & Schuurmans )02T)) Detailed configurations were provided in Appendix E.1
3.1 EXPERIMENTS WITH SYNTHETIC DATA
We first performed simulations studies by generating synthetic multimodal features with known sample labels. Here, we considered 10 ground truth classes existing in the data and generated synthetic features using multi-layer perception networks with random mapping matrices for the two modalities (Appendix [.2). Then, we used multimodal approaches learn representations in latent spaces of 10 dimensions.
3. BASELINE COMPARISONS
We set high feature drop-out rates for features modality x (90% feature missing ratio and 0.1 noise variance) and high noise levels for modality y (10% feature missing ratio and 0.5 noise variance). We observed, based on visualization of the raw feature spaces using Umap (Becht et al. (2019) ) (Fig. 2 left), that the high drop-out rates in had more significant effects on degrading the data class information than the high noise levels in y. We then tested the ability to map both modalities to the same latent space.
Modality
CCA
KCCA -plly
KCCA -RBF
DCCA
20
Modality
simCLR
Contrast AE
adCCA
Modality
Modality
Figure 2: Evaluation of cross-modality consistency. Left: visualization of raw features from two modalities using Umap (Becht et al. (2019). Color indicates the sample classes. Right: joint visu- alization of two modalities in the same latent space. Color indicates the modality that representation was learned from. KCCA-poly: CCA with polynomial kernel; KCCA-RBF: CCA with radial basis function kernel.
First, we evaluated the degree of correlation between the latent representations of the two modalities across different methods. We co-embedded the two modal representations using Umap. (We note that ACCA, VCCA(-p) and DPCCA can only output a joint representation and were not evaluated for this comparison.) For most approaches, representations from different modalities were mixed in the latent space (Fig. 2 right). Quantitative assessment in 10-dimension latent spaces using correlation coefficients and entropy of mixing (Appendix E.3) showed that CCA, DCCA and the proposed adCCA had reasonable performance for combining representations from different modalities (Table Kernel CCA also achieved good performance with appropriate kernel choices (RBF kernel in this case).
Second, evaluated whether the representations of the two modalities in the latent space reflected sample clusters present in the original data. It was evident from 2D representation embeddings with the original class labels (Fig. 3 that DCCA and adCCA best suited to preserve cluster

---

Evaluation of pre- serving class information in Under review as a conference paper at ICLR 2023
Method
Correlation coefficient
Entropy of mixing
Table 1: Evaluation of pross-modalitty
CCA
0.976 0.002
0,682 +: 0.002
consistency for latent representations learned from two modalities. Corre-
KCCA-poly
0.732 0.009
0.512 0.158
KCCA-RBF
0.978 + 0.001
0.693 +: 0.001
lation coefficients were calculated on
DCCA
0.985 0.009
0.689 = 0.005
each latent dimension and their mean and standard deviation were reported.
simCLR
0.335 0.102
0.688 0.007
Contrast AE
-0.074 = 0,396
0.185 1 0.234
adCCA
0.987 + 0.011
0,690 + 0.004
Method
ARI
Silloutte
SVM
KNN
Table 2:
CCA
0,019
-0.03
0.114 1. 0.017
0.370 + 0.008
KCCy
0.273
0.018
0.121 +0.033
0.457
0.174
the latent space. Classifica- tion accuracy was based on 5-fold cross validations.
KCCA-RBF
0.002
-0.020
0.107 1 0.002
0.233 1 0.018
DCCA
0.641
0.225
0.905 = 0.007
0,902 0.008
simCLR
0.012
-0.037
0.201 0.059
0.128 0.014
Contrast AE
0.034
-0.041
0.294 1 0.032
0.318 1 0.020
ACCA
0.462
0.113
0.408 = 0.004
0.681 = 0.028
VCCA
0.451
0.062
0,929 1 0.011
0.793 0.022
VCCA-p
0.014
-0.049
0.128 = 0.023
0.149 = 0.015
DPCCA
0.008
-0.080
0.127 = 0.018
0.140 = 0.007
adCCA
0.924
0.237
0.991
0.003
0.975
0.005
structure in the original data. We quantified the quality of the latent representations to reflect true sample classes using three measures: 1) clustering the latent representations and comparing cluster identities with the true class labels using the adjusted Rand index (ARI); 2) calculating between- and within-class distances using the true labels and quantifying sample compactness using the Sil- houette coefficient; and 3) constructing simple classifiers in the latent space using the true labels and reporting classification accuracy. Details of these measures are provided in Appendix E.3
The results of these evaluations (Ta- ble 2 confirmed that DCCA and adCCA performed well retaining class information from the original data. While some of the other ap- proaches could align multimodal rep- resentations in the latent space (i.e., high correlation; T), they did so the cost losing spatial coherency present in the original features (Fig. Table 2). In general, higher cor- relation need not lead better class predictions for classical approaches as their only learning target is the maximization of the correlation be- tween two representations. To get high correlation, the feature informa- tion from original data can be greatly compromised (An illustration in Ap- pendix Fig. D.I).
CCA
KCCA polly
KCCA-RBF
DCCA
simCLR
Contrast AE
VCCA
VCCA-p
DPCCA
ACCA
adCCA
Figure 3: Evaluation of preserving original data classes in latent space. Umap embeddings latent representations are as in Fig. [ with color annotations indicating the true sam- ple labels.
Overall, our experiments suggest that the adCCA framework can achieve a balance between cross-modality consistency (encouraged by the adversarial loss) and single-modality preservation (encouraged by the autoencoder loss). We provide a 2D illustration of how adCCA gradually aligns the two modalities in the latent space during training (Appendix D.3) well systematic structure test to highlight the impact of the different key components of the adCCA model architecture in achieving high correlation and classification prediction (Appendix Table R1
7

---

Under review as a conference paper at ICLR 2023
3.1.2 PERFORMANCE WITH INPUT TRANSFORMATIONS
A known challenge in the integrative analysis of multimodal data is balancing contributions from different modalities, especially when data are encoded in different formats or the data distributions are unknown. In the adCCA framework, to enable flexibility of input encoding, independent AEs are used to construct latent encodings for each modality, then the representations are aligned indirectly using an adversarial network.
To test the stability of performance, we considered two types of transformations to the raw features y used in the previous experiments: 1) feature distribution transformation, by applying two se- quential log transformations on entry values; and 2) feature encoding transformation, by mapping features into binary codes with local sensitive hashing (Appendix E.2). We evaluated how these transformations changed the consistency between the two modal representations (i.e., correlation) as well as the preservation of original data classes (i.e., classification accuracy).
A
B
Multiple log transform
Binary hashing code
10-
eSCGA
1.-
#800A
Figure 4: Evaluation of perfor- mance after (A) log or (B) bi- nary hashing transform of the in- put Top: Analysis of individ- ual representation learning methods. Axes: consistency (i.e., correlation) and class prediction (i.e., accuracy). Bottom: Analysis of joint represen- tation learning methods. Axis: class prediction. Colors: lighter or darker colors represent results before or af- ter transforms (resp.). Arrows: large changes.
accurary
0.0
DOGA
SVM Renuraate
0.8
DODA
0.6
0-6
6.4
SinCLR
KCCN-poy
po0y
KCCA-RFF
KCOA-RBF
0.0
0.75 1.00
Correlation
Correlation
0.00
0.25
0,50
1.00
0.30
0,25 0.50
0.75 1,00
AODA
ACCS
VODA
VDCA
VODAD
DPOGA
SVM accuracy
SVM accuracy
From Fig. 4 we observed that some methods had stable class prediction scores but unstable cor- relation (e.g. DCCA), while others had unstable class prediction scores but stable correlation (e.g., KCCA-RBF). A few were stable under all tested measurements (e.g., adCCA, CCA. KCCA-poly, VCCA-p). The joint representation learning approaches (ACCA, VCCA and DPCCA) showed strong accuracy change under log transform, implying different distributions of multimodal features can affect the effective information combination. Notably, adCCA was stable under both transforms and performed well overall (Fig. 4 upper right quadrants of top scatter plots). Full results were provided in the Appendix Tables
and
3.2 EXPERIMENTS WITH REAL DATA
Next, we performed evaluations on real datasets with multimodal features. We considered a multi- view MNIST data where two views of the digits were related through transformations and CITE- seq multimodal dataset on spleen and lymph node where gene and protein features were measured directly from single cells.
3.2.1 MNIST: DIGITS WITH TWO TRANSFORMED VIEWS
For each digit, we applied random image transformations (rotation, translation, scaling, shearing and random pixel corrptiion) to generate two views for the same digit image (Appendix E.44]. We then learned single or joint representations for the modalities. We benchmarked how representations from the two views mixed in the latent space and how these representations reflected different digit classes.
From the results (Fig. 5A). we found both the classical and kernel CCA approaches showed compro- mised performance. DCCA and adCCA still maintained high consistency between the two views as well as class information. Further, the runtime (Appendix Fig. D.4] showed adCCA was shower than VCCA, DPCCA and simCLR due to its multiple step training, but faster than classical approaches KCCA, DCCA. We selected these two top approaches and tested them with expanded trans- formation ranges of 2-fold or 3-fold (Appendix E.4). Under these increasing degradation levels, adCCA exhibited the most robust performance with less loss correlation and accuracy (Fig. 5p).
8

---

Under review as a conference paper at ICLR 2023
Figure 5: Integrative analysis of two transformed views from MNIST dataset. (a) Overall performance
(b) Increased data degradations
Method
Correlation
Class prediction
CCA
0.544 + 0.204
0.412 + 0.000
KCCA-poly
0,412 + 0.012
0.325 1 0.004
KCCA-RBF
0.528 + 0.073
0.409 0.001
SVM accuracy
0.8
1-fodd
DCCA
0.991 + 0.008
0.601 + 0.002
f
simCLR
0.845 + 0.064
0.462 + 0.001
S-kad
Told
Contrast AE
0.432 + 0.054
0.475 0.032
9-ibdd 02
ACCA
0,505 + 0.007
VCCA
0.508 + 0.006
3-64d
VCCA-p
0,409 0.003
DPCCA
0.523 1 0.042
0.00
0.25
0.50
75
1.00
Correlation
adCCA
0.970 + 0.016
0.713 + 0.002
3.2.2 CITE-SEQ: CELLS WITH TWO MODALITY PROFILES
Finally, we tested performance on human spleen and lymph node cells measured by CITE-seq [ayoso et al. (202T). This technology profiles genes and proteins simultaneously from the same cells. After data preprocessing 9,264 cells (Appendix E.5 modality x includes 2,000 genes and modality y include 112 proteins. It has been reported that the gene data follow negative binomial distributions, while the protein data follow a negative binomial mixture distribution (Gayoso et al. (202T) Transformations
data may reduce biological signals. Therefore, evaluated how well the different computational methods could directly integrate these two types of data.
SVM prediction
Table 3: Integrative analysis of
Method
Correlation
cell types
35 subtypes
gene and protein modalities from CITE-seq. Prediction results were based on 5-fold cross val- idation using linear SVM. We note that simCLR could not be evaluated because of the unequal modal dimensions.
CCA
0,877 1 0.070
0.481 1 0.001
0.295 1 0.002
KCCA-poly
0.985 + 0.007
0.413 + 0.002
0.225 0.003
KCCA-RBF
0.983 1 0.000
0.523 1 0.000
0.315 1 0.002
DCCA
0.994 1 0.004
0.819 1 0.001
0.738 1 0.003
simCLR
Contrast AE
0.452 1 0.003
0.643 1 0.030
0.389 1 0.012
ACCA
0.917 + 0.003
0.681 + 0.005
VCCA
0.919 + 0.003
0.766
0.006
VCCA-p
0.601 1 0.009
0.496 1 0.004
DPCCA
0.903 + 0.005
0.653 + 0.0TO
adCCA
0.988 1 0.015
0.930 1 0.001
0.781
0.005
We benchmarked using correlations of multimodal representations as well as their prediction ac- curacies on major classes (5) and subclasses (35) of cell types (Table 3}. For correlation, most CCA approaches showed reasonable performance in consistency of latent embeddings. However, approaches outputting joint representations (ACCA and VCCA) had better prediction performance. adCCA maintained both high correlation and prediction accuracy.
DISCUSSION AND FUTURE WORKS
In this work, we formulated the CCA task under the identical latent distribution assumption and derived adCCA framework that can be efficiently solved by standard mini-batch optimizers. The training of adCCA includes both cross-modality consistency constraint and single-modality infor- mation preservation. Analysis of both simulated and real datasets suggests that adCCA provides an effective approach to learn correlated latent representations and maintain original class informa- tion. Compared with the original CCA framework, adCCA has several limitations and directions for future exploration. First, adCCA's latent representations are output from independent encoders and therefore cannot maintain orthogonality of representations. Second, adCCA's latent representa- tions are limited by the AE framework, though the AE can be replaced with more general (such as attention-based model) or modality-specific representation learning frameworks to improve perfor- mance. Third, adCCA training involves simultancously optimization of the GAN and AE, which can be cone-connuning to reach to a balanced state. Finally, the use of inner products to constrain sim- ilarities between modalities in adCCA may be replaced by error models (such as Lg Huber loss) to enforce stronger similarity constraints. Nevertheless, adCCA provides a starting for for scalable alignment of data across modalities that preserves structure within each modality.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Galen Andrew, Raman Arora, Jeff Bilmes, and Karen Livescu. Deep canonical correlation analysis. In International conference on machine learning, pp. 1247-1255, PMLR, 2013.
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In International conference machine learning, pp. 21.2214-223 PMLR, 2017.
Etienne Becht, Leland McInnes, John Healy, Charles-Antoine Dutertre, Immanuel WH Kwok, Lai Guan Ng, Florent Ginhoux, and Evan W Newell. Dimensionality reduction for visualizing single-cell data using umap. Nature biotechnology. 37(1):38 44, 2019.
Ting Chen, Simon Korbblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference machine learning, pp. 1597-1607. PMLR, 2020.
Mayur Datar, Nicole Immorlica, Piotr Indyk, and Vahab S Mirrokni. Socality-sensiting hashing scheme based on p-stable distributions. Proceedings of the twentieth annual symposium on Computational geomelry, pp. 253-262, 2004.
Benjamin Dutton. Adversarial canonical correlation analysis, 2020.
Wenqi Fan, Yao Ma, Han Xu, Xiaorui Liu, Jianping Wang, Qing Li, and Jiliang Tang. Deep adver- sarial canonical correlation analysis. In Proceedings of the 2020 SIAM International Conference on Data Mining, pp. 352-360. SIAM, 2020.
Adam Gayoso, Zoë Steier, Romain Lopez, Jeffrey Regier, Kristopher L Nazor, Aaron Streets, and Nir Yosef. Joint probabilistic modeling of single-cell multi-omic data with totalvi. Nature meth- ods, 18(3):272-222, 2021.
Ian Goodfellow, Jean Pouget-Abadie, Mchdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
David R Hardoon, Sandor Szedmak, and John Shawe-Taylor. Canonical correlation analysis: An overview with application to learning methods. Neural computation, (6(12):26392266 2004.
Anil Jain, Karthik Nandakumar, and Arun Ross. Score normalization in multimodal biometric sys- tems. Pattern recognition, 38(12)22270-21) 2005.
Mahdi Karami and Dale Schuurmans. Deep probabilistic canonical correlation analysis. Proceed- ings of the AAAI Conference on Artificial Intelligence, 35(9):8055-8063, May 2021.
Diederik Kingma and Max Welling. arXiv:1312.6114 2013.
Auto-encoding variational bayes.
arXiv preprint
Pei Ling Lai and Colin Fyfe. Kernel and nonlinear canonical correlation analysis. International Journal of Neural Systems, 10(05):365-377 -77, 2000.
Alireza Makhzani, Jonathon Shlens, Navdeep Jaitly, Ian Goodfellow, and Brendan Frey, Adversarial autoencoders. arXiv preprint arXiv: 1.105444, 2015.
Jiquan Ngiam, Aditya Khosla, Mingyu Kim, Juhan Nam, Honglak Lee, and Andrew Y Ng. Multi- modal deep learning. In ICML, 2011.
Ronan Perry, Gavin Mischler, Richard Guo, Theo Lee, Alexander Chang, Arman Koul, Cameron Franz, Hugo Richard, lain Carmichael, Pierre Ablin, al. mvlearn: Multiview machine learning in python. J. Mach. Learn. Res., 12:10-11 2021.
Nitish Srivastava and Russ R Salakhutdinov. Multimodal learning with deep boltzmann machines. Advances in neural information processing systems, 25, 2012.
Bruce Thompson. Canonical correlation analysis: Uses and interpretation. Number 47. Sage, 1984.
10

---

Under review as a conference paper at ICLR 2023
Zhibin Wan, Changqing Zhang, Pengfei Zhu, and Qinghua Hu. Multi-view information seformation-bottleneckk representation learning. In Proceedings of the AAAI Conference on Artificial Intelligence, vol- ume 35, pp. 10092, 2021.
Weiran Wang, Raman Arora, Karen Livescu, and Jeff Bilmes. On deep multi-view representation learning. In International conference on machine learning, pp. 1083-1092. PMLR, 2015.
Weiran Wang, Xinchen Yan, Honglak Lee, and Karen Livescu. Deep variational canonical correla- tion analysis. arXiv preprint arXiv:1610.0345,, 2016.
Hok Shing Wong, Li Wang, Raymond Chan, and Tieyong Zeng. Deep tensor cea for multi-view learning. /EEE Transactions on Big Data, pp. 1-1, 2021. doi: 10.1109/BBDATA2002 307.3377344

---

Under review as a conference paper at ICLR 2023
APPENDIX
A RELATED WORKS
In the development of CCA, early improvements LLai Fyte (2000) Hardoon al. (2004) focused on employing kernels to generalize the model to non-linear transtormation. DCCA (Andrew et al. (2013) introduced deep learning to CCA and replaced the core matrix transformation with a multi- layered neural network, and DTCCA (Wong et al. (202T) further extended this to more than two views. With progress variational neural networks (Kingma Welling (2013), variational CCA (VCCA) (Wang al. (2016)) re-formulated CCA under graph model and proposed variational CCA using a generative framework. VCCA assumed multimodal data were generated from a shared latent representation and solved it under the variational autoencoder framework. Similarly, deep probabilistic CCA (DPCCA) (Karami & Schuurmans (2021)) took probabilistic interpretation of CCA and decomposed the multimodal features into shared distribution and modality-specific distribution. We note VCCA and DPCCA are different from the original CCA and DCCA as they combined all modalties into joint representation instead of learning a unique transformation for each modality. Our proposed adCCA is different from these approaches in that our model is strictly formulated within the classical CCA framework.
The generative adversarial network (GAN) (Goodfellow et al. (014)) framework was originally proposed learn generative processes underlying a data distribution. The adversarial autoencoder (Makhzani et al. (2155) was the first work to use adversarial loss to match autoencoder latent representation to any prior distribution. A combination of CCA and GAN was used for ACCA (Dutton 2020)), which can be viewed as an extension of the variational CCA. Instead of using Kullback- Leibler (KL) divergence to constrain the prior and posterior distributions in latent space, ACCA used adversarial loss to match the latent representation to arbitrary priors. DACCA (Fan ct al. (2020)) concatenates multiview generation and deep CCA for data generation and representa- tion learning. Most adversarial learning frameworks focus constraining the distributions between generated and real samples, rather than between latent representations from different modalities.
Our proposed adCCA is different from existing approaches in that adCCA directly uses the adver- sarial loss to constrain the latent distributions from multimodal features. We summarize the major differencse in representation learning and training for CCA variants Appendix Table
B MATHEMATICAL FORMULATIONS
B. OPTIMIZATION OF GENERAL CCA FRAMEWORK
Zn E Baxk and Zy E Raxk are mean-centered matrix form representations from two modalities with functions (/7 (X) and (v)) (Y) parameterized by w, and Wy. where X E Raxp and Y e Raxq The correlation can be formulated matrix trace norm:
corr Zz. Zn) S trace
(B.1)
S= S= E-1/2E,, E.ny D 1/2
where Exz - - Zn. Eng = n-T Zy and Exy n LZ Based on Andrew et al. 2Z
(003)), the optimization of w. and wy requires the gradient of correlation to be calculated:
Ōcorr Zy) Ow,
acorr Zv) OZ.
OL,
Ow.
(B.2)
0.10)
(2M Zu + May
Owz
n
where May --1//UVTT E and M,z = - 1/2 UDUTET12 U and V are from the
Singular Value Decomposition (SVD) of S by S = UDVT The term lated similarly.
(corr(Z.. can be formu- Owy
12

---

Under review as a conference paper at ICLR 2023
The calculation in Eq. B.2 requires the SVD decomposition all samples for every training itera- tion. Therefore, is not tractable for standard mini-batch training.
B.2 PROOF OF THEOREM 1
Proof. x Iº and y I represent a pair of random variables with and dimensions in two modalities. 2x fi (x) R and Zy = f2(y) R maps modality features to pair of canonical variables with transformations fi and fz. With that, a general optimization target for CCA can be formulated as:
R. arg max corr Ey)
fi.2
(1 - 1x) [z) - -by)
arg max
(B.3)
OxOy
B(
Malby
arg max f1.f2
0z0y
where Plx and o. are mean and standard deviation for 2 (Resp. My and By for zy). Here, 2y and Ey are assumed to follow the same distribution æ with shared mean and standard deviation μ and a and rewrite the target to:
2 zy)
Mu
f1. = arg max fidfa
020g
E Zny
112
(B.4)
arg max
Ji.fa
o²
arg max E[ (z
M.EEC
/1.f2
where C is the constant number set.
B.3 PROOF COROLLARY
Proof. x E EP and y 12 are random variables with p and q dimensions from modalities. Two canonical representation vectors z, and zy with k dimensions are learned from two autoencoders and constrained by the correct reconstruction of original features. The jth entries of z, and zy follow the latent distribution with fixed mean μ and variance o². Following the Theorem the learning target for CCA with multiple canonical variables be generalized as:
13

---

Under review as a conference paper at ICLR 2023
f1. = arg max fa.f2
corr (z. z (z?)
j=1
(1) (1)
= arg max f1.f2
j=1
n
(1.)) (35)
arg max
2
f1.f2
j=1
11
(B.5)
h Σ
(6,) (15)
= arg max
f1./2 7
j=1
(G:)T (1.)) y
arg max
fi.J2 n
= arg max E 2 Zyl:
f1
where 2 is the jth entry of the vector; (6J) is the jth entry of the representation learned from and zy (6.)) for
sample 2x (1.)) is the whole representation vector from sample (Resp. z 24
(1.)
modality v). We note compared with the SVD-based solution in Eq. B.2 different dimensions in the latent representations are no long orthogonal each other.
C BACKGROUND
C.I AUTOENCODERS
Autoencoders are general unsupervised representation learning framework. Given sample feature vector E IR?, autoencoder tries to learn compact feature I29 from encoder network that can be faithfully reconstructed to the original feature by decoder network. The whole framework optimized by minimizing the reconstruction error:
min E Ex~p= (go - 6,0
-x|1 x|1
(C.1)
where go and r.p are encoder and decoder networks parameterized by 0 and y, respectively; is distribution of input. The representation the output from the encoder z = (x)).
C.2 WASSERSTEIN GAN
The Wasserstein GAN is an improved GAN framework. It modifies the core classification network by using the Wasserstein distance to measure the distribution difference between the generated and realistic samples. Given sample x and a prior distribution z Pz, the Wasserstein GAN is learned a two-step minmax optimization:
min max E Ex-pa fx) - Ezzp, (z)g
(C.2)
where Io and f.r are generator and discriminator with parameter and w; Pa is the actual data distribution and p= the prior distribution to generate samples.
14

---

Under review as a conference paper at ICLR 2023
D SUPPLEMENTARY FIGURES
Modality
B
High correlation, low label accuracy
correlation, high label accuracy
High correlation, high label accuracy
Modality
e-
Modality sample
Modality y sample
Figure D.1: Cartoon illustration of different possible outcomes from canonical representation learn- ing two modalities. (A) Sample raw features from two modalities visualized in 2D. Color indicates the class labels. (B-D) Three scenarios of canonical representations learned from two modalities. (B) 2D embedding of canonical representations with high cross-modality correlation but low class information. (C) 2D embedding of canonical representations with well separation between classes but low correlation between modalities.
(D) Ideal case that canonical representations from two modalities and same classes are well mixed, meanwhile different classes well separated.
Autoencoder optimization
Diccrminator optimiztioo
Genentory optiitattion
Generator optimiztioon
ÄÄ
ÄA
X0% La
Figure D.2: Optimization steps for adCCA. Transparent structures represent modules not involved in the optimization step.
Y Fix representations of x, optimize y
AE initialization
Fix representations of optimize x
Classes
Modalities
Iterative optimization
Figure D.3: Visualization of how two modalities were aligned in the latent space during adCCA optimization. Latent representations were extracted during training and were mapped to 2D space for visualization using Umap. Samples were annotated with classes (top row) or modalities (bottom row).
15

---

Under review as a conference paper at ICLR 2023
6000
5000
4000
Timels
3000
CCA
KCCA-poly
KCCA-RBF
DCCA
emCLR
Contrast AE
ACCA
VCCA
VCCA-p
DPCCA
aiCCA
Method
Figure D.4: Runtime of comparing approaches on MNIST dataset.
1024
512
128
10
128
512
1024
Autoencoder
512
256
+
128
Adversarial
discriminator
1024
512
128
10
128
512
1024
Autoencoder
Figure D.5: Network structure adCCA experiments. Each block includes ID batch normalization and ReLU activation function.
E EXPERIMENT DETAILS
COMPARISON OF APPROACHES
Here we set the latent dimension to 10 for all approaches. For deep-learning based methods, we used the same encoding network [1024,512 256, 10) learn the latent representation.
CCA: we used the multiview CCA function in the mvlearn python package (version 0.5.0) with
regulation paramter regs=0.5.
KCCA: kernel CCA is implemented from the in KMCCA function in mvlearn python package. For Radial Basis Function (a.k.a) kernel, we set y = 1 1,regs = 0.01, sval thresh = 1e For polynomial kernel, we set kernel parameter as degree = 2, coef00 0.1 and regs 0.01.
DCCA: we used the DCCA function in mvlears and the training epoch to 500. Default values are used for all other parameters.
http:://mmllammgihhbbii
16

---

Under review as a conference paper at ICLR 2023
simCLR: made use of the NT -eet simCLR implementation and set the training epoch to 500. The learning step is chosen by grid search from [le 2, le - 3. 4].
Contrast AE:
the implementation is based the adCCA framework. The adversarial learning structure is removed and only the similarity constraint is used learning. The parameter settings and training strategy the same as in adCCA.
ACCA: we used the implementation from the authors The latent representation is generated from both modalities (g(2/2r, y) mode by setting num 2). The L2 reconstruction loss is used.
VCCA and VCCA-p: we used the implementation from ACCA. We followed the original VCCA framework and assume the joint latent representation is generated from single view.
DPCCA: we modified the model from the original paper Parameters were selected from a grid
search of learning steps [1e - 2, le 3, le 4] and epochs 5500, 1000, 2000]. À was set 1, but empirically determined it has little effect on the final results.
adCCA: the model is implemented with the pyTorch package. The network structure used in the experiment is presented in Apendix Fig. [D3] Each layer includes a Bat
and a ReLU activation function. The Adam optimizer was used for both the AE and GAN networks. 3 (coefficients for the running averages of the gradient and its square) was selected from a grid search, giving [0.9, 0.99 for the discriminator and [0.5, 0.9] for the generator and AE. The hyper- parameter À balancing the inner product term and Wasserstein distance was set to 1. Each train- each iteration, the dis-
ing step includes 500 epochs. The learning rate was set to le - criminator optimizer was repeated twice and the generator optimizer repeated 5 times for better convergence in each step. In the GAN optimization, we used the wGAN with gradient penalty 5 Source code and demonstration to run the code provided in supplementary materials.
E.2 SIMULATION DESIGN
We first pre-determined the i classes existing in the data. To get raw features X E Raxp for modality v, multivariate normal distribution with randomly determined mean generate the latent code Zz E Rgaxd Next, we projected the latent code to a new space with a multi-layer neural network:
(1) oZn ((()) ( = w1(())
(E.1)
where (.)) is the sigmoid function; w(a) E Rdxm and W(w) E Rmxp are random projection matrices. Finally, we added random noise and random feature dropouts to obtain the final feature:
x = 5(t) (x +e;E),
(E.2)
where (:: indicates the random feature sampling at a probability [0; is Gaussian noise matrix with the same shape as (2). The X is the final input feature for modality Similarly, we obtained input feature E R'** for modality x for modality y using the same framework with different random projection matrices and drop-out function.
In the experiment, we set & = 30, = = 1, 0000, and noise variances to 0.1 and 0.9 for and y, respectively. Further, 90% and 10% of features randomly missing for these two modalities.
In the experiment of feature transformations (Section 3.1.2), for the feature distribution transfor- mation, we linearly shifted features with minimal value 0 then employed two sequential log 1p
17

---

Under review as a conference paper at ICLR 2023
transformations: for the feature encoding transformation, used local sensitive hashing (LSH)
Datar et al. (2004)) implemented from GitHut to encode features into 1, 000 bits.
E.3 EVALUATION METRIC
Entropy of mixing: put latent representations from and together [z] " In the combined data, we identify its 100 nearest neighbors for each sample and calculate the proportion of neighbors from the same or different modality. The entropy of mixing is given by:
-ploog (1 - p) log(1 p)],
(E.3)
where p is the proportion of neighbors belonging modality
Classification analysis: we employed a 5-fold classification accuracy evaluation. In each data split, fit linear support vector machine (SVM) or d--nearest-eighhoo classifier (kNN) training set (80%) and calculated classification accuracies on the test set (20%). We used implementations of both classifiers from sk learn. For kNN, considered 10 nearest neighbors to build classifier.
Silhouette score:
we use the implementation from sk earn. Given the true sample classes, de the average distance between samples in the same class, dg is the average distance between one sample and its nearest samples from different classes. The score is defined as:
de da
(E.4)
max (dla dg
a positive value indicates the inner-classddistance smaller than between-chas--ditaance.
E.4 MNIST EXPERIMENTS
We applied rotation (range [-100, 10%), translation (range [0,0.1]), scaling (range [0.9,1.1]), shear- ing ([0,15)) and random pixel corruption (10%) to digit images. Each view was obtained through this transformation sequence with randomly chosen transformation parameters. After the transfor- mation, two views were input to approaches, and the results were compared with the original digit labels.
In the experiments with increased corruption levels, further increased the range of each transform to fold and fold (e.g., the rotation angle range was broaded to 2*1-10", 10 and 30--10" 10%) and then tested accuracy.
E.5 CITESEQ EXPERIMENTS
The dataset was downloaded from previous publication 7 includes 9,264 cells from spleen and lymph node in total. Each cell includes genes (> 20,000 dimensions) and surface proteins (112 dimensions). For the gene modality, we selected the top 2,000 variable genes as input; for the protein modality, we input all proteins. Cells are classified to 5 major cell types and 35 cell subtypes.
We used the DLN111-D1 batch, which
F SUPPLEMENTARY TABLES
http:///...aaali..........oe.nnn text html
18

---

Under review as a conference paper at ICLR 2023
Table F.1: Model structure analysis of adCCA. Note: removing Wasserstein distance term is equiv- alent to Contrast AE approach.
Method
Correlation
SVM
adCCA
0,987 + 0.011
0.991
0.003
Remove AE initialization
0.978 + 0.006
0.310
0.106
Remove Wasserstein distance
-0.074 + 0.396
0.294 + 0.032
Remove inner product
0.282 + 0.014
0.134
0.023
Fix decoders
0.772 + 0.045
0.327 0.090
Table F.2: Consistency evaluation of log-transformed feature.
Method
Correlation
Entropy of Mixing
CCA
0.972 = 0.002
0.692 = 0.002
KCCA-poly
0.558 +: 0.029
0.403 + 0.198
KCCA-RBF
0.993 = 0.001
0.693 = 0.000
DCCA
0.982 = 0.010
0.690 = 0.005
simCLR
0.323 + 0.131 NA
0.686 + 0.012
Contrast AE
NA
adCCA
0.998 + 0.002
0.692 + 0.002
Table F3: Class evaluation of log-transformed feature.
Method
ARI
Silhouette
SVM
ENN
CCA
0.032
-0.022
0.118 + 0.026
0.365 + 0.020
KCCA-poly
0.563
0.152
0.124 + 0.040
0.721 + 0.265
KCCA-RBF
0.000
-0.395
0.111
0.010
0.967
0.004
DCCA
0.515
0.148
0.717 + 0.013
0.710 + 0.013
simCLR
0.021
-0.038
0.223 + 0.068
0.151 + 0.033
Contrast AE
NA
NA
NA
ACCA
0.014
-0.03
0.128 1: 0.025
0.155 + 0.005
VCCA
0.000
-0.044
0.112 1 0.017
0.105 + 0.007
VCCA-p
0.013
+0.063
0.123
0.017
0.131 + 0.012
DPCCA
0.147
-0.047
0.354
1 0.018
0.333 1 0.017
adCCA
0.651
0.265
0.995 0.002
0.993 =0.002
Table F4:: Consistency evaluation of binary encoded features.
Method
Correlation
Entropy Mixing
CCA
0.972 + 0,003
0.692 = 0,002
KCCA-poly
0.762 1 0.008
0.517 1 0.140
KCCA-RBF
0.998 +: 0.001
0.693 + 0.000
DCCA
0.993 + 0.004
0.691 = 0.003
simCLR
0.274 1 0.121
0.688 + 0.006
Contrast AE
NA
NA
adCCA
0.995 # 0,005
0,692 = 0,002
Table F:5: Class of evaluation of binary encoded features.
Method
ARI
Silhouette
SVM
ANN
CCA
0.013
-0.029
0.111 0.011
0.297
0.003
KCCA-poly
0.050
-0.023
0.117 + 0.025
0.276
0.085
KCCA-RBF
0.002
-0.020
0.107 1 0.002
0.178
0.021
DCCA
0.332
0.052
0.611 0.009
0.585
0.011
simCLR
0.006
-0.033
0.151 + 0.022
0.117 0.007
Contrast AE
NA
NA
NA 0.028
NA
ACCA
0.475
0.116
0.493
0.701 0.016
VCCA
0.427
0.038
0.928 + 0.017
0.817 0.018
VCCA-pr
0.002
-0.043
0.1.3
0.017
0.123
0.014
DPCCA
0.003
-0.028
0.119
0.021
0.102 0.011
adCCA
0.530
0.100
0.943 = 0.004
0.934 0.010
19

---

Under review as a conference paper at ICLR 2023
Table F.6: A summary of representation and training for CCA variants.
Method
Representation type
Training
CCA
Modality-speciffe representation
Large/full batch
KCCA-poly
Modality-spcciftc representation
Large/full hatch
KCCA-RBF
Modality-specific representation
Large/full batch
DCCA
Modality-peccific representation
Large/full batch
ACCA
Single representations
Mini-bate batch
VCCA
Single joint representations
Mini- Mini-batch
VCCA-p
Single joint representations
Mini-batch
DPCCA
Single representations
Mini-batch
adCCA
Modality-specifie representation
Mini-batch
20


Reviews:
[{"review_id": "xDHGGKhVcTR", "reviewer": "Reviewer_yzLx", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "A method is proposed for learning correlated representations of two data views that purports to be superior to existing approaches in terms of correlation captured, utility of representations for classification, and training efficiency (since stochastic optimization can be used).", "summary_of_the_review": "There may be some value to the proposed method, but the motivation is suspect and the paper is less than clear in several places.", "strength_and_weaknesses": "The paper employs some interesting techniques and could be of value, but a few points were not clear to me.\n\n1) The proposed technique is claimed to have the advantage of not needing to \"balance the two modalities\". What is meant by that? How do existing methods for learning correlated representations suffer from \"unbalanced modalities\"? Using some modality-specific transformation certainly sounds reasonable, independent of concerns about not being \"balanced\".\n2) The derivation of Theorem 1 and (especially) corollary are suspect. How can you maximize the sum of the correlations of the components without any orthogonality/uncorrelatedness constraint? Is there anything then stopping the components from all being equal, specifically all representing the first canonical correlation? That is what the uncorrelatedness constraint is there to prevent.\n3) Eq. (5) is not clear to me. How does this represent the Wasserstein distance? Can you explain how this adversarial learning achieves what you claim?\n4) In many places, \"constraints\" are discussed (e.g., \"we constrain the latent representations for each of the two modalities to follow the same distribution\") but as far as I can tell these are not really constraints; they take the form of penalties in the joint objective.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "ovZmFIkz-P", "reviewer": "Reviewer_NwB7", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies the canonical correlation analysis under an adversarial learning framework. They propose an effective adCCA that can learn better representations. The experiments show the proposed method is competitive compared to many recent deep CCA. ", "summary_of_the_review": "This paper proposed a deep CCA model under adversarial learning framework. The proposed method is novel and the experiments are sufficient. \n\n", "strength_and_weaknesses": "The proposed method seems interesting. The experiments are sufficient to show the effectiveness of the proposed adCCA. I have some concerns listed as follows:\n\n1. The preliminary results show that latent features $z_x$ and $z_y$ have the same latent distribution, with a mean of $mu$ and a variance of $sigma2$. And section 2.2 presents this shared latent distribution is achieved by using batch normalization. This is confusing information. As illustrated in Figure 1, adCCA employs two distinct neural networks, $g(x)$ and $g(y)$, to extract the corresponding latent features. However, it's difficult to understand how to use the BN techniques to constrain the two modality features to follow the same distribution. \n2. On page 5, the authors said that ''This permits the use of the modality specific structures and removes the requirement of modality balance.'' Although this sentence is easy to follow, this statements are not supported by experiments or conclusions from previous works. I think it's better to clarify these statements (maybe from relevant literatures). Or this statements seem to be the authors' hypothesis.\n3. The descriptions of the neural networks through the whole paper are missing, which is very important for Reproducibility. \n4. According to the Figure 5 and Table 3, we find that higher correlation may not lead to better class prediction. What is the cause of this phenomenon? Is there a trade-off problem between correlation and accuracy? Moreover, we also find that most of the joint representation learning methods achieve worse results. Is this a problem inherent in this type of approach? Or is this problem due to the asymmetric information between different modalities? I think the authors should give more analysis in the experiments.\n5.In Algorithm 1, the initialization of $z_x$ and $z_y$ is link to a pre-trained model. So, how much does this initialization process affect the result? For example, do we remove it? Because in the next part, we also update the parameter of the AE network. Moreover, after we achieve two pre-trained AEs, shall we fix the parameter $\\Phi$ during the second iterative optimization?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "_mk7DrPvGFB", "reviewer": "Reviewer_fcpJ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work derived a new deep adversary CCA method, which is called adCCA, from DCCA framework. The authors frame CCA problem under the assumption that the representations of different modalities obey an identical distribution in a common latent subspace. Compared to other DNN-based CCA methods, the authors claim 1) their approach is able to learn highly correlated representations across modalities, without deforming original structure of each modal. 2) they simplified analytical solution of CCA problem with their identical distribution assumption. 3) extensive experiments valid their method and demonstrate the robustness of their model. ", "summary_of_the_review": "The current version of paper is not ready to publish since there are some concerns need to be addressed. ", "strength_and_weaknesses": "Strengths:\n- They conducted extensive experiments to demonstrate the effectiveness of their method\n\nMain concerns:\n- The paper is hard to follow due to the incoherence and imprecision in their writing. For instance, the second paragraph in introduction section \"it is temping ...\". It is unclear to me what does it refer to in this sentence. the second paragraph on page 2, \"This avoids having to balance the two modalities.\" I don't quite get what messages the authors try to convey via this sentence. \n- All theorems were built on a faithless assumption. They assume \"latent representations from modality x and y follow the same distribution $\\pi$ with the mean $\\mu$ and variance $\\sigma^2$\" which is weird to me. In practice, the distribution discrepancy is commonly occurred among data that come from different sources/views/modalities. At least, at the initial stage I don't think the representation distribution obey their assumption. However, they derived their theorems from their fundamental assumption which is untrustworthy. \n- The authors emphasized that their method is able to learn maximally correlated representations without comprising structure in each modalities. I was wondering how do the authors prove the structures among modalities were not comprised. How should I understand the correlation of representations is maximized while the structure are not deformed? I would very much appreciate the authors if they can help me understand this.\n- I don't think it is fair to claim resolving adCCA with mini-batch optimization as one major contribution of this work since  DCCA (Wang et al.) already confirmed stochastic mini-batch optimization works for DCCA based methods.\n- I would expect the authors compare their method with [1,2] from the methodology perspective. \n[1]. Benjamin Dutton, Adversarial Canonical Correlation Analysis, 2020\n[2] Fan et al., Deep Adversarial Canonical Correlation Analysis, 2020", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject