Year: 2020

Abstract:
Canonical Correlation Analysis (CCA) models can extract informative correlated representations from multimodal unlabelled data. Despite their success, CCA models may break if the number of variables exceeds the number of samples. We propose Deep Gated-CCA, a method for learning correlated representations based on a sparse subset of variables from two observed modalities. The proposed procedure learns two non-linear transformations and simultaneously gates the input variables to identify a subset of most correlated variables. The nonlinear transformations are learned by training two neural networks to maximize a shared correlation loss defined based on their outputs. Gating is obtained by adding an approximate `0 regularization term applied to the input variables. This approximation relies on a recently proposed continuous Gaussian based relaxation for Bernoulli variables which act as gates. We demonstrate the efficacy of the method using several synthetic and real examples. Most notably, the method outperforms other linear and non-linear CCA models.

heading: 1 INTRODUCTION
text: Canonical Correlation Analysis (CCA) (Hotelling, 1936; Thompson, 2005), is a classic statistical method for finding the maximally correlated linear transformations of two modalities (or views). Using modalities X ∈ RDx×N and Y ∈ RDY ×N , which are centered and have N samples with Dx and Dy features respectively. CCA seeks canonical vectors ai ∈ RDX , and bi ∈ RDY , such that ,ui = aTi X, and vi = b T i Y , i = 1, ..., N , maximize the sample correlations between ui and vi, where ui (vi) form an orthonormal basis for i = 1, ..., d, i.e.
ai, bi = argmax 〈ui,uj〉=δi,j ,〈vi,vj〉=δi,j ,i,j=1,...,d Corr(ui, vi). (1)
While CCA enjoys a closed-form solution using a generalized eigen pair problem, it is restricted to the linear transformationsA = [a1, ...,ad] andB = [b1, ..., bd].
In order to identify non-linear relations between input variables, several extensions of CCA have been proposed. Kernel methods such as Kernel CCA (Bach & Jordan, 2002), Non-paramatric CCA (Michaeli et al., 2016) or Multi-view Diffusion maps (Lindenbaum et al., 2020) learn the non-linear relations in reproducing Hilbert spaces. These methods have several shortcomings: they are limited to a designed kernel, they requireO(N2) computations for training, and they have poor interpolation and extrapolation capabilities. To overcome these limitations, Andrew et al. (2013) have proposed Deep CCA, to learn parametric non-linear transformations of the input modalities X and Y . The functions are learned by training two neural networks to maximize the total correlation between their outputs.
Linear and non-linear canonical correlation models have been widely used in the setting of unsupervised or semi-supervised learning. When d is set to a dimension satisfying d < Dx, Dy , these models find dimensional reduced representations that may be useful for clustering, classification or manifold learning in many applications. For example, in biology (Pimentel et al., 2018), neuroscience (Al-Shargie et al., 2017), medicine (Zhang et al., 2017), and engineering (Chen et al., 2017). One key limitations of these models is that they typically require more samples than features, i.e. N > Dx, Dy . However, if we have more variable than samples, the estimation based on the closed form solution of the CCA problem (in Eq. 1) breaks (Suo et al., 2017). Moreover, in high dimensional data, often some of the variables are not informative and thus should be omitted
from the transformations. For these reasons, there has been a growing interest in studying sparse CCA models.
Sparse CCA (SCCA) (Waaijenborg et al., 2008; Hardoon & Shawe-Taylor, 2011; Suo et al., 2017) uses an `1 penalty to encourage sparsity of the canonical vectors ai and bi. This can not only remove the degeneracy inherit to N < Dx, Dy , but can improve interpetability and performance. One caveat of this approach is its high computational complexity, which can be reduced by replacing the orthonormality constraints on ui and vi with orthonormality constraints on ai and bi. This procedure is known as simplified-SCCA (Parkhomenko et al., 2009; Witten et al., 2009), which enjoys a closed form solution. There has been limited work on extending these models to sparse nonlinear CCA. Specifically, there are two kernel based extensions, two-stage kernel CCA (TSKCCA) by Yoshida et al. (2017) and SCCA based on Hilbert-Schmidt Independence Criterion (SCCAHSIC) by Uurtio et al. (2018). However, these models suffer from the same limitations as KCCA and are not scalable to a high dimensional regime.
This paper presents a sparse CCA model that can be optimized using standard deep learning methodologies. The method combines the differentiable loss presented in DCCA (Andrew et al., 2013) with an approximate `0 regularization term designed to sparsify the input variables of both X and Y . Our regularization relies on a recently proposed Gaussian based continuous relaxation of Bernoulli random variables, termed gates (Yamada et al., 2020). The gates are applied to the input features to sparsify X and Y . The gates parameters are trained jointly via stochastic gradient decent to maximize the correlation between the representations of X and Y , while simultaneously selecting only the subsets of the most correlated input features. We apply the proposed method to synthetic data, and demonstrate that our method can improve the estimation of the canonical vectors compared with SCCA models. Then, we use the method to identify informative variable in multichannel noisy seismic data and show its advantage over other CCA models.

heading: 1.1 BACKGROUND

heading: 1.2 DEEP CCA
text: Andrew et al. (2013), present a deep neural network that learns correlated representations. They proposed Deep Canonical Correlation Analysis (DCCA) which extracts two nonlinear transformations of X and Y with maximal correlation. DCCA trains two neural networks with a joint loss aiming to maximize the total correlation of the network’s outputs. The parameters of the networks are learned by applying stochastic gradient decent to the following objective:
θ∗X , θ ∗ Y = argmax
θX , θY Corr(f(X; θX), g(Y ; θY )), (2)
where θX and θY are the trainable parameters, and f(X), g(Y ) ∈ Rd are the desired correlated representations.

heading: 1.3 SPARSE CCA
text: Several authors have proposed solutions for the problem of recovering sparse canonical vectors. The key advantages of sparse vectors are that they enable identifying correlated representations even in the regime of N < Dx, Dy and they allow unsupervised feature selection. Following the formulation by Suo et al. (2017), SCCA could be described using the following regularized objective
a, b = argmin [ − Cov(aTX, bTY ) + τ1‖a‖1 + τ2‖b‖1 ] ,
subject to ‖aTX‖2 ≤ 1, ‖bTY ‖2 ≤ 1,
where τ1 and τ2 are regularization parameters for controlling the sparsity of the canonical vectors a and b. Note that the relaxed inequality constrain on aTX and bTY makes the problem bi-convex, however, if ‖aTX‖2 < 1 or ‖bTX‖2 < 1, then the covariance in the objective is no longer equal to the correlation.

heading: 1.4 STOCHASTIC GATES
text: In the last few years, several methods have been proposed for incorporating discrete random variables into gradient based optimization methods. Towards this goal, continuous relaxations of discrete random variables such as (Maddison et al., 2016; Jang et al., 2017) have been proposed.
Such relaxations have been used in several applications, for example, model compression (Louizos et al., 2017), feature selection or for defining discrete activations (Jang et al., 2016). We focus on a Gaussian-based relaxation of Bernoulli variables, termed Stochastic Gates (STG) (Yamada et al., 2020), which were originally proposed for supervised feature selection. We denote the STG random vector by z ∈ [0, 1]D, where each entry is defined as
z[i] = max(0,min(1, µ[i] + [i])), (3)
where µ[i] is a trainable parameter for entry i, the injected noise [i] is drawn fromN (0, σ2) and σ is fixed throughout training. This approximation can be viewed as a clipped, mean-shifted, Gaussian random variable. In Fig. 1 we illustrate generation of the transformed random variable z[i] for µ[i] = 0.5 which represents a “fair” relaxed Bernoulli variable.

heading: 2 DEEP GATED CCA

heading: 2.1 MODEL
text: It is appealing to try to combine ideas from Sparse CCA into the rich differentiable model of Deep CCA. However, a straight forward `1 regularization of the input layer of a neural network does not work in practice because it makes the learning procedure unstable. This was observed in the supervised setting by Li et al. (2016); Feng & Simon (2017). This instability occurs because the objective is not differentiable everywhere. To overcome this limitation, we use the STG random variables (see Eq. 3) by multiplying them with the features of X and Y . Then, by penalizing for active gates using a regularization term E‖z‖0, we can induce sparsity in the input variables. We formulate the problem of sparse nonlinear CCA by regularizing a deep neural network with a correlation term. We introduce two random STG vectors into the input layers of two neural networks which are trained in tandem to maximize the total correlation. Denoting the random gating vectors zx and zy for viewX and Y respectively, the Deep Gated CCA (DG-CCA) loss is defined by
L(θ,µ) = Ezx,zy [ − Corr(f(zTxX; θX), g(zTy Y ; θY )) + λx‖zx‖0 + λy‖zy‖0 ] , (4)
where θ = (θX ,θY ), µ = (µX ,µY ) are the model parameters, and λx, λy are regularization parameters that control the sparsity of the input variables. The vectors zx and zy are random STG vectors, with elements defined based on Eq. 3.
Fig. 2 highlights the proposed architecture. Each observed modality is first passed through the gates. Then, the outputs of the gates are used as inputs to a view-specific neural sub-net. Finally, the shared loss term in Eq. 4 is minimized by optimizing the parameters of the gates and neural sub-nets.

heading: 2.2 ALGORITHM DETAILS
text: We now detail the procedure used in DG-CCA for minimizing the loss L(θ, µ) (in Eq. 4). The regularization is based on a parametric expectation and therefore, can be expressed by
Ezx‖z‖0 = Dx∑ i=1 P(zx[i] ≥ 0) = Dx∑ i=1 ( 1 2 − 1 2 erf ( −µx[i]√ 2σ )) ,
where erf() is the Gaussian error function, and is defined similarly for Ezy‖zy‖0. Denoting the centered output representations of X,Y by ΨX ,ΨY ∈ Rd×N respectively, the empirical covariance matrix between these representations can be expressed as Σ̂XY = 1N−1ΨXΨ T Y . Using a similar notations, we express the regularized empirical covariance matrices of X and Y as Σ̂X = 1 N−1ΨXΨ T X + γI and Σ̂Y = 1 N−1ΨY Ψ T Y + γI , where the matrix γI (γ > 0) is added to
stabilize the invertability of Σ̂X and Σ̂Y . The total correlation in Eq. 4 can be expressed using the trace of Σ̂ −1/2 Y Σ̂Y XΣ̂ −1 X Σ̂XY Σ̂ −1/2 Y .
To learn the parameters of the gates µ and of the representations θ we apply stochastic gradient decent to L(θ, µ). Specifically, we used Monte Carlo sampling to estimate the left part of Eq. 4. This is repeated for each batch, using one Monte Carlo sample per batch as suggested by Louizos et al. (2017) and Yamada et al. (2020), and worked well in our experiments. After training we remove the stochastic part of the gates, and use only variables ix ∈ {1, ..., DX} and iy ∈ {1, ..., Dy} such that zx[ix] > 0 and zy[iy] > 0.

heading: 3 RESULTS
text: In the following section we detail the evaluation of the proposed approach using synthetic and real datasets. We start with two linear examples, demonstrating the performance of DG-CCA when N DX , DY . Then, we use noisy images from MNIST and seismic data measured using two channels to demonstrate that DG-CCA finds meaningful representations of data even in a noisy regime. For a full description of the training procedure as well as the baseline methods, we refer the reader to the Appendix.

heading: 3.1 SYNTHETIC EXAMPLE
text: We start by describing a simple linear model also experimented by Suo et al. (2017). Consider data generated from the following distribution ( X Y ) ∼ N( ( 0 0 ) , ( ΣX ΣXY
ΣY X ΣY
) ), where ΣX =
ΣY = I800, and ΣXY = ρΣX(φηT )ΣY . Here φ,η ∈ R800, are sparse with 5 nonzero elements
and ρ = 0.9. The indices of the active elements are chosen randomly with values equal to 1/ √
5. In this setting, based on Proposition 1 in (Suo et al., 2017), the canonical vectors a and b which maximize the objective in Eq. 1 are φ and η respectively.
Using this model we generate 400 samples and estimate the canonical vectors based on CCA and DG-CCA. In Fig. 3 we present a regularization path of the proposed scheme. Specifically, we apply DG-CCA to the data described above using various values of λ = λx = λy . We present the `0 of active gates (by expectation) along with the empirical correlation between the extracted representations Σ̂XY , which is also ρ̂ = φ̂XY T η̂T . As evident from the left panel, there is a wide range of λ values such that DG-CCA converges to true number of coefficients (10) and correct correlation value (0.9). Next, we present the values of φ, the DG-CCA estimate (using λ = 30) of the canonical vector φ̂, and the CCA based estimate of the canonical vector â. The solution by CCA is wrong and not sparse, while the DG-CCA solution correctly identifies the support of φ. Finally, we evaluate the estimation error of φ using Eφ = 2(1 − |φ
T φ̂|), and Eη is defined similarly. In Table 1 we present the estimated correlation along with the estimation errors ofφ and ρ (averaged over 100 simulations). As baselines we present the results simulated by Suo et al. (2017) (mod-SCCA), comparing the performance to PMA (Witten et al., 2009) and SCCA (Chen et al., 2013).
empirical correlation between the estimated representations, i.e. ρ̂ = φ̂ T XTY η̂, where φ̂ and η̂ are the estimated canonical vectors. Dashed lines indicate the correct number of active coefficients (green) and true correlation ρ (blue). Note that for small values of λ = λx = λy the model select more variables than needed and attains a higher correlation value, this is a similar over-fitting phenomenon that CCA suffers from. Right: True canonical vector φ along with the estimated vectors using DG-CCA (φ̂) and CCA (â).

heading: 3.2 MULTI VIEW SPINNING PUPPETS
text: As an illustrative example we use a dataset collected by Lederman & Talmon (2018) for multiview learning. The authors have generated two videos capturing rotations of 3 desk puppets. One camera captures two puppets, while the other captures another two, where one puppet is shared across cameras. A snapshot from both cameras appears in the top row of Fig. 4. All puppets are placed on a spinning device that rotates the dolls at different frequencies. In both video there is a shared
parameter, namely the rotation of the common bulldog. Even thought the Bulldog is captured from a slightly different angle, we attempt to use CCA to identify a linear transformation that projects the two Bulldogs in to a common embedding. We use a subset of the spinning puppets dataset, with 400 images from each camera. Each image has 240 × 320 = 76800 pixels (using a gray scaled version of the colored image), therefore, there are more feature than samples and direct application of CCA would fail. We apply the proposed scheme using λy = λx = 50, a linear activation and embedding dimension d = 2. DG-CCA converges to embedding with a correlation of 1.99 using 372 and 403 pixels from viewsX and Y . The active gates are presented in the bottom row of Fig. 4.
In Fig. 5 we present the coupled two dimensional embedding of both videos. Both embeddings are highly correlated with the angular orientation of the Bulldog. Note that adjacent images in the embedding are not necessarily adjacent in the original ambient space, this is because the Bunny and the Yoda puppets are gated and do not affect the embedding.

heading: 3.3 NOISY MNIST
text: MNIST (LeCun et al., 2010) which consists of 28×28 grayscale digit images, with 60K/10K images for training/testing, is a well known and widely investigated dataset. We use a challenging variant of MNIST with coupled noisy views. The first view is created by adding noise drawn uniformly from [0, 1] to all pixels. The second view is created by placing a random patch from a natural image in the background of the handwritten digits. Both views are based on different digit instances drawn from the same digit class. Random samples from both views are presented in Fig. 6. Both views consists of 62, 000 samples, of which we use 40, 000 for training 12, 000 for testing and 10, 000 are used as a validation set. Here the validation set is used for early stopping of the training procedure and optimizing λ = λx = λy .
Multi-view processing of the two noisy views can generate an informative representation of the noisy MNIST data. In the following we will focus on performing unsupervised embedding of each noisy MNIST view into correlated 10 dimensional space.
By reducing the correlation cost, the DG-CCA learns which per-view pixels are relevant and informative in correlation maximization sense. In the bottom right corner of Fig 6 we present the location of the active gates. DG-CCA selects features (pixels) within an oval-like shape in the center of each view thus capturing the digit information and reducing the influence of the noise.
To measure the class separation in the learned embedding, we apply k-means clustering to the stacked embedding of the two views. We run k-means (with k = 10) using 20 random initializations and record the run with the smallest sum of square distances from centroids. Given the cluster assignment, k-means clustering accuracy (KM ACC) and mutual information (MI) are measured using the true labels. Additionally, we train a Linear-SVM (LSVM) model on our train and validation datasets. LSVM classification accuracy (LSVM ACC) is measured on the remaining test set. The performance of DG-CCA compared with several baselines appears in Table 2. In the appendix we provide all implementation details and provide an experiment demonstrating the performance for various values of λ = λx = λy .

heading: 3.4 SEISMIC EVENT CLASSIFICATION
text: Next, we evaluate the method using a dataset studied by Lindenbaum et al. (2018). The data consists of 1609 seismic events. Here, we focus on 537 explosions which are categorized into 3 quarries. The events occurred between the years 2004 − 2015, in the southern region of Israel and Jordan. Each event is recorded using two directional channels facing east (E) and north (N), these comprise the coupled views for the correlation analysis. Following the analysis by Lindenbaum et al. (2018), the input features are Sonogram representations of the seismic signal. Sonograms are time frequency
representations with bins equally tempered on a logarithmic scale. Each Sonogram z ∈ R1157 with 89 time bins and 13 frequency bins. An example of Sonograms from both channels appears in the top row of Fig. 7.
We create the noisy seismic data by adding sonograms computed based on vehicle noise from 1. Examples of noisy sonograms appear in the middle row of Fig 7. We omit 20% of the data as a validation set. Then we train DG-CCA to embed the data in 3 dimensions using several values for λ = λx = λy . Then, we use the model that attains maximal correlation on the validation set. In Table 2 we present the MI, k-means and SVM accuracies computed based on DG-CCA embedding. Furthermore, we compare the performance with several other baselines. Here, the proposed scheme improves performance in all 3 metrics while identifying a subset of 71 and 68 features from channel E and N respectively. The active gates are presented in the bottom row of Fig. 7.

heading: 4 CONCLUSION
text: In this paper we present a method for learning sparse non-linear transformations which maximize the canonical correlations between two modalities. Our method is realized by gating the input layers of two neural networks which are trained to maximize their output’s total correlations. Input variables are gated using a regularization term which encourages sparsity. This allows us to learn informative representations even when the number of variables far exceeds the number of samples. Finally, we demonstrate that the method outperforms existing methods for linear and non-linear canonical correlation analysis.
1https://bigsoundbank.com/search?q=car

heading: A APPENDIX
text: A.1 GATES INITIALIZATION
The Gaussian based stochastic gates suggested by Yamada et al. (2020) are based on trainable parameters µ and a constant parameter σ. These control the mean and standard deviation of the injected noise respectively. Yamada et al. (2020) have initialize all values of µ to 0.5, in this case, the gates approximate “fair” Bernoulli parameters. This is a reasonable choice, if no prior knowledge about the solution is available, however, we can utilize the closed form solution of the CCA problem to derive a smarter initialization procedure for the parameters of the gates. Specifically, given the empirical covariance matrixCXY = XY T
(N−1) , we denote the thresholded covariance matrix by SXY , with values defined as follows
(SXY )ij = { (CXY )i,j , if |(CXY )i,j | > δ 0, otherwise.
Where δ is a threshold value selected based on the estimated sparsity of X and Y . Specifically, if we assume that r percent of the values should be zeroed, then δ is set to be the r-th percentile of |(CXY )|. Then we compute the leading singular vectors u and v of SXY . We further threshold the absolute values of these vectors (using the same percentile used for SXY . The initial values of the parameters of the gates are then defined by µX = ū+ 0.5, and µY = v̄ + 0.5, where ū and ū are the thresholded versions of the absolute value of the singular vectors.
The standard deviation of the injected noise σ was set to 0.5 by (Yamada et al., 2020). They have selected this value as it maximized the gradient of the regularization term at initialization.
Empirically, we have observed that for DG-CCA smaller values of σ translate to improved convergence. Specifically, we have used σ = 0.25 which worked well in our experiments. Studying the effect of σ is an open question that we aim to pursue in future study.
A.2 ADDITIONAL EXPERIMENTAL DETAILS
In the following sections we provide additional experimental details required for reproduction of the experiments provided in the main text.
A.2.1 SYNTHETIC EXAMPLE
For the linear model we use a learning rate of 0.005 with 10, 000 epochs. The values of λx and λy are both set to 30. These values were obtained using a cross validation procedure. We run the method 100 times with different realizations of X and Y . Importantly, following Suo et al. (2017) we present the average errors for the estimation of the canonical vectors, however the median values are one order of magnitude better, specifically Eψ = 0.0017 and Eη = 0.0020.
A.2.2 NOISY MNIST
In this subsection we provide additional details regarding the noisy MNIST experiment. In Fig 8, we present the performance as a function of the number of active gates (pixels) controlled by λx = λy . The MI score, k-means and SVM accuracy were computed based on DG-CCA embedding with learning rate of 0.01. Furthermore, the number of epochs (∼ 4000) was tuned by early stopping using random validation of size 10000. To learn 10 dimensional correlated embedding, we use the same architecture as suggested by (Wang et al., 2015) consisting of three hidden layers with 1000 neurons each. The number of dimensions in the embedding was selected based on the number of classes in MNIST. This architecture is used for both DCCA and DG-CCA. Note that for DG-CCA using small values of the regularization parameters λx and λy , increases the number of selected features and the degrades performance. This is duo to the fact that as more features are selected more noise is introduced into the extracted representation (of size 10).It is interesting to note that the k-means was more robust to the introduced noise than the LSVM.
The regularization parameters λx and λy balances between the correlation loss and the amount of sparsification performed by the gates. These hyper parameters are tuned using the validation set in by maximizing the total correlation value. We compare DG-CCA to CCA (Chaudhuri et al., 2009)2, KCCA (Bach & Jordan, 2002)3, NCCA (Michaeli et al., 2016) 4 and DCCA (Andrew et al., 2013) 5. For all methods we use an embedding with dimension 10, and evaluate performance with k-means using 20 random initilizations, and using LSVM by performing training on the training samples and testing on the remaining samples (split defined in the main text). In this experiment we tried to train SCCA-HSIC (Uurtio et al., 2018) 6 for over two days, but it did not converge. Furthermore, we believe that the poor performance of the kernel methods are degraded due to the high level of noise in the input images.
A.2.3 SEISMIC EVENT CLASSIFICATION
Using the seismic data, we compare the performance of DG-CCA with a linear and non-linear activation. In this exaple, we use a learning rate of 0.01 with 2000 epochs. The number of neurons for each hidden layer are: 300, 200, 100, 50, 40, with a Tanh activation. he number of dimensions in the embedding (d = 3) was selected based on the number of classes in the data. Parameters are optimized manually to maximize the correlation on a validation set. In Fig. 9 we present SVM accuracy for different levels of sparsity. The presented number of features is the average over both modalities, and SVM performance is evaluated using 5-folds cross validation. We compare DGCCA to CCA (Chaudhuri et al., 2009), SCCA (Suo et al., 2017), SCCA-HSIC (Uurtio et al., 2018), KCCA (Bach & Jordan, 2002) , NCCA (Michaeli et al., 2016) and DCCA (Andrew et al., 2013). For
2https://scikit-learn.org/stable/modules/generated/sklearn.cross_decomposition.CCA.html 3https://gist.github.com/yuyay/16ce4914683da30f87d0 4https://tomer.net.technion.ac.il/files/2017/08/NCCAcode_v3.zip 5https://github.com/adrianna1211/DeepCCA_tensorflow 6https://github.com/aalto-ics-kepaco/scca-hsic
all methods we use an embedding with dimension 3, and evaluate performance with k-means using 20 random initilizations, and using linear SVM by performing a 5-folds cross validation. For the kernel methods we evaluated performance by constructing a kernel using k = 5, 10, ..., 50, nearest neighbors and selected the value which maximized performance in terms of total correlation.

Review summaries:
This paper presents a new deep CCA method to learn non-linear relationships between two modalities. It trains two neural networks each for a modality to maximize the total correlations of their output representations. Gating is applied to input variables by associating each with a latent Bernoulli variables which is then relaxed with the clipped Gaussian random variable. Experiments on one synthetic and two real datasets demonstrate the superiority of the proposed method.

This paper proposes a DL method for learning sparse non-linear transformations that maximize correlations between two views. In particular, each view is passed through a separate network. Stochastic Gating is applied to the input layer of each network. The two networks are jointly trained by maximising the correlation between their outputs. Sparsity is obtained by imposing L0 regularization terms on the Stochastic Gating variables.