A Multilinear Least-Squares Formulation for Sparse Tensor Canonical Correlation Analysis
Authors: Jun Yu
Zhaoming Kong
Kun Chen
Xin Zhang
Yong Chen
Lifang He
Venue: Accepted by TMLR ()

Abstract: Tensor data are becoming important recently in various applications, e.g., image and video recognition, which pose new challenges for data modeling and analysis approaches, such as high-order relations of large complexity, varying data scale and gross noise. In this paper, we consider the problem of sparse canonical correlation analysis for arbitrary tensor data. Although several methods have been proposed for this task, there are still limitations hindering its practical applications. To this end, we present a general Sparse Tensor Canonical Correlation Analysis (gSTCCA) method from a multilinear least-squares perspective. Specifically, we formulate the problem as a constrained multilinear least-squares problem with tensor-structured sparsity regularization based on CANDECOMP/PARAFAC (CP) decomposition. Then we present a divide-and-conquer deflation approach to tackle the problem by successive rank-one tensor estimation of the residual tensors, where the overall model is broken up into a set of unconstrained linear least-squares problems that can be efficiently solved. Through extensive experiments conducted on five different datasets for recognition tasks, we demonstrate that the proposed method achieves promising performance compared to the SOTA vector- and tensor-based canonical correlation analysis methods in terms of classification accuracy, model sparsity, and robustness to missing and noisy data. The code is publicly available at https://github.com/junfish/gSTCCA.

Published in Transactions on Machine Learning Research (01/2024)
A Multilinear Least- Formulation for Sparse Tensor Canonical Correlation Analysis
Jun Yu
juy2200lehighh.edi edu
Deportment of Computer Science and Engineering
Lehigh Universily
Zhaoming Kong
kong. cong.zmBBll amm.lmmmll.ccmmm scul. edu.cccc.o
School of Software Engineering
South China University of Technology
Kun Chen
Sun.chen@ucomee edu
Department of Statistics
University of Connecticul Xin Zhang
henry@stal.fsu edu
Department of Statistics Florida State University
Yong Chen
ychen 1
Department of Biostatistics, Epidemiology and Informatics
University of Pennsylvania
Lifang He
lih319 edu
Deportment of Computer Science and Engineering Lehigh University
Reviewed on OpenReview: https: openreview. net/ forum? id= zcOVOcAuTV
Abstract
Tensor data are becoming important recently in various applications, e.g., image and video recognition, which pose new challenges for data modeling and analysis approaches, such as high-order relations of large complexity, varying data scale and gross noise. In this paper, we consider the problem of sparse canonical correlation analysis for arbitrary tensor data. Although several methods have been proposed for this task, there are still limitations hindering its practical applications. To this end, we present general Sparse Tensor Canonical Correlation Analysis (gSTCCA) method from a multilinear least-squares perspective. Specifically, we formulate the problem as a constrained multilincar least-squares problem with tensor-structured sparsity regularization based on LANDECOMP/PARATE (CP) decomposition. Then we present a divide-and-conquer deflation approach to tackle the problem by successive rank-one tensor estimation of the residual tensors, where the overall model is broken up into set of unconstrained linear least-squares problems that can be elficiently solved. Through extensive experiments conducted on five different datasets for recognition tasks, we demonstrate that the proposed method achieves promising performance compared to the SOTA vector- and tensor-based canonical correlation analysis methods in terms of classification accuracy, model sparsity, and robustness to missing and noisy data. The code publicly available hhtpss//gthhbbbco////iii////CC
1 Introduction
Canonical Correlation Analysis (CCA) (Hotelling 1936) is a powerful tool for correlation analysis and dimensionality reduction that has been successfully applied a wide spectrum of fields, such as industrial process (Zhu et al. 2023), neuroscience (Mihalik al. 2022). speech processing (Choukri Chollet, 1986). natural language processing DDhilon et al. (011)), computer vision (Lu et al. 2022). and graph learning (Zhang et al. 2021 Suppose that two sets of measurements available the same set of observations, the classical

---

Published in Transactions on Machine Learning Research (01/2024)
CCA seeks linear combinations or loadings of all variables in each set that are maximally correlated with each other. Over the years, CCA has been extensively studied from multiple perspectives (Hardoon et al. 2004 Yang et al., 2019 Lindenbaur et al., 2021; Sanghavi & Verma 2022 Friedlander & Wolfj 2023p. In particular, CCA provides fully dense loadings, it can often lead to difficulty in interpretation of results and disturbance from noisy features (Park & Konishi 2020). ) When the number of variables far exceeds the number of subjects, classical CCA methods are no longer appropriate (Chalise Fridley, 2012). Therefore, a variety of models
have been proposed for sparse CCA (Chu et al., 2013; Wilms & Croux 2015 Gao et al., 2017). Furthermore, the sparsity of model weights provides essential interpretability in mumerous applications (Witten et al. 2009 Qiu et al. 2022). Nevertheless, these approaches have been limited to vector-type data and cannot effectively deal with high order tensor data.
In many real-world applications, data are frequently organized in the form of tensors. Specifically, a grayscale face image is transformed as a sccond-order tensor of height width and gait sequence is arranged as a third-order tensor of body height body width x gait length. A common approach for applying CCA methods tensor data is to convert. it into vectors by flattening the tensor element by element. However, such an approach ignores the latent higher-order structure of pixel arrangement, such as spatial coherence. Additionally, the high dimensionality of the resulting vectors can lead to computational challenges and inefficiency (Zouu et al. 2013
In recent years, there has been increasing interests in generalizing CCA methods to tensors. Several tensor- based methods have been proposed to overcome the above limitations (Chen et al. 2021 Gang et al. 2011 Kim al., 2007b] Lee & Choi 2007 Min et al. 2019, Wang et al. 2016). The objective is to discover relationships between two tensors while simultancously preserving multidimensional structure of the tensors and utilizing substantially fewer parameters. While these methods have shown promising results in several applications, most of them directly maximize the correlation between two tensor spaces and require input tensors having the same order or even the same size, which are not practical for general use. Furthermore, existing sparse tensor approaches (Min et al., 2019 Wang et al. 2016) focus on quantifying the sparsity of lactor vectors, where the high-order correlation ions between different modes of tensors are ignored. The use of structured sparsity in tensors is still an issue under the constraint of unit variance in CCA criteria.
In this paper, propose general sparse tensor CCA (gSTCCA) method for analyzing paired tensor data with arbilrary order, which exploits the tensor structure to encourage lensor-level sparsily. The main contributions of this paper are threefold:
We formulate the gSTCCA model as a constrained multilinear least-squares regression problem with cnnsor-structured sparsity regularization based on the CANDECOMP/PARAFAS (CP) decomposi- lion, which is both parsimonious and highly interpretable.
We present efficient divide-and-conquer approach to solve gSTCCA based successive rank-one tensor estimation of the residual tensors, and theoretically show that the whole problem boils down to a set of unconstrained alternating least-squares problems.
We conduct extensive experiments on five real-world classification datasets with varying tensor sizes and orders. The results reveal that gSTCCA outperforms SOTA vector- and tensor-based CCA methods in terms of accuracy, model sparsity, and robustness to missing and noisy data. Moreover, gSTCCA demonstrates noleworthy self-supervised capabilities within the framework of CCA, adapting effectively to downstream classification tasks.
2
Related Works
In this section, we briefly review the existing works that are relevant to our approach.
Vestor-based CCA Approaches. CCA has been extensively studied in vector space from many aspects. For example, a number of sparse CCA variants (Chu et al. 2013 Gao et al. 2017 Wilms & Croux| 2015 Witten & Tibshirani 2009) have been proposed to impose sparsity constraints on the canonical loading vectors. In particular, the f-rorm regularization is commonly used for its convexity and ease of optimization.
2

---

Published in Transactions on Machine Learning Research (01/2024)
Various nonlinear methods have been proposed to capture nonlinear dependency in the data based on kernel methods (Hardoon al. 2004 Zheng et al. 2006) and deep learning methods (Andrew et al. 2013 Yang et al. Some supervised approaches (Arandjelović, 2014 Kim et al. 2007a) have been proposed to exploit label information by adding discriminative term to the classical CCA. Several multi-sct CCA methods (Correa Parra 2018) have been proposed to inlegrate multiple datasets. In addition, has been shown that under mild condition, CCA can be formulated as a least-squares problem (Sun al.; 2008).
2017).
et al. 2009 2010
Tensor-based CCA Approaches. Several tensor-based CCA methods have recently been proposed for tensor data. For example, Lee & Choi (2007) proposed two-dimensional CCA (2DCCA) via low-rank matrix factorization. Gang et al. (2011) extended the 2DCCA to 3DCCA. Kim et al. (2007b) designed a shared-mode lensor CCA (TCCA) model to caplure the relations among tensor modes. Chen et al. (2021) applied CP decomposition and devised a deflation-base
model (dTCCA) using the higher-order power method. Yan ct al. (2011) proposod sparse 2DCCA (S2DCCA) with &r-norm regularization. Waan et al. (2016) generalized S2DCCA higher-order tensors and proposed sparse tensor CCA (STCCA) by iteratively sceking CCA components for each lensor mode. Min al. (2019) proposed population model via CP decomposition and adapted it to the sparse case with the hard-thresholding technique called the spTCCA.
Generally, most of the above tensor methods focus on extending the classical CCA to maximize the correlation coefficient between tensors, and require input tensors having the same order or size. Chen et al. 2021] solved the problem from multilinear least-squares perspective, which is the most closely related work to us, however, it is confined to tensor data of the same order and not a sparse method. Furthermore, existing sparse tensor models (Min et al., 2019; Wang al., 2016) focus on inducing the sparsity of the factor vectors, there still lacks general and effective model that directly learn tensor-level sparsity.
3 Preliminaries
Notation. Tensors are denoted by bold calligraphic letters, c.g., A E R.Ix-XXN, Matricos denoted by bold uppercase letters, e.g., A E R4xl2, Vectors are denoted by bold lowercase letters, e.g,, E R.. Indices are denoted by lowercase letters spanning the range from to the uppercase letter of the index, e.g., n l.......AA N. We denote the entries by ag 0g,j, aij,k, etc., depending on the number of dimensions. Thus, each entry of an Nth-order lensor A E Rlix--Xxx is indexed by N indices { and each in indexes the n-mode of A. Specifically, -n denotes every mode except n, Vec(-) denotes the vectorization operator, and Tr(-) denotes the trace of a matrix.
3.1 Tensor Algebra
We refer readers to Kolda & Bader (2009) for more details of tensor algebra.
Definition 3.1 (Inner product). The inner product of two tensors A, B E IR4xxxxxx is the sum of the
IN products of their entries, defined as (), ΣT, ... Ei Qig bi
It follows immodiately that the ls norm of is defined as |||||== (), The l norm of a tensor is defined as ||A|]== Ei Ei
Definition 3.2 (Tensor product). Let (n)) I be vector with dimension In The tensor product of N vectors, denoted by A (1) o.... a (N), is an (11 ... X -tensor of which the entries are given by (1)
Ag....i = 0 ...... a
(N). We call A a rank-one lensor or a unil-rank tensor.
Definition 3.3 (n-mode product). The n-mode product of tensor A xh by a vector u E R,,, denoted by A X, u, an (I, .... In-1 x In+1 ... Ix)-sens of which the entries are given by (A Xn u) 1h = -// Din=t Oi.....in Ui
Definition 3.4 (CP decomposition). For any tensor A E R1x--XXx, the CP decomposition is defined as a()) (n) R ar o... (N), where o, E RR
A E
[1)
(n) E R and ||a!") --= 1, and A (m) Ja , ag for
1, N are called factor matrices.
3

---

Published in Transactions on Machine Learning Research (01/2024)
Definition 3.5 (Tensor rank"".
The tensor rank of A, denoted by Rank(A), is the smallest number R such that the CP decomposition is exact.
Definition 3.6 (Orthogonality (Kolda 2001 iff
Two rank-one tensors A, and Ag are orthogonal (A: 1 Az),
(Ar
II N
(a("). , (n
(1)
where a and
(n) aj are unit vectors based on CP.
3.2
Canonical Correlation Analysis
Let X E PxN and Y E RQXN be two sets of N paired observations. Without loss of generality, assume both fx, and [yn]e=1 have zero mean, i.c., Em=1 Xn = 0 and Σx= Yn = O. CCA aims to maximize the correlation between the projections of two sets by
max u TUuTTYY "'XYYv. v, s.t. u U.V
to get the first pair of canonical vectors u E R. and v E R2, which are further used to obtain the first pair of canonical variales u² X and v" To obtain multiple projections of CCA, we can recursively solve the above optimization problem with additional constraint that the current canonical variates must be orthogonal to all previous ones (Chu et al. 2013). Specifically, denoting
[u,,--,, us] E RPx5 and V Vi, vs] E RQx5 Eq. (2) can be reformulatod to get multiple projections as
max Tr(V'XY"V), s.t. - U.V
One appealing property of CCA is that can be formulated from the view of least-squares problems follows Brillinger
1975 Chu al. 2013):
min | U.V
vTy y s.t. vYY''V =s.
(4)
Once we obtain canonical matrices and V, the projection of and Y into the new space can be obtained by VTY).
4 Sparse Tensor Canonical Correlation Analysis
To illustrate design, start with the first pair of canonical rank-one tensors, and then present deflation method to find more than one canonical rank-one component.
4.1 Model Formulation
Given N paired samples of tensors (K,, J.N where Xx E RPPxxxxP and Vn E RQix- Without loss of generality, assume that (Kr, and ) normalized to have mean zero and standard deviation one on cach feature. We consider the STCCA model of the form
N
Σ x,)) - (V.).)) + +.
min V. 2N
n=1
Σ (,,
N E (V,2))² 1, Rank(()) Rank(V) 1,
(5)
where the low-rank constraints are enforced through the canonical tensors 24 and V in order to reduce the complexity of the model and leverage the structural information, and the tensor-structured E norm is employed to encourage sparsity in the multilinear combination of decision variables.
There multiple ways define the tensor rank. In this paper, we define based the CP decompasition and call CP-rank.
4

---

Published in Transactions on Machine Learning Research (01/2024)
Based on the definition of tensor rank, we assume that U =
or.y')")....'''') o... o- "'..."")...... where
ou: Fv > O, || 1 and ||v[0||== 1. Note that this amounts to using an l normalization, instead of the l2 norm preferred by mathematicians, and the trade-off is in the simplicity of formulae. ou 2 0 and op > 0 can always hold by flipping the signs of the factors.
It becomes evident. that |= au [[== and || oo IT' // In other words, the sparsity of rank-one tensor directly leads the sparsity of its components. This allows us to kill multiple birds with one stone: by simply pursuing the clement-wise sparsity of the canonical rank-one tensor with only one tunable parameter À in each view, solving (5) can produce set of joint sparse factors u(+) for 1, ,, simultancously (same as v(t), t ,,
Continuing with the unified representation, let X' x= [X,, .. NyI E XRP and y = [)1," " yvl e R xx XN We denote by X x u X x1 owu ... Xs / a N-dimension vector with n-th element (U, X,),, and y x V y X1 o, v,v(1) XT v(T) a N-dimension vector with n-th element (V.J.). Eq. (5) can be writlen in a unified compact form
1 U. 2N
min
X x x ,
s.t. Var(X x u) = Var(Y' x v) = 1, Rank(U)) = Rank(V) = 1.
(6)
4.2 Bank-One Estimation
The objective function in Eq. 6 is non-convex multi-objective optimization problem, which makes it hard to solve. Inspired by He et al. 2018), we show that this problem can be broken up into set of unconstrained alternating least-squares problems which can be efficiently solved.
Suppose we have an initial value V* for the canonical lensor with respect. to V. By fixing it, the optimization problem in (6) reduues to
ūp arg min
2N x x x x 12 11]11,
s.t. Var(X x u) 1, Rank(U) =
(7)
Similarly, for a fixed value 4 the optimal value for V can also be reduced to
Diu arg min
|x x Ut y x V1
V 2N
s.t. Var x V)- = 1, Rank(V) = 1.
(8)
Let for moment assume that will solve Eqs. and by the alternating least squares (ALS) algorithm. We alternately optimize with respect to variables (ø (()) (ax, v(0) with others fixed. Let ( = o,,( X(-)) = X x1 (!)) X2 ... x.-1 =(=-1) Xs+1 ... X u (()), and let v(t) = v,)()) y(-t) = =x= (!I) X2 ... x (t-1) X XT (T), The subproblems in with respect to (a., u()) and
(a,, v(0) boil down to
min
i - 2N
-00T 1| | s.t. (ar(1(()x((-9)
(9)
(())
min
I
(( 11 (1) s.t. (O0T =1
(10)
Sen 2N
| and u = ((0)) /o, to satisfy the constraints whenever (e) of O. (e) = 0, (*) is no longer identifiable, we then set o. = 0 and terminate the algorithm.
Once we obtain the solution (s)), we can set o,
Analogously, 0e and v(() can be computed.
Inspired by the Lemma in Mai & Zhang (201)). we present the following Lemma 4.1 We accordingly get rid of the unit variance constraints in Eqs. 9 and 10)
5

---

Published in Transactions on Machine Learning Research (01/2024)
Lemma 4.1. Consider the following minimization problem
1 ú x =1
min 2N I yl@++A
s.t. Var(ū x):
(11)
N
where X and y fized, and du is hyperparameter that controls sparsity. i' (Var(ū-TX) -\)_ where ir is obtained via
The solution to Eq.
11 is
u mig min
ü x
v112
(12)
2N
The proof of Lemma (+)) (Var(nl)Tx(>>)) 1/2 n()T with fixed v(t where Ú is obtained by
4.1 above is shown in Mai & Zhang 2019
Specifically, the solution to Eq. (9) is
min
x v( A [ u( 2N
(13)
Similarly, the solution to Eq. obtained by
10 is pit) = (Var(v (V with a fixed u(s),, where v(() is
min
[+) (-)) F(OT - -2,,1
-
(14)
2N
Notice that the problem boils down to alternatively solving Eqs. 13 and which are the standard least-squares lasso problems. Here we employ the fast SURF algorithm proposed in He al. (2018) to solve it, which can trace the solution paths for A and Au in the manner similar to that of stagewise solution for sparse CCA (Sun et al. 2008). Algorithm summarizes the procedure.
Algorithm 2 Deflation for gSTCCA
Algorithm Rank- One Estimation
Input: Paired tensor data (Nu. V.
Jm21 CP-rank R. and R..
Input: Paired tensor dataset (Nes Val Wa stop
Output: Canonical tensor pair ((,,) 1: Initialize 2!" and V while not converged do
RR. and (V.)
Output: Canonical rank-one tensor pair U and V. 1: Initialize
2: Set & = O.
3: for L,.... R2 do
3: while not converged do
Compute in Eq. Compute the residual Pr by using Eq. 17 6: end for
Algorithm
4: Run SURF(() algorithm to get Ut and Auj
by using Eq. [3
5: Run SURF( algorithm to get Ve and An,
7: Update 4" == =>>>> Hr.
by using Eq.
8:
for r 1,... R. do
6: k - k 1.
9:
Compute V. in Eq.
16 by Algorithm
while
8: Compute (()) (Var(uf (1) x -)y- 2u(or == 1,
10: Compute residual q. by using Eq.
11: end for
Compute (1)) (Var(v) (TT -1) -o--------- foO I.
12: Update V >\> Ry D.
13:
end while
4.3 Sequential Pursuit of Higher Rank
After the first pair rank-one tensors is identified, it is desirable to pursue a higher-rank tensor canonical pathway for better performance. In the following, we show how more tensor components can be derived with a sequential rank-one tensor approximation algorithm based on the deflation technique (Chen al. 2021). To streamline the presentation, let assume that we solve the rank-one problem in Eq. (6) to get U, and D, each r-th step and select the tuning parameters Xr and X suitably, e.g., cross validation. general, the gSTCCA problem in the r-th step can be expressed alternatively as
1
a. = arg min ur 2N
|| x - Pr 1111++A,,
s.t. Var(X xf U Rank(U4) 1, 1,.r., Ra:
(15)

---

Published in Transactions on Machine Learning Research (01/2024)
v arg min 1 V. 2N
| x3 V. 2.11+
s.t. Var()' x V.) Rank = 1, 1,,,, Re
(16)
where P. and q, are the current residues of canonical variables with
y x V", Pr-1- - X xi Up-1
if r r=1 otherwise
p
(17)
X u u³
if r=
and q where 4d and V are the imitial values, Pr-1 and 9r-1 are the estimated values in the (r- 1)-th step. Note that, the values R2 and Rs are not necessarily the same, since input data X' and y may differ in tensor structures.
(18)
99-1 x V--1, otherwise'
It can be seen that the problems 15 and 166 are reduced to the same forms as given by Eqs. 77 and
Based Algorithm| a deflation method for solving the optimization problems of gSTCCA in (15 and 16
that aims to find canonical tensor pairs {/m Ri, and (V furthermore, is summarized Algorithm
5 Discussions
Convergence Analysis. It is proved that the SURF algorithm converges to coordinato-wise minimum point (He et al. 2018 By construction of Steps 4 and 5 in Algorithm we have f'(ds, Au,k) 2 5(0k, Ao.k) 2
f(1x+1; )x1) ) 2 f(M++1, Thus, the iterative approach in Algorithm 1 (line to line 7) monotonically decreases the objective function (6) in each iteration. Steps 8 and are equivalent to constrained optimization problems in (9) and (I0), which has been proved in Lemma 4.1 and is finite. Hence the number of iterations is finite in Algorithm| [ Morcover, Algorithm 2 repeatedly calls Algorithm to generate sequential rank-one lensors based on the deflation, which has been proved that it leads to monotonically decreasing sequence (da Silva et al. 2015). Thus, Algorithm monotonically decreases during deflation and converges in a finite number of steps.
For simplicity, we assume that the size of the first view Pix x X is larger than that of the second view yxxxx-XxxXX For each iteration, the most time-consuming part of Algorithm 2 lies in iteratively performing the tensor-vector multiplication and solving the rank-one tensor estimation by Algorithm [ Based on the results in He et al. (2018), the computational complexity of gSTCCA per iteration is O,N Es (T) Pj)). Since gSTCCA does not require explicit vectorization of input data, thus yielding the space complexity of O(N II.II..)...
Complexity Analysis.
Orthogonal Constraints. Unlike the standard CCA in Eqs. B and for the gSTCCA model we do not enforce orthogonality, as the concurrency of sparsity and orthogonality constraints makes the problem much more difficult. On the other hand, sparsity and orthogonality have been two largely incompatible goals. For instance, as mentioned Uemaasu al. 2019), a standard orthogonalization process such as QR factorization (Chan 1987) will generally destroy the sparsity pattern of a matrix. Here we adopt the tensor orthogonality given in Definition 3.6 to investigate the influence of orthogonality constraints
gSTCCA model. After solving gSTCCA in Algorithm 2 we utilize the 'CP-ORTHO' framework (Aishar et al. 2017 to orthogonalize rank-one tensors (dubbed gSTCCA.).
Feature Projection. To preserve the higher-order tensor structure input data, after obtaining a sequence of canonical rank-one tensors {14))mm ( we can project each pair of (X,, V. onto the corresponding feature spaces spanned by factor matrices (U(+)) and [V(0]² as
X ynew Xn x1 U()) ... x (S))
ynew Yn x1 ((1) x.. Xr ((T).
(19)
where U juʃ ..., RR R P. and v(1) |v (t)
(t) E R. The data pair Nne and nnew
are S-th order and T-th order tensors, respectively.

---

Published in Transactions on Machine Learning Research (01/2024)
Table 1: Statistics of five real-world datasets in our experiments. We sunmarize the number of images, classes and sizes for both the original and preprocessod ones. We can see from it that our methods have no constraint on tensor input size.
Type
Dataset
# Images
# Classes
Image size
Preprocessing
Size(A')
Size(3?)
Handwristen Digit
MNIST
1000
3'
28
Cropped
14
14
Human Face
Yale
165
15
32 x 32
Wavelet
32 32
16 x 16
Brain Network
BP
82 82
IMRIDDTI
82 82
82 82
Facial Expression
JAFFE
213
200 x 180
Gabor
200
180
x 25 x 40
Gait Sequenoe
Gait32
731
71
32 22 x 10
Optical flow
32 22 10
32 x 22 x 10
For fair comparison, use classes instead of 10 classes and follow the setsings Chen
2021 conduct experiments the
MNIST dalawel.
Table 2: Visualization of original datasets with two views across five datasets.
Dataset
MNIST
Yale
BP
JAFFE
Gait32
Original
O
2
X
O
6
Experiments
We evaluate the effectiveness and efficiency of gSTCCA through comprehensive experiments on diverse real-world tensor datasets. Our evaluation encompasses comparison with state-of-the-art methods in terms of classification accuracy, model sparsity, and training time. Furthermore, we delve into feature visualization, perform in-depth hyperparameter analysis, and examine the robustness of our approach in handling missing and noisy data. In all of our experiments, 80% of data are randomly selected for training, while the remaining 20% is used for testing. To mitigate randomness, perform 10 independent cross-validation and report the average results. All experiments run with MATLAB R2022b machine equipped with 3.6GHz CPU and 16GB RAM.
6.1 Datasets
Table I summarizes the statistics of five real-world datasets used in our experiments. The original images with two different views are visualized in Table 2 For reproducibility purpose, more details of data description and preprocessing are provided as below.
Data Preprocessing. Handwritten Digit (MNIST). MNIST database (LeCun et al. 1998 contains 60000 training and 10000 testing images of size 28 x 28 with labels from '0" to '9.. We choose a subset.
consists of 1000 images with labels '0', '1', and '2' and follow the same scttings as Chen al. (2021 conduct our experiments. The goal learn correlated representations between the upper and lower halves (two views) of the original images.
Human Face (Yale). The face images are collected from the Yale database (Cai et al. 2007). which contains 165 images of size 32 x 32. For each image, we apply wavelet transformation to generate the corresponding encoded feature image using the dw2 function in MATLAB. The original and encoded feature tensors are used as two different views.
8

---

Published in Transactions on Machine Learning Research (01/2024)
Table 3: Properties of all compared methods. A, and Ay are the regularization parameters, R is the matrix/tensor rank, K is the iteration number, N is the sample size. For brevity we assume the size of Xx E R x is larger than that of Vn E BQ x with tensor rank equal R for each view.
Method
Iupus Type Data Coustraius
Regularivet
Hpperparanetess
Time Complexity
Space Complexity
OCA
Hotelling
1936
Voctor
R
OfT R1)
OIT P. N NI 1 R)
SOCA
Avants et al
2014
Voctor
4(0)), l(v))
Au,
O Pf)
oIII.. P. N NI.. P)
DOCA
Andrew
Vector
Many"
OIII__ P?)
oilL,, P? WIIL R)
3DOCA
Gang al.
2011
Tensor
8 T
R
O(K N Ei-1 (P)
O IT--, Pt)
aTOCA
Choa et al.
2021
Tonaor
8 = T
O(K Ei PS)
O N IT A))
STOCA
Wang et
2016
Tensor
8-T T
h(u),, 4(9)
Au,
O Pf &KNEL PD)
O(N IT R))
TFTCAA
Min al.
2019
Tensor
(tt)), G(v))
Rucho,R
OAT F2 E2, KITA)
OtT. /2 N MT
RSTCCAA (Ouns)
Tensor
6,50), G(V)
Aur R
O(KNR U PM)
O IT R3)
*Masyy
there leperpaanntes.
Brain Network (BP). Brain networks play an important role in understanding brain functions. Bipolar disorder (BP) dataset Sluitiiel--Gbbiiae & Nieto- Castanon) 2012) is collected from two modalities, e.g., functional magnetic resonance imaging (fMRI), and diffusion tensor imaging (DTT). We follow [iu al. (2018) to preprocess the imaging data, including realignment, co-registration, normalization and smoothing, and then construct the brain networks from fMRI and DTI based the Brodmann template, which are treated as two views.
Facial Expression (JAFFE). The JAFFE database Lyons et al. 2020 contains female facial expressions of seven categories (neutral, happiness, sadness, surprise, anger, disgust and fear), and the number of images for each category is almost the same. We first crop each image to the size of 200 180, and then construct dataset of different. sizes and orders. Specifically; we use the cropped image as the first view, and its 3D Gabor features as the second view.
Gait Sequence (Gait32).. The Gait32 dataset (Lu et al. 2008) contains 731 video sequences with 71 subjects designed for human identification. The size of each gait vidco is 32 x 22 x 10, which can be naturally represented by a third-order tensor with the column, row, and time mode. We calculate optical flow and obtain two 3rd-order tensors, which are used as two views (Wang et al. 2016).
6.2 Experimental Settings
Baselines. Table 3 summarizes the properties of all compared methods. Specifically, we investigate the viability of the proposed gSTCCA as feature extractor for classification tasks, and compare it with the following methods: standard CCA (Hotelling) 1936), sparse CCA (SCCA) (Avants et al. 2014], deep CCA
(DCCA) (Andrew et al. 2013 three-dimensional CCA (3DCCA) (Gang et al.) 2011 Lee & Choi 2007). CP-based sparse tensor CCA (sPTCCA) (Min et al.) 2019), deflation-based tensor CCA (dTCCA) (Chen et al. 2021) and sparse tensor CCA (STCCA) (Wang et al.) 2016). For STCCA we implement it by ourselves as the source code is not publicly available. For other compared methods, we use the source codes provided by the original authors. To obtain the corresponding feature representation for classification, for all tensor-basod approaches, we use Eq. (19 to project cach pair of input data (/)
get projected tensors (X ).ee). Then (X ))new) are vectorized and concatenated to form the final feature vector f, = [Vee(XX"")) Vec()me")), which is further fed to the SVM classifier (Chang & Lin 2011). For vector-based methods, we concatenate their extracted feature vectors as the input of SVM.
Hyperparameter Settings. For a fair comparison, the parameters of all compared methods are carefully tuned using the same 5-fold cross validation strategy on the training set based on classification accuracy. Specifically, rank R chosen from {5, 10, ,0), for methods that enforce sparsity constraints, X, and X are selected from (0.001, 0.005. ............. 0.13. For DCCA, we select the number of hidden layers from to 4 and learning rate 4, from (0.001, 0.005, 0.01).
9

---

Published in Transactions on Machine Learning Research (01/2024)
Table Performance comparison different methods in terms classification accuracy (mean + std). 'n/a": results are not available due to method constraints.
Methods
CCA
SCCA
DCCA
3DCCA
d'OCA
STOCA
spICCA
@STCCA
BSTOCA
MNNST
IX1.7 1.4
97.1 4 0.9
96.9 1: 1.1
16.2 1.1
90.9 3.8
97.3 +: 1.2
90.8 = 4.2
97.1 11
97.4 1.3
Vale
89.0 9.9
89.0 + 7.9
84.6 11.9
74.6 + 16.2
75.0 + 19.8
87.0 + 8.2
73.6 + 8.9
86.6 + 12.3
89.0 9.6
61.1 9.1
58.9 10.8
51.1
12.2
55.8 4: 11.9
55.0 1: 12.6
50.5 10.8
52.1 :: 9.8
59.5 13.6
63.2 7.0
86.1 7.2
86.3 + 6.9
+ 6.8
1//a
w//
n//
81.5 12.1
85.9 = 12.2
90.0 5.7
Gait32
25.8 6.8
26.6 + 6.2
+ 4.3
35.4
14.5
34.6 6.6
16.5 = 3.1
26.8 3.6
41.4 5.2
Spanits ONL
Time Cut00
(a) Sparsity
(b) Training Time
(c) Sparsity R (d) Accuracy
Figure 1:
Comparisons among sparse models. (a) and (b): Histogram of SCCA, STCCA, spTCCA and
gSTCCA in terms of model sparsity and train time on five datasets. (c) and (d): Influence of R on sparsity and accuracy of gSTCCA.
Evaluation Metrics. To quantitatively evaluate the performance of compared methods, mainly classification accuracy and model sparsity. The model sparsity is measured by the ratio of zero elements to the total number of elements on factor matrices (U(+) and (V(0)T
6.3 Experimental Results
Table 4 presents the classification accuracy of all compared methods. The feature visualization of these compared methods on MNIST is shown in Fig. 2 The model sparsity and training time of the sparse approaches are illustrated in Figs. [(a) and (b). Based on these results, we have the following observations: Overall, the proposed gSTCCA demonstrates very competitive performance in terms of accuracy, sparsity and efficiency. Specifically, gSTCCA produces the best results on all five dalasets in terms of classification accuracy. The thet--- STCCA method also shows outstanding performance, which reflects the advantage of pursuing sparse components. However, compared to STCCA, gSTCCA is more flexible since STCCA can not handle paired tensor data with different orders. Meanwhile, gSTCCA is also able to achieve higher sparsity, as can be seen from Fig. [a). In addition, we can see that gSTCCA outperforms gSTCCA on all five datasets, indicating that the orthogonality constraints may introduce fitting noise and lead to poor approximation of the observed tensors. It noticed that some related works also discarded orthogonality
constraints of sparse components Allen 2012: Shen Huang 2008 Wang al. 2016).
For efficiency, it noticed from Fig. [bb, gSTCCA more computationally complex than STCCA, but it is also easy and efficient to train in that can select the optimal À for U and V from their corresponding solution paths, thus eliminating the need to fine tune A and A from pre-defined range. Besides, vector- based sparse method (i.e., SCCA) is very fast when the feature size is small, while its training time grows significantly as the feature size increases. The reason is that the computational burden of SCCA lies mainly in performing singular value decomposition (SVD), which is come-consuming for large matrices. To summarize, the results above indicate that our gSTCCA able effectively and efficiently find high-quality sparse tensor coefficients 4 and V that capture the most useful components of input tensor data, and then boost the classification performance.
10

---

Published in Transactions on Machine Learning Research (01/2024)
10
5
10
0
0
0
-10
-10
-10
-10
-16
21
-20
-20
-10
-10
10
20
30
CCA
(b) SCCA
(c) DCCA
(d) 3DCCA
40
30
10
20
10
10
0
-10
-10
-10
-20
-16
-30
-00
-20
-20
-10
0
10
20
-10
20
-20
10
20
-20
-10
10
20
(e) dTCCA
(1) STOCA
(8) spTCCA
(b) gSTCCA
Figure 2: The t-SNE visualization of features obtained from compared methods on MNIST dataset.
Accuenary (76)
(a) Missing data
(b) Nolsy data
(c) Accuracy (Missing data)
(d) Accuracy (Nolsy data)
Figure 3: (a) and (b): Illustrations of MNIST images with missing ratio f0.1, 0.9) and noise variance
EE 1..1, 0.9}. (c) and (d): Classification accuracy of all compared methods on missing and noisy data.
6.3.1 Feature Visualization
qualitatively assess the effectiveness gSTCCA, we use the t-SNE algorithm (Van der Maaten & Hinton
2008 to visualize the features (f,),N, learned by different methods on MNIST dataset. As shown in Fig. the results indicate that only STCCA and gSTCCA produce well-separated and compact clusters. However, from Table $ in our paper, is noticed that STCCA cannot handle paired tensor data with different orders.
6.3.2 Hyperparameter Sensitivity Analysis
The tensor rank essential for tensor methods, since it directly controls the reduced dimensionality of feature space. Here we investigate the influence tensor rank on the proposed gSTCCA over five datascts. For simplicity we assume R R. R2 and compare the sparsity and accuracy by varying from (5, 10,....30) 30). From Fig. [cc, notice that the model becomes sparse given larger R, which can be explained
11

---

Published in Transactions on Machine Learning Research (01/2024)
12-
04
(04)
JEET!
65
13
-1
2214
12
Σ E.
(a) 0.06
(b) 0.06
(c)
(d) 0.6
Figure 4: Comparison of solution paths of gSTCCA and ALS for a subset of canonical tensors U and V with small and large step sizes MNIST dataset. The z-axis denotes the sparseness coefficient, and the y-axis denotes the value of the coefficients.
by the fact that important information tends to be contained in the first few tensor components. However, increasing R does not always guarantee higher accuracy in that it may introduce some unwanted noise. For example, as can be seen from Fig. díd), the best results of gSTCCA on MNIST, Yale, BP, JAFFE and Gait32 are achieved with rank R = 5, 20, 55, 10 and 35, respectively. In practice, this interesting observation may be useful narrow the range of parameter tuning for related datasets and applications.
6.3.3 Influence of Missing and Noisy Data
In many real-world applications, images are inevitably contaminated by outliers such as missing data and noise during acquisition. Figs. aa and (b) illustrate the MNIST data corrupted with certain missing ratio y and additive white gaussian noise (AWGN) different variance V, respectively.
evaluate the robustness of all compared methods by varying y and From Figs. [e) and (d), noticed that the proposed gSTCCA is more robust to high missing data ratio and noise level, because the SVD adopted by compared methods may be more sensitive to extreme outliers.
6.3.4 Solution Path
Fig. 4 shows the collection of solution paths of gSTCCA for a subset of canonical tensors on MNIST dataset, under both big and small step sizes. The path of estimates U and V for cach À is treated of t Σ| and t ElVil,, respectively. The z-axis denotes the sparseness coefficient, and the y-axis denotes the value of the coefficients. The vertical lines denote (a subset of) the turning point of the path, the solution path for each of the coefficients is piece-wise linear. From Fig, g we can see that the gSTCCA path is almost indiscernible from that of ALS when the step size is small, which matches well with the theoretical analysis in He et al. 20188
function
6.3.5 Application on ADNI
To show the significance and benefits of integrating lensor-structured sparsity into the CCA framework, we perform the task of predicting the stage of Alzheimer's Disease (AD) using real-world data sourced from the Alzheimer's Disease Neuroimaging Initiative (ADNI) database?] The dataset comprises 116 features extracted from brain regions of interests (ROIs) across three medical imaging modalities: Voxel-Based Morphometry Magnetic Resonance Imaging (VBM-MRI), 18 "F-fuuoooeexxyllucoss Positron Emission Tomography (FDG- PET), and 18-F florbetapir PET (AV45-PET) PET). The feature grouping is performed using the MarsBaR toolbox Tzourio- Mazoyer el al. 2002). For detailed information data acquisition and preprocessing, please
refer to Yu et al. (2022)
www.adniiinfooorg
12

---

Published in Transactions on Machine Learning Research (01/2024)
@STCCA
Figure 5: Visualization of coefficient weights on the ADNI data. From top to bottom, three rows are corresponding three imaging modalitics: (i.c., VBM-MRI, FDG-PET, and AV45-PET). From left right, there 116 brain regions defined in Trourio-Mazoyerr et al. 22002). We easily find that gSTCCA pay relatively balanced attention to all three modalities. The top 10 brain regions selected by our method are: Cingulum-Post-L, Hippocampus-R, Hippocampus-L, Cingulum-Mid-RR Prontal-Med-Orb-L. Rectus-L, Cemporal-Pole-Spp-ll Rolandic- Opelii--ppr-R... Angular-L, Parietal-Inf-L. Most of these selected regions are known to be highly related to AD and Mild Cognitive Impairment (MCT) in previous studies Du et al. (2001
Miklossy 20111; Vlassenko et al. 2012); Mattis et al. (2016); Bubb et al. 2018
7 Conclusion
In this paper, we propose a novel sparse tensor CCA model for arbitrary tensor data. Specifically, we formulate the problem constrained multilinear least-squares problem with tensor-structured sparse regularization, and propose divide-and-conquer strategy based on the deflation technique to sequentially solve the problem. This solution results in set of unconstrained linear least-squares problems which can be efficiently solved. Extensive experiments conducted on five different datasets demonstrate the efficacy and robustness of our approach in terms of classification accuracy, model sparsity and having missing and noise data. Our future work will extend the proposed gSTCCA multi-vicw learning and deep learning. It is also interesting explore how to better fuse gSTCCA with orthogonality constraints, which help discover distinct and easily interpretable patterns from tensor data (Aíshar et al. 2017).
Acknowledgments
This work is partially supported by the NSF grants (MRI-2215789, IIS-1909879, IIS-2319451, DMS-2053697), NIH grants (1R01LM014344, 1R01AG077820, R01LM012607, R01AI130460, ROLAG0734355, R56AG074604, ROTLM013519, R56AG069580, UO1TR003709, RF1 G077820, R21 A11771,,, R21EY034179), and Lehigh's 's
grants under Accelerator and CORE. We also would like to acknowledge and thank all the individuals who shared their source code and datasets, or made them publicly available.
13

---

Published in Transactions on Machine Learning Research (01/2024)
References
Ardavan Afshar, Joyce C Ho, Bistra Dilkina, loakeim Perros, Elias B Khalil, Li Xiong, and Vaidy Sunderam. Cp-ortho: An orthogonal tensor factorization framework for spatio-temporal data. In ACM SIGSPATIAL, pp. 1-4, 2017.
Genevera Allen. Sparse higher-order principal components analysis. In Artificial Intelligence and Statistics, pp. 27-36. PMT.R, 2012.
Galen Andrew, Raman Arora, Jeff Bilmes, and Karen Livescu. Deep canonical correlation analysis. In ICML, pp. 1247-1255. PMLR, 2013.
Ognjen Arandjelović. Discriminative extended canonical correlation analysis for pattern set matching, Machine Learning, 94(3):353 370, 2014.
Brian B Avants, David J Libon, Katya. Rascovsky, Ashley Boller, Corey T McMillan, Lauren Massimo, H Branch Coslett, Aujan Chatterjee, Rachel G Gross, and Murray Grossman. Sparse canonical correlation analysis relates network-level atrophy to multivariate cognitive measures in a neurodegenerative population. Neuroimage, 84:698-711 711, 2014.
David R. Brillinger. Time series data analysis and theory holt. RinchartSamp Winston, New York, 1975.
Emma J Bubb, Claudia Metaler-Baddeley, and John P Aggleton. The cingulum bundle: anatomy, function, and dysfunction. Neuroscience & Biobehavioral Reviews, 92:104-127, 2018.
Deng Cai, Xiaofei He, Yuxiao Hu, Jiawei Han, and Thomas Huang. Learning a spatially smooth subspace for face recognition. In CVPR, 2007.
Prabhakar Chalise and Brooke L Fridley. Comparison of penalty functions for sparse canonical correlation analysis. Computational statistics 8 data analysis, 56(2):245-25) 254, 2012.
Tony F Chan, Rank revealing qr factorizations. Linear algebra and its applications, 88:67-82 82, 1987.
Chih-Chung Chang and Chih-Jen Lin. Libsvm: library for support vector machines. ACM Transactions on Intelligent Systems and Technology, 2(3):1 27, 2011.
You-Lin Chen, Mladen Kolar, and Ruey S Tsay. Tensor canonical correlation analysis with convergence and statistical guarantcos. Journal of Computational and Graphical Statistics, pp. 1-17, 2021.
K Choukri and G Chollet. Adaptation of automatic speech recognizers to speakers using canonical correlation analysis techniques. Computer Speech & Language, 1(2):95-107, 1986.
Delin Chu, Li-Zhi Liao, Michael K Ng, and Xiaowei Zhang. Sparse canonical correlation analysis: New formulation and algorithm. IEEE Transactions Pattern Analysis and Machine Intelligence, 35(12): 3050-3066 2013.
Nicolle M Correa, Yi-Ou Li, Tulay Adali, and Vince D Calhoun. Fusion of fmri, smri, and eeg data using canonical correlation analysis. In ICASSP, pp. 385 388. IEEE, 2009.
Nicolle M Correa, Tom Fichele, Tülay Adah, Yi-Ou Li, and Vince D Calhoun. Multi-set canonical correlation analysis for the fusion of concurrent single trial erp and functional mri. Neuroimage, 50(4):1438 1445, 2010.
Alex P da Silva, Pierre Comon, and André LF de Almeida. An iterative deflation algorithm for exact ep tensor decomposition. In ICASSP, pp. 3961 3965. IEEE, 2015.
Paramveer S Dhillon, Dean P Foster, and Lyle H Ungar. Multi-view learning of word embeddings via cca. In
NewrIPS, 2011.
AT Du, Nea Schuff, D Amend, MP Laakso, YY Hsu, WJ Jagust, K Yaffe, JH Kramer, Reed, D Norman, al. Magnetic resonance imaging of the entorhinal cortex and hippocampus in mild cognitive impairment and alzheimer's disease. Journal of Neurology, Neurosurgery & Psychiatry, 71(4):441444 2001.
14

---

Published in Transactions on Machine Learning Research (01/2024)
Tomer Friedlander and Lior Wolf. Dynamically-scalee deep canonical correlation analysis. In Pacific-Asia Conference on Knowledge Discovery and Data Mining, pp. 232-244- 222-244. Springer, 2023.
Lei Gang, Zhang Yong, Liu Yan-Lei, and Deng Jing. Three dimensional canonical correlation analysis and its application to facial expression recognition. In ICICIS, pp. 56 61. Springer, 2011.
Chao Gao, Zongming Ma, Harrison H Zhou, et al. Sparse cca: Adaptive estimation and computational barriers. The Annals of Statistics, 45(5):2074-22)) 2101, 2017.
David Hardoon, Sandor Szedmak, and John Shawe- Taylor. Canonical correlation analysis: An overview with application learning methods. Neural Computation, 16(12):2639-2664 2004.
Lifang He, Kun Chen, Wanwan Xu, Jiayu Zhou, and Fei Wang. Boosted sparse and low-rank tensor regression. In Newr/PS, pp. 1009-1018, 2018.
Harold Hotelling, Relations between two sets of variates. Biometrika, 28(3/4):321 377, 1936.
TyenKyun Kim, Josef Kittler, and Roberto Cipolla. Discriminative learning and recognition of image set classes using canonical correlations. IEEE Transactions on Pattern Analysis and Machine Intelligence, 29 (6):1005 1018, 2007a.
Tae-Kyun Kim, Shu-Fai Wong, and Roberto Cipolla. Tensor canonical correlation analysis for action classification. In CVPR, pp. 1-8. IEEE, 2007b.
Tamara G Kolda. Orthogonal tensor decompositions. SIAM Journal on Matrix Analysis and Applications, 23 (1):243-255, 2001.
Tamara G Kolda and Brett W Bader, Tensor decompositions and applications. SIAM Review, 51(3):455-50 2009.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278 2324, 1998.
Sun Ho Lee and Seungjin Choi. Two-dimensional canonical correlation analysis. IEEE Signal Processing Letters, 14(10):735-735 738, 2007.
Ofir Lindenbaum, Moshe Salhov, Amir Averbuch, and Yuval Kluger. LO-sparse canonical correlation analysis. In International Conference on Learning Representations, 2021.
Ye Liu, Lifang He, Bokai Cao, S Yu Philip, Ann B Ragin, and Alex D Leow. Multi-view multi-graph embedding for brain network clustering analysis. In AAAI, pp. 117-124, 2018.
Haiping Lu, Konstantinos N Plataniotis, and Anastasios N Venetsanopoulls.. Mpca: Multilinear principal component analysis of tensor objects. IEEE Transactions on Neural Networks, 19(1):18 39, 2008
Y Lu, W Wang, B Zeng, Z Lai, L Shen, and X Li. Canonical correlation analysis with low-rank learning for image representation. IEEE Transactions on Image Processing: Publication of the IEEE Signal Processing Society, 2022.
Michael J Lyons, Miyuki Kamachi, and Jiro Gyoba. Coding facial expressions with gabor wavelets (ive spocial issue). Xiv preprint arXim:2009.088 05938, 2020.
Qing Mai and Xin Zhang. An iterative penalized least squares approach to sparse canonical correlation analysis. Biometrics, 75(3):734 744, 2019.
Paul J Mattis, Martin Niethammer, Wataru Sako, Chris C Tang, Amir Nazem, Marc L Gordon, Vicky Brandt, Vijay Dhawan, and David Eidelberg. Distinct brain networks underlie cognitive dysfunction in parkinson and alzheimer diseases. Newrology, 97(1 1933, 2016.
15

---

Published in Transactions on Machine Learning Research (01/2024)
Agoston Mihalik, James Chapman, Rick A Adams, Nils R Winter, Fabio S Ferreira, John Shawe-Taylor, Janaina. Mourão-Miranda, Alhhemer's Disease Neuroimaging Initiative, et al. Canonical correlation analysis and partial least squares for identifying train-behaviou associations: tutorial and a comparative study. Biological Psychiatry: Cognitive Neuroscience and Neuroimaging, 2022.
Judith Miklossy. Alzheimer's discaso-a neurospirochetosis. analysis of the evidence following koch's and hill's criteria. Journal of neeuronffaammaaiinn, 8(1):1-16 16, 2011.
Eun Jeong Min, Eric Chi, and Hua Zhou. Tensor canonical correlation analysis. Stat, 8(1):e253, 2019.
Heewon Park and Sadanori Konishi. Sparse common component analysis for multiple high-dimensional datasets via noncentered principal component analysis. Statistical Papers, (1(6):2283-25 2020.
Lucas C Parra. Multi-set canonical correlation analysis simply explained. ar preprint arXi:: 15:2. 302.03759, 2018.
Lin Qiu, Vernon M Chinchilli, and Lin Lin. Variational interpretable deep canonical correlation analysis. In ICLR2022 Machine Learning for Drug Discovery, 2022.
Rushil Sanghavi and Yashaswi Verma. Multi-view multi-label canonical correlation analysis for cross-modal matching and retrieval. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 4701 -4100, 2022.
Haipeng Shen and Jianhua Z Huang. Sparse principal component analysis via regularized low rank matrix approximation. Journal of Multivariate Analysis, (9)6:001 1034, 2008.
Liang Sun, Shuiwang Ji, and Jieping Ye, A least squares formulation for canonical correlation analysis. In ICML, pp. 1024-1031, 2008.
Nathalie PRourio-Mazoyerr Brigitte Landeau, Dimitri Papathanassion, Fabrice Crivello, Olivier Etard, Nicolas Deleroix, Bernard Mazoyer, and Mare Joliot. Automated anatomical labeling of activations in spm using a macroscopic anatomical parcellation of the mui mri single-subject brain. Newroimage, 15(1):273-22) 229, 2002.
Yoshimasa Uematsu, Yingying Fan, Kun Chen, Jinchi Lv, and Wei Lin. Sofar: Large-scale association network learning. IEEE Transactions on Information Theory, (5(8))492444998) 2019.
Laurens Van der Maaten and Geoffrey Hinton. research, 9(1),, 2008.
Visualizing data using t-sne. Journal of machine learning
Andrei G Vlassenko, Tammie LS Benzinger, and John C Morris. Pet amyloid-beta imaging in preclinical alzheimer's 's disease. Biochêmica et Biophysica Acta (BBA)-Molecular Basis of Disease, 1822(3):370 379 2012.
Su-Jing Wang, Wen-Jing Yan, Tingkai Sun, Guoying Zhao, and Xiaolan Fu. Sparse tensor canonical correlation analysis for micro-expression recognition. Neurocomputing. 214:218 232, 2016.
Susan Saitfield-Gbbrid and Alfonso Nieto-Castanon. Conn: functional connectivity toolbox for correlated and anticorrelated brain networks. Brain Connectivily, 2(3): 141, 2012.
Ines Wilms and Christophe Croux. Sparse canonical correlation analysis from predictive point of view. Biometrical Journal, 57(5):834 851, 2015.
Daniela M. Witten and Robert Tibshirani. Extensions of sparse canonical correlation analysis with
applications genomic data. Statistical Applications in Genetics and Molecular Biology, 8(1), 2009.
Danicla M Witten, Robert Tibshirani, and Trevor Hastic. A penalized matrix decomposition, with applications to sparse principal components and canonical correlation analysis. Biostatistics, 5333) 2009.
Jingjie Yan, Wenming Zheng, Xiaoyan Zhou, and Zhijian Zhao. Sparse 2-d canonical correlation analysis via low rank matrix approximation for feature extraction. IEFE Signal Processing Letters, 19(1):51-5)) 54, 2011.
16

---

Published in Transactions on Machine Learning Research (01/2024)
Xinghao Yang, Weifeng Liu, Dapeng Tao, and Jun Cheng. Canonical correlation analysis networks for two-view image recognition. Information Sciences, 385:338-2 -023388--, 2017.
Xinghao Yang, Liu Weifeng, Wei Liu, and Dacheng Tao. A survey on canonical correlation analysis. IEEE Trasacttons Knowledge and Data Engineering, 2019.
Jun Yu, Benjamin Zalatan, Yong Chen, Li Shen, and Lifang He. Tensor-based multi-modal multi-target regression for alzheimer's disease prediction. In 2022 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), pp. 639-646. IEEE, 2022.
Hengrui Zhang, Qitian Wu, Junchi Yan, David Wipf, and Philip Yu. From canonical correlation analysis to self-supervised graph neural networks. Adcances in Newral Information Processing Systems, 34:76 89, 2021.
Wenming Zheng, Xiaoyan Zhou, Cairong Zou, and Li Zhao. Facial expression recognition using kernel canonical correlation analysis (koca). JEEE Transactions on Neural Nelworks, 17(1):233 238, 2006.
Hua Zhou, Lexin Li, and HongLu Zhu. Tensor regression with applications in neuroinaging data analysis. Journal of the American Statistical Association, 108(5)2):540 552, 2013.
Xiuli Zhu, Seshu Kumar Damarla, Kuangrong Hao, Biao Huang, Hongtian Chen, and Yicun Hua. Convlstm and self-attention aided canonical correlation analysis for multi-output soft sensor modeling. 1EEE Transactions on Instrumenlation and Measurement, 2023.
17


Reviews:
[{"review_id": "UuliS1IuzZ", "reviewer": "Reviewer_yh64", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: The proposed gSTCCA method seems to work well on real datasets.\n\nWeaknesses:\n\n(1) The proposed gSTCCA method seems to be a straightforward application of the results in He et al. (2018) (e.g., from Eq. (6) to Eq. (8) and the fast SURF algorithm for solving Eqs. (13) and (14)), with a combination of a lemma in Mai & Zhang (2019) (which gives Lemma 4.1 presented in this submission). \n\n(2) The settings for the experiments are vague/confusing. For example: \n\n- From the current Section 6.2, I cannot see clearly how the authors construct the input tensors $\\mathcal{X}_n$ and $\\mathcal{Y}_n$ from the datasets.\n- I am not familiar with CCA. Is it standard to compare approaches for CCA in terms of the downstream classification task (and via classification accuracy)? \n- For the classification task, why the authors choose the SVM classifier? I believe that there should be many classification algorithms (maybe random forest and XGBoost) that significantly outperform SVM in terms of both accuracy and running time. \n\n(3) Some minor problems in the writing: \n\n- It is better to use \\mathrm{\\cdot} for $Vec(\\cdot)$ and $Tr(\\cdot)$ and similar notations. The authors should clearly define the $\\\\|\\cdot\\\\|_1$ norm for a tensor. \n- To make Eq. (2) consistent with Eq. (5), it should be $\\mathbf{u}^T \\mathbf{X}\\mathbf{X}^T \\mathbf{u}   = \\mathbf{u}^T \\mathbf{Y}\\mathbf{Y}^T \\mathbf{u} = N$. \n- In the sentence before Lemma 4.1: I did not find Lemma 4 in Mai & Zhang (2019). I guess the authors are referring to Lemma 2 or 3 in Mai & Zhang (2019). Lemma 4.1 seems to be a simple adaption of the lemma in Mai & Zhang (2019). The proof in Appendix is not needed (and it is wrong; obviously, showing that $\\hat{\\mathbf{u}}^*$ satisfies the unit-variance constraint is not sufficient. You need also show its optimality). \n- Section 6: Please rephrase \"To avoid randomness\".", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this paper, the authors focus on the problem of performing sparse canonical correlation analysis (CCA) for tensor data. They pose the problem as a constrained multilinear least-squares problem with tensor structured sparse regularization, and then utilize the deflation technique to obtain multiple projections. This entire process is called the general sparse tensor CCA (gSTCCA) method. The authors present experimental results on five real datasets to demonstrate the efficacy and robustness of the proposed gSTCCA method.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See the Weaknesses above.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "vhGP7NaqpA", "reviewer": "Reviewer_dFfz", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: Overall the paper is well written, the English is of a satisfactory level, and the presentation is clear and easy to follow. The notation is clear and mostly consistent. The problem addressed in the paper is essential and has numerous applications. I find the solution presented by the authors novel and efficient. The authors use real data and several demonstrations to showcase the applicability of the method.\n\nWeaknesses: While modeling the data as tensors is well-motivated the sparsity is not motivated at all. Typically sparse CCA is motivated by the regime of high dimensional low sample size data; in this regime, CCA overfits since the problem is underdetermined. Unfotonotly, I don’t see any discussion on the sparsity and why it is useful for this method. Furthermore, I am missing quantitative and qualitative results that demonstrate the useability of sparsity. Can it help identify informative features in high-dimensional settings? \nAll datasets evaluated in the paper are not really multimodal rather the authors extract features to create coupled datasets, this is useful for evaluating a method, but I’m missing a use case for a real multimodal setting where data is samples with different measurement devices. Each device can have its own noise, and CCA-type schemes can lead to a substantial gain in fusing the shared information and attenuating the noise.\nEvaluating is limited to relatively simple datasets (N<1000). \nThe method is not compared to more recent CCA-based schemes (also, these are not mentioned):\n\n[1] Friedlander, T., & Wolf, L. (2023, May). Dynamically-Scaled Deep Canonical Correlation Analysis. In Pacific-Asia Conference on Knowledge Discovery and Data Mining (pp. 232-244). Cham: Springer Nature Switzerland.\n\n[2] Lindenbaum, Ofir, et al. \"L0-sparse canonical correlation analysis.\" International Conference on Learning Representations. 2021.\n[3] Sanghavi, Rushil, and Yashaswi Verma. \"Multi-view multi-label canonical correlation analysis for cross-modal matching and retrieval.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors address the important problem of unsupervised fusion of information collected from multiple modalities. Toward this goal, they focus on the well-celebrated Canonical Correlation Analysis (CCA) approach. CCA is a powerful multimodal dimensional reduction scheme that learns to transform coupled datasets into a low-dimensional space where the observations are maximally correlated on each axis. CCA has numerous linear and nonlinear extensions and countless applications. This manuscript presents a generalization of CCA termed general Sparse Tensor CCA (gSTCCA). The problem is framed as multilinear least squares that can be solved efficiently by breaking it up into many unconstrained linear least squares problems. They use real datasets to demonstrate the merits of the proposed approach.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "It is essential to add details about why sparsity is important.\nSome more recent sparse CCA methods are missing from the related work section.\nThe tensor product could be added to the notation to help the reader.\nCapital R is mentioned twice with different contexts; on P3 it is the rank, and on page 4 it serves as the dimension of V and U. Please change.\nIn section 4.1 the transition between the second and third paragraph is too sharp; please revise the writing to connect the two paragraphs.\nThe constraint in Eq. 6 is not motivated and explained. I understand why this is needed, but it is not detailed in the paper.\nSome dots are missing after equations, for example, 12.\nP7 “a standard orthogonalization process..” can you elaborate on this claim?\n\nWhy don’t all the methods in Table 4 also appear in Figure 1?\nCan you demonstrate the effect of sparsity on feature selection? (see [2] for examples).\n", "broader_impact_concerns": "No impact concerns or ethical implications", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "BCAraN3W1p", "reviewer": "Reviewer_Gq2a", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength: \nThis paper is easy to follow. \n\nWeakness: \nW1 (Novelty). \nThe proposed model (5) uses rank-1 tensors as the CCA axises, which seems a straightforward extension of Model (4) and the novelty is thus limited. \n\nW2 (Effectiveness).\n It seems that proposed tensor-based model has no obvious theoretical advantage in comparison with deep learning based models. \n\nW3 (Writing). \nThe writing can be improved.  It is better to use '\\in' rather than '=' before 'argmin' in. Eqs. (7), (8), (12), etc.  \n\nW4 (Experimental settings). \nThe experiments are far from satisfactory. \nW4.1. The authors did not give a convincing explanation about why the proposed model performs better.\nW4.2. The sizes of the datasets are not large enough. Thus, it is not sufficient to show the empirical effectiveness of the proposed model in real applications. \nW4.3. The proposed model has not been compared with SOTA classification methods like deep neural networks.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "A sparse tensor CCA model is proposed as a constrained multi-linear least-squares problem with tensor structured sparse regularization. The model is solved by alternative minimization. Experimental results with other tensor CCA models show the effectiveness of the proposed model for classification.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please refer to \"Weakness\".", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers agree that the paper is well written and easy to understand.

Reviewer yh64 recommends accepting the paper after the response provided by the authors. The main concerns of the reviewer were the novelty of the proposed method (more below) and some clarifications regarding the experimental setting. All points were clarified and the reviewer leaned to accept the paper.

Reviewer dFfz leans to reject the paper despite considering the problem to be important with many applications in diverse fields, and the proposed solution is novel. Their main concern is that the experimental evaluations fail to show strong evidence for the importance of incorporating the tensor-structured sparse regularization within the CCA framework. Adding such clarifications was the requirements that they established for accepting the paper. The authors provided a detailed response. To the question: “Can it help identify informative features in high-dimensional settings?” the authors provided an interesting analysis using ADNI dataset.

The AE agrees with the reviewer the importance of adding the sparsity regularization in more detail. Also considers the response provided by the authors sufficient. The sparse CCA problem (with and without tensor representations) has received attention in the literature, with several methods presented.This method could be useful for practitioners or other researchers working on the topic.

Reviewer Gq2a recommends rejecting the paper on the grounds of lack of novelty. The AE highlights that according to the evaluation criteria of TMLR, the novelty/significance should not be a reason for rejecting a paper but rather, whether or not the claims made in the submission are supported by accurate, convincing and backed by clear evidence. In addition, reviewers yh64 and dFfz disagree with this statement. The main novelty of the work has to do with providing a  formulation of the tensor-based sparse CCA problem with an efficient solution. The issue of novelty was also raised by Reviewer yh64, who later found satisfactory the explanation provided by the authors in their rebuttal.

Reviewer Gq2a also points out some limitations of the experimental evaluation (e.g. dataset size and showing why the model works) and states that the effectiveness of the method is not shown because the authors should have compared the proposed method to deep learning-based baselines. The authors clarified that the proposed model is an unsupervised approach and the classification experiments are a downstream task to show the effectiveness of the learned representations. 

The AE finds this explanation satisfactory. Having said this, the AE agrees that the paper would benefit from showing other downstream tasks in addition to classification. While the method achieves state of the art performance with respect to other CCA methods, if classification is the only talk, asking for other baselines seems a reasonable request. Furthermore, CCA is widely used in several unsupervised learning scenarios that would also be very interesting to show. The analysis added in the response to reviewer dFfz with the ADNI datasets is a great example of it.

Overall the AE considers the paper borderline but meeting the acceptance criteria of TMLR, Specifically, the proposed solution is technically strong and well presented. Being a methodological work, the experimental evidence is sufficient to back the claims in the work, and the topic is certainly of interest by many readers of TMLR. The AE agrees that the paper would have been much stronger by adding more experimental validation of the benefits of the proposed model. 

The AE asks the authors to incorporate all the feedback provided by the reviewers and the changes described in their rebuttal, putting particular attention to the comments provided by Reviewer dFfz regarding the motivation for the tensor-structured sparse regularization as well as the analysis of the ADNI dataset.
Decision: Accept (poster)