Title: On Sparse Canonical Correlation Analysis
Venue: NeurIPS 2024
Accepted: yes
Keywords: Sparse canonical correlation analysis

low rank

exact and approximation algorithms
Abstract: The classical Canonical Correlation Analysis (CCA) identifies the correlations between two sets of multivariate variables based on their covariance, which has been widely applied in diverse fields such as computer vision, natural language processing, and speech analysis. Despite its popularity, CCA can encounter challenges in explaining correlations between two variable sets within high-dimensional data contexts. Thus, this paper studies Sparse Canonical Correlation Analysis (SCCA) that enhances the interpretability of CCA. We first show that SCCA generalizes three well-known sparse optimization problems, sparse PCA, sparse SVD, and sparse regression, which are all classified as NP-hard problems. This result motivates us to develop strong formulations and efficient algorithms. Our main contributions include (i) the introduction of a combinatorial formulation that captures the essence of SCCA and allows the development of exact and approximation algorithms; (ii) the establishment of the complexity results for two low-rank special cases of SCCA; and (iii) the derivation of an equivalent mixed-integer semidefinite programming model that facilitates a specialized branch-and-cut algorithm with analytical cuts. The effectiveness of our proposed formulations and algorithms is validated through numerical experiments.

On Sparse Canonical Correlation Analysis

Abstract

The classical Canonical Correlation Analysis (CCA) identifies the correlations between two sets of multivariate variables based on their covariance, which has been widely applied in diverse fields such as computer vision, natural language processing, and speech analysis. Despite its popularity, CCA can encounter challenges in explaining correlations between two variable sets within high-dimensional data contexts. Thus, this paper studies Sparse Canonical Correlation Analysis (SCCA) that enhances the interpretability of CCA. We first show that SCCA generalizes three well-known sparse optimization problems, sparse PCA, sparse SVD, and sparse regression, which are all classified as NP-hard problems. This result motivates us to develop strong formulations and efficient algorithms. Our main contributions include (i) the introduction of a combinatorial formulation that captures the essence of SCCA and allows the development of approximation algorithms; (ii) the establishment of the complexity results for two low-rank special cases of SCCA; and (iii) the derivation of an equivalent mixed-integer semidefinite programming model that facilitates a specialized branch-and-cut algorithm with analytical cuts. The effectiveness of our proposed formulations and algorithms is validated through numerical experiments.

1 Introduction

The Canonical Correlation Analysis (CCA), proposed by H. Hotelling [23], aims to identify the correlations between two sets of multivariate variables based on their covariance. Since then, CCA has become a powerful statistical technique used for multivariate data analysis, with its applications across diverse fields such as computer vision [24], natural language processing [38], and speech analysis [21]. Despite its popularity, CCA can encounter challenges in explaining correlations between two variable sets within high-dimensional data contexts, such as genomic datasets [36]. In contrast, Sparse Canonical Correlation Analysis (SCCA), which seeks sparse linear combinations of these variable sets, offers substantially enhanced interpretability [41, 42, 44].

Formally, this paper studies the SCCA problem:

where s 1 ≤ n , s 2 ≤ m are positive integers and ( B A A ⊤ C ) denotes a covariance matrix of ( n + m )

random variables. Specifically, B and C are the covariance matrices of the n and m random variables, respectively, and A ∈ R n × m is the cross-covariance matrix between n and m random variables.

Hence, ( B A A ⊤ C ) , B , C are positive semidefinite matrices of size ( n + m ) , n , and m , respectively.

Here, matrices B , C can be singular, i.e., some random variables may be dependent on others. In fact, the covariance matrices B , C are often low-rank, especially within the high-dimension low-sample size data context (see, e.g., the gene expression data in [41]).

The SCCA problem generalizes three widely-studied sparsity-constrained optimization problems as special cases, which are sparse PCA [2, 13, 27], sparse SVD [28, 41], and sparse regression [22, 3]. To be specific, when n = m , s 1 = s 2 , B , C are identity matrices, and A is a positive semidefinite matrix, SCCA reduces to the classic sparse PCA problem; when B , C are identity matrices, SCCA becomes the sparse SVD problem; and when A is rank-one, Section 3 shows that SCCA is equivalent to two sparse linear regression subproblems.

1.1 Main contributions

SCCA is generally NP-hard, given that its special cases, sparse PCA, sparse SVD, and sparse regression are all classified as NP-hard problems. We are motivated to develop efficient formulations and algorithms for SCCA through a mixed-integer optimization lens. The main contributions, along with the structure of the remainder of this paper, are the following:

(i) In Section 2, we present an exact semidefinite programming (SDP) reformulation and derive a closed-form optimal value of classic CCA problem. We also develop an equivalent combinatorial formulation of SCCA, which allows the development of approximation algorithms;

(ii) When the covariance matrix ( B A A ⊤ C ) is low-rank, Section 3 studies the complexity of two special cases of SCCA. This motivates us to develop a polynomial-time exact algorithm of complexity O ( n 3 + m 3 ) for solving SCCA to global optimiality when the sparsity levels (i.e., s 1 and s 2 ) meet or exceed the ranks of B and C ;

(iii) Section 4 derives an equivalent mixed-integer SDP (MISDP) reformulation for SCCA. When applying the Benders decomposition approach, instead of solving the large-scale SDPs, we design a customized branch-and-cut algorithm with closed-form cuts, which can successfully solve SCCA to optimality; and

(iv) Section 5 numerically test the proposed formulations and algorithms. It is noted that our polynomial-time exact algorithm can solve real-world instances with n = 19 , 672 and m = 2 , 149 variables in seconds, provided that both s 1 and s 2 are at least the ranks of B and C .

Our analyses and results can be extended to SCCA with multiple pairs of basis vectors ( x , y ) , allowing for a more flexible and comprehensive exploration of correlations among data sets. The detailed formulations of multiple SCCA, along with the computational results, are provided in Appendix F.

1.2 Relevant literature

SCCA. To the best of our knowledge, the work [36] was the first paper that introduced the concept of SCCA to select only small subsets of variables to better explain the relationship between many genetic loci and gene expression phenotypes. A handful subset of features enhances interpretability, a desirable property, especially in complex data analysis, which has been successfully demonstrated in Sparse PCA [25]. To obtain sparse canonical loadings ( x , y ) , [39] first applied elastic net penalty to the classical CCA via an iterative regression procedure. In a seminal work on SCCA [41], the authors proposed a rigorous formulation by enforcing the ℓ 1 constraints on variables ( x , y ) and developed a penalized matrix decomposition method to solve the penalized CCA problem. Then, extensive research has focused on various penalty norm functions to obtain sparse canonical loadings (see, e.g., [20, 26, 39, 42, 10]). In particular, [10] penalized multiple canonical loadings by ℓ 1 norm and computed the sparse solution by the linearized Bregman method. It should be noted that under the assumption that the leading canonical loadings are sparse, [7, 17, 18] established theoretical guarantees of iterative approaches for estimating sparse solutions. Another research direction in SCCA introduced penalty functions based on group structural information of input data and developed group SCCA methods [29, 30]. For a comprehensive overview of CCA and SCCA methods, we refer readers to the survey by [44] and the references therein. These approaches, however, do not strictly enforce the exact sparsity requirement but only approximate the sparsity requirement (i.e., the ℓ 0 norm) by a convex function. Another relevant work [40] introduced binary variables to recast SCCA as a mixed-integer nonconvex program under the assumption of positive definite matrices B , C , based on which they designed a branch-and-bound algorithm. Different from the literature, our work does not require positive definiteness assumption of matrices B , C , and we are able to

obtain mixed-integer conic and semidefinite programming reformulations, allowing for better exact and approximation algorithms.

Connections to and differences with sparse PCA and sparse SVD. Analogous to SCCA, both sparse PCA [13, 25] and sparse SVD [28] select small subsets of variables to improve the interpretability of dimensionality reduction methods: PCA and SVD. Considerable investigation has been conducted on solving sparse PCA and sparse SVD from three angles: convex relaxations [12-14], approximation algorithms [6, 9, 28], and exact algorithms [2, 27, 28]. As mentioned before, in sparse PCA and sparse SVD, the covariance matrices B , C are identity. Such a setting dramatically simplifies the subset selection problems of sparse PCA and sparse SVD compared to that of SCCA, as in these problems, it suffices to focus on the selection of a submatrix of the matrix A . Specifically, it is shown in [11, 27, 35] that sparse PCA reduces to selecting a principal submatrix of A to maximize the largest eigenvalue(s) and sparse SVD reduces to selecting a possibly non-symmetric submatrix of A to maximize the largest singular value(s) [28]. Quite differently, the combinatorial reformulation (1) of SCCA aims to simultaneously select a sized-( s 1 × s 1 ) principal submatrix of B , a sized-( s 2 × s 2 ) principal submatrix of C , and a sized-( s 1 × s 2 ) submatrix of A . These fundamental differences in the underlying formulations of sparse PCA and sparse SVD preclude the direct application of their existing algorithms to the SCCA.

Notations: The following notation is used throughout the paper. We use bold lower-case letters (e.g., x ) and bold upper-case letters (e.g., X ) to denote vectors and matrices, respectively, and we use corresponding non-bold letters (e.g., x i ) to denote their components. We let S n , S n + , S n ++ denote the set of all the n × n symmetric real matrices, the set of all the n × n symmetric positive semidefinite matrices, and the set of all the n × n symmetric positive definite matrices, respectively. We let I denote the identity matrix and let 0 denote the vector or matrix with all-zero entries. We let R n + denote the set of all n -dimensional nonnegative vectors. We let [ n ] = { 1 , 2 , · · · , n } , [ s, n ] = { s, s +1 , · · · , n } . Given a matrix A ∈ R n × m and two subsets S ⊆ [ n ] , T ⊆ [ m ] , we let A † denote the pseudo inverse of matrix A , let A S,T denote a submatrix of A with rows and columns indexed by sets S, T , respectively, and let ( A S,T ) † denote the pseudo inverse of submatrix A S,T . For a set S and an integer k , we define the set S + k = { i + k | i ∈ S } . Given a vector a ∈ R n and a subset S ⊆ [ n ] , we let a S denote a subvector of a in the subset S . We define [ λ ] + = max { λ, 0 } . We let σ max ( · ) denote the largest singular value function and let λ max ( · ) denote the largest eigenvalue value function.

2 A combinatorial reformulation of SCCA

This section introduces an equivalent combinatorial optimization reformulation of SCCA. This reformulation serves as the foundation for developing two effective approximation algorithms.

2.1 An exact semidefinite programming representation of CCA

To begin with, let us focus on the classic CCA problem, which refers to SCCA without zero-norm constraints, as defined below:

This formulation of CCA can be regarded as a quadratically constrained quadratic program concerning the variables ( x y ) ∈ R n × m . We next define three-block matrices of size ( n + m ) below that aid in the presentation of our results.

By introducing a size-( n + m ) matrix variable X = ( x y )( x y ) ⊤ and removing the rank-one constraint on X , we can obtain an SDP relaxation of (CCA), as described below

Next, let us present a key lemma regarding properties of block matrices being positive semidefinite, fundamental for reformulating the SCCA.

Lemma 1 ([16]) For any symmetric matrix ( B A A ⊤ C ) ∈ S n + m , the followings are equivalent:

(i) The block matrix is positive semidefinite;

(ii) B ∈ S n + , ( I -BB † ) A = 0 , C -A ⊤ B † A ∈ S m + ; and

Inspired by Lemma 1, we hereby establish the equivalence between CCA and its SDP Relaxation. Remarkably, both of these problems achieve the same optimal value, namely σ max ( √ B † A √ C † ) .

Proposition 1 For the CCA problem, we have the following results.

(i) Both CCA and its SDP Relaxation have an optimal value σ max ( √ B † A √ C † ) ;

(ii) A pair of optimal solutions ( x ∗ , y ∗ ) to CCA satisfies

where q ∈ R n , p ∈ R m denote a pair of leading singular vectors of matrix √ B † A √ C † ; and

(iii) An optimal solution X ∗ to the SDP Relaxation is

□

Proposition 1 motivates the following observation on the optimal values of CCA and SCCA.

Observation 1 The optimal value of CCA is upper bounded by 1 , so is the optimal value of SCCA.

It is noteworthy that the results presented in Proposition 1 are established through a distinct methodology. This methodology leverages the positive semidefinite condition of block matrices, as shown in Lemma 1, and incorporates duality theory. This approach differs from most prior research [31, 37, 44], which proved Part (i) of Proposition 1 by relying on the singular value decomposition and assuming that matrices B and C are positive definite (i.e., full rank). To the best of our knowledge, [10] showed parts (i) and (ii) of Proposition 1 for a special low-rank CCA problem, where the authors assumed that the covariance matrices are defined as A = UV ⊤ , B = UU ⊤ , and C = V V ⊤ . Remarkably, Proposition 1 extends this result to a more general scenario where B and C are not constrained to be strictly positive definite and A is not constrained to directly depend on B , C , allowing for rank deficiencies and flexible data structure.

2.2 An equivalent formulation of SCCA

In this subsection, we transform SCCA into a combinatorial optimization problem, according to the insights provided by Proposition 1.

Theorem 1 SCCA is equivalent to the following combinatorial optimization:

The combinatorial formulation (1) presents significant computational difficulties when attempting to solve SCCA. The primary obstacles are two-fold: first, simultaneously selecting submatrices from the matrices A , B , C requires a sophisticated optimization across multiple dimensions. Second,

the selection criterion is particularly complex, as it involves optimizing the largest singular value of the product of the selected submatrix of A and the square root of pseudo-inverse submatrices of B and C . These complexities necessitate effective optimization solution procedures to address the high-dimensional and non-convex nature of the problem.

Motivated by Theorem 1, we customize the greedy and local search algorithms for SCCA (1) that has been widely used in the literature to solve special cases of SCCA, such as sparse PCA and sparse SVD in literature (see, e.g., [27, 28]). The detailed implementations can be found in Appendix B.

3 Low-rank SCCA

In practice, it is common that the sample covariance matrix ( B A A ⊤ C ) exhibits low-rank characteris- tics. This phenomenon is especially prominent when dealing with high-dimensional, low-sample size data, e.g., the real gene expression data in [41]. In this section, we study two special cases of low-rank SCCA and their computational complexities. Specifically, we develop a polynomial-time exact algorithm of complexity O ( n 3 + m 3 ) for solving SCCA to global optimiality when sparsity levels (i.e., s 1 and s 2 ) exceed or equal the ranks of B and C . Besides, we recast SCCA into mixed-integer convex quadratic programming when matrix A is rank-one.

3.1 Special Case I: SCCA with low-rank covariance matrices

In this section, we show that the computational complexity of SCCA is contingent upon the ranks of the covariance matrices B and C . To be more precise, when the sparsity level s 1 (or s 2 ) is equal to or greater than the rank r (or ˆ r ) of the covariance matrix B (or C ), the imposition of a zero-norm constraint over x (or y ) in SCCA becomes redundant. Consequently, lower ranks in the covariance matrices correspond to better computational complexity in solving SCCA.

Theorem 2 Suppose r = rank( B ) and ˆ r = rank( C ) , then SCCA takes a complexity of O ( n r -1 m ˆ r -1 + n r -1 + m ˆ r -1 ) . The following results hold:

(i) When s ≥ r and s ≥ ˆ r

(ii) When s 1 ≥ r and s 2 < ˆ r , the SCCA problem can be reduced to

(iii) When s 1 < r and s 2 ≥ ˆ r , the SCCA problem can be reduced to

The results in Theorem 2 build on the covariance structure of the data matrix. Specifically, if B and C are of rank r and ˆ r , respectively, there are only r and ˆ r linearly independent vectors in the subspaces corresponding to B and C . Thus, the cosine of the principal angle can always be represented by these r and ˆ r vectors. As a result, the canonical directions of CCA consist of only r and ˆ r nonzero elements. We further make the following remarks about Theorem 2:

(i) Theorem 2 implies the complexity of solving SCCA, as summarized in the corollary below.

(ii) Inspired by Part (i) of Theorem 2, we also develop a polynomial-time exact algorithm yielding an optimal solution to SCCA (1) when s 1 ≥ r and s 2 ≥ ˆ r . The detailed implementation can be found in Algorithm 1, which successfully solves some large instances with up to n = 19 , 672 and m = 2 , 149 variables in seconds in our numerical experiments; and

(iii) The proof of Theorem 2 implies that CCA always admits an optimal sparse solution ( x ∗ , y ∗ ) satisfying ∥ x ∗ ∥ 0 ≤ r and ∥ y ∗ ∥ 0 ≤ ˆ r . We show in Proposition 1 that the SDP Relaxation of CCA is exact. Therefore, as a side product, we provide the first-known sufficient condition about (i.e., s 1 ≥ r and s 2 ≥ ˆ r ) when the convex SDP Relaxation matches SCCA.

Corollary 1 Suppose r = rank( B ) and ˆ r = rank( C ) . There exists an algorithm that can find an optimal solution to SCCA in O ( n r -1 m ˆ r -1 ) time complexity.

Proposition 2 Suppose r = rank( B ) and ˆ r = rank( C ) . Then Algorithm 1 returns an optimal solution to SCCA (1) in O ( n 3 + m 3 ) time complexity when s 1 ≥ r and s 2 ≥ ˆ r .

Proof. Following the proof of Theorem 2, we can show that the output solution of Algorithm 1 is optimal to SCCA (1). In addition, the Step 2 of Algorithm 1 needs computing the eigendecomposition of matrix B ∈ S n + , which takes a time of O ( n 3 ) . Given a matrix Q ∈ R n × ( n -r ) , it also takes a time of O ( n 3 ) to find its ( n -r ) linearly independent rows at Step 3 through the QR decomposition [19]. Hence, Algorithm 1 takes a complexity of O ( n 3 + m 3 ) . □

Algorithm 1 An exact algorithm for SCCA (1) when s 1 ≥ r and s 2 ≥ ˆ r

1: Input: Matrices A ∈ R m × n , B ∈ S m + , C ∈ S m + and integers s 1 ∈ [ r, n ] , s 2 ∈ [ˆ r, m ]

2: Compute the eigenvectors Q ∈ R n × ( n -r ) of B that correspond to its ( n -r ) zero eigenvalues,

3: Find ( n -r ) linearly independent rows in Q , and collect their indices into a subset T ∗ 1 ⊆ [ n ]

5: Define the subsets S ∗ 1 = [ n ] \ T ∗ 1 and S ∗ 2 = [ m ] \ T ∗ 2 , and compute

4: Perform the same procedure on matrix C to obtain the subset T ∗ 2 ⊆ [ m ]

6: Output: An optimal solution ( S ∗ 1 , S ∗ 2 ) and optimal value v ∗

3.2 Special Case II: SCCA with a rank-one cross-covariance matrix

In this subsection, we study the other interesting low-rank special case of SCCA where the crosscovariance matrix A is rank-one. For this special case, we prove its NP-hardness with reduction to the sparse regression problem. We further demonstrate that rank-one SCCA can be simplified to solving two Mixed-Integer Convex Quadratic Programs (MICQPs), which can be more scalable than directly solving SCCA. Our numerical findings confirm this improved scalability.

We observe that SCCA can be separable over variables x and y for the rank-one A . In fact, suppose that A = ab ⊤ , then SCCA is equivalent to

which can be equivalently the product of the optimal values of the following two subproblems:

That is, the identity v ∗ = v x v y holds. Next, we show that each subproblem in (6) can be reduced to the classic sparse regression problem [1, 33] and is thus NP-hard as shown below.

Theorem 3 When matrix A = a ⊤ b is rank-one, each maximization problem in (6) is NP-hard.

Theorem 3 links the maximization problem (6) and the well-known sparse regression problem, implying that even solving the rank-one SCCA problem (5) is NP-hard. However, it also motivates us to adapt existing mixed-integer optimization techniques from sparse regression (see, e.g., [1, 4, 43]) to tackle each subproblem in (6). By introducing binary variables to model the zero-norm constraint, we derive equivalent MICQP formulations for subproblems (6) in Appendix C. There are two types of formulations depending on whether matrices B and C are positive definite, which build on the Big-M and perspective techniques, respectively.

4 Reformulating SCCA as a mixed-integer semidefinite program (MISDP)

While the combinatorial formulation (1) is elegant in its structure, it poses significant challenges when attempting to solve it to optimality using branch-and-bound based methods. To fill this gap, in this section, we derive an equivalent MISDP formulation for SCCA, amenable for developing exact methods.

First, it is noted that an optimal solution ( x ∗ , y ∗ ) to SCCA is always bounded that satisfies ∥ x ∗ ∥ 2 2 ≤ M 1 and ∥ y ∗ ∥ 2 2 ≤ M 2 , where we specify the construction of the coefficients M 1 and M 2 in Appendix C.1. Such bounds are essential to the derivation of the MISDP. It is convenient to introduce the following notation about { M ii } i ∈ [ n + m ] :

Theorem 4 The SCCA is equivalent to the following MISDP:

Note that the proposed MISDP formulation (7) is of size ( n + m ) × ( n + m ) since our matrix variable

We have formulated SCCA as a mixed-integer convex optimization problem in Theorem 4. Unfortunately, no commercial solvers can efficiently solve MISDP problems. We derive an equivalent mixed-integer linear program of SCCA with exponentially many linear constraints and an efficient separation oracle based on the approach introduced by [15], which allows us to develop a tailored branch-and-cut algorithm. First, by separating the binary variables z , we rewrite the MISDP (7) as

where the function f ( z ) is defined as

For any feasible solution ˆ z ∈ Z of the problem (8), by leveraging the concavity of function f ( · ) , the linear inequality

cuts off the solution ˆ z unless it happened to be optimal in (8), which paves the way for a delayed cut-generation procedure within a branch-and-bound framework. As the linear inequality of the above type needs to be added dynamically given different solutions ˆ z at each iteration, it calls for an efficient evaluation of function f (ˆ z ) and its subgradient. To speed up the computation, we derive the closed-form expression for both of them. The detailed derivations can be found in Appendix D.

Strategies to improve computational speed in practice: First, we provide a variable-fixing method that can identify some binary variables of the MISDP (7) being one at optimality. Removing these pre-selected variables from the feasible set reduces the problem size of SCCA. Second, we enhance the branch-and-cut algorithm with a high-quality warm start solution obtained from the local search algorithm. Third, by relaxing the binary variables in the MISDP (7) to be continuous or computing CCA, we can obtain an upper bound of SCCA, and the gap between this bound and the local search output gives an initial gap at the root node. Finally, at each iteration, the branching node is selected based on its potential to decrease the current upper bound instead of random branching.

5 Numerical results

This section tests the numerical performance of our formulations and algorithms on synthetic and real data. All the experiments are conducted in Python 3.6 with calls to Gurobi 9.5.2 and MOSEK 10.0.29 on a PC with 10-core CPU, 16-core GPU, and 16GB of memory. The codes and data used in our experiments are available at https://github.com/yongchunli-13/SCCA.git .

5.1 Experimental setup

Synthetic data generation: Before we present the empirical results, we first describe the properties of the synthetic data which shall be used throughout this section. By following [32], given parameters ( n, m, s 1 , s 2 ) , we first synthetically generate positive definite matrices B ∗ ∈ S n ++ and C ∗ ∈ S n ++ by B ∗ = ˆ B ˆ B ⊤ + I and C ∗ = ˆ C ˆ C ⊤ + I , respectively, where ˆ B and ˆ C consist of elements generated from a normal distribution N (0 , 1) . Then, we let A ∗ ∈ R n × m = λ B ∗ uv ⊤ C ∗ , where we generate λ uniformly from (0 , 1) , and vectors u , v are generated from a normal distribution N (0 , 1) that satisfy ∥ u ∥ 0 = s 1 , ∥ v ∥ 0 = s 2 , u ⊤ B ∗ u = 1 and v ⊤ C ∗ v = 1 . Next, we sample N = 5 , 000 data samples from a normal distribution N ( 0 , ( B ∗ A ∗ ( A ∗ ) ⊤ C ∗ )) and compute their sample covariance matrix to .

obtain the testing data ( B A A ⊤ C )

Real data: To obtain a comprehensive understanding of the overall performance of our algorithms, we further conduct experiments on six UCI datasets [5] with sizes ranging from 34 to 385 variables. The dataset is split into the first n variables and the remaining m variables to construct the sample covariance matrices A , B , C . Besides, we examine the performance of the proposed algorithms on the real breast cancer dataset [8] that contain n = 19 , 672 and m = 2 , 149 variables. The information on each dataset is summarized in Appendix E.

Throughout, the computational time is in seconds, the time limit is one hour, and the dashed line '-" denotes the unsolved case within the time limit. Note that we let LB denote the lower bound obtained from the approximation algorithm, and we let UB denote the upper bound obtained from convex relaxations of SCCA. Besides, we define gap(%) = 100 × ( UB -v ∗ ) /v ∗ to be the optimality gap, and we replace v ∗ with the best lower bound when v ∗ is not available. We define MIPGap(%) to be the gap of exact algorithms at termination. Notably, the complexity analysis of the SCCA problems in Section 3 indicates that its solution process depends on the ranks of the data matrices. Therefore, we present the numerical results under both full-rank and low-rank cases for a comprehensive evaluation.

5.2 Illustration of the impact sparsity levels on SCCA

In this subsection, we apply the local search algorithm to evaluating the performance of SCCA against different sparsity levels s 1 and s 2 . Specifically, for a given dataset, we compute the ratio of correlations between SCCA and CCA for various s 1 , s 2 parameters. We test the real UCI data and synthetic data, and the results are displayed in Figure 1 and Figure 2. This visualization provides insights for the maximum sparsity SCCA can achieve while maintaining the correlation of full data. For the real UCI data, SCCA almost recovers the correlation of CCA when s 1 ≈ n/ 2 and s 2 ≈ m/ 2 .

2660

866'0

1.000

1.000

1.000

0.998

0.992

0.998

0.999

1.000

1.000

0.996

2660

660

0.999

0.999

666'0

0.994

[66°0

0.96

2660

0.997

0.997

0.992

0.989

0.992

0.993

0.994

0.994

0.990

10

15

20

25

S1

5.3 Solving SCCA with full-rank matrices

The numerical results on synthetic and real data are presented in Table 1 and Table 2, respectively, which include multiple instances with various parameters ( n, m, s 1 , s 2 ) . First, we observe that the greedy and local search algorithms are scalable, and their outputs match the optimal values for most solved testing cases. That is, they achieve zero optimality gaps on these cases. In the 'Convex relaxation" column, we compute an upper bound by solving either the continuous relaxation of MISDP (7) or CCA, and we use CCA for n ≥ 40 and m ≥ 40 cases for efficiency. It is seen that the upper bound maintains an optimality gap at most 2 . 78% . Then, we apply the branch-and-cut algorithm to solve SCCA to optimality, which can handle the case up to a size of n = m = 120 in

1.0000

860

0.993

0.995

0.997

0.997

0.999

1.000

1.000

0.9975

2

0.989

0.993

0.994

0.996

0.997

0.999

1.000

1.000

0.9950

0.992

0.994

0.994

0.996

0.999

0.999

0.999

1.000

0.9925

-

0.988

0.993

0.989

0.995

0.997

0.998

0.999

0.999

S

0.9900

0.988

0.990

0.992

0.993

0.996

0.997

0.998

0.997

0.9875

-0.981

0.990

0.992

0.994

0.996

660

0.997

0.996

- 0.9850

0.986

0.987

0.990

0.990

0.993

0.994

0.994

0.992

- 0.9825

-0.900

0.963

0.985

0.968

0.969

0.992

0.991

0.991

10

20

30

40

50

60

70

80

S1

5.4 Solving SCCA with low-rank matrices

Despite the high dimensions of the breast cancer dataset [8], the resulting sample covariance matrices B and C have a rank of 89 , i.e., r = ˆ r = 89 . When s 1 ≥ r and s 2 ≥ ˆ r , we apply Algorithm 1 to solving SCCA that returns optimal solutions and values in one second, as shown in Table 3. If s 1 < r and s 2 < ˆ r , Algorithm 1 cannot applied and the proposed branch-and-cut algorithm is hard to scale to this dataset. Hence, we only consider using approximation algorithms to solve SCCA for small s 1 and s 2 cases in Table 4. To be specific, we randomly sample n and m variables from the breast cancer data to construct testing cases. As displayed in Table 4, the greedy and local search algorithms run fast, and the local search algorithm slightly outperforms the greedy output.

The SCCA (5) problem with a rank-one matrix A can be more tractable, as we can equivalently decompose it into two MICQPs (see Appendix C). By approximating A with a rank-one matrix

including leading singular value and vectors, Table 5 presents the numerical results for solving rank-one SCCA (5). The continuous relaxation of the MICQPs also provides an upper bound and is denoted by the Perspective relaxation in Table 5. We see that the perspective relaxation is computationally efficient and yields small optimality gaps. Besides, we can directly solve two MICQPs below via Gurobi to find the optimal value of rank-one SCCA (5), i.e., v ∗ = v x v y , where the performance can be found in the last column of Table 5. We can address the rank-one SCCA (5) problem up to size 200 × 200 within one hour, improving the problem-solving capacity compared to the size120 × 120 full-rank SCCA (5) in Table 1. However, it should be pointed out that the SCCA may not be mixed-integer convex quadratic representable in general.

5.5 Experimental comparison of SCCA algorithms

This section compares the proposed local search algorithm with the SCCA methods of [10, 37, 41] in correlation (Corr) value, the zero norm of x (denoted by S. x ), the zero norm of y (denoted by S. y ), and running time. The computational results on synthetic, UCI, and breast cancer data are presented in Table 6, where we highlight the best correlation and sparsity results in bold. Unlike the local search algorithm, these existing methods do not strictly enforce the exact sparsity requirement, i.e., the zero-norm constraints on variables x , y . Consequently, the local search algorithm achieves the best sparsity for nearly all testing cases. More importantly, the local search algorithm yields a larger correlation value than these existing methods in 16 out of 22 testing cases. Finally, the running time of the local search algorithm dominates that of [10, 37].

Acknowledgments and Disclosure of Funding

Santanu S. Dey would like to gratefully acknowledge the support from ONR grant # N00014-22-12632. Yongchun Li (when was a graduate student) and Weijun Xie were supported in part by the National Science Foundation grant # 2246417 and ONR grant # N00014-24-1-2066. The authors also would like to gratefully acknowledge undergraduate researcher Quill O Healey for his help with the initial version of branch-and-bound codes.

References

Alper Atamturk and Andres Gomez. Rank-one convexification for sparse regression. arXiv preprint arXiv:1901.10334 , 2019.

Dimitris Bertsimas and Ryan Cory-Wright. Solving large-scale sparse PCA to certifiable (near) optimality. The Journal of Machine Learning Research , 23(1):566-600, 2022.

Dimitris Bertsimas, Angela King, and Rahul Mazumder. Best subset selection via a modern optimization lens. The Annals of Statistics , 44(2):813 - 852, 2016.

Dimitris Bertsimas, Jean Pauphilet, and Bart Van Parys. Rejoinder: Sparse regression: Scalable algorithms and empirical performance. Statistical Science , 35(4):623-624, 2020.

Catherine L Blake. UCI repository of machine learning databases. http://www. ics. uci. edu/˜ mlearn/MLRepository. html , 1998.

Siu On Chan, Dimitris Papailliopoulos, and Aviad Rubinstein. On the approximability of sparse PCA. In Conference on Learning Theory , pages 623-646. PMLR, 2016.

Mengjie Chen, Chao Gao, Zhao Ren, and Harrison H Zhou. Sparse cca via precision adjusted iterative thresholding. arXiv preprint arXiv:1311.6186 , 2013.

Koei Chin, Sandy DeVries, Jane Fridlyand, Paul T Spellman, Ritu Roydasgupta, Wen-Lin Kuo, Anna Lapuk, Richard M Neve, Zuwei Qian, Tom Ryder, et al. Genomic and transcriptional aberrations linked to breast cancer pathophysiologies. Cancer cell , 10(6):529-541, 2006.

Agniva Chowdhury, Petros Drineas, David P Woodruff, and Samson Zhou. Approximation algorithms for sparse principal component analysis. arXiv preprint arXiv:2006.12748 , 2020.

Delin Chu, Li-Zhi Liao, Michael K Ng, and Xiaowei Zhang. Sparse canonical correlation analysis: New formulation and algorithm. IEEE transactions on pattern analysis and machine intelligence , 35(12):30503065, 2013.

Alexandre d'Aspremont, Francis Bach, and Laurent El Ghaoui. Optimal solutions for sparse principal component analysis. Journal of Machine Learning Research , 9(7), 2008.

Alexandre d'Aspremont, Laurent Ghaoui, Michael Jordan, and Gert Lanckriet. A direct formulation for sparse PCA using semidefinite programming. Advances in neural information processing systems , 17, 2004.

Santanu S Dey, Rahul Mazumder, and Guanyi Wang. Using ℓ 1 -relaxation and integer programming to obtain dual bounds for sparse PCA. Operations Research , 70(3):1914-1932, 2022.

Santanu S Dey, Marco Molinaro, and Guanyi Wang. Solving sparse principal component analysis with global support. Mathematical Programming , 199(1-2):421-459, 2023.

Marco A Duran and Ignacio E Grossmann. An outer-approximation algorithm for a class of mixed-integer nonlinear programs. Mathematical programming , 36:307-339, 1986.

Jean Gallier et al. The schur complement and symmetric positive semidefinite (and definite) matrices (2019). URL https://www. cis. upenn. edu/jean/schur-comp. pdf , 2020.

Chao Gao, Zongming Ma, Zhao Ren, and Harrison H Zhou. Minimax estimation in sparse canonical correlation analysis. 2015.

Chao Gao, Zongming Ma, and Harrison H Zhou. Sparse cca: Adaptive estimation and computational barriers. 2017.

Colin R Goodall. 13 computation using the qr decomposition. 1993.

David R Hardoon and John Shawe-Taylor. Sparse canonical correlation analysis. Machine Learning , 83:331-353, 2011.

Hynek Hermansky and Nelson Morgan. Rasta processing of speech. IEEE transactions on speech and audio processing , 2(4):578-589, 1994.

Ronald R Hocking and RN Leslie. Selection of the best subset in regression analysis. Technometrics , 9(4):531-540, 1967.

Harold Hotelling. The most predictable criterion. Journal of educational Psychology , 26(2):139, 1935.

Hua Huang, Huiting He, Xin Fan, and Junping Zhang. Super-resolution of human face image using canonical correlation analysis. Pattern Recognition , 43(7):2532-2543, 2010.

John NR Jeffers. Two case studies in the application of principal component analysis. Journal of the Royal Statistical Society: Series C (Applied Statistics) , 16(3):225-236, 1967.

Kim-Anh Lê Cao, Pascal GP Martin, Christèle Robert-Granié, and Philippe Besse. Sparse canonical methods for biological data integration: application to a cross-platform study. BMC bioinformatics , 10:1-17, 2009.

Yongchun Li and Weijun Xie. Exact and approximation algorithms for sparse PCA. arXiv preprint arXiv:2008.12438 , 2020.

Yongchun Li and Weijun Xie. Beyond symmetry: Best submatrix selection for the sparse truncated svd. arXiv preprint arXiv:2105.03179 , 2021.

Dongdong Lin, Vince D Calhoun, and Yu-Ping Wang. Correspondence between fmri and snp data by group sparse canonical correlation analysis. Medical image analysis , 18(6):891-902, 2014.

Dongdong Lin, Jigang Zhang, Jingyao Li, Vince D Calhoun, Hong-Wen Deng, and Yu-Ping Wang. Group sparse canonical correlation analysis for genomic data integration. BMC bioinformatics , 14(1):1-16, 2013.

Yichao Lu and Dean P Foster. Large scale canonical correlation analysis with iterative least squares. Advances in Neural Information Processing Systems , 27, 2014.

Qing Mai and Xin Zhang. An iterative penalized least squares approach to sparse canonical correlation analysis. Biometrics , 75(3):734-744, 2019.

Alan Miller. Subset selection in regression . CRC Press, 2002.

Balas Kausik Natarajan. Sparse approximate solutions to linear systems. SIAM journal on computing , 24(2):227-234, 1995.

Dimitris Papailiopoulos, Alexandros Dimakis, and Stavros Korokythakis. Sparse PCA through low-rank approximations. In International Conference on Machine Learning , pages 747-755. PMLR, 2013.

Elena Parkhomenko, David Tritchler, and Joseph Beyene. Genome-wide sparse canonical correlation of gene expression with genotypes. In BMC proceedings , volume 1, pages 1-5. BioMed Central, 2007.

Elena Parkhomenko, David Tritchler, and Joseph Beyene. Sparse canonical correlation analysis with application to genomic data integration. Statistical applications in genetics and molecular biology , 8(1), 2009.

Alexei Vinokourov, Nello Cristianini, and John Shawe-Taylor. Inferring a semantic representation of text via cross-language correlation analysis. Advances in neural information processing systems , 15, 2002.

Sandra Waaijenborg, Philip C Verselewel de Witt Hamer, and Aeilko H Zwinderman. Quantifying the association between gene expressions and dna-markers by penalized canonical correlation analysis. Statistical applications in genetics and molecular biology , 7(1), 2008.

Akihisa Watanabe, Ryuta Tamura, Yuichi Takano, and Ryuhei Miyashiro. Branch-and-bound algorithm for optimal sparse canonical correlation analysis. Expert Systems with Applications , 217:119530, 2023.

Daniela M Witten, Robert Tibshirani, and Trevor Hastie. A penalized matrix decomposition, with applications to sparse principal components and canonical correlation analysis. Biostatistics , 10(3):515534, 2009.

Daniela M Witten and Robert J Tibshirani. Extensions of sparse canonical correlation analysis with applications to genomic data. Statistical applications in genetics and molecular biology , 8(1), 2009.

Weijun Xie and Xinwei Deng. Scalable algorithms for the sparse ridge regression. SIAM Journal on Optimization , 30(4):3359-3386, 2020.

Xinghao Yang, Weifeng Liu, Wei Liu, and Dacheng Tao. A survey on canonical correlation analysis. IEEE Transactions on Knowledge and Data Engineering , 33(6):2349-2368, 2019.

Appendices

A Proofs of technical results

A.1 Proof of Proposition 1

Proof. The proof includes three parts.

Part (i). To prove the equivalence between CCA and its SDP Relaxation, let us introduce the Lagrangian multiplies θ 1 ≥ 0 , θ 2 ≥ 0 corresponding to two constraints in SDP Relaxation, which leads to the following Lagrangian dual problem

̸

where the equation results from the definition of block matrices ˜ A , ˜ B , and ˜ C . Given the nonzero matrices A = 0 , B = 0 , C = 0 and positive semidefinite matrices B ⪰ 0 , C ⪰ 0 , following Lemma 1, we must have θ 2 C -A ⊤ ( θ 1 B ) † A / 4 ⪰ 0 and θ 1 B -A ( θ 2 C ) † A ⊤ / 4 ⪰ 0 , implying that either θ 1 = 0 or θ 2 = 0 is infeasible to the minimization problem above. That is, θ 1 > 0 and θ 2 > 0 must hold.

̸

̸

According to Lemma 1, the block matrix ( B A A ⊤ C ) is positive semidefinite, implying that ( I -CC † ) A ⊤ = 0 , ( I -BB † ) A = 0 . Then, it is easy to show

Given θ 1 , θ 2 > 0 and using Lemma 1, the result above allows us to further simplify the right-hand side minimization problem in (10) to

where the first equation is because

where we let B = Q Λ Q ⊤ denote the eigendecomposition of matrix B with Λ containing all the positive eigenvalues.

As a result, the dual problem of SDP Relaxation admits an optimal value of σ max ( √ B † A √ C † ) , which gives an upper bound of the CCA and its SDP Relaxation. Next, we construct their optimal solutions, which exactly attain this upper bound. Thus, this upper bound is achievable and equals their optimal values.

Part (ii). For the CCA, let us consider a part of optimal solutions ( x ∗ , y ∗ ) below

with q ∈ R n , p ∈ R m denoting a pair of leading singular vectors of matrix √ B † A √ C † .

First, ( x ∗ , y ∗ ) is feasible to the CCA as

where the inequalities stem from the facts that I ⪰ B † B B † and I ⪰ C † C C † .

On the other hand, according to the definitions of q , p , we can show that ( x ∗ , y ∗ ) is optimal to the CCA, i.e.,

Part (iii). In a similar vein, we can show that X ∗ = ( x ∗ y ∗ )( x ∗ y ∗ ) ⊤ is optimal to SDP Relaxation with the optimal value σ max ( √ B † A √ C † ) . □

A.2 Proof of Theorem 1

Proof. By introducing the subsets ( S 1 , S 2 ) to denote the supports of variables ( x , y ) in SCCA, then we can remove the zero-norm constraints on ( x , y ) and reformulate SCCA as

Following from the Part (i) in Proposition 1, we can show that for any subsets S 1 ⊆ [ n ] , S 2 ⊆ [ m ] , the following identity holds.

Plugging the result above into the inner maximization problem in (11), we complete the proof. □

A.3 Proof of Theorem 2

Proof. The proof is split into three parts.

Part (i). It suffices to prove that CCA admits an optimal solution ( x ∗ , y ∗ ) satisfying ∥ x ∗ ∥ 0 ≤ r and ∥ y ∗ ∥ 0 ≤ ˆ r . Then, ( x ∗ , y ∗ ) is also feasible and optimal to SCCA, which implies the equivalence between SCCA and CCA.

First, according to Part (ii) in Proposition 1, we can obtain a closed-form optimal solution (ˆ x , ˆ y ) for the CCA. By adjusting (ˆ x , ˆ y ) , we will construct a new optimal sparse solution ( x ∗ , y ∗ ) satisfying ∥ x ∗ ∥ 0 ≤ r and ∥ y ∗ ∥ 0 ≤ ˆ r .

For matrix B ∈ S n + , we let { q i } i ∈ [ n -r ] ∈ R n denote the eigenvectors corresponding to ( n -r ) zero eigenvalues of B . Thus, { q i } i ∈ [ n -r ] are orthonormal. There exists a size-( n -r ) subset S ⊆ [ n ] such that the subvectors { ( q i ) S } i ∈ [ n -r ] are linearly independent, where ( q i ) S denotes the subvector of q i indexed by S for each i ∈ [ n -r ] . As a result, there exist a vector ( γ 1 , · · · , γ n -r ) ⊤ such that

Let us now construct solution x ∗

where x ∗ i = 0 for all i ∈ S based on the equation (12) and | S | = n -r , implying ∥ x ∗ ∥ 0 ≤ r . In addition, we show that the new solution x ∗ is still optimal to CCA. First, x ∗ is feasible since

where the equation is due to Bq i = 0 for all i ∈ [ n -r ] .

Given the positive semidefinite block matrix ( B A A ⊤ C ) , using Part (ii) of Lemma 1, the identity ( I -BB † ) A = 0 is equivalent to ∑ i ∈ [ n -r ] q i q ⊤ i A = 0 . Then, for each i ∈ [ n -r ] , multiplying

q ⊤ i on both sides of this equation leads to

̸

where the result follows from q ⊤ i q j = 0 for any i = j . Then, we can show the optimality of the new solution x ∗ :

Similarly, we can also construct an optimal sparse solution y ∗ by leveraging ˆ y and eigenvectors of zero eigenvalues of C such that ∥ y ∗ ∥ 0 ≤ s 2 .

Therefore, there exists an optimal solution ( x ∗ , y ∗ ) to the CCA whose zero norms are bounded from above by r, ˆ r , respectively. Adding the constraints ∥ x ∥ 0 ≤ r, ∥ y ∥ 0 ≤ ˆ r to the CCA does not affect the optimality, which gives an equivalent formulation (2) of CCA.

Part (ii). Suppose that (˜ x , ˜ y ) denotes an optimal solution to problem (3). When s 1 ≥ r , following the proof of Part (I), ˜ x , we can construct another optimal solution x ∗ whose zero norm is bounded by r and ( x ∗ , ˜ y ) is feasible and optimal to SCCA.

Part (iii). Similarly, we can reduce SCCA to problem (4). We thus complete the proof. □

A.4 Proof of Theorem 3

Proof. Let us first consider the maximization problem over x in (6), i.e.,

Then, we derive a combinatorial optimization reformulation of problem (13) based on the result below.

Claim 1 For any subset S ⊆ [ n ] , max x ∈ R | S | { a ⊤ S x : x ⊤ B S,S x ≤ 1 } = √ a ⊤ S ( B S,S ) † a S .

Proof. Given A = ab ⊤ , since the matrix ( B ab ⊤ b ⊤ a C ) is positive semidefinite, using Lemma 1, the identity ( I -B S,S B † S,S ) a S b ⊤ = 0 must hold for any subset S . As a result, we have a S -B S,S B † S,S a S = 0 as vector b is nonzero.

Next, the Lagrangian dual of the problem max x ∈ R | S | { a ⊤ S x : x ⊤ B S,S x ≤ 1 } can be written as

where the second equation builds on the identity a S -B S,S B † S,S a S = 0 and optimal solution

Suppose that an optimal solution to problem (13) admits the support S ∗ . According to Claim 1, we have

On the other hand, the Lagrangian dual of problem (13) can be written as

where the first equation is due to Claim 1, the second inequality is by plugging the feasible solution λ ∗ = √ a ⊤ S ∗ ( B S ∗ ,S ∗ ) † a S ∗ 2 into minimization problem, and the last equation is from the optimality of subset S ∗ . Since both left-hand and right-hand sides above equal v x , the strong duality of problem (13) holds, and all the inequalities above must attain the equalities. That is, problem (13) is equivalent to

Since the outer minimization is a one-dimensional convex program that can be solved efficiently, as a result, for any given λ > 0 , the inner maximization is equivalent to solving

Next, let us consider the NP-hard sparse regression problem (see, e.g., [34]), which admits

where data matrix U consists of observations of n variables and vector v denotes the corresponding response variables.

Suppose that in the problem (14), let us define λ B = U ⊤ U and a = 2 U ⊤ v . Then using the singular value decomposition of matrix U , we see that the following equation still holds.

Thus, for any given λ > 0 , the maximization problem (14) is equivalent to the sparse regression problem (15). This shows that problem (13) is NP-hard.

Similarly, the maximization problem over y in (6) can also be reduced to the sparse regression problem. □

A.5 Proof of Theorem 4

Proof. For the SCCA (11), according to Proposition 1, the inner maximization problem admits an exact semidefinite programming formulation. Using the variables z ∈ Z to describe the set constraints in SCCA (11), we can reformulate it as

Proposition 3 shows that there is an optimal solution ( x ∗ , y ∗ ) to SCCA that satisfies ∥ x ∗ ∥ 2 2 ≤ M 1 and ∥ y ∗ ∥ 2 2 ≤ M 2 . Based on this, we can construct an optimal solution ( z ∗ , X ∗ ) for SCCA (16) by letting

̸

where the optimal solution X ∗ satisfies the following inequalities

This allows us to recast the SCCA (16) into an MISDP formulation (7). □

B Implementations of greedy and local search algorithms

This section presents the detailed implementations of greedy and local search algorithms based on the combinatorial formulation (1) of SCCA.

C Mixed-integer convex quadratic programming reformulations

This section shows that each subproblem in (6) can be equivalently formulated by a MICQP. Therefore, SCCA (5) is mixed-integer convex quadratic representable when matrix A is rank-one.

̸

Algorithm 3 Local search algorithm for SSVD (1) 1: Input: Matrices A ∈ R m × n , B ∈ S m + , C ∈ S m + and integers s 1 ∈ [ n ] , s 2 ∈ [ m ] 2: Initialize ( ˆ S 1 , ˆ S 2 ) as the output of greedy Algorithm 2 3: do 4: for each pair ( i 1 , j 1 ) ∈ ˆ S 1 × ([ n ] \ ˆ S 1 ) do 5: if σ max (√ ( B ˆ S 1 ∪{ j 1 }\{ i 1 } , ˆ S 1 ∪{ j 1 }\{ i 1 } ) † A ˆ S 1 ∪{ j 1 }\{ i 1 } , ˆ S 2 √ ( C ˆ S 2 , ˆ S 2 ) † ) > σ max (√ ( B ˆ S 1 , ˆ S 1 ) † A ˆ S 1 , ˆ S 2 √ ( C ˆ S 2 , ˆ S 2 ) † ) then 6: Update ˆ S 1 = ˆ S 1 ∪ { j 1 } \ { i 1 } 7: end if 8: end for 9: for each pair ( i 2 , j 2 ) ∈ ˆ S 2 × ([ m ] \ ˆ S 2 ) do 10: if σ max (√ ( B ˆ S 1 ∪{ j 1 }\{ i 1 } , ˆ S 1 ∪{ j 1 }\{ i 1 } ) † A ˆ S 1 ∪{ j 1 }\{ i 1 } , ˆ S 2 √ ( C ˆ S 2 , ˆ S 2 ) † ) > σ max (√ ( B ˆ S 1 , ˆ S 1 ) † A ˆ S 1 , ˆ S 2 √ ( C ˆ S 2 , ˆ S 2 ) † ) then 11: Update ˆ S 2 = ˆ S 2 ∪ { j 2 } \ { i 2 } 12: end if 13: end for 14: while there is still an improvement 15: Output: ˆ S 1 , ˆ S 2

C.1 Valid inequalities for SCCA

Before deriving the formulations, we first prove that there exists a bounded optimal solution ( x ∗ , y ∗ ) of the SCCA. To be specific, we show that there exists an optimal solution ( x ∗ , y ∗ ) of the SCCA satisfying the constraints ∥ x ∗ ∥ 2 2 ≤ M 1 and ∥ y ∗ ∥ 2 2 ≤ M 2 , where M 1 and M 2 are finite-valued parameters.

Proposition 3 The SCCA admits an optimal solution ( x ∗ , y ∗ ) satisfying ∥ x ∗ ∥ 2 2 ≤ M 1 and ∥ y ∗ ∥ 2 2 ≤ M 2 , where M 1 = 1 /λ r ( B ) + 1 / ( λ r ( B ) s min ( B )) and M 2 = 1 /λ ˆ r ( C ) + 1 / ( λ ˆ r ( C ) s min ( C )) with λ r ( B ) , λ ˆ r ( C ) being the smallest nonzero eigenvalues of matrices B , C and s min ( R ) being the smallest nonzero singular value of all the submatrices of the zero eigenvectors of matrix R .

Proof. Let ( x ∗ , y ∗ ) denote an optimal solution to SCCA. We bound ∥ x ∗ ∥ 2 first and the same technique can be also straightforwardly applied to bound ∥ y ∗ ∥ 2 .

For matrix B ∈ S n + of rank r , we let { q i } i ∈ [ n ] ∈ R n denote the eigenvectors corresponding to n eigenvalues λ of B such that λ 1 ≥ . . . ≥ λ r > λ r +1 = . . . = λ n = 0 . Thus, { q i } i ∈ [ n ] are orthonormal and span the space of R n . Hence, there exists α ∈ R n such that x ∗ = ∑ i ∈ [ n ] α i q i . Given that ( x ∗ ) ⊤ Bx ∗ ≤ 1 , we have

Hence, the values of { α i } i ∈ [ r ] are bounded. On the other hand, let us define a subset S ⊆ [ n ] of size at most s 1 such that x ∗ i = 0 for each i ∈ S and x ∗ j = 0 for each j ∈ [ n ] \ S . Then for each j ∈ [ n ] \ S , we arrive at the following linear system:

where ˆ q i denote a subvector of q i with indices [ n ] \ S for each i ∈ [ n ] . For a fixed { α i } i ∈ [ r ] , since the linear system (17) is nonempty, we let ¯ Q ¯ α = ¯ q denote its minimal linear subsystem such that a submatrix ¯ Q is non-singular and the index set ˆ S of ¯ α is a subset of [ n ] \ S . Thus, we can construct an alternative solution ˆ α such that

and ˆ x = ∑ i ∈ [ n ] ˆ α i q i . According to Lemma 1, we have

i.e., (ˆ x , y ∗ ) is also optimal to SCCA. Hence,

̸

Note that ∑ i ∈ [ r ] α 2 i ≤ 1 /λ r and

where s min ( B ) denotes the smallest nonzero singular values of all the submatrices of [ q r +1 , . . . , q n ] . In summary, we have

This completes the proof.

The proof of Proposition 3 is straightforward in the case when B and C are of full rank as in this case the feasible region is a bounded set. In order to prove the result in the case when B is not full-rank, one has to show that it is possible to construct sparse solutions that are not 'too far" away.

In fact, the bounds M 1 , M 2 in Proposition 3 also hold for any given feasible subsets ( S 1 , S 2 ) of SCCA (1).

Corollary 2 For any given feasible subsets ( S 1 , S 2 ) of SCCA 1, there exists a SCCA feasible solution ( x , y ) such that the supports of x , y are S 1 , S 2 , respectively and we have that ∥ x ∥ 2 2 ≤ M 1 and ∥ y ∥ 2 2 ≤ M 2 , where M 1 , M 2 are defined in Proposition 3.

C.2 Equivalent mixed-integer convex quadratic program of rank-one SCCA

When matrix A is rank-one, let us focus on analyzing the subproblem over x in (6), i.e.,

Then the second subproblem over y in (6) simply follows.

According to Corollary 2, introducing the binary variables z 1 ∈ { 0 , 1 } n can reformulate the problem (18) as

When matrix B is positive definite, there is a positive vector b ∈ R n ++ and a positive semidefinite matrix ˆ B such that B = ˆ B +Diag( b ) . Given this equation, by leveraging the perspective techniques (see, e.g., [1, 43]), we can derive another equivalent MICQP formulation of the problem (18):

which is often stronger than the above formulation.

D A branch-and-cut algorithm with closed-form cuts

By dualizing the inner maximization problem over X in the MISDP (7), in this subsection, we derive an equivalent mixed-integer linear program for SCCA, which motivates us to develop a branch-and-cut algorithm.

By introducing the Lagrangian multipliers ( θ 1 , θ 2 , λ ) , the Lagrangian dual of the maximization problem (9) can be written as

where the strong duality holds due to the function f ( z ) being concave, bounded, and thus continuous in the set ˆ Z and Slater condition holds for any interior point z in the set ˆ Z .

Below, we derive the closed-form expression of the function f ( z ) with the given binary variable z ∈ Z . This allows us to reformulate SCCA (8) as a mixed-integer linear program with exponentially many linear constraints and an efficient separation oracle.

Proposition 4 The SCCA (8) is equivalent to

where for a pair of subsets ( S 1 , S 2 ) , the scalar λ ∗ is defined as the largest positive eigenvalue of matrix D ⊤ 2 D -1 1 D 2 -D 3 with

Proof. First, for any binary variable z ∈ Z , suppose S 1 = { i : z i = 1 , ∀ i ∈ [ n ] } , S 2 = { i -n : z i = 1 , ∀ i ∈ [ n + 1 , n + m ] } , and T ⊆ [ n + m ] denotes the support of z . Then following the proof of Proposition 1, we can construct a rank-one optimal solution X ∗ = ( x ∗ y ∗ )( x ∗ y ∗ ) ⊤ to the maximization problem below that admits the optimal value σ max ( √ ( B S 1 ,S 1 ) † A S 1 ,S 2 √ ( C S 2 ,S 2 ) † ) , i.e.,

where the inequality is because the maximization problem above relaxes the valid constraints X ii ≤ M ii for all i ∈ T in maximization problem (9). The result in Corollary 2 suggests that x ∗ , y ∗ can be bounded and their two norms must not exceed M 1 , M 2 , which means that the optimal solution X ∗ satisfies the X ii ≤ M ii for all i ∈ T . Therefore, X ∗ is feasible and optimal to maximization problem (9) and we have that

According to strong duality, the minimization problem (19) admits an optimal value σ max ( √ ( B S 1 ,S 1 ) † A S 1 ,S 2 √ ( C S 2 ,S 2 ) † ) . Next, we construct its optimal solution ( θ ∗ 1 , θ ∗ 2 , λ ∗ ) .

For any given ϵ > 0 , we let θ ∗ 1 = f ( z ) / 2 , θ ∗ 2 = f ( z ) / 2 , ˆ λ i ( ϵ ) = ϵ M ii | T | for all i ∈ T , and ˆ λ i ( ϵ ) = λ ∗ ( ϵ ) for all i ∈ [ n ] \ T , where

It is easy to compute that θ ∗ 1 + θ ∗ 2 + ∑ i ∈ [ n + m ] ˆ λ i ( ϵ ) M ii z i = f ( z ) + ϵ . Thus, for any ϵ > 0 , if ( θ ∗ 1 , θ ∗ 2 , ˆ λ ( ϵ )) were feasible, then it is an ϵ -optimal solution to the minimization problem (19). It remains to verify the feasibility of the solution ( θ ∗ 1 , θ ∗ 2 , ˆ λ ( ϵ )) , i.e., checking the constraint below

By performing the permutation of the rows and columns of the above matrix, it is sufficient to show that the new block matrix

is positive semidefinite.

Since ( B S 1 ,S 1 -A S 1 ,S 2 / 2 -A ⊤ S 1 ,S 2 / 2 C S 2 ,S 2 ) is a principal submatrix of a positive semidefinite matrix ( B -A / 2 -A ⊤ / 2 C ) , it is also positive semidefinite. According to Lemma 1 and the fact that θ ∗ 1 = θ ∗ 2 = σ max ( √ ( B S 1 ,S 1 ) † A S 1 ,S 2 √ ( C S 2 ,S 2 ) † ) / 2 , the matrix D 1 is also positive semidefinite. As ϵ > 0 , the matrix D 1 +Diag ( ˆ λ T ( ϵ ) ) must be positive definite, which means that

Besides, according to the definition of λ ∗ ( ϵ ) , we obtain

Taking these results together, according to Lemma 1, the constraint in (21) must hold for a given solution ( θ ∗ 1 , θ ∗ 2 , ˆ λ ( ϵ )) . Since the objective value corresponding to ( θ ∗ 1 , θ ∗ 2 , ˆ λ ( ϵ )) is at most ϵ larger than the optimal value of problem (19), letting ϵ → 0 and using the closedness of the feasible set in problem (19), we can confirm the optimality of ( θ ∗ 1 , θ ∗ 2 , λ ∗ ) with λ ∗ i = 0 for all i ∈ T and λ ∗ i = λ ∗ for all i ∈ [ n ] \ T .

Given the closed-form optimal solution to problem (19), the rest of the proof follows from [28, theorem 7]. □

We note that SCCA (20) can be implemented via a delayed cut-generation procedure. That is, at each feasible branch-and-bound node with a binary solution ˆ z , let S 1 = { i : ˆ z i = 1 , ∀ i ∈ [ n ] } and S 2 = { i -n : ˆ z i = 1 , ∀ i ∈ [ n +1 , n + m ] } . Then we can compute the corresponding scalar λ ∗ and generate the following valid inequality based on (20):

E Data description

F Multiple Sparse Canonical Correlation Analysis

The multiple CCA problem can be formulated as follows:

where k denotes the number of pairs of basis vectors and I k denotes the identity matrix of size k .

As x , y can be matrices, we propose adding row sparse constraints to extend SCCA for multiple vectors, which is defined as:

where we let ∥ x ∥ 0 and ∥ y ∥ 0 denote the number of nonzero rows of x and y , respectively.

This multiple SCCA model can (i) compute the multiple weight vectors ( x , y ) simultaneously and (ii) enforce the sparsity and orthogonality strictly. To be specific, the constraints x ⊤ Bx = I k , y ⊤ Cy = I k ensure the orthogonal left- and right-canonical loading vectors in multiple SCCA. By the definition of row sparsity, the resultant multiple left- and right-basis vectors, i.e., the columns of x and y , share the same nonzero rows, respectively.

More importantly, the row-sparsity enables us to readily extend the proposed algorithms to solve multiple SCCA. We have tested them on UCI data, and the computational results are presented in Table 8. As k increases, it takes branch-and-cut a longer time to return an optimal solution.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The abstract and/or introduction should clearly state the claims made, including the629 contributions made in the paper and important assumptions and limitation

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The paper discusses the limitations of the proposed algorithms when analyzing the numerical results of each table in Section 5. The paper points out the preconditions to apply Algorithm 1 and to reduce SCCA to MICQPs in Subsection 3.2.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: The proofs can either appear in the main paper or the supplemental material.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Section 5 discusses the generation of experimental results. The supplementary material also contains necessary codes and data used for result reproducibility.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The supplementary material contains necessary codes and data.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We describe the experimental setting in Subsection 5.1.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: The paper focus on solving an optimization problem with deterministic data and evaluates the optimality performance of the algorithms. Hence, the paper does not include error bars or other statistical significance measures of the experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We provide the details on the compute resources in the beginning of Section 5. Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have reviewed the NeurIPS Code of Ethics and make sure that we have conformed with it.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: The paper improves the interpretability of the widely-used CCA method, making it more convincing when applied to real data analytics, e.g., genetic analytics of breast cancer data. We do not foresee any negative societal impact of this work.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We have cited the original paper that produced the code package or dataset.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: The paper does not release new assets

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
comments: SUMMARY:
The paper presents an efficient MISDP algorithm for sparse CCA.

STRENGTHS:
Sparse CCA is an important and interesting problem. The analysis is useful for this area.

WEAKNESSES:
- Typos: optimiality -> optimality line 167
- Limitations are not fully addressed
- No separate evaluation of sparsity in the experiments (see Questions)
- Most experiments have running time of exactly 1 second, it would be clearer to show more decimal digits.

QUESTIONS:
- Your analysis is to compute the first pair of basis vectors, how does your algorithm work to compute multiple pairs?
- Why is the MIPGap always 0 (except when timed out)? Does a dataset exist where it is not zero?
- Related to above, can the authors expand on the limitations of the proposed method?
- In experiments, to evaluate the different algorithms, you use the distance with the optimum. Do you also compare sparsity explicitly?

LIMITATIONS:
See above
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes several algorithms for the Sparse Canonical Correlation Analysis (SCCA) problem.\n\nFirst, the authors present an exact semidefinite programming representation of the classic CCA problem. These results are then used to derive an equivalent combinatorial formulation of the Sparse CCA problem, which can be optimized using greedy and local search algorithms.\n\nFor the case that the sample covariance matrix has low rank, the paper then derives a polynomial time algorithm to solve SCCA to global optimality. Next, the authors show the NP-hardness for the case where the cross-covariance matrix has rank one, and then discuss how the problem can be solved using two Mixed-Integer Convex Quadratic Programs.\n\nThe authors then derive an equivalent mixed integer semidefinite programming reformulation for the SCCA problem, which can be simplified so that it allows them to develop a tailored branch-and-cut algorithm to solve SCCA to global optimality.\n\nFinally, the paper performs numerical evaluation of the proposed algorithms on synthetic data as well as several real-world datasets. It is shown that their approximate algorithms can solve small-to medium scale problems in a few seconds, while achieving the optimal value in most cases. In the rank-one case, they can globally solve problems up to size 200x200 within one hour.\n\nSTRENGTHS:\nThe paper proposed several new algorithms for the sparse CCA problem in different scenarios. The authors prove the equivalence of their reformulations as well as the convergence of their exact algorithm for SCCA in the low rank case in O(n^3+m^3). In the experiments the approach is shown to be applicable to several synthetic and real-world datasets.\n\nWEAKNESSES:\nAn experimental comparison of the approach to other state-of-the-art methods for sparse CCA is missing and should be added to the paper, e.g. the methods introduced by Witten et al. [41] or Parkhomenko et al.[37].\n\nQUESTIONS:\nWhat are the results of the buzz dataset, which is included in the data description in Appendix E but not included in the results in Table 2?\n\nLIMITATIONS:\nAs also discussed by the authors, the proposed algorithms have certain preconditions such as low rank which does not make them applicable to all cases. Moreover, the exact branch-and-cut based algorithms do not scale to large datasets.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-07"}

comments: SUMMARY:
To enhance the interpretability of CCA, the authors explore sparse CCA. It is interesting to note that sparse CCA generalizes sparse PCA, sparse SVD, and sparse regression. The authors derive efficient algorithms to solve sparse CCA and perform the theoretical analysis for sparse CCA. The effectiveness of the proposed method is verified through numerical experiments.

STRENGTHS:
1.The paper is well-written and well-organized.
2.The paper works on sparse CCA and gives a derivation of an equivalent mixed-integer semi-definite programming model. The authors verify the effectiveness of the proposed method via numerical experiments.

WEAKNESSES:
I have several minor concerns and hope to hear from the authors,
1. The work in [10] also addresses the sparsity of CCA. Why not give experimental results on the reference [10] or other sparse methods?  
2. In section 4, the authors give a mixed-integer semidefinite program. In toy examples, it is much better to discuss this model.

QUESTIONS:
Please see the above weaknesses.

LIMITATIONS:
The authors have explained some of the limitations in the article, and others can be seen from weaknesses.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-09

comments: SUMMARY:
This paper presented analyses on sparse canonical correlation analysis (SCCA) with optimization tools, including an SDP relaxation formulation. The authors discussed the theoretical properties and demonstrated numerical experiments.

STRENGTHS:
The paper presented detailed theoretical developments supported by numerical evaluations. The presentation is generally easy to follow.

WEAKNESSES:
1. The current manuscript focused mostly on the mathematical properties of SCCA with optimization tools. Though such analyses are valuable, it would be more helpful to the community if the authors could provide connections to learning practices/applications, i.e., how the algorithms help understand the real datasets.
2. Technically, it would be valuable to have more discussions on the covariance structure of the matrix, instead of treating them as $A, B, C$ blocks. For example, line 136 (observation 1) becomes a trivial fact if we treat CCA as finding the principal angle/vector of two subspaces, so the cosine of the principal angle cannot exceed one.

QUESTIONS:
1. Sec 2.1, the **exact** *semidefinite programming (SDP) reformulation* seems to be in contradiction to the fact that this is a **relaxation** of the original problem.

LIMITATIONS:
None.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-13