# A Jacobi-Davidson Method for Large Scale Canonical Correlation Analysis
## 2 Preliminaries  
Throughout this paper, \(\mathbb{R}^{m\times n}\) is the set of all \(m\times n\) real matrices, \(\mathbb{R}^{m}=\mathbb{R}^{m\times 1}\), and \(\mathbb{R}=\mathbb{R}^{1}\). \(I_{n}\) is the \(n\times n\) identity matrix. The superscript \"\(\cdot\),\" takes transpose only, and \(\|\cdot\|_{1}\) denotes the \(\ell_{1}\)-norm of a vector or matrix. For any matrix \(N\in\mathbb{R}^{m\times n}\) with \(m\geq n\), \(\sigma_{i}(N)\) for \(i=1,\ldots,n\) is used to denote the singular values of \(N\) in descending order. For vectors \(x\), \(y\in\mathbb{R}^{n}\), the usual inner product and its induced norm are conveniently defined by  
\[\langle x,y\rangle:=y^{\mathrm{T}}x,\quad\|x\|_{2}:=\sqrt{\langle x,x\rangle}.\]  
With them, the usual acute angle \(\angle(x,y)\) between \(x\) and \(y\) can then be defined by  
\[\angle(x,y):=\arccos\frac{|\langle x,y\rangle|}{\|x\|_{2}\|y\|_{2}}.\]  
Similarly, given any symmetric positive definite \(W\in\mathbb{R}^{n\times n}\), the \(W\)-inner product and its induced \(W\)-norm are defined by  
\[\langle x,y\rangle_{w}:=y^{\mathrm{T}}Wx,\quad\|x\|_{w}:=\sqrt{\langle x,x \rangle}_{w}.\]  
If \(\langle x,y\rangle_{w}=0\), then we say \(x\perp_{w}y\) or \(y\perp_{w}x\). The \(W\)-acute angle \(\angle_{w}(x,y)\) between \(x\) and \(y\) can then be defined by  
\[\angle_{w}(x,y):=\arccos\frac{|\langle x,y\rangle_{w}|}{\|x\|_{w}\|y\|_{w}}.\]  
Let the singular value decomposition of \(\widetilde{C}\) be \(\widetilde{C}=P\Lambda Q^{\mathrm{T}}\) where \(P=[p_{1},\ldots,p_{m}]\in\mathbb{R}^{m\times m}\) and \(Q=[q_{1},\ldots,q_{n}]\in\mathbb{R}^{n\times n}\) are orthonormal, i.e., \(P^{\mathrm{T}}P=I_{m}\) and \(Q^{\mathrm{T}}Q=I_{n}\), and \(\Lambda=\mathrm{diag}(\lambda_{1},\ldots,\lambda_{n})\in\mathbb{R}^{m\times n}\) with \(\lambda_{1}\geq\lambda_{2}\geq\cdots\geq\lambda_{n}\geq 0\) is a leading diagonal matrix. The singular value decomposition of \(\widetilde{C}\) closely relates to the eigendecomposition of the following symmetric matrix [30] (p. 32):  
\[ \begin{bmatrix}0&\widetilde{C}\\ \widetilde{C}^{\mathrm{T}}&0\end{bmatrix} , \tag{8}\]whose eigenvalues are \(\pm\lambda_{i}\) for \(i=1,\ldots,n\) plus \(m-n\) zeros, i.e.,  
\[-\lambda_{1}\leq\cdots\leq-\lambda_{n}\leq\underbrace{0\leq\cdots\leq 0}_{m-n} \leq\lambda_{n}\leq\cdots\leq\lambda_{1}, \tag{9}\]  
with associated eigenvectors are  
\[ \begin{bmatrix}p_{i}\\ \pm q_{i}\end{bmatrix} ,\;i=1,2,\ldots,n,\quad\text{and}\quad \begin{bmatrix}p_ {i}\\ 0\end{bmatrix} ,\;i=n+1,\ldots,m,\]  
respectively. The equivalence between (4) and (6) leads that the eigenvalues of CCGEP (4) are \(\pm\lambda_{i}\) for \(i=1,\ldots,n\) plus \(m-n\) zeros, and the corresponding eigenvectors are  
\[ \begin{bmatrix}x_{i}\\ \pm y_{i}\end{bmatrix} ,\;i=1,2,\ldots,n,\quad\text{and}\quad \begin{bmatrix}x_ {i}\\ 0\end{bmatrix} ,\;i=n+1,\ldots,m,\]  
respectively, where  
\[x_{i}=R_{\Lambda}^{-1}p_{i}\quad\text{and}\quad y_{i}=R_{\Lambda}^{-1}q_{i}. \tag{10}\]  
Let \(X=[x_{1},\ldots,x_{m}]\) and \(Y=[y_{1},\ldots,y_{n}]\). Then, the \(A\)- and \(B\)-orthonormal constraints of \(X\) and \(Y\), respectively, i.e.,  
\[X^{\mathrm{T}}AX=I_{m}\quad\text{and}\quad Y^{\mathrm{T}}BY=I_{n} \tag{11}\]  
are followed by \(P^{\mathrm{T}}P=I_{m}\) and \(Q^{\mathrm{T}}Q=I_{n}\). Here, the first few \(x_{i}\) and \(y_{i}\) for \(i=1,2,\ldots,k\) with \(k<n\) are wanted canonical correlation weight vectors. Furthermore, their corresponding eigenvalues satisfy the following maximization principle which is critical to our later developments. For the proof see Appendix A.1.