SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Zhuoping Zhou $^{\pi}$ , Davoud Ataee Tarzanagh $^{\pi}$ , Bojian Hou $^{\pi}$ , Boning Tong, Jia Xu, Yanbo Feng, Qi Long $^{\sigma}$ , Li Shen $^{\sigma}$
{zhuopinz@sas.,tarzanaq@,boningt@seas.,jiaxu7@,yanbof@seas.}upenn.edu {bojian.hou,qlong,li.shen}@pennmedicine.upenn.edu
This paper investigates fairness and bias in Canonical Correlation Analysis (CCA), a widely used statistical technique for examining the relationship between two sets of variables. We present a framework that alleviates unfairness by minimizing the correlation disparity error associated with protected attributes. Our approach enables CCA to learn global projection matrices from all data points while ensuring that these matrices yield comparable correlation levels to group-specific projection matrices. Experimental evaluation on both synthetic and real-world datasets demonstrates the efficacy of our method in reducing correlation disparity error without compromising CCA accuracy.
Canonical Correlation Analysis (CCA) is a multivariate statistical technique that explores the relationship between two sets of variables [30]. Given two datasets  $\mathbf{X} \in \mathbb{R}^{N \times D_x}$  and  $\mathbf{Y} \in \mathbb{R}^{N \times D_y}$  on the same set of  $N$  observations, CCA seeks the  $R$ -dimensional subspaces where the projections of  $\mathbf{X}$  and  $\mathbf{Y}$  are maximally correlated, i.e. finds  $\mathbf{U} \in \mathbb{R}^{D_x \times R}$  and  $\mathbf{V} \in \mathbb{R}^{D_y \times R}$  such that
maximize trace  $\left(\mathbf{U}^{\top}\mathbf{X}^{\top}\mathbf{Y}\mathbf{V}\right)$  subject to  $\mathbf{U}^{\top}\mathbf{X}^{\top}\mathbf{X}\mathbf{U} = \mathbf{V}^{\top}\mathbf{Y}^{\top}\mathbf{Y}\mathbf{V} = \mathbf{I}_R$ . (CCA)
CCA finds applications in various fields, including biology [51], neuroscience [2], medicine [79], and engineering [14], for unsupervised or semi-supervised learning. It improves tasks like clustering, classification, and manifold learning by creating meaningful dimensionality-reduced representations [70]. However, CCA can exhibit unfair behavior when analyzing data with protected attributes, like sex or race. For instance, in Alzheimer's disease (AD) analysis, CCA can establish correlations between brain imaging and cognitive decline. Yet, if it does not consider the influence of sex, it may result in disparate correlations among different groups because AD affects males and females differently, particularly in cognitive decline [36, 81].
The influence of machine learning on individuals and society has sparked a growing interest in the topic of fairness [42]. While fairness techniques are well-studied in supervised learning [5, 18, 20], attention is shifting to equitable methods in unsupervised learning [11, 12, 15, 34, 49, 55, 64]. Despite extensive work on fairness in machine learning, fair CCA (F-CCA) remains unexplored. This paper investigates F-CCA and introduces new approaches to mitigate bias in (CCA).
For further discussion, we compare CCA with our proposed F-CCA in sample projection, as illustrated in Figure 1. In Figure 1(a), we have samples  $\mathbf{x}_1$  and  $\mathbf{x}_2$  from matrix  $\mathbf{X}$ , and in Figure 1(b), their corresponding samples  $\mathbf{y}_1$  and  $\mathbf{y}_2$  are from matrix  $\mathbf{Y}$ . CCA learns  $\mathbf{U}$  and  $\mathbf{V}$  to maximize correlation,
$\pi$  Equal contribution  $\sigma$  Corresponding authors
1 The columns of  $\mathbf{X}$  and  $\mathbf{Y}$  have been standardized.
Figure 1: Illustration of CCA and F-CCA, with the sensitive attribute being sex (female and male). Figures (a)-(c) demonstrate the general framework of CCA, while Figures (d)-(i) provide a comparison of the projected results using various strategies. It is important to note that the correlation between two corresponding samples is inversely associated with the angle formed by their projected vectors. F-CCA aims to equalize the average angles among different groups.
inversely related to the angle between the sample vectors. Figure 1(c) demonstrates the proximity within the projected sample pairs  $(\mathbf{U}^{\top}\mathbf{x}_1,\mathbf{V}^{\top}\mathbf{y}_1)$  and  $(\mathbf{U}^{\top}\mathbf{x}_2,\mathbf{V}^{\top}\mathbf{y}_2)$ . In Figure 1(d)-(i), we compare the results of different learning strategies. There are five pairs of samples, with female pairs highlighted in red and male pairs shown in blue. Random projection (Figure 1(e)) leads to randomly large angles between corresponding sample vectors. CCA reduces angles compared to random projection (Figure 1(f)), but significant angle differences between male and female pairs indicate bias. Using sex-based projection matrices heavily biases the final projection, favoring one sex over the other (Figures 1(g) and 1(h)). To address this bias, our F-CCA maximizes correlation within pairs and ensures equal correlations across different groups, such as males and females (Figure 1(i)). Note that while this illustration represents individual fairness, the desired outcome in practice is achieving similar average angles for different groups.
Contributions. This paper makes the following key contributions:
- We introduce fair CCA (F-CCA), a model that addresses fairness issues in (CCA) by considering multiple groups and minimizing the correlation disparity error of protected attributes. F-CCA aims to learn global projection matrices from all data points while ensuring that these projection matrices produce a similar amount of correlation as group-specific projection matrices.
- We propose two optimization frameworks for F-CCA: multi-objective and single-objective. The multi-objective framework provides an automatic trade-off between global correlation and equality in group-specific correlation disparity errors. The single-objective framework offers a simple approach to approximate fairness in CCA while maintaining a strong global correlation, requiring a tuning parameter to balance these objectives.
- We develop a gradient descent algorithm on the generalized Stiefel manifold to solve the multi-objective problem, with convergence guarantees to a Pareto stationary point. This approach extends Riemannian gradient descent [8, 9] to multi-objective optimization, accommodating a broader range of retraction maps than exponential retraction [23, 6]. Furthermore, we provide a similar algorithm for single-objective problems, also with convergence guarantees to a stationary point.
- We provide extensive empirical results showcasing the efficacy of the proposed algorithms. Comparison against the CCA method on synthetic and real datasets highlights the benefits of the F-CCA approach, validating the theoretical findings<sup>2</sup>.
Organization: Section 2 covers related work. Our proposed approach is detailed in Section 3, along with its theoretical guarantees. Section 4 showcases numerical experiments, while Section 5 discusses implications and future research directions.
$^{2}$ Code is available at https://github.com/PennShenLab/Fair_CCA.
Canonical Correlation Analysis (CCA). CCA was first introduced by [28, 29]. Since then, it has been utilized to explore relations between variables in various fields of science, including economics [72], psychology [19, 27], geography [45], medicine [39], physics [76], chemistry [69], biology [62], time-series modeling [26], and signal processing [57]. Recently, CCA has demonstrated its applicability in modern fields of science such as neuroscience, machine learning, and bioinformatics [59, 60]. CCA has been used to explore relations for developing brain-computer interfaces [10, 46] and in the field of imaging genetics [22]. CCA has also been applied for feature selection [47], feature extraction and fusion [61], and dimension reduction [71]. Additionally, numerous studies have applied CCA in bioinformatics and computational biology, such as [54, 56, 58]. The broad range of application domains highlights the versatility of CCA in extracting relations between variables, making it a valuable tool in scientific research.
Fairness. Fairness in machine learning has been a growing area of research, with much of the work focusing on fair supervised methods [5, 16, 18, 20, 67, 78]. However, there has also been increasing attention on fair methods for unsupervised learning tasks [11, 12, 15, 34, 33, 49, 55, 64, 50, 66]. In particular, Samadi et al. [55] proposed a semi-definite programming approach to ensure fairness in PCA. Kleindessner et al. [33, 34] focused on fair PCA formulation for multiple groups and proposed a kernel-based fair PCA. Kamani et al. [32] introduced an efficient gradient method for fair PCA, addressing multi-objective optimization. In this paper, we propose a novel multi-objective framework for F-CCA, converting constrained F-CCA problems to unconstrained ones on a generalized Riemannian manifold. This framework enables the adaptation of efficient gradient techniques for numerical optimization on Riemannian manifolds.
Riemannian Optimization. Riemannian optimization extends Euclidean optimization to smooth manifolds, enabling the minimization of  $f(\mathbf{x})$  on a Riemannian manifold  $\mathcal{M}$  and converting constrained problems into unconstrained ones [1, 8]. It finds applications in various domains such as matrix/tensor factorization [31, 63], PCA [21], and CCA [77]. Specifically, CCA can be formulated as Riemannian optimization on the Stiefel manifold [13, 43]. In our work, we utilize Riemannian optimization to develop a multi-objective framework for F-CCAs on generalized Stiefel manifolds.
3 Fair Canonical Correlation Analysis
This section introduces the formulation and optimization algorithms for F-CCA.
Real numbers are represented as  $\mathbb{R}$ , with  $\mathbb{R}_+$  for nonnegative values and  $\mathbb{R}_{++}$  for positives. Vectors and matrices use bold lowercase and uppercase letters (e.g.,  $\mathbf{a}$ ,  $\mathbf{A}$ ) with elements  $a_i$  and  $a_{ij}$ . For  $\mathbf{x}, \mathbf{y} \in \mathbb{R}^m$ ,  $\mathbf{x} \prec \mathbf{y}$  and  $\mathbf{x} \preceq \mathbf{y}$  mean  $\mathbf{y} - \mathbf{x} \in \mathbb{R}_{++}^m$  and  $\mathbf{y} - \mathbf{x} \in \mathbb{R}_+^m$ , respectively. For a symmetric matrix  $\mathbf{A} \in \mathbb{R}^{N \times N}$ ,  $\mathbf{A} \succ 0$  and  $\mathbf{A} \succeq 0$  denote positive definiteness and positive semidefiniteness (PSD), respectively.  $\mathbf{I}_D$ ,  $\mathbf{J}_D$ , and  $\mathbf{0}_D$  are  $D \times D$  identity, all-ones, and all-zeros matrices.  $\Lambda_i(\mathbf{A})$  stands for the  $i$ -th singular values of  $\mathbf{A}$ . Matrix norms are defined as  $\| \mathbf{A} \|_1 = \sum_{ij} |a_{ij}|$ ,  $\| \mathbf{A} \| = \max_i \Lambda_i(\mathbf{A})$ , and  $\| \mathbf{A} \|_F := (\sum_{ij} |a_{ij}|^2)^{1/2}$ . We introduce some preliminaries on manifold optimization [1, 6, 8]. Given a PSD matrix  $\mathbf{B} \in \mathbb{R}^{D \times D}$ , the generalized Stiefel manifold is defined as
$$
\mathbf {S t} (D, R, \mathbf {B}) = \left\{\mathbf {Z} \in \mathbb {R} ^ {D \times R} \mid \mathbf {Z} ^ {\top} \mathbf {B} \mathbf {Z} = \mathbf {I} _ {R} \right\}. \tag {1}
$$
The tangent space of the manifold  $\mathcal{M} = \mathsf{St}(D,R,\mathbf{B})$  at  $\mathbf{Z}\in \mathcal{M}$  is given by
$$
\mathcal {T} _ {\mathbf {Z}} \mathcal {M} = \left\{\mathbf {W} \in \mathbb {R} ^ {D \times R} \mid \mathbf {Z} ^ {\top} \mathbf {B} \mathbf {W} + \mathbf {W} ^ {\top} \mathbf {B} \mathbf {Z} = \mathbf {0} _ {R} \right\}. \tag {2}
$$
The tangent bundle of a smooth manifold  $\mathcal{M}$ , which consists of  $\mathcal{T}_{\mathbf{Z}}\mathcal{M}$  at all  $\mathbf{Z} \in \mathcal{M}$ , is defined as
$$
\mathcal {T M} = \left\{\left(\mathbf {Z}, \mathbf {W}\right) \mid \mathbf {Z} \in \mathcal {M}, \mathbf {W} \in \mathcal {T} _ {\mathbf {Z}} \mathcal {M} \right\}. \tag {3}
$$
Definition 1. A retraction on a differentiable manifold  $\mathcal{M}$  is a smooth mapping from its tangent bundle  $\mathcal{T}\mathcal{M}$  to  $\mathcal{M}$  that satisfies the following conditions, with  $R^{\mathbf{z}}$  being the retraction of  $R$  to  $\mathcal{T}_{\mathbf{Z}}\mathcal{M}$ :
1.  $R^{\mathbf{z}}(\mathbf{0}) = \mathbf{Z}$ , for all  $\mathbf{Z} \in \mathcal{M}$ , where  $\mathbf{0}$  denotes the zero element of  $\mathcal{T}_{\mathbf{Z}}\mathcal{M}$ .
2. For any  $\mathbf{Z} \in \mathcal{M}$ , it holds that  $\lim_{\mathcal{T}_{\mathbf{Z}}\mathcal{M} \ni \pmb{\xi} \to 0} \frac{\|R^{\mathbf{z}}(\pmb{\xi}) - (\mathbf{Z} + \pmb{\xi})\|_F}{\|\pmb{\xi}\|_F} = 0$ .
In the numerical experiments, this work employs a generalized polar decomposition-based retraction. Given a PSD matrix  $\mathbf{B} \in \mathbb{R}^{D \times D}$ , for any  $\boldsymbol{\xi} \in \mathcal{T}_{\mathbf{Z}}\mathcal{M}$  with  $\mathcal{M} = \mathrm{St}(D,R,\mathbf{B})$ , it is defined as:
$$
R ^ {\mathbf {z}} (\boldsymbol {\xi}) = \bar {\mathbf {U}} \left(\mathbf {Q} \boldsymbol {\Lambda} ^ {- \frac {1}{2}} \mathbf {Q} ^ {\top}\right) \bar {\mathbf {V}} ^ {\top}, \tag {4}
$$
where  $\bar{\mathbf{U}}\Sigma \bar{\mathbf{V}}^{\top} = \pmb{\xi}$  is the singular value decomposition of  $\pmb{\xi}$ , and  $\mathbf{Q},\Lambda$  are obtained from the eigenvalue decomposition  $\mathbf{Q}\Lambda \mathbf{Q}^{\top} = \bar{\mathbf{U}}^{\top}\mathbf{B}\bar{\mathbf{U}}$ . Further details on retraction choices are in Appendix A.1.
3.2 Correlation Disparity Error
As previously mentioned, applying CCA to the entire dataset could lead to a biased result, as some groups might dominate the analysis while others are overlooked. To avoid this, we can perform CCA separately on each group and compare the results. Indeed, we can compare the performance of CCA on each group's data with the performance of CCA on the whole dataset, which includes all groups' data. The goal is to find a balance between the benefits and sacrifices of different groups so that each group's contribution to the CCA analysis is treated fairly. In particular, suppose the datasets  $\mathbf{X} \in \mathbb{R}^{N \times D_x}$  and  $\mathbf{Y} \in \mathbb{R}^{N \times D_y}$  on the same set of  $N$  observations, belong to  $K$  different groups  $\{(\mathbf{X}^k, \mathbf{Y}^k)\}_{k=1}^K$  with  $\mathbf{X}^k \in \mathbb{R}^{N_k \times D_x}$  and  $\mathbf{Y}^k \in \mathbb{R}^{N_k \times D_y}$ , based on demographics or some other semantically meaningful clustering. These groups need not be mutually exclusive; each group can be defined as a different weighting of the data.
To determine how each group is affected by F-CCA, we can compare the structure learned from each group's data  $(\mathbf{X}^k,\mathbf{Y}^k)$  with the structure learned from all groups' data combined  $(\mathbf{X},\mathbf{Y})$ . A fair CCA approach seeks to balance the benefits and drawbacks of each group's contribution to the analysis. Specifically, if we train global subspaces  $\mathbf{U}\in \mathbb{R}^{D_x\times R}$  and  $\mathbf{V}\in \mathbb{R}^{D_y\times R}$  on  $k$  -th group dataset  $(\mathbf{X}^k,\mathbf{Y}^k)$ , we can identify the group-specific (local) weights represented by  $(\mathbf{U}^k,\mathbf{V}^k)$  that has the best performance on that dataset. Thus, F-CCA algorithm should be able to learn global weights  $(\mathbf{U},\mathbf{V})$  on all data points while ensuring that each group's correlation on the CCA learned by the whole dataset is equivalent to the group-specific subspaces learned only by its own data.
To define these fairness criteria, we introduce correlation disparity error as follows:
Definition 2 (Correlation Disparity Error). Consider a pair of datasets  $(\mathbf{X},\mathbf{Y})$  with  $K$  sensitive groups with data matrix  $\{(\mathbf{X}^k,\mathbf{Y}^k)\}_{k = 1}^K$  representing each sensitive group's data samples. Then, for any  $(\mathbf{U},\mathbf{V})$ , the correlation disparity error for each sensitive group  $k\in [K]$  is defined as
$$
\mathcal {E} ^ {k} (\mathbf {U}, \mathbf {V}) := \operatorname {t r a c e} \left(\mathbf {U} ^ {k, \star} ^ {\top} \mathbf {X} ^ {k} ^ {\top} \mathbf {Y} ^ {k} \mathbf {V} ^ {k, \star}\right) - \operatorname {t r a c e} \left(\mathbf {U} ^ {\top} \mathbf {X} ^ {k} ^ {\top} \mathbf {Y} ^ {k} \mathbf {V}\right), \quad 1 \leq k \leq K. \tag {5}
$$
Here,  $(\mathbf{U}^{k,\star},\mathbf{V}^{k,\star})$  is the maximizer of the following group-specific CCA problem:
$$
\text {m a x i m i z e} \quad \operatorname {t r a c e} \left(\mathbf {U} ^ {k ^ {\top}} \mathbf {X} ^ {k ^ {\top}} \mathbf {Y} ^ {k} \mathbf {V} ^ {k}\right) \quad \text {s u b j . t o} \quad \mathbf {U} ^ {k ^ {\top}} \mathbf {X} ^ {k ^ {\top}} \mathbf {X} ^ {k} \mathbf {U} ^ {k} = \mathbf {V} ^ {k ^ {\top}} \mathbf {Y} ^ {k ^ {\top}} \mathbf {Y} ^ {k} \mathbf {V} ^ {k} = \mathbf {I} _ {R}. \tag {6}
$$
This measure shows how much correlation we are suffering for any global  $(\mathbf{U},\mathbf{V})$ , with respect to the loss of optimal local  $(\mathbf{U}^{k,\star},\mathbf{V}^{k,\star})$  that we can learn based on data points  $(\mathbf{X}^k,\mathbf{Y}^k)$ .
Using Definition 2, we can define F-CCA as follows:
Definition 3 (Fair CCA). A CCA pair  $(\mathbf{U}^{\star},\mathbf{V}^{\star})$  is called fair if the correlation disparity error among  $K$  different groups is equal, i.e.,
$$
\mathcal {E} ^ {k} \left(\mathbf {U} ^ {\star}, \mathbf {V} ^ {\star}\right) = \mathcal {E} ^ {s} \left(\mathbf {U} ^ {\star}, \mathbf {V} ^ {\star}\right), \quad \forall k \neq s, \quad k, s \in [ K ]. \tag {7}
$$
A CCA pair  $(\mathbf{U}^{\star},\mathbf{V}^{\star})$  that achieves the same disparity error for all groups is called a fair CCA.
Next, we introduce the concept of pairwise correlation disparity error for CCA, which measures the variation in correlation disparity among different groups.
Definition 4 (Pairwise Correlation Disparity Error). The pairwise correlation disparity error for any global  $(\mathbf{U},\mathbf{V})$  and group-specific subspaces  $\{(\mathbf{U}^{k,\star},\mathbf{V}^{k,\star})\}_{k = 1}^{K}$ , is defined as
$$
\Delta^ {k, s} (\mathbf {U}, \mathbf {V}) := \phi \left(\mathcal {E} ^ {k} (\mathbf {U}, \mathbf {V}) - \mathcal {E} ^ {s} (\mathbf {U}, \mathbf {V})\right), \quad \forall k \neq s, \quad k, s \in [ K ]. \tag {8}
$$
Here,  $\phi :\mathbb{R}\to \mathbb{R}_+$  is a penalty function such as  $\phi (x) = \exp (x)$ ,  $\phi (x) = x^{2}$ , or  $\phi (x) = |x|$ .
Algorithm 1: A Multi-Objective Gradient Method for F-CCA (MF-CCA)
1:	Input: (X, Y), (U0, V0), (R, T) ∈ N × N; and stepsizes { (ηt u, ηt v)T}k=1;
2:	Find the R-rank subspaces for each group, { (Uk*, Vk*)K}k=1 using (6).
3:	for t = 0, ..., T-1 do
4:	Find (Ptu, Ptv) by solving (10).
5:	Ut+1← Ru (ηt uPtu)
6:	Vt+1← Rv (ηt vPv)
7:	end for
8:	Output: (UT, VT), { (Uk*, Vk*)}k=1.
Algorithm 2: A Single-Objective Gradient Method for F-CCA (SF-CCA)
1:	Input: (X, Y), (U0, V0), (R, T) ∈ N × N; λ ∈ R++; and stepsizes { (ηt u, ηt v)Tt=1;
2:	Find the R-rank subspaces for each group, {(Uk*, Vk*)}Kk=1 using (6).
3:	for t = 0, ..., T-1 do
4:	Find (Gut, Gv) by solving (12).
5:	Ut+1← Ru(ηt uGt u)
6:	Vt+1← Rv(ηt vGt v)
7:	end for
8:	Output: (UT, VT), {(Uk*, Vk*)}Kk=1.
The motivation for incorporating pairwise correlation disparity error in our approach can be attributed to the work by [40, 55] in the context of PCA. To facilitate convergence analysis, we will primarily consider smooth penalization functions, such as squared or exponential penalties.
3.3 A Multi-Objective Framework for Fair CCA
In this section, we introduce an optimization framework for balancing correlation and disparity errors. Let  $f_{1}(\mathbf{U},\mathbf{V}) \coloneqq -\mathrm{trace}\left(\mathbf{U}^{\top}\mathbf{X}^{\top}\mathbf{Y}\mathbf{V}\right)$ ,  $f_{2}(\mathbf{U},\mathbf{V}) \coloneqq \Delta^{1,2}(\mathbf{U},\mathbf{V}),\ldots ,f_{M}(\mathbf{U},\mathbf{V}) \coloneqq \Delta^{K - 1,K}(\mathbf{U},\mathbf{V})$ . The optimization problem of finding an optimal Pareto point of  $\mathbf{F}$  is denoted by
$$
\underset {\mathbf {U}, \mathbf {V}} {\text {m i n i m i z e}} \quad \mathbf {F} (\mathbf {U}, \mathbf {V}) := \left[ f _ {1} (\mathbf {U}, \mathbf {V}), f _ {2} (\mathbf {U}, \mathbf {V}), \dots , f _ {M} (\mathbf {U}, \mathbf {V}) \right], \tag {9}
$$
$$
\text {s u b j . t o} \quad \mathbf {U} \in \mathcal {U}, \quad \mathbf {V} \in \mathcal {V},
$$
where  $\mathcal{U} \coloneqq \{\mathbf{U} \in \mathbb{R}^{D_x \times R} | \mathbf{U}^\top \mathbf{X}^\top \mathbf{X}\mathbf{U} = \mathbf{I}_R\}$  and  $\mathcal{V} \coloneqq \{\mathbf{V} \in \mathbb{R}^{D_y \times R} | \mathbf{V}^\top \mathbf{Y}^\top \mathbf{Y}\mathbf{V} = \mathbf{I}_R\}$ .
A point  $(\mathbf{U},\mathbf{V})\in \mathcal{U}\times \mathcal{V}$  satisfying  $\operatorname {Im}(\nabla \mathbf{F}(\mathbf{U},\mathbf{V}))\cap (-\mathbb{R}_{+ + }^{M}) = \emptyset$  is called critical Pareto. Here,  $\operatorname {Im}$  denotes the image of Jacobian of  $\mathbf{F}$ . An optimum Pareto point of  $\mathbf{F}$  is a point  $(\mathbf{U}^{\star},\mathbf{V}^{\star})\in \mathcal{U}\times \mathcal{V}$  such that there exists no other  $(\mathbf{U},\mathbf{V})\in \mathcal{U}\times \mathcal{V}$  with  $\mathbf{F}(\mathbf{U},\mathbf{V})\prec \mathbf{F}(\mathbf{U}^{\star},\mathbf{V}^{\star})$ . Moreover, a point  $\mathbf{U}^{\star},\mathbf{V}^{\star}\in \mathcal{U}\times \mathcal{V}$  is a weak optimal Pareto of  $\mathbf{F}$  if there is no  $(\mathbf{U},\mathbf{V})\in \mathcal{U}\times \mathcal{V}$  with  $\mathbf{F}(\mathbf{U},\mathbf{V})\preceq \mathbf{F}(\mathbf{U}^{\star},\mathbf{V}^{\star})$ . The multi-objective framework (9) addresses the challenge of handling conflicting objectives and achieving optimal trade-offs between them.
To effectively solve Problem (9), we propose utilizing a gradient descent method on the manifold  $\mathcal{U} \times \mathcal{V}$  that ensures convergence to a Pareto stationary point. The proposed gradient descent algorithm for solving (9) is provided in Algorithm 1. For each  $(\mathbf{U}, \mathbf{V}) \in \mathcal{U} \times \mathcal{V}$ , let  $\mathbf{P} := (\mathbf{P}^{\mathbf{u}}, \mathbf{P}^{\mathbf{v}})$  with  $\mathbf{P}^{\mathbf{u}} \in \mathcal{T}_{\mathbf{U}}\mathcal{U}$  and  $\mathbf{P}^{\mathbf{v}} \in \mathcal{T}_{\mathbf{V}}\mathcal{V}$ . The iterates  $(\mathbf{P}_t^{\mathbf{u}}, \mathbf{P}_t^{\mathbf{v}})$  in Step 4 are obtained by solving the following subproblem in the joint tangent plane  $\mathcal{T}_{\mathbf{U}}\mathcal{U} \times \mathcal{T}_{\mathbf{V}}\mathcal{V}$ :
$$
\min  _ {\mathbf {P} \in \mathcal {T} _ {\mathbf {U}} \mathcal {U} \times \mathcal {T} _ {\mathbf {V}} \mathcal {V}} Q _ {t} (\mathbf {P}), \text {w h e r e} Q _ {t} (\mathbf {P}) := \left\{\max  _ {i \in [ M ]} \operatorname {t r a c e} \left(\mathbf {P} ^ {\top} \nabla f _ {i} ((\mathbf {U} _ {t}, \mathbf {V} _ {t}))\right) + \frac {1}{2} \| \mathbf {P} \| _ {\mathrm {F}} ^ {2} \right\}. \tag {10}
$$
If  $(\mathbf{U}_t,\mathbf{V}_t)\notin \mathcal{U}\times \mathcal{V}$  is not a Pareto stationary point, Problem (10) has a unique nonzero solution  $\mathbf{P}_t$  (see Lemma 7), known as the steepest descent direction for  $\mathbf{F}$  at  $(\mathbf{U}_t,\mathbf{V}_t)$ . In Steps 5 and 6,  $R^{\mathbf{u}}$  and  $R^{\mathbf{v}}$  denote the retractions onto the tangent spaces  $\mathcal{T}_{\mathbf{U}}\mathcal{U}$  and  $\mathcal{T}_{\mathbf{V}}\mathcal{V}$ , respectively; refer to Definition 1.
Assumption A. For a given subset  $\mathcal{S}$  of the tangent bundle  $\mathcal{T}\mathcal{U} \times \mathcal{T}\mathcal{V}$ , there exists a constant  $L_{F}$  such that, for all  $(\mathbf{Z},\mathbf{P}) \in \mathcal{S}$ , we have  $\mathbf{F}(R^{\mathbf{z}}(\mathbf{P})) \preceq \mathbf{F}(\mathbf{Z}) + \nabla + (L_{F}/2)\|\mathbf{P}\|_{\mathrm{F}}^{2}\mathbf{1}_{M}$ , where  $\nabla_{i} := \langle \nabla f_{i}(\mathbf{Z}),\mathbf{P}\rangle$ ,  $\nabla := [\nabla_{1},\dots,\nabla_{M}]^{\top} \in \mathbb{R}^{M}$ , and  $R^{\mathbf{z}}$  is the retraction.
The above assumption extends [8, A 4.3] to multi-objective optimization, and it always holds for the exponential map (exponential retraction) if the gradient of  $\mathbf{F}$  is  $L_{F}$ -Lipschitz continuous [23, 6].