SlowGuess/NeurIPS_2023_OCR huggingface dataset:
To solve the optimization problem (9), we introduce a method for finding the descent direction on the joint manifold, as described in Equation (10). In this section, we will provide a detailed explanation of the relevant calculations. We expand the optimization problem into two sub-optimization problems, one for updating the canonical weight  $\mathbf{U}$  and the other for updating the canonical weight  $\mathbf{V}$ . Here, we focus on outlining the process of updating  $\mathbf{U}$ , which is described below.
In the  $t$ -th iteration, the gradient of the multi-objective function (9) with respect to  $\mathbf{U}_t$  is computed as follows:
$$
\nabla_ {\mathbf {U}} f _ {1} \left(\mathbf {U} _ {t}, \mathbf {V} _ {t}\right) = \frac {\partial f _ {1} \left(\mathbf {U} _ {t} , \mathbf {V} _ {t}\right)}{\partial \mathbf {U} _ {t}} = - \mathbf {X} ^ {\top} \mathbf {Y} \mathbf {V} _ {t},
$$
$$
\begin{array}{l} \nabla_ {\mathbf {U}} f _ {j} \left(\mathbf {U} _ {t}, \mathbf {V} _ {t}\right) = \frac {\partial \Delta^ {k , s} \left(\mathbf {U} _ {t} , \mathbf {V} _ {t}\right)}{\partial \mathbf {U} _ {t}} \tag {15a} \\ = \phi^ {\prime} \left(\mathcal {E} ^ {k} (\mathbf {U}, \mathbf {V}) - \mathcal {E} ^ {s} (\mathbf {U}, \mathbf {V})\right) \cdot \left(\frac {\partial \mathcal {E} ^ {k} (\mathbf {U} _ {t} , \mathbf {V} _ {t})}{\partial \mathbf {U} _ {t}} - \frac {\partial \mathcal {E} ^ {s} (\mathbf {U} _ {t} , \mathbf {V} _ {t})}{\partial \mathbf {U} _ {t}}\right), \\ \end{array}
$$
for all  $M \geq j \geq 2$ ,  $k, s \in [K]$ ,  $k \neq s$ , where
$$
\begin{array}{l} \frac {\partial \mathcal {E} ^ {k} \left(\mathbf {U} _ {t} , \mathbf {V} _ {t}\right)}{\partial \mathbf {U} _ {t}} = \operatorname {t r a c e} \left(\mathbf {U} _ {t} ^ {k, \star} ^ {\top} \mathbf {X} ^ {k ^ {\top}} \mathbf {Y} ^ {k} \mathbf {V} _ {t} ^ {k, \star}\right) - \mathbf {X} ^ {k ^ {\top}} \mathbf {Y} ^ {k} \mathbf {V} _ {t}, \tag {15b} \\ \frac {\partial \mathcal {E} ^ {s} (\mathbf {U} _ {t} , \mathbf {V} _ {t})}{\partial \mathbf {U} _ {t}} = \operatorname {t r a c e} \left(\mathbf {U} _ {t} ^ {s, \star^ {\top}} \mathbf {X} ^ {s ^ {\top}} \mathbf {Y} ^ {s} \mathbf {V} _ {t} ^ {s, \star}\right) - \mathbf {X} ^ {s ^ {\top}} \mathbf {Y} ^ {s} \mathbf {V} _ {t}. \\ \end{array}
$$
Similarly, we can compute the gradient of  $f$  w.r.t  $\mathbf{V}_t$ .
Lemma 7, inspired by [23, 6], states that the direction  $\mathbf{P}_t$  can be expressed as a linear combination of gradients  $\{\nabla f_1(\mathbf{U}_t,\mathbf{V}_t),\nabla f_2(\mathbf{U}_t,\mathbf{V}_t),\ldots ,\nabla f_M(\mathbf{U}_t,\mathbf{V}_t)\}$ .
Lemma 7. The unconstrained optimization problem in (10) has a unique solution. Moreover,  $\mathbf{P}_t$  is the solution of the problem in (10) if only if there exist  $\mu_i \geq 0, i \in [M]$ , such that
$$
\mathbf {P} _ {t} = - \sum_ {i \in [ M ]} \mu_ {i} \nabla f _ {i} \left(\mathbf {U} _ {t}, \mathbf {V} _ {t}\right), \quad \sum_ {i \in [ M ]} \mu_ {i} = 1, \quad \mu_ {i} \geq 0, \text {f o r} 1 \leq i \leq M. \tag {16}
$$
Proof. Recall that for each  $(\mathbf{U},\mathbf{V})\in \mathcal{U}\times \mathcal{V}$ , we have  $\mathbf{P} = (\mathbf{P}^{\mathbf{u}},\mathbf{P}^{\mathbf{v}})$  with  $\mathbf{P}^{\mathbf{u}}\in \mathcal{T}_{\mathbf{U}}\mathcal{U}$  and  $\mathbf{P}^{\mathbf{v}}\in \mathcal{T}_{\mathbf{V}}\mathcal{V}$ . It can be easily seen that the first term of  $Q(\mathbf{P})$  (defined in (10)) is a summation of the maximum of linear functions in the linear space  $\mathcal{T}_{\mathbf{U}}\mathcal{U}\times \mathcal{T}_{\mathbf{V}}\mathcal{V}$ . Hence, it is convex, which implies that  $Q(\mathbf{P})$  is strongly convex. Thus, the solution is unique.
From the convexity of the function, it is well known that  $\mathbf{P}_t$  is the solution of (10) if only if
$$
\mathbf {0} \in \partial Q (\mathbf {P}) \mathbf {P} _ {t},
$$
$$
\left. - \mathbf {P} _ {t} \in \partial \left(\max  _ {i \in [ M ]} \operatorname {t r a c e} \left(\mathbf {P} ^ {\top} \nabla f _ {i} (\mathbf {U} _ {t}, \mathbf {V} _ {t})\right)\right) \mathbf {P} _ {t}. \right.
$$
Therefore, the second statement follows from the formula for the subdifferential of the maximum of convex functions.
The (sub)-gradient formula (15) can be substituted into Equation (10), and subsequently, the fminmax function from the MATLAB optimization toolbox can be applied to solve for the descent direction  $\mathbf{P}_t$ . We found that the Goal Attainment Method [24] gives a more stable and accurate solution for our subproblem  $^3$ . Given  $\mathbf{P}_t^{\mathbf{u}}$ ,  $\mathbf{U}_t$  can be updated by:
$$
\mathbf {U} _ {t + 1} = R ^ {\mathbf {u}} \left(\eta_ {t} \mathbf {P} _ {t} ^ {\mathbf {u}}\right), \tag {17}
$$
where  $\eta_t$  is the adaptive stepsize and  $R^{\mathbf{u}}$  is the retraction operator onto  $\mathcal{U} = \{\mathbf{U} \in \mathbb{R}^{D_x \times R} | \mathbf{U}^\top \mathbf{X}^\top \mathbf{X}\mathbf{U} = \mathbf{I}_R\}$ .
Various retraction options are discussed in Sections 3.1 and A.1. Specifically, the generalized polar decomposition method defined in (4) is employed in our numerical experiments described in Section 4. The symmetric nature of the update process for canonical weights  $\mathbf{U}$  and  $\mathbf{V}$  in CCA allows us to employ identical procedures for updating  $\mathbf{V}$ . By substituting  $\mathbf{X}$  with  $\mathbf{Y}$  and  $\mathbf{U}$  with  $\mathbf{V}$  in the aforementioned steps, we obtain the process for updating  $\mathbf{V}$ . In each iteration,  $\mathbf{U}$  and  $\mathbf{V}$  are updated once in sequence.
A.3 Subproblem Solver for SF-CCA
To solve the optimization problem (11), the method of finding the descent direction is introduced in Equation (12). Detailed calculations will be explained below. We expand it into two sub-optimization problems to update the canonical weight  $\mathbf{U}$  and the canonical weight  $\mathbf{V}$ , respectively. First, to update the canonical weight  $\mathbf{U}_t$  in iteration  $t$ , we can take gradient of  $f(\mathbf{U}_t,\mathbf{V}_t)$  with respect  $\mathbf{U}_t$  as follows:
$$
\begin{array}{l} \nabla f \left(\mathbf {U} _ {t}, \mathbf {V} _ {t}\right) = \frac {\partial f \left(\mathbf {U} _ {t} , \mathbf {V} _ {t}\right)}{\partial \mathbf {U} _ {t}} \\ = - \mathbf {X} ^ {\top} \mathbf {Y} \mathbf {V} _ {t} + \lambda \frac {\partial \Delta \left(\mathbf {U} _ {t} , \mathbf {V} _ {t}\right)}{\partial \mathbf {U} _ {t}} (10) \\ = - \mathbf {X} ^ {\top} \mathbf {Y} \mathbf {V} _ {t} (18a) \\ + \lambda \left(\sum_ {k, s \in [ K ], k \neq s} \phi^ {\prime} \left(\mathcal {E} ^ {k} (\mathbf {U}, \mathbf {V}) - \mathcal {E} ^ {s} (\mathbf {U}, \mathbf {V})\right) \cdot \left(\frac {\partial \mathcal {E} ^ {k} \left(\mathbf {U} _ {t} , \mathbf {V} _ {t}\right)}{\partial \mathbf {U} _ {t}} - \frac {\partial \mathcal {E} ^ {s} \left(\mathbf {U} _ {t} , \mathbf {V} _ {t}\right)}{\partial \mathbf {U} _ {t}}\right)\right), \\ \end{array}
$$
$$
\frac {\partial \mathcal {E} ^ {k} \left(\mathbf {U} _ {t} , \mathbf {V} _ {t}\right)}{\partial \mathbf {U} _ {t}} = \operatorname {t r a c e} \left(\mathbf {U} _ {t} ^ {k, \star} ^ {\top} \mathbf {X} ^ {k} ^ {\top} \mathbf {Y} ^ {k} \mathbf {V} _ {t} ^ {k, \star}\right) - \mathbf {X} ^ {k} ^ {\top} \mathbf {Y} ^ {k} \mathbf {V} _ {t}, \tag {18b}
$$
$$
\frac {\partial \mathcal {E} ^ {s} (\mathbf {U} _ {t} , \mathbf {V} _ {t})}{\partial \mathbf {U} _ {t}} = \operatorname {t r a c e} \left(\mathbf {U} _ {t} ^ {s, \star^ {\top}} \mathbf {X} ^ {s ^ {\top}} \mathbf {Y} ^ {s} \mathbf {V} _ {t} ^ {s, \star}\right) - \mathbf {X} ^ {s ^ {\top}} \mathbf {Y} ^ {s} \mathbf {V} _ {t}.
$$
<sup>3</sup>https://ww2.mathworks.cn/help/optim/ug/multiobjective-optimization-algorithms.html
Given the subgradients (18) and the single-objective subproblem (12), the direction  $\mathbf{G}_t = (\mathbf{G}_t^{\mathbf{u}},\mathbf{G}_t^{\mathbf{v}})$  can be effectively computed using subgradient-based methods such as the fmincon function from the MATLAB optimization toolbox. Given  $\mathbf{G}_t^{\mathbf{u}}$ , the update for  $\mathbf{U}_t$  can be obtained as follows:
$$
\mathbf {U} _ {t + 1} = R ^ {\mathbf {u}} \left(\eta_ {t} \mathbf {G} _ {t} ^ {\mathbf {u}}\right). \tag {19}
$$
Here,  $\eta_t$  is the adaptive stepsize and  $R^{\mathbf{u}}$  is the retraction operator onto  $\mathcal{U} = \{\mathbf{U} \in \mathbb{R}^{D_x \times R} | \mathbf{U}^\top \mathbf{X}^\top \mathbf{X}\mathbf{U} = \mathbf{I}_R\}$ .
In our numerical experiments, we utilize polar decomposition as the retraction operator; see (4). Similarly, the canonical weight  $\mathbf{V}$  can be updated using the same procedure.
A.4 Auxiliary Lemmas
Lemma 8. Suppose Assumption A holds. For any  $(\mathbf{U}_t,\mathbf{V}_t)\in \mathcal{U}\times \mathcal{V}$ , let  $\mathbf{P}_t$  be the solution of (10). Then, for any  $\eta_t\geq 0$ , we have the following:
$$
\mathbf {F} (\mathbf {U} _ {t + 1}, \mathbf {V} _ {t + 1}) \preceq \mathbf {F} (\mathbf {U} _ {t}, \mathbf {V} _ {t}) + \eta_ {t} \pmb {\nabla} _ {t} + \eta_ {t} ^ {2} \frac {L _ {F}}{2} \| \mathbf {P} _ {t} \| _ {\mathrm {F}} ^ {2} \mathbf {1} _ {M}, \qquad \mathbf {P} _ {t} \in \mathcal {T} _ {\mathbf {U}} \mathcal {U} \times \mathcal {T} _ {\mathbf {V}} \mathcal {V},
$$
where  $\nabla_{it} \coloneqq \langle \nabla f_i(\mathbf{U}_t, \mathbf{V}_t), \mathbf{P}_t \rangle$  and  $\nabla_t \coloneqq [\nabla_{1t}, \dots, \nabla_{Mt}]^\top \in \mathbb{R}^M$ .
Proof. The proof is a straightforward consequence of Assumption A.  $\square$
Lemma 9. For any  $(\mathbf{U}_t,\mathbf{V}_t)\in \mathcal{U}\times \mathcal{V}$ , let  $\mathbf{P}_t$  be the solution of (10). Then,
$$
\max  _ {i \in [ M ]} \operatorname {t r a c e} \left(\mathbf {P} _ {t} ^ {\top} \nabla f _ {i} \left(\mathbf {U} _ {t}, \mathbf {V} _ {t}\right)\right) = - \| \mathbf {P} _ {t} \| _ {\mathrm {F}} ^ {2}. \tag {20}
$$
Hence, trace  $\left(\mathbf{P}_t^\top \nabla f_i(\mathbf{U}_t,\mathbf{V}_t)\right) \preceq -\|\mathbf{P}_t\|_{\mathrm{F}}^2$ . Further,  $\mathbf{P}_t$  is critical Pareto point of  $\mathbf{F}$  if, and only if,  $\|\mathbf{P}_t\|_{\mathrm{F}} = 0$ .
Proof. It follows from (10) that
$$
- \| \mathbf {P} _ {t} \| _ {\mathrm {F}} ^ {2} = \operatorname {t r a c e} \left(\mathbf {P} _ {t} ^ {\top} \sum_ {i \in [ M ]} \mu_ {i} \nabla f _ {i} (\mathbf {U} _ {t}, \mathbf {V} _ {t})\right) = \sum_ {i \in [ M ]} \mu_ {i} \operatorname {t r a c e} \left(\mathbf {P} _ {t} ^ {\top} \nabla f _ {i} (\mathbf {U} _ {t}, \mathbf {V} _ {t})\right).
$$
Hence, by the second equality in (16), it is easy to verify that (20) holds. The second statement follows by using the definitions of  $\mathrm{trace}(\mathbf{P}_t^\top \nabla F((\mathbf{U}_t,\mathbf{V}_t))$ . We proceed with the proof of the third statement of the lemma. Assuming that  $\mathbf{P}_t$  is a critical Pareto, it follows from the definition that there exists  $i\in [M]$  such that  $\mathrm{trace}(\mathbf{P}_t^\top \nabla f_i(\mathbf{U}_t,\mathbf{V}_t)\geq 0$ . Then, by the first part of the lemma, we have  $\| \mathbf{P}_t\| = 0$ . The converse (only if) follows from the application of [6, Lemma 4.2].
The following result provides an estimate for the decrease of a function  $\mathbf{F}$ , along  $\mathbf{P}$ . This result is crucial in determining the iteration-complexity bounds for the gradient method on a general Riemannian manifold.
Lemma 10 (Descent Property of MF-CCA). Suppose Assumption A holds. For any  $(\mathbf{U}_t,\mathbf{V}_t)\in \mathcal{U}\times \mathcal{V}$  let  $\mathbf{P}_t$  be the solution of (10). Then, for any  $\eta_t\geq 0$ , we have
$$
\mathbf {F} (\mathbf {U} _ {t + 1}, \mathbf {V} _ {t + 1}) \preceq \mathbf {F} (\mathbf {U} _ {t}, \mathbf {V} _ {t}) + \left(\frac {L _ {F} \eta_ {t} ^ {2}}{2} - \eta_ {t}\right) \| \mathbf {P} _ {t} \| _ {\mathrm {F}} ^ {2} \mathbf {1} _ {M}.
$$
Proof. The proof is a straight combination of Lemmas 8 and 9.
A.5 Proof of Theorem 5
Proof. It follows from Lemma 10 that
$$
\left(\frac {L _ {F} \eta_ {t} ^ {2}}{2} - \eta_ {t}\right) \| \mathbf {P} _ {t} \| _ {\mathbf {F}} ^ {2} \mathbf {1} _ {M} \preceq \mathbf {F} (\mathbf {U} _ {t}, \mathbf {V} _ {t}) - \mathbf {F} (\mathbf {U} _ {t + 1}, \mathbf {V} _ {t + 1}),
$$
for all  $t = 0,1,\ldots ,T - 1$
Setting  $\eta_t = \eta$ , and summing both sides of this inequality for  $t = 0,1,\ldots ,T - 1$ , we get
$$
\left(\frac {L _ {F} \eta^ {2}}{2} - \eta\right) \sum_ {t = 0} ^ {T - 1} \| \mathbf {P} _ {t} \| _ {\mathrm {F}} ^ {2} \mathbf {1} _ {M} \preceq \mathbf {F} (\mathbf {U} _ {0}, \mathbf {V} _ {0}) - \mathbf {F} (\mathbf {U} _ {T}, \mathbf {V} _ {T}).
$$
Thus, by the definition of  $i_{*}$ , we obtain
$$
\left(\frac {L _ {F} \eta^ {2}}{2} - \eta\right) T \min  \left\{\| \mathbf {P} _ {t} \| _ {\mathrm {F}} ^ {2}: t = 0, 1, \dots , T - 1 \right\} \leq f _ {i _ {*}} \left(\mathbf {U} _ {0}, \mathbf {V} _ {0}\right) - f _ {i _ {*}} ^ {*},
$$
which together with  $\eta \leq \frac{1}{L_F}$  implies
$$
\min  \left\{\| \mathbf {P} _ {t} \| _ {\mathrm {F}}: t = 0, \dots , T - 1 \right\} \leq \frac {2}{\eta} \left[ \frac {f _ {i _ {*}} \left(\mathbf {U} _ {0} , \mathbf {V} _ {0}\right) - f _ {i _ {*}} ^ {*}}{T} \right] ^ {\frac {1}{2}}.
$$
Remark 11. As a consequence, if we consider a tolerance level of  $\epsilon$ , we can bound the number of iterations required by the gradient method to obtain  $\mathbf{P}_T$  such that  $\| \mathbf{P}_T \|_{\mathrm{F}} \leq \epsilon$ . This bound can be expressed as  $\mathcal{O}\left((f_{i_*}(\mathbf{U}_0, \mathbf{V}_0) - f_{i_*}^*) / \epsilon^2\right)$ , where  $f_{i_*}(\mathbf{U}_0, \mathbf{V}_0) - f_{i_*}^* = \min \{f_i(\mathbf{U}_0, \mathbf{V}_0) - f_i^* : i \in [M]\}$ . In other words, the number of iterations needed by the gradient method to achieve a solution  $(\mathbf{U}_T, \mathbf{V}_T)$  with  $\| \mathbf{P}_T \|_{\mathrm{F}} \leq \epsilon$  is proportional to the difference between the initial objective function value and the optimal objective function value, divided by the square of the tolerance level  $\epsilon$ . The "big O" notation,  $\mathcal{O}$ , indicates that the bound is an upper bound, providing an estimate of the worst-case behavior of the algorithm. This result showcases the convergence rate of the gradient method and provides a useful guideline for choosing the tolerance level and estimating the computational resources required to achieve the desired accuracy.
Proof. This proof is a slightly modified version of the proof of Theorem 5. Using Assumption B and a single-objective variant of Lemma 10, we obtain
$$
\begin{array}{l} f \left(\mathbf {U} _ {t + 1}, \mathbf {V} _ {t + 1}\right) - f \left(\mathbf {U} _ {t}, \mathbf {V} _ {t}\right) \\ \leq \left\langle \nabla f \left(\mathbf {U} _ {t}, \mathbf {V} _ {t}\right), \left(\left(\mathbf {U} _ {t + 1}, \mathbf {V} _ {t + 1}\right) - \left(\mathbf {U} _ {t}, \mathbf {V} _ {t}\right)\right) \right\rangle + \frac {L _ {f}}{2} \| \left(\mathbf {U} _ {t + 1}, \mathbf {V} _ {t + 1}\right) - \left(\mathbf {U} _ {t}, \mathbf {V} _ {t}\right) \| _ {\mathrm {F}} ^ {2} \\ = - \eta \left\| \nabla f \left(\mathbf {U} _ {t}, \mathbf {V} _ {t}\right) \right\| _ {\mathrm {F}} ^ {2} + \frac {L _ {f} \eta^ {2}}{2} \left\| \nabla f \left(\mathbf {U} _ {t}, \mathbf {V} _ {t}\right) \right\| _ {\mathrm {F}} ^ {2} \\ = - \eta \left(1 - \frac {L _ {f} \eta}{2}\right) \| \nabla f (\mathbf {U} _ {t}, \mathbf {V} _ {t}) \| _ {\mathrm {F}} ^ {2}. \\ \end{array}
$$
Summing both sides of this inequality for  $t = 0,1,\ldots ,T - 1$ , we get
$$
\left(\frac {L _ {f} \eta^ {2}}{2} - \eta\right) \sum_ {t = 0} ^ {T - 1} \| \mathbf {G} _ {t} \| _ {\mathrm {F}} ^ {2} \leq f (\mathbf {U} _ {0}, \mathbf {V} _ {0}) - f (\mathbf {U} _ {T}, \mathbf {V} _ {T}).
$$
Hence, using the fixed stepsize  $\eta_t = \eta \leq \frac{1}{L_f}$ , we get
Remark 12. As a result, given a tolerance  $\epsilon$ , the number of iterations required by the gradient method to obtain  $(\mathbf{U}_T,\mathbf{V}_T)$  such that  $\| \mathbf{G}_T\|_{\mathrm{F}}\leq \epsilon$  is bounded by  $\mathcal{O}\left((f(\mathbf{U}_0,\mathbf{V}_0) - f^*) / \epsilon^2\right)$ .
Table 3: Statistic summary of real datasets. In the ADNI database, AV45 is used twice in two different datasets but with a different number of features. In AV45 vs. AV1451, we use the shared ROI (region of interest), thus, the two views have the same number of features. While in AV45 vs Cognitive Score, AV45 contains the total number of features.
Database	Dataset	Number of
Features	Number of
Samples	Sensitive
Attribute	Group
Distribution
NHANES	Phenotypic Measures	96	8843	Education	<High School: 2495 = High School: 2203 >High School: 4145
Environmental Measures	55
NHANES	Phenotypic Measures	96	8136	Race	White: 4576 Black: 1793 Mexican: 1767
Environmental Measures	55
MHAAPS	Psychological Performance	3	600	Sex	Male: 273 Female: 327
Academic Performance	4
ADNI	AV45	52	496	Sex	Male: 241 Female: 255
AV1451	52
ADNI	AV45	68	785	Sex	Male: 431 Female: 354
Cognitive Score	46
B Addendum to Section 4
B.1 Detailed Description of Datasets
Table 3 provides a comprehensive overview of the relevant details associated with each real dataset. These details include the number of samples, the number of features, sensitive attributes, and group distribution. Notably, within the group distribution, it is important to acknowledge that in certain datasets, the total number of samples within different sensitive attributes may not be equivalent. This discrepancy arises due to the presence of missing information, as well as the insufficient sample sizes in certain subgroups. Figure 5 visualizes the distribution of all the groups in each dataset. We can see that the groups are imbalanced in synthetic data, NHANES (Education) and NHANES (Race), while balanced in MHAAPS (Sex), ADNI AV45/AV1451 (Sex), and ADNI AV45/Cognition (Sex). Imbalanced groups can make the problem more complicated, which is a good challenge to validate the effectiveness of our methods.
In the following, we provide comprehensive descriptions of all the datasets we used.
B.1.1 National Health and Nutrition Examination Survey (NHANES)
The NHANES dataset is extensively used for CCA analysis [48] and fairness studies [17, 52]. The dataset's diverse attributes give rise to fairness considerations, underscoring the importance of conducting equitable CCA experiments to tackle discrepancies in sample representation stemming from these inequalities.
In our study, we narrow our focus to the 2005-2006 subset of the NHANES database available at https://www.cdc.gov/nchs/nhanes. This subset comprises physical measurements and self-reported questionnaires from participants. To address missing data concerns, we employ the Multiple Imputation by Chained Equations (MICE) Forest methodology. Afterward, we divide the data into two distinct subsets: the "Phenotypic-Measure" dataset and the "Environmental-Measure" dataset, based on the inherent nature of the features. This stratified approach enables us to leverage F-CCA to gain a nuanced understanding of how phenotypic and environmental factors interact in influencing health outcomes while also accounting for the potential influence of education and race.
In our numerical experiments, we utilize education and race as sensitive attributes to partition the two datasets. In the initial experiment, we split the dataset into three subgroups based on participants' educational attainment. These subgroups comprise 2495, 2203, and 4145 observations, respectively. For the subsequent experiment, we work with a total of 8136 observations, dividing them into three
(b) NHANES (Education)
(c) NHANES (Race)
(d) MHAAPS (Sex)
(e) ADNI AV45/AV1451 (Sex)
(f) ADNI AV45/Cognition (Sex)
Figure 5: Group distributions of the studied datasets.
subgroups based on racial categories. Specifically, there are 4576 white subjects, 1793 black subjects, and 1767 Mexican subjects. The "Phenotypic-Measure" dataset encompasses 96 distinct features, while the "Environmental-Measure" dataset includes 55 features.
B.1.2 Mental Health and Academic Performance Survey (MHAAPS)
The dataset employed in this study is obtained from the online repository available at https: //github.com/marks/convert_to_csv/tree/master/sample_data. This particular dataset includes three distinct psychological variables and four academic variables in the form of standardized test scores, as well as gender information for a cohort of 600 individuals classified as college freshmen. The primary objective of this investigation revolves around examining the interrelationship between the aforementioned psychological variables and academic indicators, with careful consideration given to the potential influence exerted by gender. Specifically, the dataset consists of 327 female samples and 273 male samples.
B.1.3 Alzheimer's Disease Neuroimaging Initiative (ADNI)
We utilize AV45 (amyloid), AV1451 (tau) positron emission tomography (PET), and Cognitive Score data from the Alzheimer's Disease Neuroimaging Initiative (ADNI) database (http://adni.loni.usc.edu) as three of our experiment datasets. Over- or under-representation of age, sex, or biometric groups might affect ADNI data fairness. Disease distribution among sensitivity features, such as higher AD occurrence in women, can also impact fairness [41]. The unfairness in sex representation within the CCA study of the ADNI dataset arises from an imbalance in the number of male and female participants. This disparity undermines the generalizability and validity of the study findings, as it fails to account for potential sex-related differences in Alzheimer's disease progression and response to treatments. The skewed representation limits the ability to draw accurate conclusions
Figure 6: Aggregate disparity of 1st projection dimension on synthetic data comprising varying numbers of subgroups  $(K)$ .
and develop tailored interventions for both sexes, perpetuating sex bias in research and healthcare. Consequently, it becomes imperative to utilize F-CCA as a means to acknowledge and integrate sex disparities within research initiatives.
In the conducted numerical study, two distinct experiments are carried out utilizing the ADNI dataset, where the sensitive attribute considered is the sex of the participants. The first experiment aims to explore the correlation between AV45 and AV1451, whereas the second experiment focuses on investigating the correlation between AV45 and Cognitive Scores. In the first experiment, both AV45 and AV1451 include 52 regional features and comprise a total of 496 observations. Among these observations, 255 belong to the female subgroup, and 241 belong to the male subgroup. In the analysis involving the correlation between AV45 and Cognitive Scores, a subset of 785 common samples is extracted, consisting of 431 male samples and 354 female samples. This analysis employs 68 regional features associated with AV45 and 46 cognitive score features.
B.2 Trade-off Analysis of Correlation and Disparity
B.2.1 Sensitivity of Correlation and Disparity Error to  $K$
We conducted an analysis to examine the sensitivity of fairness in relation to the number of subgroups  $K$ . To investigate this, we utilized synthetic data consisting of varying numbers of subgroups and compared the resulting aggregate disparity on the first projection dimension.
Figure 6 provides a visual representation of our findings. The results clearly demonstrate that as the number of subgroups increases, there is a substantial amplification in the aggregate disparity when employing CCA. However, both MF-CCA and SF-CCA effectively address this issue and mitigate the observed phenomenon. Specifically, they successfully reduce the level of disparity even as the number of subgroups grows.