SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Junghyun Lee* Hanseul Cho* Se-Young Yun Chulhee Yun Kim Jaechul Graduate School of AI, KAIST
{jh_lee00, jhs4015, yunseyoung, chulhee.yun}@kaist.ac.kr
Fair Principal Component Analysis (PCA) is a problem setting where we aim to perform PCA while making the resulting representation fair in that the projected distributions, conditional on the sensitive attributes, match one another. However, existing approaches to fair PCA have two main problems: theoretically, there has been no statistical foundation of fair PCA in terms of learnability; practically, limited memory prevents us from using existing approaches, as they explicitly rely on full access to the entire data. On the theoretical side, we rigorously formulate fair PCA using a new notion called probably approximately fair and optimal (PAFO) learnability. On the practical side, motivated by recent advances in streaming algorithms for addressing memory limitation, we propose a new setting called fair streaming PCA along with a memory-efficient algorithm, fair noisy power method (FNPM). We then provide its statistical guarantee in terms of PAFO-learnability, which is the first of its kind in fair PCA literature. Lastly, we verify the efficacy and memory efficiency of our algorithm on real-world datasets.
Algorithmic fairness ensures that machine learning algorithms do not propagate nor exacerbate bias, which may lead to discriminatory decision-making (Barocas and Selfst, 2016) and thus has been a very active area of research. This has direct implications in our everyday life, including but not limited to criminal justice (Kirchner et al., 2016), education (Kizilcec and Lee, 2021), and more. See Mehrabi et al. (2021) for a comprehensive survey of bias and fairness in machine learning.
Often, one needs to consider fairness for a large number of high-dimensional data points. One of the standard tools for dealing with such high-dimensional data is PCA (Hotelling, 1933; Pearson, 1901), a classical yet still one of the most popular algorithms for performing interpretable dimensionality reduction. It has been adapted as a baseline and/or standard tool in exploratory data analysis, whose application ranges from natural sciences, engineering (Abdi and Williams, 2010; Jolliffe and Cadima, 2016), and even explainable AI (Li et al., 2023; Tjoa and Guan, 2021). Due to its ubiquity and wide applicability, several works study defining fairness in PCA and developing a fair variant of it. A recent line of research (Kleindessner et al., 2023; Lee et al., 2022; Olfat and Aswani, 2019) defines PCA fairness in the context of fair representation (Zemel et al., 2013) in that the projected group conditional distributions should match.
However, existing fair PCA approaches suffer from two problems. Theoretically, they provide no statistical foundation of fair PCA or guarantees for their algorithms. By statistical foundation, we mean the usual PAC-learnability (Shalev-Schwartz and Ben-David, 2014) guarantees, e.g., sample complexity for ensuring optimality in explained variance and fairness constraint with high probability.
On top of that, the second problem arises from a practical viewpoint: memory limitation. All the aforementioned fair PCA algorithms assume that the learner can store the entire data points and incurs memory complexity of order at least  $\mathcal{O}(d\max (N,d))$ , where  $d$  is the dimensionality of the data and  $N$  is the number of data points. As memory limitation is often a critical bottleneck in deploying machine learning algorithms (Mitliagkas et al., 2013), as much as fairness is important, it is also paramount that imposing fairness to PCA does not incur too much memory overhead. A popular approach to mitigate such memory limitation for PCA is to consider the one-pass, streaming setting. In this setting, each data point is revealed to the learner sequentially, each point is irretrievably gone unless she explicitly stores it, and she can use only  $\mathcal{O}(dk)$  memory, with  $k$  being the target dimension of projection. Indeed, without the fairness constraint, streaming PCA has been studied extensively; see Balzano et al. (2018) and references therein.
In this work, we address both problems in a principled manner. Our contributions are as follows:
- We provide an alternative formulation of fair PCA based on the "Null It Out" approach (Section 3). Based on the new formulation, we introduce the concept of probably approximately fair and optimal (PAFO)-learnability to formalize the problem of fair PCA (Section 4).
- To address the memory limitation, we propose a new problem setting called fair streaming PCA, as well as fair noisy power method (FNPM), a simple yet memory-efficient algorithm based on the noisy power method. (Section 5). We note that our algorithm incurs a much lower memory complexity even compared to the most efficient variant of fair PCA proposed by Kleindessner et al. (2023).
- We then prove that our algorithm achieves the PAFO-learnability for fair streaming PCA (Section 6). Such statistical guarantee is the first of its kind in fair PCA literature.
- Lastly, we empirically validate our algorithm on CelebA and UCI datasets. Notably, we run FNPM on the original full-resolution CelebA dataset on which existing fair PCA algorithms fail due to high memory requirements. It shows turning such a non-streaming setting into a streaming setting and applying our algorithm allows one to bypass the memory limitation (Section 7).
Notation. For  $\ell \geq 1$ , let  $I_{\ell}$  be the identity matrix of size  $\ell \times \ell$ . For  $k < d$ , we bring the Stiefel manifold  $\mathrm{St}(d,k) = \{\pmb{A} \in \mathbb{R}^{d \times k} : \pmb{A}^{\top}\pmb{A} = \pmb{I}_{k}\}$ , which is the collection of all rank- $k$  semi-orthogonal matrices. We denote an orthonormal column basis of a (full column rank) matrix  $M \in \mathbb{R}^{d \times k}$  obtained by QR decomposition as  $\mathrm{QR}(M) \in \mathrm{St}(d,k)$  and denote its column space by  $\operatorname{col}(M)$ . Also, for  $A \in \mathrm{St}(d,k)$ , we denote the orthogonal projection matrix to  $\operatorname{col}(A)^{\perp} = \mathrm{null}(A^{\top})$  as  $\Pi_A^\perp = I_d - AA^{\top} = I_d - \Pi_A$ . Moreover, we denote the collection of all possible  $d$ -dimensional probability distributions as  $\mathcal{P}_d$ . For a zero-mean random matrix  $Z$ , its (scalar-valued) variance is defined as  $\operatorname{Var}(Z) = \max(\|\mathbb{E}[ZZ^{\top}]|_2, \|\mathbb{E}[Z^{\top}Z]\|_2)$ . In general,  $\operatorname{Var}(Z) = \operatorname{Var}(Z - \mathbb{E}[Z])$ . Lastly, we use the usual  $\mathcal{O}$ ,  $\Omega$ , and  $\Theta$  notations for asymptotic analyses, where tildes ( $\tilde{\mathcal{O}}$ ,  $\tilde{\Omega}$ , and  $\tilde{\Theta}$ , resp.) are used for hiding logarithmic factors.
Setup. Assume that the sensitive attribute variable, which we will be imposing fairness, is binary, denoted by  $a \in \{0,1\}$ . For each group  $a$ , let  $\mathcal{D}_a$  be a  $d$ -dimensional distribution of mean  $\pmb{\mu}_a$  and covariance  $\pmb{\Sigma}_a$ , both of which are assumed to be well-defined. We often call them group-conditional mean and covariance, respectively. With a fixed, unknown mixture parameter  $p \in (0,1)$ , let us denote the total data distribution as  $\mathcal{D} := p\mathcal{D}_0 + (1 - p)\mathcal{D}_1$ . Equivalently, the sensitive attribute follows  $a \sim \text{Bernoulli}(p)$ , and the conditional random variable  $\pmb{x}|a$  is sampled from  $\mathcal{D}_a$ . In that case,  $\pmb{\mu}_a = \mathbb{E}[\pmb{x}|a]$  and  $\pmb{\Sigma}_a = \mathbb{E}[\pmb{x}\pmb{x}^\top |a] - \pmb{\mu}_a\pmb{\mu}_a^\top$ . We often write  $p_0 = 1 - p$  and  $p_1 = p$  for brevity. We also define the mean difference  $\pmb{f} := \pmb{\mu}_1 - \pmb{\mu}_0$  and the second moment difference  $\pmb{S} := \mathbb{E}[\pmb{x}\pmb{x}^\top |a = 1] - \mathbb{E}[\pmb{x}\pmb{x}^\top |a = 0] = \pmb{\Sigma}_1 - \pmb{\Sigma}_0 + \pmb{\mu}_1\pmb{\mu}_1^\top - \pmb{\mu}_0\pmb{\mu}_0^\top$ . Accordingly, denote the true mean and covariance of  $\mathcal{D}$  as  $\pmb{\mu}$  and  $\pmb{\Sigma}$ , respectively. For simplicity, let us assume that  $\mathcal{D}$  is centered, i.e.,  $\pmb{\mu} = \mathbf{0}$ ; note that this does not mean that the group conditional distributions  $\mathcal{D}_a$ 's are centered.
PCA. In the offline setting, the full covariance matrix  $\pmb{\Sigma}$  is given which is often a sample covariance matrix  $\frac{1}{n}\sum_{i=1}^{n}\pmb{x}_i\pmb{x}_i^\top$  for  $n$  data points  $\pmb{x}_1,\dots,\pmb{x}_n$ . The goal of vanilla (offline) PCA (Hotelling, 1933; Pearson, 1901) is to compute the loading matrix  $\pmb{V} \in \mathbb{R}^{d \times k}$  that preserves as much variance as possible after projecting  $\pmb{\Sigma}$  via  $\pmb{V}$ , i.e., maximize  $\mathrm{tr}(\pmb{\Pi}_{\pmb{V}}\pmb{\Sigma}) = \mathrm{tr}(\pmb{V}^\top\pmb{\Sigma}\pmb{V})$ . Here,  $k < d$  is the
target dimension to which the data's dimensionality  $d$  is to be reduced and is chosen by the learner. We additionally consider the constraint  $V \in \mathrm{St}(d, k)$  to ensure that the resulting coordinate after the transformation is orthogonal and thus amenable to various statistical interpretations (Johnson and Wichern, 2008). Without any fairness constraint, Eckart-Young theorem (Eckart and Young, 1936) implies that the solution is characterized as a matrix whose columns are the top- $k$  eigenvectors of  $\boldsymbol{\Sigma}$ .
Fair PCA. Recently, it has been suggested that performing vanilla PCA on real-world datasets may exhibit bias, making the final outputted projection "unfair". As is often the case, there can be multiple definitions of fairness in PCA, but the following two are the most popular: equalizing reconstruction losses (Kamani et al., 2022; Samadi et al., 2018; Tantipongpipat et al., 2019; Vu et al., 2022), or equalizing the projected distributions (Kleindessner et al., 2023; Lee et al., 2022; Olfat and Aswani, 2019) from the perspective of fair representation (Zemel et al., 2013); we focus on the latter one.
3 An Alternative Approach to Fair PCA
3.1 "Null It Out" Formulation of Fair PCA
In this work, we consider fair PCA as learning fair representation (Zemel et al., 2013). The goal is to preserve as much variance as possible while obfuscating any information regarding the sensitive attribute. To this end, we take the "Null It Out" approach as proposed in Ravfogel et al. (2020). Intuitively, we want to nullify the directions in which the sensitive attribute  $a$  can be inferred, and in this work, we consider two such unfair directions: mean difference  $f$  and eigenvectors of second moment difference  $S$ . To give the learner flexibility in choosing the trade-off between fairness and performance (measured in explained variance), let  $m \geq 1$  be the number of top eigenvectors of  $S$  to nullify. Thus, the learner is nullifying at most  $(m + 1)$ -dimensional subspace that is unfair with respect to  $a$ , which we refer to as the unfair subspace. Precisely, we formulate our fair PCA as follows:
$$
\max  _ {\boldsymbol {V} \in \operatorname {S t} (d, k)} \operatorname {t r} \left(\boldsymbol {V} ^ {\top} \boldsymbol {\Sigma} \boldsymbol {V}\right), \quad \text {s u b j e c t} (\boldsymbol {V}) \subset \operatorname {c o l} ([ \boldsymbol {P} _ {m} | \boldsymbol {f} ]) ^ {\perp}, \tag {1}
$$
where  $d$  is the data dimensionality,  $k$  is the target dimension, and the columns of  $\pmb{P}_m\in \mathrm{St}(d,m)$  is top-  $m$  orthonormal eigenvectors of  $\pmb{S}$ .
3.2 An Explicit Characterization for Solution of Fair PCA
To first construct the unfair subspace that is spanned by  $\mathbf{f}$  as well as  $P_{m}$ , let us define  $U \in \mathrm{St}(d, m')$  to be the orthogonal matrix whose columns form a basis of  $\operatorname{col}([P_m | f])$ . Then,  $U$  has a closed form as follows:  $m' = m$  if  $\mathbf{f} \in \operatorname{col}(P_{m})$  and  $m' = m + 1$  otherwise, and
$$
\boldsymbol {U} = \left\{ \begin{array}{l l} \boldsymbol {P} _ {m}, & \text {i f} \boldsymbol {f} \in \operatorname {c o l} \left(\boldsymbol {P} _ {m}\right), \\ \mathbb {Q R} ([ \boldsymbol {P} _ {m} | \boldsymbol {f} ]) = \left[ \boldsymbol {P} _ {m} \mid \frac {\boldsymbol {g}}{\| \boldsymbol {g} \| _ {2}} \right], & \text {o t h e r w i s e}, \end{array} \right. \tag {2}
$$
where  $\pmb{g} = \Pi_{P_m}^{\perp}\pmb{f} \in \mathrm{col}(P_m)^{\perp}$ . Note that  $\pmb{g}$  is a vector in a direction that  $\pmb{f}$  is projected onto  $\mathrm{col}(P_m)^{\perp} = \mathrm{null}(P_m^{\top})$ . For this  $\pmb{U}$ , our constraint in (1) can be interpreted as an equivalent nullity constraint  $\Pi_U V = 0$ :
$$
\max  _ {\boldsymbol {V} \in \operatorname {S t} (d, k)} \operatorname {t r} \left(\boldsymbol {V} ^ {\intercal} \boldsymbol {\Sigma} \boldsymbol {V}\right), \quad \text {s u b j e c t} \quad \boldsymbol {\Pi} _ {\boldsymbol {U}} \boldsymbol {V} = \boldsymbol {0}. \tag {3}
$$
The above is equivalent to the following problem without any constraint other than semi-orthogonality:
$$
\max  _ {\boldsymbol {V} \in \operatorname {S t} (d, k)} \operatorname {t r} \left(\boldsymbol {V} ^ {\intercal} \boldsymbol {\Pi} _ {\boldsymbol {U}} ^ {\perp} \boldsymbol {\Sigma} \boldsymbol {\Pi} _ {\boldsymbol {U}} ^ {\perp} \boldsymbol {V}\right), \tag {4}
$$
which is basically the vanilla  $k$ -PCA problem of a matrix  $\Pi_U^\perp \Sigma \Pi_U^\perp$ . Therefore, a top- $k$  orthonormal column basis of this matrix is indeed a solution of our problem (4).
3.3 Comparison to the Existing Covariance Matching Constraint
Previous works on fair PCA (Kleindessner et al., 2023; Olfat and Aswani, 2019) consider an exact covariance-matching constraint (namely,  $V^{\top}(\Sigma_1 - \Sigma_0)V = 0$ ). In fact, this is equivalent to the condition  $V^{\top}SV = 0$  under the mean-matching constraint  $f^{\top}V = 0$ , which can be derived as
$$
\begin{array}{l} \boldsymbol {V} ^ {\intercal} \left(\boldsymbol {\Sigma} _ {1} - \boldsymbol {\Sigma} _ {0}\right) \boldsymbol {V} = \boldsymbol {V} ^ {\intercal} \left(\mathbb {E} [ \boldsymbol {x x} ^ {\intercal} | a = 1 ] - \mathbb {E} [ \boldsymbol {x x} ^ {\intercal} | a = 0 ]\right) \boldsymbol {V} - \boldsymbol {V} ^ {\intercal} \left(\boldsymbol {\mu} _ {1} \boldsymbol {\mu} _ {1} ^ {\intercal} - \boldsymbol {\mu} _ {0} \boldsymbol {\mu} _ {0} ^ {\intercal}\right) \boldsymbol {V} \\ = \boldsymbol {V} ^ {\intercal} \boldsymbol {S} \boldsymbol {V} - \boldsymbol {V} ^ {\intercal} \left(\boldsymbol {f} \boldsymbol {\mu} _ {1} ^ {\intercal} + \boldsymbol {\mu} _ {0} \boldsymbol {f} ^ {\intercal}\right) \boldsymbol {V} = \boldsymbol {V} ^ {\intercal} \boldsymbol {S} \boldsymbol {V} = \boldsymbol {0}. \\ \end{array}
$$
One immediate problem with this is that the constraint may be infeasible depending on the choice of  $\Sigma_0$ ,  $\Sigma_1$ , or  $S$ ; e.g., when  $\Sigma_1 - \Sigma_0$  is positive definite. For this reason, Kleindessner et al. (2023); Olfat and Aswani (2019) propose relaxations of the fairness constraints but provide no further discussions on its impact on statistical guarantees. On the contrary, our formulation is always feasible without any need for relaxation, allowing us to consider a rigorous definition of fair PCA (Definition 2) for the first time in fair PCA literature.
4 Statistical Viewpoint: PAFO-Learnability of PCA
As all the distribution statistics  $(\Sigma, p, \dots)$  are unknown, the learner, given some finite number of samples, must learn all of them and solve fair PCA. In supervised learning, such a problem is often formalized in a PAC-learnability framework (Shalev-Schwartz and Ben-David, 2014). In the context of PAC-learnability for unsupervised learning settings, TV-learning, which is the task of learning distribution, has been mainly considered so far (Ananthakrishnan et al., 2021; Hopkins et al., 2023). However, unlike TV-learning, it is unnecessary to learn the whole distribution in fair PCA; moreover, fair PCA has the fairness constraint  $\Pi_U V = 0$  to be satisfied. Inspired by the unsupervised PAC-learnability as well as constrained PAC-learnability (Chamon and Ribeiro, 2020), we propose a new notion of learnability for fair PCA, called PAFO (Probably Approximately Fair and Optimal) learnability, as follows:
Definition 1 (Projection Learner). A projection learner is a function that takes  $k \geq 1$  and  $d$ -dimensional samples as input and outputs a loading matrix  $V \in \mathrm{St}(d, k)$ .
Definition 2 (PAFO-Learnability of PCA). Let  $d, k, m$  be integers such that  $1 \leq k < d$  and  $m < d$ . We say that  $\mathcal{F}_d \subset \mathcal{P}_d \times \mathcal{P}_d \times (0,1)$  is PAFO-learnable for PCA if there exists a function  $N_{\mathcal{F}_d} : (0,1)^3 \to \mathbb{N}$  and a projection learner  $\mathcal{A}$  satisfying the following:
For every  $\varepsilon_{\mathrm{o}},\varepsilon_{\mathrm{f}},\delta \in (0,1)$  and  $(\mathcal{D}_0,\mathcal{D}_1,p)\in \mathcal{F}_d$  , when running  $\mathcal{A}$  on  $N\geq N_{\mathcal{F}_d}(\varepsilon_{\mathrm{o}},\varepsilon_{\mathrm{f}},\delta)$  i.i.d. samples from  $\mathcal{D}\coloneqq p\mathcal{D}_1 + (1 - p)\mathcal{D}_0$  of the form  $(a,\pmb {x})$  ,  $\mathcal{A}$  returns  $\mathbf{V}$  s.t., with probability at least  $1 - \delta$  (over the draws of the  $N$  samples),
$$
\operatorname {t r} \left(\boldsymbol {V} ^ {\intercal} \boldsymbol {\Sigma} \boldsymbol {V}\right) \geq \operatorname {t r} \left(\boldsymbol {V} _ {\star} ^ {\intercal} \boldsymbol {\Sigma} \boldsymbol {V} _ {\star}\right) - \varepsilon_ {\mathrm {o}}, \quad \| \boldsymbol {\Pi} _ {\boldsymbol {U}} \boldsymbol {V} \| _ {2} \leq \varepsilon_ {\mathrm {f}}, \tag {5}
$$
where  $\mathbf{U}$  is as defined in Eqn. (2) and  $\mathbf{V}_{\star}$  is any solution to Eqn. (4) (with prescribed  $k$  and  $m$ ).
Like in the usual PAC-learnability,  $N_{\mathcal{F}_d}$  is referred to as the sample complexity of fair PCA. Observe how the optimality is measured w.r.t. the optimal solution of fair PCA, not the vanilla PCA. Also, the two conditions are not overlapping: vanilla PCA (overly) satisfies  $\varepsilon_{\mathrm{o}}$ -optimality in explained variance but does not satisfy  $\varepsilon_{\mathrm{f}}$ -optimality in fairness, and vice versa for a poorly chosen  $V \in St(d,k)$  with  $\operatorname{col}(V) \subseteq \operatorname{col}([P_m|\pmb {f}]^\perp)$ .
5 Algorithmic Viewpoint: Fair Streaming PCA
We now introduce a new problem setting, fair streaming PCA. In this setting, the learner receives a stream of pairs  $(a_{t},\pmb{x}_{t})\in \{0,1\} \times \mathbb{R}^{d}$  sequentially. Note that the sensitive attribute information  $a_{t}$  is also available at each time-step; this is commonly assumed when considering fairness in streaming setting (Bera et al., 2022; El Halabi et al., 2020). Precisely, we assume the following model of the data generation process: at each time-step  $t$  , a sensitive attribute is chosen as  $a_{t}\sim \mathrm{Bernoulli}(p)$  , then the data is sampled from the corresponding sensitive group's conditional distribution  $\pmb {x}_t\mid a_t\sim \mathcal{D}_{s_t}$  Importantly, as done in previous streaming PCA literature (Mitliagkas et al., 2013), we assume that the learner has only  $\mathcal{O}(dk)$  memory, where  $d$  is the data dimension and  $k$  is the target dimension. We can formally define the PAFO-learnability in this streaming setting:
Definition 3. We say that  $\mathcal{F}_d \subseteq \mathcal{P}_d \times \mathcal{P}_d \times (0,1)$  is PAFO-learnable for streaming PCA if the projection learner  $\mathcal{A}$  for which Definition 2 holds uses only  $\mathcal{O}(dk)$  memory for streaming data.
5.1 Our Algorithm: Fair Noisy Power Method (FNPM)
One only needs to estimate  $\pmb{U}$  to use the off-the-shelf streaming PCA algorithm. As  $\pmb{U}$  is of size  $d \times m$ , storing its estimate is no problem for the memory constraint as long as  $m = \mathcal{O}(k)$ . Naturally, we proceed via a two-stage approach; first, estimate  $\pmb{U}$  sufficiently well, then with the fixed estimate of  $\pmb{U}$ , apply the noisy power method (Hardt and Price, 2014; Mitliagkas et al., 2013) for  $\pmb{V}$ .
For estimating  $\pmb{U}$ , one needs to estimate  $\pmb{f}$  and  $P_{m}$ . Estimating  $\pmb{f}$  can be done using the usual cumulative averaging. As for  $P_{m}$ , we can consider the two main approaches for streaming PCA: Oja's method (Huang et al., 2021; Oja, 1982; Oja and Karhunen, 1985) and noisy power method (NPM) (Hardt and Price, 2014; Mitliagkas et al., 2013). We first show that Oja's method is inapplicable for our purpose, as it may ignore some eigenvectors corresponding to negative (but large in magnitude) eigenvalues of  $S$ . For instance, if  $S = -2e_{1}e_{1}^{\top} + e_{2}e_{2}^{\top} + 4e_{3}e_{3}^{\top}$  with  $e_{i}$  being the standard basis vectors, then Oja's method with  $m = 2$  would yield  $[e_2|e_3]$  when we actually want  $[e_1|e_3]$ . For the same reason, simply shifting the eigenvalue spectrum by considering  $S + \| S\| _2I$  does not work. Thus we apply NPM for estimating  $P_{m}$  in our case, which is known to converge as long as the singular value gap of  $P_{m}$  is large enough and norms of the noise matrices at each iterate are properly bounded (Balcan et al., 2016; Hardt and Price, 2014).
Algorithm 1:UnfairSubspace
1 Input:  $m$ , Block size  $b$ , Number of iterations  $T$ ;
2 Output: A matrix  $\widehat{U}$  with orthonormal columns;
3  $W_{0} = \mathbb{Q}\mathbb{R}(\mathcal{N}(0,1)^{d\times m})$
4  $(\overline{\mathbf{m}}^{(0)},\overline{\mathbf{m}}^{(1)},B^{(0)},B^{(1)}) = (\mathbf{0}_d,\mathbf{0}_d,0,0);$
5 for  $t\in [T]$  do
$$
\begin{array}{l} \mathbf {6} \\ \mathbf {7} \\ \mathbf {8} \\ \mathbf {9} \\ \mathbf {1 0} \\ \mathbf {1 1} \end{array} \left\lfloor \begin{array}{c} \text {R e c e i v e} \{(a _ {i}, \boldsymbol {x} _ {i}) \} _ {i = (t - 1) b} ^ {t b} \\ \text {f o r e a c h} a \in \{0, 1 \} \text {d o} \\ \left\lfloor \begin{array}{c} \text {C o m p u t e} b _ {t} ^ {(a)}, \boldsymbol {m} _ {t} ^ {(a)}, C \\ \overline {{\boldsymbol {m}}} ^ {(a)} \leftarrow \frac {B ^ {(a)}}{B ^ {(a)} + b _ {t} ^ {(a)}} \overline {{\boldsymbol {m}}} ^ {(a)} \\ B ^ {(a)} \leftarrow B ^ {(a)} + b _ {t} ^ {(a)}; \end{array} \right. \\ W _ {t} = \mathbb {Q R} \left(\boldsymbol {C} _ {t} ^ {(1)} - \boldsymbol {C} _ {t} ^ {(0)}\right);
$$
12  $\widehat{f}\gets \overline{m}^{(1)} - \overline{m}^{(0)}$
13  $\widehat{\pmb{g}}\gets \widehat{\pmb{f}} -\pmb{W}_T\pmb{W}_T^\top \widehat{\pmb{f}};$
14 if  $\|\widehat{g}\|_2 = 0$  then
$$
\left. \begin{array}{l l} \text {1 5} & \widehat {\boldsymbol {U}} = \boldsymbol {W} _ {T} \end{array} \right.
$$
16 else
$$
\left\lfloor \widehat {\boldsymbol {U}} = \left[ \boldsymbol {W} _ {T} \mid \frac {\widehat {\boldsymbol {g}}}{\| \widehat {\boldsymbol {g}} \| _ {2}} \right] \right. \tag {17}
$$
18 return  $\widehat{U}$
Algorithm 2: Fair NPM
1 Input:  $k$ , Block sizes  $\mathcal{B}, b$ , Numbers of iterations  $\mathcal{T}, T$ ;
2 Output:  $V_{\mathcal{T}} \in \mathrm{St}(d, k)$ ;
3  $\widehat{U} \leftarrow$  UnfairSubspace(b,T); // Algorithm 1
4  $V_{0}\gets \mathbb{QR}(\mathcal{N}(0,1)^{d\times k})$
5 for  $\tau \in [\mathcal{T}]$  do
6  $\mathbf{V}_{\tau}\gets \mathbf{V}_{\tau -1} - \widehat{\mathbf{U}}\widehat{\mathbf{U}}^{\intercal}\mathbf{V}_{\tau -1};$
7 Receive  $\{(*, \tilde{x}_j)\}_{j = (\tau - 1)B + 1}^{\tau B}$ ;
8  $V_{\tau}\gets \frac{1}{\mathcal{B}}\sum_{j = 1}^{\mathcal{B}}\tilde{x}_{j}\tilde{x}_{j}^{\top}V_{\tau};$
9  $\begin{array}{r}\mathbf{V}_{\tau}\gets \mathbb{QR}\left(\bar{\mathbf{V}}_{\tau} - \hat{\mathbf{U}}\hat{\mathbf{U}}^{\top}\mathbf{V}_{\tau}\right); \end{array}$
10 return  $V_{T}$