SlowGuess/NeurIPS_2023_OCR huggingface dataset:
For example, in binary classification with a binary group attribute, the above optimization only has 8 variables, 14 linear constraints + a single convex constraint  $P \in \mathcal{C}$ . Hence, standard convex optimization solvers can directly compute its optimal value as long as we know how to characterize  $\mathcal{C}$ .
Remark 2. Note that Kim et al. (2020) investigated fairness Pareto frontiers via confusion matrices. The main difference is that Definition 1 in Kim et al. (2020) relaxed the constraint (5c) to  $P \in \mathcal{T}(C|AC)$  where  $\mathcal{T}(C|AC)$  represents all transition matrices from  $[AC]$  to  $[C]$ . This leads to a loose approximation of the frontier because  $\mathcal{C}$  is often a strict subset of  $\mathcal{T}(C|AC)$ . To demonstrate this point, consider the scenario where  $X \perpline (S, Y)$ . Then  $\hat{Y} \perpline (S, Y)$  by data processing inequality so
$$
\mathcal {C} = \left\{\boldsymbol {P} \in \mathcal {T} (C | A C) \mid \text {e a c h r o w o f} \boldsymbol {P} \text {i s t h e s a m e} \right\}. \tag {6}
$$
Optimizing over  $\mathcal{C}$  rather than  $\mathcal{T}(C|AC)$  can significantly tighten the fairness Pareto frontier.
Before diving into the analysis, we first introduce a function  $g: \mathcal{X} \to \Delta_{AC}$  defined as  $g(x) = (P_{\mathrm{S,Y|X}}(1,1|x), \dots, P_{\mathrm{S,Y|X}}(A,C|x))$ . To obtain this function in practice, a common strategy among various post-processing fairness interventions (see e.g., Menon and Williamson, 2018; Alghamdi et al., 2022) is to train a probabilistic classifier that uses input features  $\mathbf{X}$  to predict  $(\mathrm{S}, \mathrm{Y})$ . The output probability generated by this classifier is then utilized as an approximation of the function  $g$ .
The following theorem is the main theoretical result in this paper. It provides a precise characterization of the set  $\mathcal{C}$  through a series of convex constraints.
Theorem 1. The set  $\mathcal{C}$  is the collection of all transition matrices  $P \in \mathcal{T}(C|AC)$  such that the following condition holds:
For any  $k\in \mathcal{N}$  and any  $\{\pmb {a}_i\mid \pmb {a}_i\in [-1,1]^{AC},i\in [k]\}$
$$
\sum_ {\hat {y} = 1} ^ {C} \max  _ {i \in [ k ]} \left\{\boldsymbol {a} _ {i} ^ {T} \boldsymbol {\Lambda} _ {\mu} \boldsymbol {p} _ {\hat {y}} \right\} \leq \mathbb {E} \left[ \max  _ {i \in [ k ]} \left\{\boldsymbol {a} _ {i} ^ {T} \boldsymbol {g} (\mathrm {X}) \right\} \right], \tag {7}
$$
where  $\pmb{p}_{\hat{y}}$  is the  $\hat{y}$ -th column of  $\pmb{P}$  and  $\Lambda_{\mu} = \mathrm{diag}(\mu_{1,1},\dots ,\mu_{A,C})$ .
Intuitively, (7) uses piecewise linear functions to approximate the boundary of the convex set  $\mathcal{C}$  where  $k$  represents the number of linear pieces. Unfortunately, replacing  $P\in \mathcal{C}$  with this series of constraints in (5) may result in an intractable problem since standard duality-based approaches will lead to infinitely many dual variables. To resolve this issue, we first fix  $k$  and let  $\mathcal{C}_k$  be the set of  $\pmb{P}$  such that (7) holds under this fixed  $k$ . Accordingly, we define  $\mathrm{FairFront}_k(\alpha_{\mathrm{SP}},\alpha_{\mathrm{EO}},\alpha_{\mathrm{OAE}})$  as the optimal value of (5) when replacing  $\mathcal{C}$  with  $\mathcal{C}_k$ . Since  $\mathcal{C}_1\supseteq \mathcal{C}_2\supseteq \dots \supseteq \mathcal{C}$ , we have  $\mathrm{FairFront}_1(\alpha_{\mathrm{SP}},\alpha_{\mathrm{EO}},\alpha_{\mathrm{OAE}})\geq \mathrm{FairFront}_2(\alpha_{\mathrm{SP}},\alpha_{\mathrm{EO}},\alpha_{\mathrm{OAE}})\geq \dots \geq \mathrm{FairFront}(\alpha_{\mathrm{SP}},\alpha_{\mathrm{EO}},\alpha_{\mathrm{OAE}})$ . However, computing  $\mathrm{FairFront}_k(\alpha_{\mathrm{SP}},\alpha_{\mathrm{EO}},\alpha_{\mathrm{OAE}})$  still involves infinitely many constraints.
Next, we introduce a greedy improvement algorithm that consists of solving a sequence of tractable optimization problems for approximating  $\mathsf{FairFront}_k(\alpha_{\mathrm{SP}},\alpha_{\mathrm{EO}},\alpha_{\mathrm{OAE}})$ . We use  $\mathcal{A}$  to collect the constraints of  $P$  and set  $\mathcal{A} = \emptyset$  initially. At each iteration, our algorithm solves a convex program to find an optimal  $P$  that maximizes the accuracy while satisfying the desired group fairness constraints and the constraints in  $\mathcal{A}$ ; then we verify if this  $P$  is within the set  $\mathcal{C}_k$  by solving a DC (difference of
Algorithm 1 Approximate the fairness Pareto frontier.
Input:  $\mathcal{D} = \{(x_i,y_i,s_i)\}_{i = 1}^N$ , max number of iterations  $T$ ; max pieces  $k$ ; classifier  $g(x)$ ;  $\alpha_{\mathrm{SP}}$ ,  $\alpha_{\mathrm{EO}}$ ,  $\alpha_{\mathrm{OAE}}$ .
Initialize: set  $\mathcal{A} = \emptyset$ ;  $\mu_{s,y} = \frac{|\{i|s_i = s,y_i = y\}|}{N}$ ;  $t = 1$ .
Repeat:
Solve a convex program:
$$
\max  _ {P} \sum_ {s = 1} ^ {A} \sum_ {y = 1} ^ {C} \mu_ {s, y} P _ {(s, y), y}
$$
$$
\mathrm {s . t .} \mathbf {P} \in \mathcal {T} (C | A C), \mathrm {S P} \leq \alpha_ {\mathrm {S P}}, \mathrm {E O} \leq \alpha_ {\mathrm {E O}}, \mathrm {O A E} \leq \alpha_ {\mathrm {O A E}}
$$
$$
\sum_ {\hat {y} = 1} ^ {C} \max  _ {i \in [ k ]} \left\{\boldsymbol {a} _ {i} ^ {T} \boldsymbol {\Lambda} _ {\mu} \boldsymbol {p} _ {\hat {y}} \right\} \leq \mathbb {E} \left[ \max  _ {i \in [ k ]} \left\{\boldsymbol {a} _ {i} ^ {T} \boldsymbol {g} (\mathrm {X}) \right\} \right] \quad \forall (\boldsymbol {a} _ {1}, \dots , \boldsymbol {a} _ {k}) \in \mathcal {A}.
$$
Let  $v^t$  and  $P^t$  be the optimal value and optimal solution.
Solve a DC program:
$$
\min_{\substack{\boldsymbol {a}_{i}\in [ - 1,1]^{A C}\\ i\in [k]}}\mathbb{E}\left[ \max_{i\in [k]}\{\boldsymbol{a}_{i}^{T}\boldsymbol {g}(\mathrm{X})\} \right] - \sum_{\hat{y} = 1}^{C}\max_{i\in [k]}\left\{\boldsymbol{a}_{i}^{T}\boldsymbol{\Lambda}_{\mu}\boldsymbol{p}_{\hat{y}}^{t}\right\} .
$$
If the optimal value is  $\geq 0$  or  $t = T$ ,
stop;
otherwise,
add the optimal  $(a_{1},\dots ,a_{k})$  to  $\mathcal{A}$  and  $t = t + 1$
return:  $v^t, P^t, \mathcal{A}$ .
convex) program (Shen et al., 2016; Horst and Thoai, 1999). If  $P \in \mathcal{C}_k$ , then the algorithm stops. Otherwise, the algorithm will find the constraint that is mostly violated by  $P$  and add this constraint to  $\mathcal{A}$ . Specifically, we determine a piecewise linear function that divides the space into two distinct regions: one containing  $P$  and the other containing  $\mathcal{C}_k$ . By "mostly violated", we mean the function is constructed to maximize the distance between  $P$  and the boundary defined by the function. We describe our algorithm in Algorithm 1 and establish a convergence guarantee below.
Theorem 2. Let  $T = \infty$ . If Algorithm 1 stops, its output  $P^t$  is an optimal solution of  $\text{FairFront}_k(\alpha_{\text{SP}}, \alpha_{\text{EO}}, \alpha_{\text{OAE}})$ . Otherwise, any convergent sub-sequence of  $\{P^t\}_{t=1}^{\infty}$  converges to an optimal solution of  $\text{FairFront}_k(\alpha_{\text{SP}}, \alpha_{\text{EO}}, \alpha_{\text{OAE}})$ .
Note that the output  $v^{t}$  from Algorithm 1 is always an upper bound for FairFront  $(\alpha_{\mathrm{SP}}, \alpha_{\mathrm{EO}}, \alpha_{\mathrm{OAE}})$ , assuming the estimation error is sufficiently small. The tightness of this upper bound is determined by  $k$  (i.e., how well the piecewise linear functions approximate the boundary of  $\mathcal{C}$ ),  $T$  (i.e., the total number of iterations). On the other hand, running off-the-shelf in-processing and post-processing fairness interventions can only yield lower bounds for FairFront  $(\alpha_{\mathrm{SP}}, \alpha_{\mathrm{EO}}, \alpha_{\mathrm{OAE}})$ .
In this section, we demonstrate the tightness of our upper bound approximation of FairFront, apply it to benchmark existing group fairness interventions, and show how data biases, specifically missing values, impact their effectiveness. We find that given sufficient data, SOTA fairness interventions are successful at reducing epistemic discrimination as their gap to (our upper bound estimate of) FairFront is small. However, we also discover that when different population groups have varying missing data patterns, aleatoric discrimination increases, which diminishes the performance of fairness intervention algorithms. Our numerical experiments are semi-synthetic since we apply fairness interventions to train classifiers using the entire dataset and resample from it as the test set. This setup enables us to eliminate the estimation error associated with Algorithm 1 (see Appendix E for a discussion). We provide additional experimental results and details in Appendix C.
Figure 1: We compare Reduction and FairProjection with (our upper bound estimate of) FairFront on the Adult (Left) and COMPAS (Right) datasets. We train a classifier that approximates the Bayes optimal and use it as a basis for Reduction and FairProjection. This result not only demonstrates the tightness of our approximation but also shows that SOTA fairness interventions have already achieved near-optimal fairness-accuracy curves.
4.1 Benchmark Fairness Interventions
Setup. We evaluate our results on the UCI Adult dataset (Bache and Lichman, 2013), the ProPublica COMPAS dataset (Angwin et al., 2016), the German Credit dataset (Bache and Lichman, 2013), and HSLS (High School Longitudinal Study) dataset (Ingels et al., 2011; Jeong et al., 2022). We recognize that Adult, COMPAS, and German Credit datasets are overused and acknowledge the recent calls to move away from them (see e.g., Ding et al., 2021). We adopt these datasets for benchmarking purposes only since most fairness interventions have available code for these datasets. The HSLS dataset is a new dataset that first appeared in the fair ML literature last year and captures a common use-case of ML in education (student performance prediction, see Jeong et al., 2022). It has multi-class labels and multiple protected groups. We apply existing (group) fairness interventions to these datasets and measure their fairness violations via Max equalized odds:
$$
\max  | \operatorname * {P r} (\hat {\mathbf {Y}} = \hat {y} | \mathbf {S} = s, \mathbf {Y} = y) - \operatorname * {P r} (\hat {\mathbf {Y}} = \hat {y} | \mathbf {S} = s ^ {\prime}, \mathbf {Y} = y) |
$$
where the max is taken over  $y, \hat{y}, s, s'$ . We run Algorithm 1 with  $k = 6$  pieces, 20 iterations, and varying  $\alpha_{\mathrm{EO}}$  to estimate FairFront on each dataset. We compute the expectations and the  $g$  function from the empirical distributions and solve the DC program by using the package in Shen et al. (2016). The details about how we pre-process these datasets and additional experimental results on the German Credit and HSLS datasets are deferred to Appendix C.
Group fairness interventions. We consider five existing fairness-intervention algorithms: Reduction (Agarwal et al., 2018), EqOdds (Hardt et al., 2016), CalEqOdds (Pleiss et al., 2017), LevEqOpp (Chzhen et al., 2019), and FairProjection Alghamdi et al. (2022). Among them, Reduction is an in-processing method and the rest are all post-processing methods. For the first three benchmarks, we use the implementations from IBM AIF360 library (Bellamy et al., 2018); for LevEqOpp and FairProjection, we use the Python implementations from the Github repo in Alghamdi et al. (2022). For Reduction and FairProjection, we can vary their tolerance of fairness violations to produce a fairness-accuracy curve; for EqOdds, CalEqOdds, and LevEqOpp, each of them produces a single point since they only allow hard equality constraint. We note that FairProjection is optimized for transforming probabilistic classifier outputs (see also Wei et al., 2021), but here we threshold the probabilistic outputs to generate binary predictions which may limit its performance. Finally, we train a random forest as the Baseline classifier.
Results. We observe that if we run Algorithm 1 for a single iteration, which is equivalent to solving Proposition 1 without (5c), its solution is very close to 1 for all  $\alpha_{\mathrm{EO}}$ . This demonstrates the benefits of incorporating Blackwell's conditions into the fairness Pareto frontier.
Figure 2: We benchmark existing fairness interventions using (our upper bound estimate of) FairFront. We use FairFront to quantify aleatoric discrimination and measure epistemic discrimination by comparing a classifier's accuracy and fairness violation with FairFront. The results show that SOTA fairness interventions are effective at reducing epistemic discrimination.
We train a classifier that approximates the Bayes optimal and use it as a basis for both Reduction and FairProjection, which are SOTA fairness interventions. We then apply these two fairness interventions to the entire dataset and evaluate their performance on the same dataset. Figure 1 shows that in this infinite sample regime, the fairness-accuracy curves produced by Reduction and FairProjection can approach our upper bound estimate of FairFront. This result not only demonstrates the tightness of our approximation (recall that Algorithm 1 gives an upper bound of FairFront and existing fairness interventions give lower bounds) but also shows that SOTA fairness interventions have already achieved near-optimal fairness-accuracy curves.
Recall that we use FairFront to quantify aleatoric discrimination since it characterizes the highest achievable accuracy among all classifiers satisfying the desired fairness constraints. Additionally, we measure epistemic discrimination by comparing a classifier's accuracy and fairness violation with FairFront. Given that our Algorithm 1 provides a tight approximation of FairFront, we use it to benchmark existing fairness interventions. Specifically, we first train a base classifier which may not achieve Bayes optimal accuracy. Then we use it as a basis for all existing fairness interventions. The results in Figure 2 show that SOTA fairness interventions remain effective at reducing epistemic discrimination. In what follows, we demonstrate how missing values in data can increase aleatoric discrimination and dramatically reduce the effectiveness of SOTA fairness interventions.
4.2 Fairness Risks in Missing Values
Real-world data often have missing values and the missing patterns can be different across different protected groups (see Jeong et al., 2022, for some examples). There is a growing line of research (see e.g., Jeong et al., 2022; Fernando et al., 2021; Wang and Singh, 2021; Subramonian et al., 2022; Caton et al., 2022; Zhang and Long, 2021; Schelter et al., 2019) studying the fairness risks of data with missing values. In this section, we apply our result to demonstrate how disparate missing patterns influence the fairness-accuracy curves.
Setup. We choose sex (group 0: female, group 1: male) as the group attribute for the Adult dataset, and race (group 0: African-American, group 1: Caucasian) for the COMPAS dataset. To investigate the impact of disparate missing patterns on aleatoric discrimination, we artificially generate missing values in both datasets. This is necessary as the datasets do not contain sufficient missing data. The missing values are generated according to different probabilities for different population groups. For each data point from group 0, we erase each input feature with a varying probability  $p_0 \in \{10\%, 50\%, 70\% \}$ , while for group 1, we erase each input feature with a fixed probability  $p_1 = 10\%$ . We then apply mode imputation to the missing values, replacing them with the mode of non-missing values for each feature. Finally, we apply Algorithm 1 along with Reduction and Baseline to the imputed data. The experimental results are shown in Figure 3.
Figure 3: Fairness risks of disparate missing patterns. The missing probabilities of group 0 (female in Adult/African-American in COMPAS) and group 1 (male in Adult/Caucasian in COMPAS) are varying among  $\{(10\%, 10\%), (50\%, 10\%), (70\%, 10\%)\}$ . We apply Reduction and Baseline to the imputed data and plot their fairness-accuracy curves against FairFront. As shown, the effectiveness of fairness interventions substantially decrease with increasing disparate missing patterns in data.
Results. As we increase the missing probability of group 0, (our upper bound estimate of) FairFront decreases since it becomes more difficult to accurately predict outcomes for group 0. This in turn affects the overall model performance, since the fairness constraint requires that the model performs similarly for both groups. We also observe the fairness-accuracy curves of Reduction decrease as the missing data for group 0 become more prevalent. In other words, as the missing data for group 0 increase, it becomes more difficult to maintain both high accuracy and fairness in the model's prediction.
5 Final Remarks
The past years have witnessed a growing line of research introducing various group fairness-intervention algorithms. Most of these interventions focus on optimizing model performance subject to group fairness constraints. Though comparing and benchmarking these methods on various datasets is valuable (e.g., see benchmarks in Friedler et al., 2019; Bellamy et al., 2019; Wei et al., 2021), this does not reveal if there is still room for improvement in their fairness-accuracy curves, or if existing methods approach the information-theoretic optimal limit when infinite data is available. Our results address this gap by introducing the fairness Pareto frontier, which measures the highest possible accuracy under a set of group fairness constraints. We precisely characterize the fairness Pareto frontier using Blackwell's conditions and present a greedy improvement algorithm that approximates it from data. Our results show that the fairness-accuracy curves produced by SOTA fairness interventions are very close to the fairness Pareto frontier on standard datasets.
Additionally, we demonstrate that when data are biased due to missing values, the fairness Pareto frontier degrades. Although existing fairness interventions can still reduce performance disparities, they come at the cost of significantly lowering overall model accuracy. The methods we present for computing the fairness Pareto frontier can also be applied to analyze other sources of aleatoric discrimination, such as when individuals may misreport their data or when there are measurement errors. Overall, the fairness Pareto frontier can serve as a valuable framework for guiding data collection and cleaning. Our results indicate that existing fairness interventions can be effective in reducing epistemic discrimination, and there are diminishing returns in developing new fairness interventions focused solely on optimizing accuracy for a given group fairness constraint on pristine data. However, existing fairness interventions have yet to effectively provide both fair and accurate classification when additional sources of aleatoric discrimination are present (such as missing values in data). This suggests that there is still significant need for research on handling aleatoric sources of discrimination that appear throughout the data collection process.
We provide an in-depth discussion on future work in Appendix E.
This material is based upon work supported by the National Science Foundation under grants CAREER 1845852, CIF 2312667, FAI 2040880, CIF 1900750.
Agarwal, A., Beygelzimer, A., Dudík, M., Langford, J., and Wallach, H. (2018). A reductions approach to fair classification. In International Conference on Machine Learning, pages 60-69. PMLR.
Alghamdi, W., Asoodeh, S., Wang, H., Calmon, F. P., Wei, D., and Ramamurthy, K. N. (2020). Model projection: Theory and applications to fair machine learning. In 2020 IEEE International Symposium on Information Theory (ISIT), pages 2711-2716. IEEE.
Alghamdi, W., Hsu, H., Jeong, H., Wang, H., Michalak, P. W., Asoodeh, S., and Calmon, F. P. (2022). Beyond Adult and COMPAS: Fair multi-class prediction via information projection. In Advances in Neural Information Processing Systems.
Angwin, J., Larson, J., Mattu, S., and Kirchner, L. (2016). Machine bias. ProPublica.
Bache, K. and Lichman, M. (2013). UCI Machine Learning Repository.
Bellamy, R. K., Dey, K., Hind, M., Hoffman, S. C., Houde, S., Kannan, K., Lohia, P., Martino, J., Mehta, S., Mojsilovic, A., et al. (2018). Ai fairness 360: An extensible toolkit for detecting, understanding, and mitigating unwanted algorithmic bias. arXiv preprint arXiv:1810.01943.
Bellamy, R. K., Dey, K., Hind, M., Hoffman, S. C., Houde, S., Kannan, K., Lohia, P., Martino, J., Mehta, S., Mojsilovic, A., et al. (2019). Ai fairness 360: An extensible toolkit for detecting and mitigating algorithmic bias. IBM Journal of Research and Development, 63(4/5):4-1.
Berk, R., Heidari, H., Jabbari, S., Kearns, M., and Roth, A. (2021). Fairness in criminal justice risk assessments: The state of the art. Sociological Methods & Research, 50(1):3-44.
Blackwell, D. (1951). Comparison of experiments. Proceedings of the Second Berkeley Symposium on Mathematical Statistics and Probability, pages 93-102.
Blackwell, D. (1953). Equivalent comparisons of experiments. The annals of mathematical statistics, pages 265-272.
Blodgett, S. L., Barocas, S., Daumé III, H., and Wallach, H. (2020). Language (technology) is power: A critical survey of "bias" in nlp. arXiv preprint arXiv:2005.14050.
Blum, A. and Stangl, K. (2019). Recovering from biased data: Can fairness constraints improve accuracy? arXiv preprint arXiv:1912.01094.
Calmon, F., Wei, D., Vinzamuri, B., Natesan Ramamurthy, K., and Varshney, K. R. (2017). Optimized pre-processing for discrimination prevention. Advances in neural information processing systems, 30.
Cam, L. L. (1964). Sufficiency and approximate sufficiency. The Annals of Mathematical Statistics, pages 1419-1455.
Caton, S., Malisetty, S., and Haas, C. (2022). Impact of imputation strategies on fairness in machine learning. Journal of Artificial Intelligence Research, 74:1011-1035.
Celis, L. E., Huang, L., Keswani, V., and Vishnoi, N. K. (2019). Classification with fairness constraints: A meta-algorithm with provable guarantees. In Proceedings of the conference on fairness, accountability, and transparency, pages 319-328.
Chen, I., Johansson, F. D., and Sontag, D. (2018). Why is my classifier discriminatory? Advances in neural information processing systems, 31.
Chouldechova, A. (2017). Fair prediction with disparate impact: A study of bias in recidivism prediction instruments. *Big data*, 5(2):153-163.
Zhang, B. H., Lemoine, B., and Mitchell, M. (2018). Mitigating unwanted biases with adversarial learning. In Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society, pages 335-340.
Zhang, Y. and Long, Q. (2021). Assessing fairness in the presence of missing data. Advances in neural information processing systems, 34:16007-16019.
A Technical Background
In this section, we extend some results in Blackwell (1951, 1953) to our setting. For a random variable  $X$ , we denote its probability distribution by  $\mathcal{L}(X)$ . A conditional distribution  $P_{\mathrm{X|A}}: [n] \to \mathcal{P}(\mathcal{X})$  can be equivalently written as  $P \triangleq (P_1, \dots, P_n)$  where each  $P_i = P_{\mathrm{X|A} = i} \in \mathcal{P}(\mathcal{X})$ . Let  $\mathcal{A}$  be a closed, bounded, convex subset of  $\mathbb{R}^n$ . A decision function is a mapping  $f: \mathcal{X} \to \mathcal{A}$ , which can also be written as  $f(x) = (a_1(x), \dots, a_n(x))$ . A decision function is associated a loss vector:
$$
\boldsymbol {v} (\boldsymbol {f}) = \left(\int a _ {1} (x) \mathrm {d} P _ {1} (x), \dots , \int a _ {n} (x) \mathrm {d} P _ {n} (x)\right). \tag {9}
$$
The collection of all  $\pmb{v}(\pmb{f})$  is denoted by  $\mathcal{B}(P_{\mathrm{X|A}},\mathcal{A})$  or  $\mathcal{B}(P,\mathcal{A})$ .
For a vector  $\pmb{\lambda} \in \Delta_{n}$  such that  $\pmb{\lambda} > 0$ , we define a function  $p_{\pmb{\lambda}}(x): \mathcal{X} \to \Delta_{n}$ :
$$
\boldsymbol {p} _ {\boldsymbol {\lambda}} (x) = \left(\frac {\lambda_ {1} \mathrm {d} P _ {1}}{\lambda_ {1} \mathrm {d} P _ {1} + \cdots + \lambda_ {n} \mathrm {d} P _ {n}}, \dots , \frac {\lambda_ {n} \mathrm {d} P _ {n}}{\lambda_ {1} \mathrm {d} P _ {1} + \cdots + \lambda_ {n} \mathrm {d} P _ {n}}\right). \tag {10}
$$