Title: Demographic Parity Constrained Minimax Optimal Regression under Linear Model
Venue: NeurIPS 2023
Accepted: yes
Keywords: demographic parity

regression

minimax optimal
Abstract: We explore the minimax optimal error associated with a demographic parity-constrained regression problem within the context of a linear model. Our proposed model encompasses a broader range of discriminatory bias sources compared to the model presented by Chzhen and Schreuder. Our analysis reveals that the minimax optimal error for the demographic parity-constrained regression problem under our model is characterized by $\Theta(\frac{dM}{n})$, where $n$ denotes the sample size, $d$ represents the dimensionality, and $M$ signifies the number of demographic groups arising from sensitive attributes. Moreover, we demonstrate that the minimax error increases in conjunction with a larger bias present in the model.

Demographic Parity Constrained Minimax Optimal Regression under Linear Model

Abstract

We explore the minimax optimal error associated with a demographic parityconstrained regression problem within the context of a linear model. Our proposed model encompasses a broader range of discriminatory bias sources compared to the model presented by Chzhen and Schreuder [6]. Our analysis reveals that the minimax optimal error for the demographic parity-constrained regression problem under our model is characterized by Θ( dM / n ) , where n denotes the sample size, d represents the dimensionality, and M signifies the number of demographic groups arising from sensitive attributes. Moreover, we demonstrate that the minimax error increases in conjunction with a larger bias present in the model.

1 Introduction

Machine learning techniques have been incorporated into numerous automated decision-making systems, spanning critical domains such as employment, credit assessment, insurance, and security. Nevertheless, these systems can exhibit discriminatory behavior towards specific demographic groups, including gender, race, and ethnicity, potentially causing significant societal ramifications. This issue, known as the fairness problem, has attracted substantial attention within the machine learning research community. The growing focus on the fairness problem primarily arises from reported instances of unfair behavior in real-world systems, encompassing recidivism risk prediction [3], hiring practices [10], facial recognition [9, 17], and credit scoring [24].

Motivated by these concerns, a considerable body of research has explored regression problems subject to fairness constraints [13, 25, 16, 2, 7, 8, 14, 18]. Numerous regression algorithms incorporating various fairness constraints have been developed to accommodate diverse contexts, with demographic parity [21] and equalized odds [11] being the predominant fairness constraints adopted by these methods.

In this study, we focus on the regression problem under the fairness constraint of demographic parity [21]. Existing literature primarily concentrates on the development of fair regression algorithms, and their performance evaluation predominantly relies on empirical analyses. Such evaluations, however, only offer performance guarantees for specific scenarios explored in the experiments, which may result in poor performance in unexamined situations. To ensure the algorithm's robust performance across a wider range of contexts and obtain a comprehensive understanding of the fair regression problem, a theoretical analysis of statistical efficiency is indispensable.

Several studies have introduced fair regression algorithms accompanied by theoretical analyses of their statistical efficiency in terms of accuracy and fairness. Agarwal et al. [2] designed a demographic parity-based fair regression algorithm using reduction methods [1] and established upper bounds on its empirical approximation errors for accuracy and fairness using Rademacher complexity. Chzhen

et al. [8] proposed a discretization-based fair regression algorithm, deriving upper bounds on the mean squared excess risk for accuracy and a Kolmogorov distance-based score for demographic parity as fairness guarantees. Chzhen et al. [7] derived the Bayes optimal regressor under a demographic parity constraint, providing upper bounds on the mean absolute deviation from the Bayes optimal regressor for accuracy and a Kolmogorov distance-based score for demographic parity as fairness guarantees. Despite ensuring low error and fair treatment even in non-linear models, it remains unclear if these guarantees represent optimal performance among possible algorithms.

Minimax optimal fair regression. Numerous researchers have investigated minimax optimal regression algorithms, the best possible algorithm, without addressing fairness considerations [22, 23, 19, 15]. In contrast to standard regression problems, minimax optimality in fair regression problems remains relatively unexplored, with a notable exception being the recent work by Chzhen and Schreuder [6]. They examine minimax optimality in fair regression problems, incorporating demographic parity constraints within the following linear model:

In this model, Y , X , and S represent the outcome, non-sensitive features, and a sensitive attribute, respectively. ⟨· , ·⟩ denotes the inner product, ξ represents zero-mean noise, and Σ is an arbitrary covariance matrix. For example, in salary calculations, X and S correspond to working hours and gender, respectively, with b S and β ∗ signifying the base salary and hourly wage. In Eq (1), the salary Y is determined by the base salary b S and the product of working hours X and an hourly wage β ∗ .

The model in Eq (1) exhibits limitations pertaining to its applicability across various scenarios. We elucidate these limitations by discussing the notion of direct discrimination and indirect discrimination , summarized succinctly in the second row of Table 1. Direct discrimination occurs when the sensitive attribute influences the outcome, regardless of non-sensitive features. The model in Eq (1) can treat direct discrimination resulting from the dependency of the intercept b S on S ; for example, it can capture discrimination due to basing base salary on gender (third column in Table 1). However, it is imperative to underscore that the model in Eq (1) fails to handle direct discrimination arising from the partial (regression) coefficients β ∗ , as these are independent of S ; for instance, it cannot accommodate discrimination due to gender-dependent hourly wages (second column in Table 1).

Indirect discrimination (or redlining effect [4]) constitutes another source of unfair bias, arising when the sensitive attribute influences the outcome through its correlation with non-sensitive features. The presence of the dependency between non-sensitive features and the sensitive attribute signifies indirect discrimination. In the model in Eq (1), non-sensitive features X is independet from the sensitive attribute S , thereby implying an absence of indirect discrimination (forth column in Table 1).

Chzhen and Schreuder [6] effectively revealed the minimax optimal error for fair regression problems involving direct discrimination due to varying intercepts associated with sensitive attributes. However, their research does not address direct discrimination from partial coefficients and indirect discrimination through non-sensitive features.

Our model and contributions. In this study, we investigate the minimax optimality of the fair regression problem in the context of the following model:

where σ X > 0 , and I denotes the identity matrix. The subscript in β ∗ S and µ S signifies that our model varies regression coefficients and the mean of non-sensitive features based on the sensitive attribute.

Compared to the model proposed by Chzhen and Schreuder [6], our model accommodates a broader range of direct and indirect discrimination. These discriminations can be characterized as follows:

(Direct discrimination) Our model accommodates direct discrimination through discrepancies in β ∗ S concerning S , as the regression coefficients β ∗ S hinge on the sensitive attribute S (second and third columns on the third row in Table 1). This includes, for instance, discrimination arising from varying base salaries and hourly wages. Divergent partial coefficients yield varied outcome variance amongst S , while disparate intercepts relative to S merely alter the outcome's mean. Hence, our model introduces an additional challenge of attenuating direct discrimination through disparate variance, alongside mitigating direct discrimination through disparate mean. This presents a stark contrast to Chzhen and Schreuder [6]'s model, which solely focuses on mitigating discrimination via the mean without considering the variance.

(Indirect discrimination) The sensitive attribute S affects the mean of non-sensitive features X , as denoted by the subscript of µ S . Our model thereby introduces indirect discrimination through variations in µ S with respect to S (e.g., disparate working hours by gender). To alleviate this form of indirect discrimination, µ S needs to be estimated to adjust the learned regressor, thereby ensuring its output remains invariant to differing µ S . Therefore, our model presents an additional complexity in estimating µ S for mitigating indirect discrimination.

Overall, our model demonstrates an expanded dependency of partial coefficients (direct discrimination) and non-sensitive features (indirect discrimination) on the sensitive attribute (second and fourth columns of Table 1).

The principal contribution of this paper lies in the establishment of matching upper and lower bounds on the minimax optimal error (i.e., the error corresponding to the minimax optimal regression algorithm) and the proposition of a regression algorithm that achieves this optimal error under Eq (2). The optimal error elucidates several insights:

(Direct discrimination) The optimal error comprises a term reflecting the outcome's variance heterogeneity but excludes that of the outcome's mean. This insight implies that mitigating direct discrimination due to the outcome's variance sacrifices statistical efficiency, whereas addressing direct discrimination due to the outcome's mean does not entail this cost. This term effectively quantifies the cost of mitigating direct discrimination in variance and is absent from the optimal error of the Chzhen and Schreuder [6]'s model. Its identification, thus, signifies a crucial contribution of our research.

(Indirect discrimination) Our lower bound is independent of the term associated with indirect discrimination. Although this evidence is not definitive, it hints at the potential for mitigating indirect discrimination without additional costs under certain conditions. This observation sets the stage for future research focused on developing cost-effective strategies to tackle indirect discrimination.

Our technical contributions to establish these bounds are detailed in Section 4.

Notations. Given a positive integer m , define [ m ] = { 1 , ..., m } . For a finite set A , denote its cardinality by | A | . Given an event E , its complement is represented as E c , and its probability is denoted by P {E} . For a random variable X , its expectation is E [ X ] , and its associated sigma-algebra is σ ( X ) . For two real values a and b , the notations a ∨ b = max { a, b } and a ∧ b = min { a, b } are used. For a square matrix A ∈ R d × d , its maximum and minimum eigenvalues are denoted by λ max ( A ) and λ min ( A ) , respectively, and its transpose is represented by A ⊤ . The set of unit vectors is given by S d -1 . For a sequence a t indexed by t ∈ T , the notation a · denotes the sequence ( a t ) t ∈T .

2 Problem Setup

2.1 Model and Learning Algorithm

Model. The proposed model, described in the introduction, is formulated according to Eq (2). We consider X ∈ R d and S ∈ [ M ] where M ≥ 2 . The noise variable, ξ , is assumed to follow a Gaussian distribution with zero mean and variance σ 2 ξ > 0 . We define p s = P { S = s } for all s ∈ [ M ] , and the optimal regression function is denoted as f ∗ ( x, s ) = ⟨ β ∗ s , x ⟩ .

Learning algorithm. Given n i.i.d. copies of the tuple ( X,S,Y ) , denoted as D n = { ( X 1 , S 1 , Y 1 ) , ..., ( X n , S n , Y n ) } , the goal is to construct a regression function f that maps ( X,S ) to Y , represented as ˆ f n . The learner seeks to optimize the accuracy of ˆ f n while satisfying a fairness constraint. The definitions of fairness and accuracy are provided in subsequent subsections.

2.2 Fairness

Demographic parity. We utilize demographic parity [21] as our fairness criterion. A regressor f adheres to demographic parity if its output distribution is invariant when conditioned on S = s .

Definition 1. A regressor f satisfies (strong) demographic parity if, for all s, s ′ ∈ [ M ] , and for all E ∈ σ ( f ( X,S )) , P { f ( X,S ) ∈ E | S = s } = P { f ( X,S ) ∈ E | S = s ′ } .

Denote the set of all regressors fulfilling demographic parity for a given distribution of X , parameterized by µ · , as F DP ( µ · ) .

Fairness consistency. Instead of enforcing strict demographic parity (Definition 1), which results in the regressor to be a constant function due to the unknown ( X,S ) distribution, we introduce fairness consistency (Definition 2). This concept demands the learned regressor to converge to a fair regressor as the sample size n approaches infinity.

To define 'convergence', we introduce the unfairness score U ( f ) ≥ 0 , where a lower U ( f ) indicates a higher fairness level. U ( f ) = 0 if and only if f achieves demographic parity (Definition 1). We claim the learned regressor ˆ f n converges to an exactly fair regressor when U ( ˆ f n ) → 0 as n →∞ .

Definition 2. A learning algorithm is ( α, δ ) -consistently fair for an unfairness score U if there exist constants n 0 ≥ 0 and C > 0 , independent of n , such that P { U ( ˆ f n ) > Cn -α } ≤ δ for all n ≥ n 0 , with randomness arising from the training sample via ˆ f n .

Note that an ( α, δ ) -consistently fair regressor ˆ f n exhibits ( α ′ , δ ) -consistent fairness for any α ′ ∈ (0 , α ] .

We adopt a specific unfairness score using the Wasserstein distance. Given two probability measures ν and ν ′ over R , Π( ν, ν ′ ) denotes the set of all coupling measures π satisfying π ( A × R ) = ν ( A ) and π ( R × A ′ ) = ν ′ ( A ′ ) for every measurable sets A,A ′ ⊂ R . The 2-Wasserstein distance W 2 between ν and ν ′ is expressed as W 2 2 ( ν, ν ′ ) = inf π ∈ Π( ν,ν ′ ) ∫ ( z -z ′ ) 2 π ( dz, dz ′ ) . Our unfairness score is then formulated as:

where ν f | s represents the distribution of f ( X,S ) conditioned on S = s . Prior works, including [2, 7, 8, 6], have adopted different unfairness scores (see the appendix for details 2 ).

2.3 Accuracy

Under the fairness consistency constraint, the learner's objective is to obtain a fair approximation of f ∗ , denoted as f ∗ DP , which is the closest regressor to f ∗ within F DP ( µ · ) using the L 2 distance:

To evaluate the inaccuracy of a regressor f , we compute the mean squared deviation from f ∗ DP :

Chzhen et al. [7, 8] employ similar definitions, differing only in the choice of deviation metric.

This paper aims to identify the minimax optimal regression algorithm, which minimizes Eq (3) while maintaining fairness consistency. Given parameters α > 0 and δ ∈ (0 , 1) , the optimal error is formulated as:

where the infimum is taken over all ( α, δ ) -consistently fair algorithms, and B and M represent the sets of possible β ∗ · and µ · , respectively.

3 Main Results

Our main result is to establish the minimax optimal error bound, delineating the dependency on the diversity of conditional outcome variances concerning the sensitive attribute. This diversity of the variances is quantified via a parameter B > 0 , which is defined such that it satisfies:

The left-hand side of the second inequality in Eq (4) forms as a product of two factors: the weighted average norms, ( ∑ s p s ∥ β ∗ s ∥ ) 2 , and the averaged inverse norms, 1 M ∑ s 1 ∥ β ∗ s ∥ 2 . As the norms increase, the first factor (weighted average norms) has the propensity to grow, while the second factor (averaged inverse norms) tends to rise when the norms decrease. Maximizing the product of these two elements involves a delicate balancing act: the norms of some groups need to be large, while the norms of other groups need to be smaller. As such, the left-hand side of the second inequality in Eq (4) can increase when the norms ∥ β ∗ s ∥ display diversity.

We adopt mild assumptions on β ∗ · and µ · . Let B denote the set of β ∗ · satisfying Eq (4). Assume there exists a finite universal constant U > 0 such that ∥ µ s ∥ ≤ U for all s ∈ [ M ] , leading to M = { µ · ∈ R d × M : ∀ s ∈ [ M ] , ∥ µ s ∥ ≤ U } . Our analysis relies on these assumptions.

Our main results are as follows:

Theorem 1. Given α ∈ (0 , 1 / 2 ] and δ ∈ (0 , 1) , suppose M ( d -1) > 16 and n ≥ 12(3 d ∨ 4 ln( M/δ )) / min s ∈ [ M ] p s . Then, there exist universal constants C > 0 and c > 0 such that

Theorem 1 illustrates that the optimal error is σ 2 ξB 2 dM / n up to a constant factor which may potentially depend on σ X and U . The implications of Theorem 1 can be summarized as follows:

The optimal error for the standard linear regression problem can be denoted as d / n [15]. The dependency on n and d is consistent with the standard case, provided α ∈ (0 , 1 / 2 ] .

The term dM denotes the number of unknown parameters in Eq (2), comprising β ∗ 1 , .., β ∗ M ∈ R d and µ 1 , ..., µ M ∈ R d . This dependency on the number of unknown parameters is a common characteristics observed in statistical estimation problems.

(Direct discrimination) The minimax error delineated in Theorem 1 demonstrates a dependency on parameter B . As the variation of ∥ β ∗ s ∥ with respect to s increases, so does the magnitude of B . Hence, B serves as a measure of the difficulty in mitigating direct discrimination due to the outcome's variance. This unique quantification of difficulty is absent in standard regression problems and specific to fair regression problems.

(Indirect discrimination) The lower bound precludes parameters associated with indirect discrimination. It is conceivable that biases arising from indirect discrimination can be reduced without extra costs, provided the dependence of X on S exists only in its mean. Investigating and clarifying this aspect offers a promising direction for future research.

The minimax error is invariant to α and δ , implying that the learning process does not introduce unfair bias for α ∈ (0 , 1 / 2 ] . However, the case for α ≥ 1 / 2 remains unexplored and poses a significant research challenge.

The gap between the upper and lower bounds regarding σ X and U remains, making narrowing this gap an essential future research direction.

Remark 1. Direct comparison of the minimax error between our model and that of Eq (1) is not feasible due to the differing f ∗ DP across the models. However, the emergence of the fairness-specific term B can be unequivocally identified as a novel contribution in our study. Notably, the minimax error validated by Chzhen and Schreuder [6] is congruent with the minimax optimal error of standard linear regression within their model, a contrast to our findings.

To prove Theorem 1, we initiate by constructing the estimator detailed in Section 5. We then prove in Section 6 that the estimator satisfies 1) ( α, δ ) -fairness consistency for α ∈ (0 , 1 / 2 ] , and 2) the error aligns with the upper bound specified in Theorem 1. Subsequently, we present a sketch of the proof for the lower bound in Theorem 1 in Section 7. All omitted proofs can be found in the appendices.

4 Technical Difficulties in Minimax Optimality Analyses

In this section, we expound on the challenges arising from the analysis of minimax optimality for our problem. First, we introduce the closed-form expression for the Bayes optimal fair regressor f ∗ DP . We then outline the technical difficulties encountered during the analysis.

Bayes optimal fair regressor under Eq (2). Chzhen et al. [7] present a characterization of regression error and the corresponding regressor minimizing the mean squared error under the demographic parity constraint. Building upon the results from Chzhen et al. [7], we derive the closed-form expression for f ∗ DP in the following lemma.

Lemma 1. Given the model in Eq (2) , the Bayes optimal regressor adhering to the demographic parity constraint can be formulated as

Technical difficulty in deriving the upper bound in Theorem 1. To obtain the upper bound in Theorem 1, we first construct an estimator for the regression function in Eq (5) and analyze its regression error. This entails developing estimators for individual components in Eq (5) (e.g., ∥ β ∗ · ∥ , β ∗ s / ∥ β ∗ s ∥ , µ s , etc.) and substituting them into Eq (5). The upper bound on E n ( α, δ ) is derived by combining estimation error bounds for each component's estimator. However, to our best knowledge, no existing estimators provide bounds for the norm ( ∥ β ∗ · ∥ ) and direction ( β ∗ s / ∥ β ∗ s ∥ ) of regression coefficients. A direct approach involves computing the norm and direction of the OLS estimator, but standard analyses for OLS do not yield bounds on the estimation errors.

The main challenge in deriving the upper bound of Theorem 1 lies in analyzing the following problem: given X following a non-isotropic Gaussian distribution with mean µ , find upper bounds on E [( X/ ∥ X ∥ -µ/ ∥ µ ∥ ) 2 ] and E [( ∥ X ∥ - ∥ µ ∥ ) 2 ] . Solving this problem provides estimation errors for the norm and direction estimators, as the OLS estimator is an unbiased estimator with noise following the non-isotropic Gaussian distribution. Our key technical contribution is the derivation of these bounds (Theorems 4 and 5).

Technical difficulty in deriving the lower bound in Theorem 1. The minimax optimal error characterizes the intrinsic complexity of the regression problem, as no algorithm can surpass this error. In our analysis of the lower bound presented in Theorem 1, we demonstrate that the fair regression problem's complexity, under the model Eq (2), is characterized by the complexity in estimating the direction β ∗ s / ∥ β ∗ s ∥ . The primary challenge lies in establishing this characterization.

To overcome this challenge, we investigate the geometric structure of the error term E n ( f ; β ∗ · , µ · ) concerning the parameters β ∗ · and µ · . We then reveal that the geometric structure of E n ( f ; β ∗ · , µ · ) is characterized by the geometric structure of the direction β ∗ s / ∥ β ∗ s ∥ (Theorem 7).

5 Estimator

In this section, we present a detailed construction of the estimators that attain the minimax error as delineated in Theorem 1. Existing theoretical results, such as those found in Agarwal et al. [2], Chzhen et al. [7, 8], are incapable of addressing unbounded non-sensitive features X or unbounded outcomes Y , rendering them inapplicable to our problem. Consequently, we have developed a novel estimator accompanied by rigorous analytical techniques.

Estimator construction. In constructing the optimal regressor for model Eq (2), we leverage the results from Lemma 1 and employ a plugin estimator. The method involves estimating the components of terms in Eq (5) and substituting the obtained estimates into the same equation. Concretely, we derive estimators ̂ ∥ β · ∥ , ˜ β s , ˆ µ s , ˆ p s , ˆ β ′ s , and ˆ µ ′ s , with the following correspondence:

For technical reasons, we partition the sample to calculate each estimand. Each estimator is assigned a corresponding subset, as shown in Table 2. Under specific conditions, n s > 18 d or n s > 12 d , estimators may exhibit altered behavior, primarily as technical considerations for subsequent analyses. We detail the partitioning process as follows. First, we create a histogram of the sensitive attribute S i , denoted as n · = ( n 1 , ..., n M ) , with n s = |{ i ∈ [ n ] : S i = s }| . Simultaneously, we form group-wise samples D s = { ( X i , Y i ) : i ∈ [ n ] , S i = s } . For each s ∈ [ M ] , we partition D s into D 1 ,s , D 2 ,s , and D 3 ,s , ensuring | D b,s | := n b,s ≥ ⌊ n s / 3 ⌋ for b ∈ [3] . Using n · , D 1 ,s , D 2 ,s , and D 3 ,s , we estimate ˆ p s , ̂ ∥ β s ∥ , ˜ β s , and ˆ µ s , respectively. The combination of ˆ p s and ̂ ∥ β s ∥ yields ̂ ∥ β · ∥ . Furthermore, we generate a duplicate of D s , denoted as D ′ s , and partition it into D ′ 1 ,s and D ′ 2 ,s , satisfying | D ′ b,s | := n ′ b,s ≥ ⌊ n s / 2 ⌋ for b ∈ [2] . We then use D ′ 1 ,s and D ′ 2 ,s to estimate ˆ β ′ s and ˆ µ ′ s . Precise definitions of the estimator construction and subset partitioning can be found in the appendices.

Incorporating the derived estimators, we construct the final regressor as:

6 Upper Bound Analyses

In this section, we demonstrate the achievability of the upper bound presented in Theorem 1 utilizing the estimator delineated in Section 5. Initially, we conduct an analysis of the estimator's fairness guarantee, subsequently progressing to an examination of the estimator's mean squared deviation.

6.1 Analysis of Fairness

For our fairness guarantee on ˆ f n , we demonstrate the following theorem.

Theorem 2. If n ≥ 48 ln( M/δ ) / min s p s , we have for δ ∈ (0 , 1) ,

By proving Theorem 2, we can immediately confirm that the estimator adheres to ( α, δ ) -fairness consistency with α ∈ (0 , 1 / 2 ] .

6.2 Analysis of Estimation Error

In this subsection, we derive an upper bound for the estimation error presented in Theorem 1, focusing on the estimator introduced in Section 5. To derive the upper bound in Theorem 1, we begin by decomposing the mean squared deviation of the estimator in Eq (6) as follows:

Theorem 3. For the estimator defined in Eq (6) , the mean square deviation from f ∗ DP is bounded above by

In Eq (7), the terms correspond to the estimation errors of ˆ µ s , ̂ ∥ β · ∥ , ˜ β s , ˆ β ′ s , ˆ µ ′ s , and ˆ p s , respectively. Standard techniques for the OLS estimator and empirical average yield upper bounds for the first, fourth, fifth, and sixth terms. Nevertheless, the second and third terms in Eq (7) involve non-linear transformations of the OLS estimator (i.e., taking the norm or dividing by the norm), complicating their error analysis. This section's primary technical contributions involve establishing tight upper bounds for the second and third terms in Eq (7).

Estimation error of norm and direction of β ∗ s . Consider X 1 , ..., X n iid ∼ N ( µ, σ 2 X I ) , β ∗ ∈ R d with ∥ β ∗ ∥ ≤ B for some B > 0 , and ξ 1 , ..., ξ n iid ∼ N (0 , σ 2 ξ ) . Define Y i = ⟨ β ∗ , X i ⟩ + ξ i . The OLS estimator of β ∗ is given by ˆ β = ( 1 n X ⊤ X ) -1 ( 1 n X ⊤ Y ) , where X = ( X 1 · · · X n ) ⊤ and Y = ( Y 1 · · · Y n ) ⊤ . The direction estimator is ˆ β/ ∥ ˆ β ∥ , while the norm estimator is ∥ ˆ β ∥ .

We present the estimation errors for direction and norm in Theorems 4 and 5:

Theorem 4. For n > 6 d , we have

Theorem 5. For n > 6 d , we have

The direction's estimation error (Theorem 4) is O ( σ 2 ξ d / σ 2 X ∥ β ∗ ∥ 2 n ) , while the norm's estimation error (Theorem 5) is O ( B 2 σ 2 ξ d / σ 2 X n ) . Integrating Theorems 3 to 5 yields the σ 2 ξ B 2 dM / n term in the upper bound in Theorem 1. The remaining part, Uσ 2 ξ / n , arises from the estimation error of ˆ β ′ s (the third term in Eq (7)), dominating other terms in Eq (7).

7 Lower Bound Analyses

In this section, we provide a proof sketch for the lower bound, outlined in Theorem 1. To facilitate a clear and concise presentation of the proof sketch, we introduce several notations. Let θ denote the tuple of distribution parameters ( β · , µ · ) , and let Θ represent the set of all such parameters, defined as Θ = B × M . We use P θ and E θ to denote the probability and expectation operators, respectively, given X ∼ N ( µ S , σ 2 X I ) and Y = ( β S , X ) + ξ , where ξ ∼ N (0 , σ 2 ξ ) . We adopt the shorthand E ( f ; θ ) = E ( f ; β · , µ · ) for θ = ( β · , µ · ) . Moreover, we define f θ = arg min f ∈F DP R ( f ; β · , µ · ) for θ = ( β · , µ · ) . For two probability distributions π and π ′ , the Kullback-Leibler (KL) divergence is denoted as D KL ( π, π ′ ) = ∫ ln( dπ dπ ′ ( z )) π ( dz ) . Finally, we denote the set of all L 2 integrable functions f : R d × [ M ] → R as L 2 .

By utilizing Fano's inequality, we establish a lower bound for the minimax error as presented in Theorem 1. Due to the invariance of the distribution of S 1 , ..., S n under parameter alterations θ ,

Fano's inequality can be applied after conditioning on S 1 , ..., S n , or equivalently, n · . Consequently, we derive the following theorem:

Theorem 6. Let ˆ Θ ⊆ Θ be a finite set of the parameters such that there exists ϵ > 0 such that for any θ, θ ′ ∈ ˆ Θ , inf f E ( f ; θ ) ∨E ( f ; θ ′ ) ≥ ϵ , where ˆ Θ and ϵ is possibly dependent on n · . Let | ˆ Θ | = K . Then, for arbitrary α > 0 and δ ∈ (0 , 1) , we have

where π θ | n · denotes the distribution of D n conditioned on n · with parameter θ , and the expectation is taken over n · .

As demonstrated in Theorem 6, the lower bound for the minimax error can be obtained by constructing ˆ Θ such that: 1) inf f E ( f ; θ ) ∨E ( f ; θ ′ ) ≥ ϵ for any θ, θ ′ ∈ ˆ Θ , and 2) inf π 1 K ∑ θ ∈ ˆ Θ D KL ( π θ | n · , π ) ≤ ln( K/ 4) / 2 . With the construction of such a ˆ Θ , a lower bound of E [ ϵ 2 ] is attained.

We present a theorem that establishes a tight lower bound on inf f E ( f ; θ ) ∨ E ( f ; θ ′ ) .

Theorem 7. Let θ and θ ′ be the parameters of the distributions such that 1 2 σ 2 X ∥ µ s -µ ′ s ∥ 2 := d s < 1 for all s ∈ [ M ] . Then, we have

The term ∥ ∥ β · ∥ β s / ∥ β s ∥ -∥ β ′ · ∥ β ′ s / ∥ β ′ s ∥ ∥ 2 characterizes the lower bound, which is different from the characteristic term in standard linear regression, ∥ β s -β ′ s ∥ 2 .

Wenext present the construction of ˆ Θ . Weconstruct ˆ Θ such that each of its elements corresponds to an index from the set V = {-1 , 1 } M × ( d -1) , denoted by θ v = { β v, · , µ v, · } ∈ ˆ Θ , where β v,s is controlled such that its norm is equivalent to a specified value B s , i.e., ∥ β v,s ∥ = B s . This construction ensures that ˆ Θ ⊂ B × M . Given positive values ϵ 1 , ..., ϵ M and B 1 , ..., B M , we construct ˆ Θ as follows:

We demonstrate the following properties for ˆ Θ defined in Eq (8).

Theorem 8. Given ϵ 1 , ..., ϵ M > 0 and B 1 , ..., B M > 0 , let ˆ Θ ⊂ Θ represent the set of parameters defined in Eq (8) . Let π θ | n · be the distribution of the sample D n conditioned on n · with the distribution parameter θ . Then, we have 1) for any v, v ′ ∈ V ,

and 2) for v, v ′ ∈ V ,

By integrating Theorems 6 to 8 and employing the renowned Varshamov-Gilbert bound, we derive the lower bound in Theorem 1.

8 Conclusion

This paper investigates a regression problem with ( α, δ ) -fairness consistency as a fairness constraint. Specifically, we demonstrate that, under the constraint of ( α, δ ) -fairness, the minimax optimal error

scales as σ 2 ξ B 2 dM / n up to a constant factor, when α ∈ (0 , 1 / 2 ] . Additionally, we provide the fair regressor that achieves this optimal error.

Potential negative societal impacts. Our study aims to mitigate the negative impact of regression models on social groups, rather than to cause harm. However, our results are only valid for linear models, as defined in Eq (2). Misapplication of our findings to other models may result in discriminatory treatment, which should be avoided.

Acknowledgments and Disclosure of Funding

This work was partly supported by JSPS KAKENHI Grant Numbers JP23K13011 and JP23H00483.

References

Alekh Agarwal, Alina Beygelzimer, Miroslav Dudík, John Langford, and Hanna Wallach. A Reductions Approach to Fair Classification. In the 35th International Conference on Machine Learning , pages 60-69, 2018.

Alekh Agarwal, Miroslav Dudík, and Zhiwei Steven Wu. Fair Regression: Quantitative Definitions and Reduction-based Algorithms. In 36th International Conference on Machine Learning, ICML 2019 , volume 2019-June, pages 120-129. International Machine Learning Society (IMLS), 2019.

Julia Angwin, Jeff Larson, Surya Mattu, and Lauren Kirchner. Machine Bias, 2016. URL https://www.propublica.org/article/machine-bias-risk-assessments-incriminal-sentencing .

Toon Calders and Sicco Verwer. Three naive Bayes approaches for discrimination-free classification. Data Mining and Knowledge Discovery , 21(2):277-292, 2010.

M. T. Chao and W. E. Strawderman. Negative Moments of Positive Random Variables. Journal of the American Statistical Association , 67(338):429, 6 1972. ISSN 01621459. doi: 10.2307/2284399.

Evgenii Chzhen and Nicolas Schreuder. A minimax framework for quantifying risk-fairness trade-off in regression. The Annals of Statistics , 50(4):2416-2442, 8 2022. ISSN 0090-5364. doi: 10.1214/22-AOS2198.

Evgenii Chzhen, Christophe Denis, Mohamed Hebiri, and Massimiliano Pontil. Fair Regression with Wasserstein Barycenters. In Advances in Neural Information Processing Systems , volume 33, pages 7321-7331. Curran Associates, Inc., 2020.

Evgenii Chzhen, Christophe Denis, Mohamed Hebiri, and Massimiliano Pontil. Fair Regression via Plug-In Estimator and Recalibration. In Advances in Neural Information Processing Systems , volume 33, pages 19137-19148. Curran Associates, Inc., 2020.

Kade Crockford. How is Face Recognition Surveillance Technology Racist?, 2020. URL https://www.aclu.org/news/privacy-technology/how-is-face-recognitionsurveillance-technology-racist/ .

Jeffrey Dastin. Amazon scraps secret AI recruiting tool that showed bias against women, 2018. URL https://www.reuters.com/article/us-amazon-com-jobs-automationinsight/amazon-scraps-secret-ai-recruiting-tool-that-showed-biasagainst-women-idUSKCN1MK08G .

Moritz Hardt, Eric Price, and Nathan Srebro. Equality of Opportunity in Supervised Learning. In D D Lee, M Sugiyama, U V Luxburg, I Guyon, and R Garnett, editors, Advances in Neural Information Processing Systems 29 , pages 3315-3323, Barcelona, Spain, 12 2016. Curran Associates, Inc.

Dragana Jankov Maširevi´ c. On New Formulas for the Cumulative Distribution Function of the Noncentral Chi-Square Distribution. Mediterranean Journal of Mathematics , 14(2):1-13, 4 2017. ISSN 16605454. doi: 10.1007/S00009-017-0874-1/METRICS. URL https://link.springer.com/article/10.1007/s00009-017-0874-1 .

Junpei Komiyama, Akiko Takeda, Junya Honda, and Hajime Shimao. Nonconvex Optimization for Regression with Fairness Constraints. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning , volume 80 of Proceedings of Machine Learning Research , pages 2737-2746, Stockholmsmässan, Stockholm Sweden, 2018. PMLR.

Jeremie Mary, Clément Calauzènes, and Noureddine El Karoui. Fairness-Aware Learning for Continuous Attributes and Treatments. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning , volume 97 of Proceedings of Machine Learning Research , pages 4382-4391, Long Beach, California, USA, 2019. PMLR.

Jaouad Mourtada. Exact minimax risk for linear least squares, and the lower tail of sample covariance matrices. The Annals of Statistics , 50(4):2157-2178, 8 2022. ISSN 0090-5364. doi: 10.1214/22-AOS2181. URL https://projecteuclid.org/journals/annals-ofstatistics/volume-50/issue-4/Exact-minimax-risk-for-linear-leastsquares-and-the-lower/10.1214/22-AOS2181.full .

Daniel Moyer, Shuyang Gao, Rob Brekelmans, Greg Ver Steeg, and Aram Galstyan. Invariant Representations without Adversarial Training. In Advances in Neural Information Processing Systems , volume 31, pages 9084-9093, 2018.

Alex Najibi. Racial Discrimination in Face Recognition Technology, 2020. URL https://sitn.hms.harvard.edu/flash/2020/racial-discrimination-in-facerecognition-technology/ .

Harikrishna Narasimhan, Andrew Cotter, Maya R Gupta, and Serena Wang. Pairwise Fairness for Ranking and Regression. In The Thirty-Fourth AAAI Conference on Artificial Intelligence , pages 5248-5255. AAAI Press, 2020.

Roberto Imbuzeiro Oliveira. The lower tail of random quadratic forms with applications to ordinary least squares. Probability Theory and Related Fields , 166(3-4):1175-1194, 12 2016. ISSN 01788051. doi: 10.1007/S00440-016-0738-9/METRICS.

I. Olkin and F. Pukelsheim. The distance between two random vectors with given dispersion matrices. Linear Algebra and its Applications , 48(C):257-263, 12 1982. ISSN 0024-3795. doi: 10.1016/0024-3795(82)90112-4.

Dino Pedreshi, Salvatore Ruggieri, and Franco Turini. Discrimination-aware data mining. In Proceeding of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining - KDD 08 , pages 560-568, 2008. ISBN 9781605581934. doi: 10.1145/1401890.1401959.

Charles J Stone. Optimal rates of convergence for nonparametric estimators. The annals of Statistics , pages 1348-1360, 1980.

Alexandre B. Tsybakov. Optimal rates of aggregation. Lecture Notes in Artificial Intelligence (Subseries of Lecture Notes in Computer Science) , 2777:303-313, 2003. ISSN 03029743. doi: 10.1007/978-3-540-45167-9{\_}23/COVER.

Neil Vigdor. Apple Card Investigated After Gender Discrimination Complaints, 2019. URL https://www.nytimes.com/2019/11/10/business/Apple-credit-cardinvestigation.html .

Qizhe Xie, Zihang Dai, Yulun Du, Eduard Hovy, and Graham Neubig. Controllable Invariance through Adversarial Feature Learning. In Advances in Neural Information Processing Systems , volume 30, pages 585-596, 2017.

A Comparison of Existing and Our Unfairness Scores

This section compares our unfairness score with existing ones. Recall that our unfairness score is defined as the maximum Wasserstein distance between any two distributions ν f | s and ν f | s ′ over all pairs of groups s and s ′ , as follows:

In contrast, Agarwal et al. [2], Chzhen et al. [7, 8] use the Kolmogorov distance D Kol to measure unfairness, which is defined as:

The difference between our score and U Kol ( f ) is solely the choice of distance metric. Our score utilizes the Wasserstein distance, while U Kol ( f ) uses the Kolmogorov distance. This difference arises mainly from technical reasons.

In addition, Chzhen and Schreuder [6] proposed another unfairness score, denoted by U Avg W 2 ( f ) , which is defined as the average of the Wasserstein distance, as follows:

Here, the score places more emphasis on the major groups, as reflected by the weight of p s . This may not be desirable if the unfairness is more prevalent in the minority groups, which may be common in real-world scenarios.

B Estimator Details

This section describes the construction of our optimal estimator in detail. Recall that our estimator is a plugin estimator in which we first estimate the parts of the terms in Eq (5) and then substitute them into Eq (5). Specifically, we construct estimators for ̂ ∥ β · ∥ , ˜ β s , ˆ p s ′ , ˆ β ′ s ′ , and ˆ µ ′ s ′ , where they correspond to the terms in Eq (5) as follows:

For analysis purposes, we split the sample into several subsets and pass each subset to the corresponding estimator (the correspondence is explained later). Figure 1 shows an overview of the sample splitting and the correspondence between the subsets and estimators. We construct the histogram of the sensitive attribute S i from the sample D n , denoted as n · = ( n 1 , ..., n M ) , where n s = |{ i ∈ [ n ] : S i = s }| (upper left in Figure 1). We also construct group-wise samples D s = { ( X i , Y i ) : i ∈ [ n ] , S i = s } . For each s ∈ [ M ] , we divide D s into D 1 ,s , D 2 ,s , and D 3 ,s such that | D b,s | := n b,s ≥ ⌊ n s / 3 ⌋ for b ∈ [3] (lower left in Figure 1). We use n · , D 1 ,s , D 2 ,s , and D 3 ,s to estimate ˆ ps , ̂ ∥ β s ∥ , ˜ β s , and ˆ µ s , respectively, where ̂ ∥ β s ∥ is the estimator for ∥ β ∗ s ∥ . We obtain ̂ ∥ β · ∥ from the combination of ˆ p s and ̂ ∥ β s ∥ (middle in Figure 1). Furthermore, for each s ∈ [ M ] , we create a copy of D s , denoted as D ′ s , and divide it into D ′ 1 , s and D ′ 2 , s such that | D ′ b, s | := n ′ b, s ≥ ⌊ n s / 2 ⌋ for b ∈ [2] . We use D ′ 1 , s and D ′ 2 ,s to estimate ˆ β ′ s and ˆ µ ′ s , respectively (right in Figure 1).

We describe the construction of each estimator below. We first define some notations. Let the i th element of D b,s and D ′ b,s be denoted as ( X b,s,i , Y b,s,i ) and ( X ′ b,s,i , Y ′ b,s,i ) , respectively. We use the matrix notations X b,s = ( X b,s, 1 · · · X b,s,n b,s ) ⊤ , X ′ b,s = ( X ′ b,s, 1 · · · X ′ b,s,n ′ b,s ) ⊤ , Y b,s = ( Y b,s, 1 · · · Y b,s,n b,s ) ⊤ , and Y ′ b,s = ( Y ′ b,s, 1 · · · Y ′ b,s,n ′ b,s ) ⊤ . We define the ordinary least square estimators for the subset of the sample D b,s and D ′ b,s as follows:

We construct each estimator as follows:

( ˆ ps ) We use the empirical mean defined as ˆ p s = n s / n .

( ̂ ∥ β s ∥ ) We use the norm of the OLS estimator. We define ̂ ∥ β s ∥ = ∥ ˆ β 1 ,s ∥ if n s > 18 d , and ̂ ∥ β s ∥ = 0 otherwise.

( ̂ ∥ β · ∥ ) Since ∥ β ∗ · ∥ = ∑ s ∈ [ M ] p s ∥ β ∗ s ∥ , we construct its estimator as ̂ ∥ β · ∥ = ∑ s ∈ [ M ] ˆ p s ̂ ∥ β s ∥ .

( ˜ β s ) We use the normalized ordinary least square estimator; ˜ β s = ˆ β 2 ,s / ∥ ˆ β 2 ,s ∥ if n s > 18 d , and ˜ β s = 0 otherwise.

( ˆ µ s ) We use the empirical mean; ˆ µ s = 1 n 3 ,s ∑ n 3 ,s i =1 X 3 ,s,i .

( ˆ β ′ s ) We employ the ordinary least square estimator; ˆ β ′ s = ˆ β ′ 1 ,s if n s > 12 d , and ˆ β ′ s = 0 otherwise.

( ˆ µ ′ s ) We use the empirical mean; ˆ µ ′ s = 1 n ′ 2 ,s ∑ n ′ 2 ,s i =1 X ′ 2 ,s,i if n s > 12 d , and ˆ µ ′ s = 0 otherwise.

Some estimators change their behavior based on the condition n s > 18 d or n s > 12 d , which is done for technical purposes in later analyses.

Recall that the final regressor is constructed as follows:

Algorithm 1 shows the algorithm for our estimator.

C Bayes Optimal Regressor under Our Modell

This section presents the proof of Lemma 1, demonstrating the Bayes optimal regressor under the model Eq (2). To establish this, we make use of a key result from the work of Chzhen et al. [7]:

Theorem 9 (Chzhen et al. [7]) . Assume, for each s ∈ [ M ] , ν f ∗ | s has a density. Then,

where the infimum is taken over all the regressors that satisfy the demographic parity. Moreover, letting f ∗ DP and ν ∗ be the minimizer of the lhs and rhs, respectively, we have ν f ∗ DP = ν ∗ and

Here, we denote ν f ∗ as the distribution of f ∗ ( X,S ) , F f | s as the cumulative distribution function of f ( X,S ) conditioned on S = s , and F -1 f | s as the inverse cumulative distribution function, given by F -1 f | s ( t ) = inf { y ∈ R | F f | s ( y ) ≥ t } .

Building upon the results of Theorem 9, we establish the proof of Lemma 1.

Proof of Lemma 1. Building upon Theorem 9, we can derive the Bayes optimal regressor under the model Eq (2) by obtaining closed expressions of the cumulative and inverse cumulative distribution functions F f ∗ | s and F -1 f ∗ | s . To obtain these closed forms, we apply certain transformations to f ∗ ( X,S ) that render it a random variable following a standard normal distribution. Let Φ and Φ -1 be the CDF and inverse CDF of the standard normal distribution, respectively. Through elementary calculations, we have:

Here, we can readily observe that 1 σ X ∥ β ∗ s ∥ ⟨ β ∗ s , X -µ s ⟩ follows the standard normal distribution under conditioned on S = s , as X ∼ N ( µ s , σ X I ) conditioned on S = s . Consequently, we have

The inverse function of F -1 f ∗ | s ( t ) can be obtained by equating the right-hand side to p and solving the resulting equation for t , which leads to

By substituting Eqs (10) and (11) into Eq (9) in Theorem 9, we obtain the desired claim.

D Details of Fairness Analysis

In this section, we provide evidence of the guarantee of our estimator's fairness consistency. Specifically, we present the following theorem.

Theorem 10. For any δ ∈ (0 , 1] , the regressor in Eq (6) is ( 1 / 2 , δ ) -consistently fair.

We prove the above claim by utilizing Theorem 2, which is shown in the main body, as follows:

Proof of Theorem 10. We can confirm the claim by comparing the bound obtained in Theorem 2 with the definition of ( 1 2 , δ ) -consistent fairness in Definition 2. In particular, we can set C = 4 Bσ X √ 48 ln( M/δ ) min s ∈ [ M ] p s and n 0 = n ≥ 48 ln( M/δ ) / min s p s to satisfy the definition of ( 1 2 , δ ) -consistent fairness.

Next, we provide the proof of Theorem 2. To this end, we prove the following two theorems:

Theorem 11. Let ˆ f n be the estimator of f ∗ DP defined in Eq (6) . Then, almost surely, we have

Theorem 12. If n ≥ (48 ln( M/δ ) -36 d ) / min s p s , we have for δ ∈ (0 , 1) ,

Combining Theorems 11 and 12 immediately yields Theorem 2.

Proof of Theorem 11. This proof investigates the distribution of ν ˆ fn | s . It is straightforward to verify that, conditioned on S = s , ˆ f n ( X,S ) follows the Gaussian distribution with mean

We can thus evaluate the Wasserstein distance between the distributions ν ˆ fn | s and ν ˆ fn | s ′ using the Wasserstein distance between Gaussian distributions. Given two Gaussian distributions N ( µ, σ 2 ) and N ( µ ′ , σ ′ 2 ) , the 2-Wasserstein distance between them are obtained [20] as

and variance

Therefore, we have

which concludes the claim.

Proof of Theorem 12. We start by deriving the concentration inequality for ⟨ ˜ β s , µ s -ˆ µ s ⟩ conditioned on ˜ β s and n · . Note that ˜ β s = 0 if n s ≤ 18 d . Conditioning on ˜ β s and n · , we observe that ⟨ ˜ β s , µ s -ˆ µ s ⟩ follows a Gaussian distribution with mean zero and variance σ 2 X /n 3 ,s . Therefore, for any s ∈ [ M ] and t > 0 ,

Taking the expectation with respect to ˜ β s and using the fact that n 3 ,s ≥ ⌊ n s / 3 ⌋ ≥ n s / 6 for n s ≥ 6 , we obtain the following inequality for s ∈ [ M ] and t > 0 :

Using the union bound for t > 0 , we have

We now derive a sufficient condition on t such that the expectation of the right-hand side in Eq (12) is less than δ . First, we note that

Taking the expectation and substituting the moment-generating function of the binomial distribution, we obtain

Since 1 -e -x ≥ (1 -e -1 ) x for x ∈ [0 , 1] , if t 2 / 24 σ 2 X ≤ 1 , we have

Hence, E [exp( -( n s +18 d ) t 2 24 σ 2 X )] ≤ δ / M if t ≥ σ X √ 24 ln( M/δ ) (1 -e -1 ) np s +18 d ≤ σ X √ 48 ln( M/δ ) min s np s +36 d because (1 -e -1 ) ≥ 1 / 2 . To ensure t 2 / 24 σ 2 X ≤ 1 , we require n ≥ (48 ln( M/δ ) -36 d ) / min s p s .

E Proofs for Norm and Direction Estimators

This section presents the proofs for Theorem 4 and Theorem 5. Our strategy for proving these theorems is to use the hyperellipsoid to interpret the distribution of the OLS estimator. Specifically, we begin by defining Σ n = 1 n X ⊤ X and expressing the OLS estimator ˆ β as

where ξ follows a zero-mean Gaussian distribution. Eq (13) shows that, conditioned on X , ˆ β follows a multivariate Gaussian distribution with mean β ∗ and covariance matrix σ 2 ξ n Σ -1 n . We establish that, under the condition ∥ ˆ β ∥ = r , ˆ β is supported on a hyperellipsoid E ( r, β , n σ 2 ξ Σ n ) , where E ( r, c, A ) = { x ∈ R d : ( x -c ) ⊤ A ( x -c ) ≤ r } denotes the hyperellipsoid with r > 0 , c ∈ R d , and a symmetric and positive-definite matrix A ∈ R d × d .

To prove Theorem 4 and Theorem 5, we adopt the following strategy. First, we provide an approximation of the hyperellipsoid E ( r, c, A ) using the maximum eigenvalue of A -1 , i.e., λ max ( A -1 ) . In our context, A = n σ 2 ξ Σ n , and we then focus on the concentration inequalities regarding λ max (Σ -1 n ) . Finally, we combine these tools to prove both theorems.

Lemmas regarding hyperellipsoid. We present two lemmas that relate to the approximation of the hyperellipsoid E ( r, c, A ) . Specifically, we demonstrate the following two lemmas:

Lemma 2. For r > 0 , c ∈ R d , and a symmetric and positive-definite matrix A ∈ R d × d , we have E ( r, c, A ) ⊆ E ( rλ max ( A -1 ) , c, I ) .

Lemma 3. For r > 0 , c ∈ R d , and a symmetric and positive-definite matrix A ∈ R d × d , if rλ max ( A -1 ) ≤ ∥ c ∥ 2 , we have

These lemmas provide insight into the approximation of the hyperellipsoid E ( r, c, A ) for a given positive value of r , vector c in R d , and positive-definite symmetric matrix A in R d × d . Lemma 2 states that the hyperellipsoid E ( r, c, A ) is contained within a hyperellipsoid E ( rλ max ( A -1 ) , c, I ) . Lemma 3 shows that, under certain conditions, the minimum angle between a point in E ( r, c, A ) and the vector c is bounded below by a quantity that depends on r , c , and A .

Proof of Lemma 2. It is trivial that A -λ min ( A ) I is positive semi-definite. Equivalently, we have for any x ∈ R d ,

From Eq (14), for any x ∈ E ( r, c, A ) , we have

Hence, for any x ∈ E ( r, c, A ) , we have

which indicates x ∈ E ( rλ max ( A -1 ) , c, I ) .

Proof of Lemma 3. Let ¯ c = c/ ∥ c ∥ , and define a set ¯ E ( r, c, A ) = { x ∈ S d -1 : ∃ γ > 0 , γx ∈ E ( r, c, A ) } . Then, x ∈ ¯ E ( r, c, A ) if and only if

We can rewrite the left-hand side of Eq (15) as

Hence,

Consequently, x ∈ ¯ E ( r, c, A ) if and only if

By Eq (16), x ∈ ¯ E ( λ max ( A -1 ) r, c, I ) if and only if

Combining Eqs (17) and (18) and the assumption yields the claim.

Least eigenvalue of the empirical covariance matrix. The previous lemmas, Lemmas 2 and 3, provide valuable insight into analyzing the randomness regarding ξ . However, to account for the randomness of X , we must also control the lower bound on the least eigenvalue of A in Lemmas 2 and 3, which corresponds to the least eigenvalue of 1 n X ⊤ X in our context. To this end, we leverage the high probability bound presented by Mourtada [15] based on the small-ball condition. We state the following probabilistic bound and expectation bound.

Lemma 4. For µ ∈ R d and σ 2 X > 0 , let X 1 , ..., X n iid ∼ N ( µ, σ 2 X I ) , and let X = ( X 1 · · · X n ) ⊤ . Then, for n > 6 d , we have

Lemma 5. For µ ∈ R d and σ 2 X > 0 , let X 1 , ..., X n iid ∼ N ( µ, σ 2 X I ) , and let X = ( X 1 · · · X n ) ⊤ . Then, for n > 6 d , we have

To prove Lemma 4, we utilize Corollary 3 in Mourtada [15]. Specifically, we use the following theorem.

Theorem 13 (Corollary 3 in Mourtada [15]) . Let X be a random vector in R d such that E [ ∥ X ∥ 2 ] < + ∞ , and let Σ = E [ XX ⊤ ] . Let ˆ Σ n = 1 n ∑ n i =1 X i X ⊤ i , where X i are i.i.d. copies of X . Given C > 0 and α ∈ (0 , 1] , assume that for every θ ∈ R d \ { 0 } and t > 0 ,

Then, if d / n ≤ α / 6 , for every t > 0 ,

with probability at least 1 -( C ′ t ) αn / 6 , where C ′ = 3 C 4 e 1+ 9 / α .

Eq (19) is known as the small-ball condition.

Proof of Lemma 4. To take an advantage of Theorem 13, we need to ensure that X i satisfies the small-ball condition in Eq (19). Let Σ n = 1 n X ⊤ X . Then, the expected value of Σ n is equal to σ 2 X I + µµ ⊤ := Σ , i.e., E [Σ n ] = Σ . Given θ ∈ R d \{ 0 } , ⟨ θ, X i ⟩ 2 /σ 2 X ∥ θ ∥ 2 follows the non-central χ 2 distribution with degree of freedom 1 and non-centrality parameter ⟨ θ, µ ⟩ 2 /σ 2 X ∥ θ ∥ 2 . Consequently, we verify the satisfication of the small-ball condition of X i by confirming that for a random variable Z following the non-central χ 2 distribution with degree of freedom 1 and non-centrality parameter λ 2 , there exists C and α ∈ (0 , 1] such that

The cumulative distribution fucntion of the non-central χ 2 distribution with degree of freedom 1 has a closed-form using the error function (See [12] and references therein). Specifically, letting erf( z ) be the error function, defined as

the cumulative distribution function of Z is obtained as

Since e -x 2 is an even function, we have

Noting that e -x 2 / 2 ≤ e -(0 ∨ ( λ -t )) 2 2 for x ∈ ( λ -t, λ + t ) , we have

We verify that X i satisfies the small-ball condition by utilizing Eq (20). Recall that ⟨ θ, X i ⟩ 2 /σ 2 X ∥ θ ∥ 2 follows the non-central χ 2 distribution with degree of freedom 1 and non-centrality parameter λ 2 = ⟨ θ, µ ⟩ 2 /σ 2 X ∥ θ ∥ 2 for any θ ∈ R d \ { 0 } . By Eq (20), we have

Noting that ∥ ∥ Σ 1 / 2 θ ∥ ∥ 2 = σ 2 X ∥ θ ∥ 2 + ⟨ θ, µ ⟩ 2 , we have

We divide into two cases, λ > t and λ ≤ t , to derive an upper bound on Eq (21).

(Case λ > t ) Since ( λ -t ) 2 = λ 2 -2 λt + t 2 ≥ λ 2 -2 t 2 + t 2 = λ 2 -t 2 , an upper bound on Eq (21) is obtained as

For positive numbers a and b , √ a + b ≤ √ a + √ b . Using this fact, we have

Since a function x → xe -x 2 / 2 admits a maximum on x ∈ (0 , ∞ ) of e -1 / 2 , we have √ λ 2 -t 2 e -λ 2 -t 2 2 ≤ e -1 / 2 . Consequently, we have

where we use the fact (1 + e -1 / 2 ) 2 ≤ 8 e -1 .

(Case λ ≤ t ) We can easily verify that

Combining Eqs (22) and (23), we have for every t > 0 ,

For t 2 ∈ (0 , -1 2 + 1 2 √ 1 + πe 4 ] , we have

For t ≥ -1 2 + 1 2 √ 1 + πe 4 ,

Hence, for every t > 0 , we have

It confirms X i satisfies the small-ball condition with C = 7 1 / 4 and α = 1 . Application of Theorem 13 yields the desired claim.

Proof of Lemma 5. For a positive random variable X , we can express the expected value of X as E [ X ] = ∫ ∞ 0 P { X > t } dt . Applying this to our problem, we obtain

Now, let us set C = 21 e 10 σ 2 X . Using the previous result, we can rewrite the expectation of interest as

which yields the claim.

Proofs of theorems. By utilizing the results of Lemmas 2 to 5, we provide the complete proofs for both Theorem 4 and Theorem 5.

Proof of Theorem 4. We begin by demonstrating that obtaining an upper bound on the expected error of the direction estimator can be reduced to finding a lower bound on the inner product A n = ⟨ ˆ β ∥ ˆ β ∥ , β ∗ ∥ β ∗ ∥ ⟩ . Specifically, a straightforward calculation yields

Therefore, it suffices to establish a lower bound on E [ A n ] .

Taking advantage of Lemma 3, we derive a lower bound on A n . Let r = ( ˆ β -β ∗ ) ⊤ ( n σ 2 ξ Σ n )( ˆ β -β ∗ ) . From Lemma 3, it follows that

provided that r ≤ n ∥ β ∗ ∥ 2 /σ 2 ξ λ min (Σ -1 n ) . Since 1 - √ 1 -x ≤ x for x ∈ [0 , 1] , it follows that

as long as r ≤ n ∥ β ∗ ∥ 2 /σ 2 ξ λ max (Σ -1 n ) .

Next, we derive an upper bound on the expectation of 1 -A n . Noting that conditioned on X , r follows the χ 2 distribution with degree of freedom d , we have

where we use the Markov inequality to obtain Eq (25).

By utilizing Eq (24), an upper bound on the expected error can be obtained by deriving an upper bound on the expectation of Eq (26). The random variable in Eq (26) is λ max ( Σ -1 n ) , which allows us to derive the upper bound on the expected error by obtaining an upper bound on the expectation of λ max ( Σ -1 n ) . To accomplish this, we apply Lemma 5. The upper bound from Lemma 5 can be substituted into Eq (26), resulting in the claimed upper bound.

Proof of Theorem 5. We first utilize Lemma 2 to get an upper bound on the squared error of the norm estimator. Let r = ( ˆ β -β ∗ ) ⊤ ( n σ 2 ξ Σ n )( ˆ β -β ∗ ) . From Lemma 2, we have

Application of the triangle and reverse triangle inequality yields

equivalently

Taking expectation conditioned on X yields

where we use the fact that r follows the χ 2 distribution with degree of freedom d to obtain the last line. Again, application of Lemma 5 into expectation of Eq (27) yields the claim.

F Details of Upper Bound Analyses

This section presents a detailed proof of the upper bound stated in Theorem 1, which is achieved through an analysis of the estimator constructed in Section 5. Specifically, we establish the following theorem:

Theorem 14. Let ˆ β n be the estimator constructed in Section 5. Then, there exists a universal constant C > 0 such that for any δ ∈ (0 , 1) and n ≥ 12(3 d ∨ 4 ln( M/δ )) / min s ∈ [ M ] p s ,

To establish the validity of Theorem 14, we begin by proving Theorem 3, which demonstrates that the estimation error can be decomposed into the sum of errors associated with individual components. Subsequently, we derive upper bounds for the estimation errors of each component. Finally, we synthesize these results to provide a proof of Theorem 14.

F.1 Proof of Theorem 3

We commence the error analysis of our estimator by decomposing the estimation error, as presented in Theorem 3. Recall the statement of Theorem 3

Theorem 15. For the estimator defined in Eq (6) , the mean square deviation from f ∗ DP is bounded above by

We provide a proof of Theorem 15 as follows:

Proof of Theorem 15. We begin by decomposing ˆ f n ( X,S ) -f ∗ DP ( X,S ) into six terms. Recall the definitions of ˆ f n ( x, s ) and f ∗ DP ( x, s ) :

Through elementary calculations, we obtain:

By the Cauchy-Schwarz inequality, for two random variable Z 1 and Z 2 , we have E [( Z 1 + Z 2 ) 2 ] 1 / 2 ≤ E [ Z 2 1 ] 1 / 2 + E [ Z 2 2 ] 1 / 2 . By applying this fact into the expectation of Eq (28) conditioned on S and n · multiple times, we have

In the subsequent analyses, we derive upper bounds for each term in Eq (29).

(First term in Eq (29)) Due to the splitting of the sample, ̂ ∥ β · ∥ , ˜ β s , and ˆ µ s are independent conditioned on n · . Thus, we have:

This term matches the first term of the desired bound.

(Second term in Eq (29)) Since ̂ ∥ β · ∥ , ˜ β s , and X are independent conditioned on n · , we have

where we use the fact that X -µ s ∼ N (0 , σ 2 X I ) conditioned on S = s , and ˜ β s ∈ S d -1 almost surely. This result corresponds to the second term of the desired bound.

(Third term in Eq (29)) Since X -µ s ∼ N (0 , σ 2 X I ) , we have

which corresponds to the third term of the desired bound.

(Forth and fifth terms in Eq (29)) These terms are independent of S , so we can omit S = s from the condition, resulting in the fourth and fifth terms of the desired bound.

(Sixth term in Eq (29)) This term does not contain any random variable when n · is fixed. Thus, we can remove the expectation, yielding the sixth term of the desired bound.

F.2 Estimation Error Analyses for Each Component

This subsection presents an analysis of the estimation errors associated with each component estimator. In particular, we investigate the estimation errors of ˆ µ s , ̂ ∥ β · ∥ , ˜ β s , ˆ β ′ s , and ˆ µ ′ s .

F.2.1 Estimation Error Analysis for ˆ µ s

Here, we presents the proof of the following theorem.

Theorem 16. Given s ∈ [ M ] , if n s > 18 d , we have

Proof of Theorem 16. Given v ∈ S d -1 , we have

According to the definition, ˆ µ s is an average of n 3 ,s i.i.d. random variables following N ( µ s , σ 2 X I ) . Hence, we have µ s -ˆ µ s ∼ N (0 , σ 2 X n 3 ,s I ) , which implies E [( µ s -ˆ µ s )( µ s -ˆ µ s ) ⊤ | n · ] = σ 2 X n 3 ,s I . Consequently, we obtain:

Since n 3 ,s ≥ ⌊ n s / 3 ⌋ ≥ n s / 6 for n s ≥ 6 , the claim follows.

F.2.2 Estimation Error Analysis for ̂ ∥ β · ∥

Here, we present the proof of the following theorem.

Theorem 17. For any s ∈ [ M ] , we have

Proof of Theorem 17. By combining the definitions of ̂ ∥ β · ∥ and ∥ β ∗ · ∥ and utilizing the CauchySchwarz inequality, we obtain

Next, we derive an upper bound for the first term in Eq (30). Applying Jensen's inequality, we have:

Using the fact that n 1 ,s > 6 d for n s > 18 d and employing Theorem 5, we obtain:

where the last line follows from the fact that n 1 ,s ≥ ⌊ n s / 3 ⌋ ≥ n s / 6 for n s ≥ 6 , 6 n 1 ,s -6 ≤ 1 / 2 for n s ≥ 18 , and ∥ β ∗ s ∥ ≤ B .

F.2.3 Estimation Error Analysis for ˜ β s

Here, we will prove the following theorem.

Theorem 18. For any s ∈ [ M ] , we have

Proof of Theorem 18. If n s ≤ 18 d , ˜ β s = 0 , and we thus have E [ ∥ ˜ β s -β ∗ s ∥ β ∗ s ∥ ∥ 2 | n · ] = ∥ β ∗ s ∥ β ∗ s ∥ ∥ 2 = 1 . For n s > 18 d , we have n 2 ,s > 6 d . Application of Theorem 4 yields

We get the claim in the same manner as the proof of Theorem 17.

F.2.4 Estimation Error Analysis for ˆ β ′ s ′

Here, we will prove the following theorem.

Theorem 19. Given s ∈ [ M ] , let Σ s = E [ X s X ⊤ s ] for X s ∼ N ( µ s , σ 2 X I ) . Then, if n s > 12 d , we have

To prove Theorem 19, we utilize the following theorem presented by Mourtada [15].

Theorem 20 (Theorem 3 in [15]) . Let X be a random vector in R d such that it statisfies the smallball condition of Eq (19) and E [ ∥ Σ -1 / 2 X ∥ 4 ] ≤ κd for some κ > 0 , where Σ = E [ XX ⊤ ] . Let ˆ Σ n = 1 n ∑ n i =1 X i X ⊤ i , where X i are i.i.d. copies of X . If n ≥ 6 α -1 d ∧ 12 α -1 ln(12 α -1 ) ,

where α and C ′ are as in Theorem 13.

Proof of Theorem 19. We can easily confirm that ˆ β ′ s ∼ N ( β ∗ s , σ 2 ξ n ′ 1 ,s (Σ ′ 1 ,s ) -1 ) conditioned on n · and X ′ 1 ,s , where Σ ′ 1 ,s = 1 n ′ 1 ,s X ′ 1 ,s ( X ′ 1 ,s ) ⊤ . Noting that E [( ˆ β ′ s -β ∗ s )( ˆ β ′ s -β ∗ s ) ⊤ | X,n · ] = σ 2 ξ n ′ 1 ,s (Σ ′ 1 ,s ) -1 , we have

We apply Theorem 20 to the expected trace term in Eq (31). To do so, we need to check X ′ 1 ,s satisfies the small-ball condition of Eq (19) and the kurtosis condition E [ ∥ Σ -1 / 2 s X ′ 1 ,s ∥ 4 ] ≤ κd .

The small-ball condition is confirmed by the same manner in the proof of Lemma 5, with C = 7 1 / 4 and α = 1 . Here, we prove the satisfication of the kurtosis condition. For a multivariate Gaussian random variable X ∼ N ( µ, Λ) such that λ min (Λ) > 0 , Σ := E [ XX ⊤ ] = µµ ⊤ +Λ , and ∥ Σ -1 / 2 X ∥ 4 = ⟨ X, Σ -1 X ⟩

Since we have

Hence, the kurtosis condition satisfies with κ = 3 .

Application of Theorem 20 into Eq (31) yields

provided that n s ≥ 12 d . We get the claim from the fact that n ′ 1 ,s ≥ ⌊ n s / 2 ⌋ ≥ n s / 4 for n s ≥ 4 ,.

F.2.5 Estimation Error Analysis for ˆ µ ′ s

Here, we will prove the following theorem.

Theorem 21. Given s ∈ [ M ] and v ∈ R d , if n s > 12 d , we have

Proof of Theorem 21. By definition, we have ˆ µ ′ s ∼ N ( µ s , σ 2 X n ′ 2 ,s I ) conditioned on n · for n s > 12 d . Hence, we have

We get the claim following the same manner of the proof of Theorem 19.

F.3 Some Auxiliary Lemmas

This subsections introduce some auxiliary lemmas for use to prove Theorem 14. Specifically, we demonstrate the following lemmas:

Lemma 6. Let a 1 , ..., a M ∈ R be arbitrary numbers. Then, we have

Lemma 7. For a constant c > 0 , we have for any s ∈ [ M ]

Lemma 8. Let c > 0 be a constant. If n > 2 c/ min s ∈ [ M ] p s ,we have for any s ∈ [ M ]

Proof of Lemma 6. Since n ˆ p · follows the multinomial distribution with the parameters n and p · , using the variance and covariance of the multinomial distribution, we have

̸

Let ¯ a = ∑ s ∈ [ M ] p s a s . Then, we have

Hence,

Proof of Lemma 7. For a random variable X following the binomial distribution with the parameters n and p , E [ 1 X +1 ] = 1 p ( n +1) (1 -(1 -p ) n +1 ) [5]. Since n s follows the binomial distribution with the parameters n and p s , we have We have

Proof of Lemma 8. From the Chernoff bound, we have

Under the assumption, we have c ≤ np s / 2 . Then, we have n 2 p s ( c n -p s ) 2 ≥ n/ 8 , which gives the claim.

F.4 Proof of Theorem 14

Proof of Theorem 14. We begin by characterizing the estimation error by each component's estimation error shown in Theorems 16 to 19 and 21. Specifically, we characterize the estimation error using the following error terms:

where Σ x = E [ X s X ⊤ s ] for X s ∼ N ( µ s , σ 2 X I ) .

We analyze each term in Eq (7) one by one.

(First term in Theorem 3) Recall the first term in Theorem 3

From the Cauchy-Schwarz inequality, we have

By definition, ˜ β s = 0 for n s ≤ 18 d , which indicates that

In the case of n s > 18 d , by utilizing the fact ˜ β s and ˆ µ s are independent conditioned on n · due to the sample spilitting and ˜ β s ∈ S d -1 , we obtain

Consequently, we have

E [ ̂ ∥ β · ∥ 2 ∣ ∣ ∣ ∣ n · ] 1 / 2 E [ 〈 ˜ β s , µ s -ˆ µ s 〉 2 ∣ ∣ ∣ ∣ n · ] 1 / 2 ≤ ( B + e norm ) e mean ,s ✶ { n s > 18 d } . (32) (Second term in Theorem 3) Recall the second term in Theorem 3

Using the notation of e norm , we have

(Third term in Theorem 3) Recall the third term in Theorem 3

Substituting e coef ,s yields

(Fourth term in Theorem 3) Recall the fourth term in Theorem 3

Due to the sample splitting, ˆ β ′ s and ˆ µ ′ s are mutually independent. Also, we have E [ ⟨ ˆ β ′ s -β ∗ s , ˆ µ ′ s ⟩ ] = 0 . Hence,

Since E [(ˆ µ ′ s )(ˆ µ ′ s ) ⊤ ] = µµ ⊤ + σ 2 X n ′ 2 ,s I = Σ s -(1 -1 n ′ 2 ,s ) σ 2 X I , we have

where we use the fact Σ s ⪰ Σ s -(1 -1 n ′ 2 ,s ) σ 2 X I , and for symmetric matrices A and B such that A ⪰ B , ⟨ v, Av ⟩ ≥ ⟨ v, Bv ⟩ for any v ∈ R . For n s ≤ 18 d , the error is zero because ˆ µ ′ s = 0 (Fifth term in Theorem 3) Recall the fifth term in Theorem 3

Since ˆ µ ′ s are independent, we have

(Sixth term in Theorem 3) Recall the sixth term in Theorem 3

which is equivalent to e prob .

By combining Theorem 3 and Eqs (32) to (36), we get

The triangle inequality gives that

By applying Theorem 16 and Lemma 7, we have

E [ e 2 mean ,s ✶ { n s > 18 d } ] ≤ E [ 6 σ 2 X n s ✶ { n s > 18 d } ] ≤ 6 σ 2 X p s ( n +1) ( 1 + 1 18 d ) . Also, from Theorem 17 and Lemmas 6 and 8, we have

provided that n > 36 d/ min s ∈ [ M ] p s . By utilizing Theorem 18 and Lemmas 7 and 8, we have

provided that n > 36 d/ min s ∈ [ M ] p s . Application of Theorem 19 gives

Application of Lemma 6 with a s = ⟨ β ∗ s , µ s ⟩ into e 2 prob yields

Synthesizing the results so far, there exists an universal constant C > 0 such that

Consequently, there exists an universal constant C > 0 such that

Then, the dominating terms match the claim.

G Details of Lower Bound Analyses

This section provides the proofs of the lower bound analyses results.

G.1 Proof of Lower Bound in Theorem 1

Theorem 22. If M ( d -1) > 16 , there exists an universal constant C > 0 such that for any α > 0 and δ ∈ (0 , 1) ,

Proof of Theorem 22. The Varshamov-Gilbert bound guarantees that there exists a subset V ′ ⊆ V such that |V ′ | ≥ 2 M ( d -1) / 8 and d H ( v s , v ′ s ) ≥ ( d -1) / 8 for any v, v ′ ∈ V ′ . With the choice of ϵ 2 s = ( d -1 16 -1 M ) σ 2 ξ / 2 σ 2 X B 2 s n s , we confirm by Theorem 8 that inf π 1 K ∑ v ∈V ′ D KL ( π θ v | n · , π ) ≤ max v,v ′ ∈V ′ D KL ( π θ v ′ | n · , π ) ≤ ln( |V ′ | / 4) / 2 ≥ M ( d -1) / 16 -1 . From Theorem 8 and the fact d H ( v s , v ′ s ) ≥ ( d -1) / 8 , we can apply Theorem 6 with ϵ = ∑ s ∈ [ M ] p s ( ∑ s ′ ∈ [ M ] p s B s ) 2 B 2 s ( d -1 16 -1 M ) σ 2 ξ / 2 n s . From the fact that E [ 1 n s +1 ] = 1 p s ( n +1) (1 -(1 -p s ) n ) due to [5], there exists an universal constant C > 0 such that E [ ϵ 2 ] ≥ C ( 1 M ∑ s ∈ [ M ] ( ∑ s ′ ∈ [ M ] p s ′ B s ′ ) 2 B 2 s ) σ 2 ξ Md n -o ( 1 n ) We can get the claim by confirming that there exists B 1 , ..., B M such that ( 1 M ∑ s ∈ [ M ] ( ∑ s ′ ∈ [ M ] p s ′ B s ′ ) 2 B 2 s ) = B 2 and B s ≤ B . Because for B 2 = ... = B M = B , tending B 1 to 0 results in ( 1 M ∑ s ∈ [ M ] ( ∑ s ′ ∈ [ M ] p s ′ B s ′ ) 2 B 2 s ) goes infinity, it is confirmed.

G.2 Proof of Theorem 6

Proof of Theorem 6. Since the distribution of n · is invariant against θ ∈ Θ , we have

̸

Given ϵ possibly dependent on n · , application of the Markov inequality yields

If inf f E ( f ; θ ) ∨E ( f ; θ ′ ) ≥ ϵ for any θ, θ ′ ∈ ˆ Θ , E ( f ; θ ) < ϵ implies E ( f ; θ ′ ) ≥ ϵ for any θ ′ ∈ ˆ Θ such that θ = θ ′ . Hence, there exists a partion {F θ } θ ∈ ˆ Θ of all the measurable functions f : R d × [ M ] → R such that { f : E ( f ; θ ) < ϵ } ⊆ F θ for all θ ∈ ˆ Θ . Consequently, we have

Application of the Fano's inequality and data processing inequality yields the claim.

G.3 Proof of Theorem 7

To prove Theorem 7, we show the following more tight lower bound.

Theorem 23. Let θ and θ ′ be the parameters of the distributions such that 1 2 σ 2 X ∥ µ s -µ ′ s ∥ 2 := d s < 1 for all s ∈ [ M ] . Then, we have

Theorem 23 immediately gives Theorem 7.

We utilize the sufficient condition for the constrained optimization problem over a Banach space. Let Z be a Banach space. We say a function f : Z → R is Gateaux differentiable if the limit lim τ → 0 f ( z + τu ) -f ( z ) τ exists for any open set U ⊆ Z , any z ∈ U , and any u ∈ Z . We denote the Gateaux derivative of f at z ∈ Z , a linear mapping from u ∈ Z to lim τ → 0 f ( z + τu ) -f ( z ) τ , as D G f ( z ) . We abuse 0 to denote the mapping that always outputs 0 .

Proof of Theorem 23. Let q s and q ′ s be the density function of X s with the parameters ( β · , µ · ) and ( β ′ · , µ ′ · ) , respectively, regarding the base measure λ . Since X s follows the Gaussian distribution, we can choose λ as the Lebesgue measure. Given η ∈ [0 , 1] , we have

Because η E ( f ; β · , µ · )+(1 -η ) E ( f ; β ′ · , µ ′ · ) is convex for f , and L 2 is a Banach space, it is minimized if for any u ∈ L 2 ,

The dominated convergence theorem gives

Consequently, η E ( f ; β · , µ · ) + (1 -η ) E ( f ; β ′ · , µ ′ · ) is minimized at

Hence,

With η = 1 / 2 , we have

Let ¯ µ s = 1 2 ( µ s + µ ′ s ) . Then, we have √ q s ( x ) q ′ s ( x ) = 1 √ (2 π ) d σ 2 d X exp ( -1 4 σ 2 X ∥ x -µ s ∥ 2 -1 4 σ 2 X ∥ x -µ ′ s ∥ 2 ) = 1 √ (2 π ) d σ 2 d X exp ( -1 2 σ 2 X ∥ x -¯ µ s ∥ 2 -1 2 σ 2 X ∥ ∥ ∥ ∥ µ s -µ ′ s 2 ∥ ∥ ∥ ∥ 2 -1 2 σ 2 X (〈 x -¯ µ s , µ ′ s -µ s 2 + µ s -µ ′ s 2 〉) ) = 1 √ (2 π ) d σ 2 d X exp ( -1 2 σ 2 X ∥ x -¯ µ s ∥ 2 -1 2 σ 2 X ∥ ∥ ∥ ∥ µ s -µ ′ s 2 ∥ ∥ ∥ ∥ 2 ) . Also, we have q s ( x ) q ′ s ( x )

Let ¯ X s ∼ N (¯ µ s , σ 2 X I ) . Then, we have

where the last line is obtained from the Cauchy-Schwarz inequality.

By definition, we have

Conditioned on ∥ ¯ X s -¯ µ s ∥ = r for r > 0 , ¯ X s follows the uniform distribution over the ( d -1) -sphere centered at ¯ µ s . For a random variable U uniformly distributed over the ( d -1) -sphere centered at origin with the radius r , E [ U ] = 0 and E [ UU ⊤ ] = r 2 / d I . Hence, for a vector v ∈ R d and a scalar c ∈ R , we have

An elementary analysis yields that ∥ ¯ X s -¯ µ s ∥ 2 ∼ Gamma( d 2 , 2 σ 2 X ) , where Gamma( k, θ ) denotes the Gamma distribution with the shape parameter k and scale parameter θ . From the upper bound of the hyperbolic cosine as cosh( x ) ≤ e x 2 / 2 , for a vector v ∈ R d , scalars c ∈ R and c ′ > 0 , and a random variable γ ∼ Gamma( k, θ ) , we have

where we use the fact that the hypergeometric function 2 F 1 ( a, b, b ; z ) = ∑ ∞ m =0 Γ( a + m ) Γ( a ) z m m ! = (1 -z ) -a for some b , provided | z | < 1 . By setting

Combining Eqs (37) and (38) yields the claim.

G.4 Proof of Theorem 8

Proof of Theorem 8. It is easy to check that d s = 0 , and for any v, v ′ ∈ V ,

Since the density function of the Gaussian distribution is L 2 integrable, E ( f ; θ ) = ∞ if f is not L 2 integrable. Hence, inf f E ( f ; θ v ) ∨ E ( f ; θ v ′ ) = inf f ∈L 2 E ( f ; θ v ) ∨ E ( f ; θ v ′ ) , and we thus can apply Theorem 7. Then, we have

Conditioned on n · , the KL-divergence between π θ v | n · and π θ v ′ | n · is obtained as

Hence, we have

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the linear regression problem under a new definition of $(\\alpha, \\delta)$-fairness consistency as a fairness constraint.\nIn particular, the authors derived that under the constraint of $(\\alpha, \\delta)$-fairness, the minimax optimal error is $dM/n$ when $\\alpha$ is less than 1/2. Moreover, they provided an estimator that achieves this optimal rate of convergence.\n\nSTRENGTHS:\nThis paper established matching upper and lower bounds on the minimax optimal error and proposed a regression algorithm that achieves this optimal error for the linear regression model under a specific fairness constraint. The theory and method extends the current literature to cover partial coefficients and non-sensitive features on the sensitive attribute. The paper is well presented and articulated. The authors also provided detailed proofs which hightlighted the technical difficulties in the minimax analysis, as well as a novel estimator for achieving the minimax optimal bound.\n\nWEAKNESSES:\nIt would be great if the authors could discuss and provide more understanding on how restrictive the assumption that $\\alpha <= 1/2$ is. Also, how should one think about solving for the case that $\\alpha > 1/2$? \nThe paper is lacking empirical study/examples which I feel would be nice given the subject under study.\n\nQUESTIONS:\nIt would be great if the authors could discuss and provide more understanding on how restrictive the assumption that $\\alpha <= 1/2$ is. \nIs it a necessary condition for achieving the minimax rate of dM/n? \nAlso, how should one think about solving for the case that $\\alpha > 1/2$? \nHow does the fairness score proposed in the paper compare with other fairness scores in the existing literature?\nThe authors mentioned that misapplication of the findings in the paper to other models might result in discriminatory treatment. Any discussion on the robustness of the proposed algorithms, especially under misspecified models?\n\nLIMITATIONS:\nThe authors have pointed out the restriction of the proposed models and cautioned against misusing the algorithms for non-linear models.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

comments: SUMMARY:
This paper investigates the minimax error in regression under the constraint of demographic parity (DP). In contrast to earlier works, this paper studies both direct and indirect discrimination and their effects on the estimator. In addition to proposing an algorithm to construct a regressor achieving a desirable error, they provide extensive theoretical analysis characterizing upper and lower bounds under the DP constraint.

STRENGTHS:
1. The paper is nicely written, with a clear structure and a comprehensive explanation of a large number of notations.

2. It studies a simple but interesting, important problem, which indeed catches attention and makes a good contribution to the community.

3. The introduction of direct and indirect discrimination is novel and realistic, and the examples are helpful for readers to understand those notions.

4. The theoretical analysis is comprehensive. The discussion in Section 4 is interesting to read and may provide insights for future work in this area.

WEAKNESSES:
I did not find particular weaknesses for this paper.

QUESTIONS:
In your model, you mentioned that indirect discrimination mainly impacts $\mu_S$ while direct discrimination is reflected via $\beta_S^*$. My understanding is that, direct discrimination is for the “model” itself, i.e. $\beta$, while indirect discrimination has influence on the data set. Is my description accurate?

LIMITATIONS:
The authors addressed them at the end of Section 8.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-07

comments: SUMMARY:
The authors study the problem of minimax optimality for the linear regression model under demographic parity constraints from the fairness literature. They provided matching upper and lower bounds for the model that is considered in the paper.

STRENGTHS:
Overall:

The paper is well-motivated by presenting both direct and indirect discrimination that occurs in real-world settings and the proposed model accounts for it. The theory is well-laid out and the difficulty is elucidated in the main paper with details left to the appendices. However, there isn't any experimental work.

Pros:

(A) The model differs from the prior literature (Chzhen and Schreuder) and is well motivated in Section 1 and Table 1. 
(B) The technical difficulty of both the norm estimators and the direction estimators is presented in equation 7. 
(C) The key takeways that the optimal error is independent of mean but depends on the variance is intriguing.

WEAKNESSES:
(1) How does this compare to the results of Chzen and Schreuder is unclear. There is no unified setting which captures both models.
(2) Also, there is no experimental work which would help us know the tightness of the proposed upper bounds (constants). 
(3) Some of the writing can be sharpened and there are multiple typos.

QUESTIONS:
Please check the cons. Slightly lower score because of lack of experimental work.

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-10

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the statistical minimax error of building a linear regression model under the constraint of demographic parity. \nIt generalizes the result in Chzhen and Schreuder [5] by allowing the slope coefficients to change with sensitive attribute rather than just\nthe intercepts. The authors provide both upper and lower bounds of the minimax error, and show that they match in the dependency on $n$.\n\nSTRENGTHS:\nAlthough this paper uses a framework similar to  Chzhen and Schreuder [5], it covers a very important setting where there exists both direct and indirect discriminations. The analysis begins with Lemma 1, which can be easily derived from Chzhen and Schreuder [5]. However, to obtain the upper bound of the minimax error, the authors need to characterize the errors of approximating each component in (5), which is one of the main challenges in this paper. They develop new techniques in Theorems 4 and 5 to handle this challenge. I think it has made significant contribution on top of   Chzhen and Schreuder [5].\n\nWEAKNESSES:\n1. This paper assumes the covariance matrix of X is an identity matrix for each $s$. It will be better to allow a more general matrix $\\Sigma_s$. \n2. It only covers the case where $\\alpha\\leq 1/2$.\n\nQUESTIONS:\n1. The statement made in line 136-137 should be explained more clearly. In particular, why does enforcing strict demographic parity result in a constant function?\n\n2. The authors consider $(\\alpha, \\delta)$-consistently fair regressor while Chzhen and Schreuder [5] consider $(\\alpha, t)$-valid estimators (see DEFINITION 5.2 in [5]). Are they equivalent? If not, why do the authors consider a different type of fair regressor in the definition of the minimax error. \n\n3. Equation (4) needs to be explain, especially the second inequality. Also, I think $\\beta$ here should be $\\beta^*$.\n\nLIMITATIONS:\nSee weaknesses above.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-26"}