On Optimal Steering to Achieve Exact Fairness
Authors: mohit sharma
Amit Deshpande
Chiranjib Bhattacharyya
Rajiv Ratn Shah
Venue: NeurIPS 2025 poster ()
Keywords: steering
fairness
representation
bayes optimal classifiers
data bias
ideal distributions
Primary area: social_and_economic_aspects_of_machine_learning
TL;DR: Steering given distributions towards ideal distributions, where fairness and accuracy are not at a tradeoff.

Abstract: To fix the `bias in, bias out' problem in fair machine learning, it is important to steer feature distributions of data or internal representations of Large Language Models (LLMs) to \emph{ideal} ones that guarantee group-fair outcomes. Previous work on fair generative models and representation steering could greatly benefit from provable fairness guarantees on the model output. We define a distribution as \emph{ideal} if the minimizer of any cost-sensitive risk on it is guaranteed to have exact group-fair outcomes (e.g., demographic parity, equal opportunity)---in other words, it has no fairness-utility trade-off. We formulate an optimization program for optimal steering by finding the nearest \emph{ideal} distribution in KL-divergence, and provide efficient algorithms for it when the underlying distributions come from well-known parametric families (e.g., normal, log-normal).

Empirically, our optimal steering techniques on both synthetic and real-world datasets improve fairness without diminishing utility (and sometimes even improve utility). We demonstrate affine steering of LLM representations to reduce bias in multi-class classification, e.g., occupation prediction from a short biography in Bios dataset (De-Arteaga et al.). Furthermore, we steer internal representations of LLMs towards desired outputs so that it works equally well across different groups.

On Optimal Steering to Achieve Exact Fairness
Mohit Sharma"
Amit Jayant Deshpande Microsoft Research India ami tdesh@m tdeshhmicroooof.t. crwsoft.
Department of Computer Science IIIT Delhi, India mohits@iiitd.ac
com
Chiranjib Bhattacharyya
Rajiv Ratn Shah Department of Computer Science IIT Delhi, India rajivratn@iiitd.aa ac.
Department of Computer Science and Automation Indian Institute of Science, Bengaluru, India chirušiisc.
Abstract
To fix the 'bias in, bias out' problem in fair machine learning, is important to steer feature distributions of data internal representations Large Language Models (LLMs) to ideal ones that guarantee group-fair outcomes. Previous work on fair generative models and representation steering could greatly benefit from provable fairness guarantees the model output. We define distribution ideal if the minimizer of any cost-sensitive risk on it is guaranteed have exact group-fair outcomes (e.g., demographic parity, equal apportunity)-1l other words, it has no huimess-utility, trade-off. We formulate an optimization program for optimal steering by finding the nearest ideal distribution in KL-divergence, and provide efficient algorithms for when the underlying distributions come from well-known parametric families (e.g., normal, log-normal).
Empirically, our optimal steering techniques both synthetic and real-world datasets improve fairness without diminishing utility (and sometimes even improve utility). We demonstrate affine steering of LLM representations reduce bias in multi-class classification, e.g., occupation prediction from short biography in Bios dataset (De-Arteaga al.). Furthermore, we steer internal representations of LLMSs towards desired outputs so that it works equally well across different groups.
Introduction
The importance of clean or ideal data in fair machine learning cannot be overstated. The principle of bias in, bias out is widely recognised a root cause of unfair outcomes in ML systems [16 52 62 26 Models trained on biased data tend to learn, perpetuate, and often amplify such biases. Importantly, the problem of biased data extends beyond training: cainess-constrained training on biased data does not guarantee fairness on (unbiased) test sets, and post-processing using biased validation data fails ensure fairness at deployment. Moreover, fairness audits based on biased assessment data can be misleading and difficult to reverse
Fairness metries such demographic parity and equal opportunity are inherently functions of both the model and the data distribution. Thus, unfair outcomes may be addressed either by adjusting the model by altering the data distribution. While prior work fair in-processing aims construct an ideal model under fairess constraints [ 29), our work focuses instead on identifying an ideal data distribution that supports fairness. In this respect, our approach aligns most closely with the fair pre-proces literature.
Part the work was done when author was an intern at Microsoft Research India.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

1,6
DC-6.666 (32-0.00 (EO 6:14
1.
DE: 0E-0..0 (CP .0.00 (00: 0.31
1.0
0.000 6SP 5,00 650: 0.69
34
DC: 60P-600000000000 650-030
KL:b23
0.1
0.0
K:5A1
03
6.6.56
De
044
0.2
0.2
0.2
022
0.5
0.
60
(a) Original
(b) Idcal (Affirmative)
(c) Ideal (Changing All)
(d) Mcan Matching
Figure 1: Comparison of different interventions for changing Data Distributions for Exact Fairness.
Figure (1a) captures the original distribution, its Bayes error (BE), and the unfairness differences
(ADP and ΔEO). In Figure (Tb), we only change the under-privileged group using Corollary 4.2 and in Figure (cc) we change all four subgroups using Proposition 4.3 Finally, in Figure [T]. we match the means of the two groups. Figures (Ib) and (Ic) show that it possible to construct 'ideal' distributions that are close to the given distribution where both the BE and ADP/AEO are small.
Early work by Kamiran and Calders 43 introduced heuristic reweighting methods for binary classification, adjusting the weight of instances from class i and group a using Pr (class i) Pr (group a) /Pr (class i, group a). However, this approach ignores the feature distribution and offers no fairness guarantees when combined with accuracy maximisation. Calmon al. [88] framed fair pre-proceste pre-prceesiing as an optimisation problem that balances distance to the original distribution with group and individual fairness constraints. While convex in some settings, their approach may be infeasible when group and individual fairness are incompatible
34
41 address label bias through reweighting; Plečko and Meinshausen (60), Plečko al. [6]] propose fair adaptation methods based on causal models; and Xiong et al. [75 reformulate pre-processing
Other work explores alternative mechanisms: Jiang and Nachum
large-scale mixed-integer program, solved via a cutting-plane method. Fair pre-processing remains widely used practice, often integrated into fairness toolkits alongside in- and post-processing methods [7] Despite its heuristic nature, the reweighing method of Kamiran and Calders [43 continues to perform well in mitigating bias across standard benchmarks [65 12 (75.
common goal of all fair pre-processing methods is to find an ideal data distribution close to the given distribution so that any downstream model trained on it must have guaranteed fairness. A stronger requirement that this should hold for downstream models optimized for multiple tasks leads to impossibility results [47). If all downstream classifiers are required to be fair, then the group-wise distributions must nearly identical, which is absurd. Thus, we restrict our downstream models only to Bayes optimal classifiers for cost-sensitive risks. Our first contribution is to formally define an ideal distribution as the one where the Bayes optimal classifier for any cost-sensitive risk satisfies exact fairness (e.g., satisfies equal opportunity perfectly).
Bayes optimal classifier maximizes accuracy on given distribution, and have been an important object of study in statistical machine learning (28). Fair Bayes optimal classifier maximizes accuracy subject to fairness constraints, and its mathematical characterization for binary fair classification
78). Blum and Stangl [13] introduce data bias model that injects under-representation and label bias in an original unbiased distribution create biased data. They show that, for stylized distribution under some conditions, the fair Bayes optimal classifier on the biased distribution recovers the Bayes optimal classifier on the original unbiased distribution. Their unbiased distribution is ideal by construction, i.e., the Bayes optimal classifier on their unbiased distribution is guaranteed to be perfectly fair. Sharma and Deshpande [64] extend this observation to general hypothesis classes and distributions beyond the stylized setting of Blum and Stangl [13]. J. Blum et al. [44 study fair Bayes optimal classifier whether its accuracy is robust to malicious corruptions in data distribution. In contrast to these results, our focus is not on finding the ideal classifier but on finding the nearest ideal distribution.
has been important in fair classification [54] 20
By definition, our ideal distribution has no trade-off between accuracy (or cost-sensitive utility) and fairness. If we find an ideal distribution close to our original distribution, we can steer our distribution towards reducing fairness-accuracy trade-off. Moreover, if the ideal distribution offers better accuracy, suggests that we can steer our distribution to improve both accuracy and fairness simultaneously. Fig. ! gives such an example where the pre-processing of [43), in contrast, leaves the

---

original distribution unchanged and our method provides direction to steer to distribution towards better accuracy and fairness simultaneously.
Dutta et al. 30 characterize a similar objective using Chernoff Information (see Cover [25 and formulate optimization program find the nearest distribution in KI KL--divergence on which the Chernoff Information gap between two group-conditional feature distributions vanishes. Their optimization problem is not known to be efficiently solvable and the fairness guarantees in terms of Chernoff Information gap does not translate easily to standard fairness metrics such as demographic parity, equal opportunity etc. In contrast, we formulate an optimization problem find the nearest ideal distribution in KL-divergence to given distribution and give efficient algorithms to solve it for various parametric families of distributions. To put our results to the test, we also apply our interventions to steer the representations of an LLM (72) [6] for fair multi-class classification [27 67 and emotion steering [7].. We are able to improve the effectiveness of steering without significantly affecting the accuracy and are able to demonstrate how our notion of ideal distributions can help
guide interventions for practical applications.
For completeness, we want to also make the reader aware of long line of work on fair representation learning where the data transformations can map the distributions another space 777 50
53
49 211
Our work is not directly related but can potentially be used to refine fair representations to achieve provable and exact fairness guarantees.
1.1 Our Results
We summarize our key contributions as follows.
We define ideal distribution (Definition 3.1 for fair classification as the one on which the Bayes optimal classifier for any cost-sensitive risk satisfies exact fairness (e.g., exact demographic parity, exact equal opportunity).
When group and class-conditioned distributions belong to well-known parametric families of distributions (e.g.. Gaussian, log-normal), we can succinctly rewrite the property of being ideal distribution as a parametric condition (Proposition 3.3)
The problem of finding the nearest ideal distribution to a given distribution is intractable an optimization problem of KL-divergence subject to the
in general. We formulate parametric conditions required for being ideal (Section 4 As stated, it is non-convex optimization problem (Proposition (..3 but we show how to solve it efficiently. We also provide a closed form optimal solution in special cases (Theorem 4.1 Corollary 4.2). When better accuracy is achievable on the nearest ideal distribution, it suggests a direction steer the original distribution in to improve both accuracy and fairness.
To illustrate the effect of different interventions, we show the effect of using Affirmative Action (Corollary(4.2) changing all subgroups (Proposition 4.3) and matching the means of sensitive groups on different univariate distribution where we can compute the Bayes Optimal Group-aware classifiers analytically in Figures
To demonstrate how our guarantees can aid practical applications we also performs experi- ments to steer LLM representations to reduce bias in multi-class classification (Figure and effective group-level emotion steering (Figure 4
2 Problem Setup and Preliminaries
Let (X,A,Y) be a random data point from a joint distribution D over X x x V. where X. A. denote the sets of features, sensitive attributes, and class labels, respectively. Let tha = Pr = i, = a) and P,, denote the distribution X || Y = i, A a with the probabil- ity density Pia (x) = Pr [X- = x
Y = i, A = a). When Pia 's come from parametric families of distributions, we assume X = Rdd. We work with the following well-known definitions of fairnsss in classficiation B1 38 6).
Definition 2.1. For exact fairness, the case of multiple classes ()) > 2) and multiple protected groups (IAA 2). classifier h: X x A ++ satisfies:
3

---

1. Demographic Parity [31] if the positive rate for class across groups is zero, i.e., max [Pr (h A)= = y y a) Pr (h(X, A) = y A a') 0, vy ),
a.'EAA
2. Equal Opportunity
if the true positive rates for a class across groups is zero, i... max [Pr (h(X, = y/Y = y. a) Pr (h( = y = y, A a.''EA
38
V.
These lead to quantitative metrics of unfairness, e.g., (h, D) denotes the absolute value of difference between Pr (r(h(X, A) =|A a) and Pr (h(X, (X, A) A)= 1| = a'). Similarly, AEO.y (h, D) denotes the absolute value of difference between Pr (h(X, A) y/Y = A = a) and
Pr (h(X,, A) = y/Y = y, = d'). Whenever we are dealing with binary classification, we will omit the use of y in the subscript.
We consider group-aware classifiers. For binary classification tasks, we are particularly in- terested in threshold classifiers hz a) that apply group and feature dependent threshold t(z, a) the class probability of an example: (x. a) (n)2,, a) 2 t(x, a)) where n)(2,a)
Pr Pr(Y- = 1|x = x. A ==A=aa a). It is well-known that the Bayes optimal classifier for given dis- tribution has the form ((x, a)
1/2 (28). For a cost matrix C E TR2x2 and the associated cost sensitive loss lc. the Bayes optimal classifier is defined T (n)(,, a) 2 (c), for threshold
to (C10 Coo) (C10- - Coo + co1 c11) ) E [0, 1], where cjy denote the entries of the cost matrix
C' g2x2
66
Ideal Distributions for Fair Classification
We define a data distribution as ideal when minimizing any cost-sensitive risk on it is guaranteed to give exact fairness (e.g., demographic parity, equal opportunity). In practice, downstream models trained on distribution are typically optimized for some performance or utility metric that may not be known in advance. Our definition of ideal distribution allows the flexibility to choose any cost-sensitive risk as the performance metric for downstream models and still gives exact fairness guarantee for any optimal model downstream.
Definition 3.1. Let H be a hypothesis class of group-aware classifiers h : X x A - y and let (ht, D) be a given unfairness metric, e.g., App, Δgo- Given a distribution D over x x y and a cost-sensitive risk C' E RDix Vi, let hig = argminPr (fx(h(X A), We call D an ideal NCH
distribution if A(ke D) O, for all C' RDIXD
Examples of fairness metrics include Demographic Parity, Equal Opportunity, and Equalized Odds (Definition 2.1 and Hardt al. (38)) and examples of cost-sensitive risk include the usual 0 loss and different performance metrics which are functions of the confusion matrix metrics 33 46
66
Our definition gets around the impossibility theorems about fair representation for multiple tasks 47
However, we need to careful of two things. First, our definition should not be too restrictive to just force the group-conditionee distributions to be similar identical, that would be impractical. Second, we need an efficient and equivalent way of expressing the constraint of being ideal. We show how express it parametric condition when the group and class-conditioned distributions belong to certain well-known parametric families of distributions. This helps in checking if a given distribution is ideal, and otherwise, finding its nearest ideal distribution.
3.1 Parametric Conditions for Ideal Distributions
Borrowing a simple setup of parametric distributions from previous work on fair machine learning (59), we assume that the class and group-conditioned feature distributions X y A = a belong to parametric family of distributions, e.g., univariate or multivariate Gaussians, log-normal. In that case, we show that the property of being ideal (Definition 3.1 can be equivalently expressed as certain parametric conditions. To begin, we will first show the set of parametric conditions for multivariate normal distributions and a multi-class,
setting.
Proposition 3.2. Let (XYY, A) denote the features, class label, and group membership, respectively, of a random data point from any data distribution D with qio = Pr (Y i, = o), for y

---

and a E A. Let Xe = -, ~ (H Ena) be multivariate Normal distributions with mean lio E TRd and covariance matrix Dia E IRdxd for E y and a E A. If the means Pio and the covariance matrices Eia satisfy
DT (Pia Mja) Lia" 1/2 (Mia' Hja') and lia Nio',
1/2 = to
11/2 T jja' and
Vi.j E V,a, E A.
9ja
Y3a'
then the group-aware Bayes optimal classifier on D satisfies equal opportunity.
The proof for Proposition 3.2 is given Section of the Appendix. When we are dealing with binary class labels and group membership, we can show necessary and sufficient condition for univariate normal distributions.
Let (X,Y, A) denote the features, binary class label, and binary group membership, respectively, of a random data point from any data distribution D with qia Pr (Y A = a), for i E {0,1} and a E {0, 1}, and let X|Y = i, A = a ((uaa daa) be univariate normal distributions. for i {0, 1} and {0, 13.
Proposition 3.3.
Then the distribution D is ideal for equal opportunity
(see Definition 3.1 if and only if
P41 P111
1400
Mio
011
910
q11
O1
001
doo
yoo
qu1
Proposition 3.3 shows that our parametric condition equivalent to (c D) = 0, for all cost matrices C R2x2 When optimal classifier in Proposition 3.2 our parametric condition is sufficient but not always necessary. However, the same condition ensures the Bayes optimal classifier to satisfy multiple fairness criteria simultaneously, viz., demographic parity, equal opportunity, equalized odds.
fixed cost matrix for 0-1 loss, and consider the Bayes
The proof for Proposition 3.3 is given in Section 1 of the supplementary material. It is interesting to note that the same parametric conditions imply that the Bayes optimal classifier on the corresponding distribution simultaneously satisfies multiple fairness criteria, viz., demographic parity, equal oppor- tunity, and equalized odds. Moreover, the same condition works for both univariate Gaussian and log- log-normal distributions. Using our proof technique. it is easy to derive similar conditions for other parametric families too. We now present a small proposition to link our intervention to a widely used reweighing intervention in the fairness literature 43].
Remark 3.4. (Our conditions imply Kamiran and Calders 43 Intervention) Kamiran and
Calders
43 reweighing method essentially reweighs Qia by multiplicative factor of
Pr (Y Pr (A a) /Pr (Y = i, Let us call the resulting probabilities qie- Using qa. PY E Pio and Pr (r
Σiey qao we get yta oc Di (E.eA (En q.)/q... Hence, 9ta/āje = 9tae/áge/ E.e 950 9te/EEE 9sa; Vi, E y and a, a' e A. is the same condition on qia 's stated in Proposition 3.2 Thus, our result can be thought of as a second stage pre-proceesing of Pi. distributions after applying the reweighing of Kamiran and Calders (43) gia 's in the first stage.
Pr Pr = a)
Remark 3.5. (Limitation of (30 As an interesting consequence, our conditions on pia and Ei imply When the classes are balanced, the error rate of the Bayes optimal classifier on group A = a in Đ equals 0.5(1 - drv (Poa. Pia)), where dry denotes the total variation distance [56]. Thus, achieving drv(Pon. Pio) = drv (Po,, P11) ensures equal error rates across both the groups However, there is closed form expression for the total variation distance between two univariate Gaussians, and KL-divergence can be thought of as a proxy using Pinsker's inequality [19]. This is similar to information theoretic argument used by Dutta et al. [30], which is why their optimization cannot guarantee outcome fairness in the way we do.
DKL Poo DKL
4 Finding The Nearest Optimal Distribution
When given distribution D is not ideal, then a natural question is find its nearest distribution ò that is ideal. We formulate this problem as follows.
minimize DKL. (DIID)
D is ideal
5

---

In the above optimization problem, the KL-divergence objective is well-known and convex but the constraint of D' being ideal is extremely non-trivial to express. We show that when the group and class-conditioned distributions X Y = i, A = a come from certain well-know parametric families be equivalently expressed
of distributions, this constraint parameters. We now give a concrete formulation of the optimization problem described in Section using the constraints derived in Proposition 3.2
constraint the distribution
min + 2
Σ
qia (Pta - Usa) 2To (jta - Hia)
2
Σ
qia
tr --1
ia Sia - log det (2,
(Dia) Sia)
2
(1,a)
($.a)
subject to E
qia' 1/2 (pin Mja) = Lna (Pia' Hja'): E11²
2 D!l? D - 1/2 2nd and
1/2
qia
Vi,j E V.a.a E.A
qia qja
For readability, we will work with binary class labels and binary group attributes in this section. However, all of our results can also be written down for multi-class and multi-group settings. In its full generality, the above problem is non-convex, and therefore, we will first propose reducing this to a convex optimization problem and then propose solving it in full generality by using line search.
4.1 A ffirmative Action
We first focus on class of interventions for which solving the optimization program efficient. We define Affirmative Action changing the underprivileged group to obtain the ideal distributions where fairness and accuracy are accord.
Theorem 4.1. Let (X,Y, A) denote the features, binary class label, and binary group membership. respectively, of a random data point from any data distribution D with qia Pr (Y = A a), for i {0, 1} and a E {0, 1]. such that q10/ fo/ooo = q11 //o-- Let X|Y = i, = a ~ N(pras Eia) be multivariate Normal distributions, with mean Pia IR and covariance matrix Eie E RRdd for i {0, 13 and a {0, 13. Let D denote a distribution obtained by keeping (Y, A) unchanged and only changing X|Y = i,A a to X i,A N(iee E,a). Then in the case of Affirmative action (changing only Pıo and Σng). we can efficiently minimize DxL.
function of the variables P:o and Σg subject to the constraints in Proposition 3.2 so that the Bayes optimal classifier on the optimal D guaranteed to be EO-fair.
(DID)
The proof for Theorem 4.1
given in Section of the Appendix. While show that the optimization program is convex, obtaining a closed-fomm expression for the change in means and covariances is extremely cumbersome for the general case. However, we show how the closed form expressions for t.o and ō look like for the univariate case in the following Corollary:
Corollary 4.2. (Univariate Affirmative Action) For the case where X|Y = i, a N (Pia a are univariate normal distributions, for i, E 10, the optimal distribution D from Theorem 4.1 with 7" being a function of the original distribution parameters, can be written down as:
P00 7" (Pot
M11)
110 + 910
900
o
a = on. Po00 = Foo0 + ( - p11), and Piu
qoo
910
+
0
4.2 Changing all Subgroups
Another intervention we can follow is to change all the subgroups of the given distribution. However, a quick check through the proof of Theorem shows that this will lead to a non-convex program. However, just like Corollary 4.2 we can show a reasonable intervention for the univariate case, where we change all four subgroups and search over a non-convex function using line search over a fairly large grid size.

---

Propasition 4.3. (All subgroup change for Exact Fairness) Let (X, Y, A) denote the features, binary class label, and binary group membership, respectively, of a random data point from any data distribution D with qia = Pr (Y = =. A == a) i (0,1) } and a E {0, 13, such that quo goo = 911/qox and let X|Y = = a Nipia o%/a) be univariate normal distributions, for i {0. 13 and a E 10.1}. Let denote a distribution obtained by keeping (Y, unchanged and only changing X|Y i, a to X = i,
[ira: R.a). Then minimizing DKL (DIID)
as a function of the variables Pio and os subject to the constraints in Proposition 3.2 leads non-convex program. Furthermore, let r = arg min for some non-convex fnncciooexiunncionn of that is -E(U,>))
only dependent on the original distribution parameters. Then, all the new distribution parameters jio and oi can be expressed as a function of and the original distribution parameters pia and dza-
The proof of Proposition 4.3 is provided in Section of the Appendix. We can also derive worst-case upper bounds on the error rate and the unfairness gap A of the Bayes optimal classifier h on D with respect the original distribution D. These bounds show that both the accuracy loss and the fairness gap depend only on the KL divergence between D and D. It also shows that the optimal value of our optimization problem be used to approximately translate the accuracy guarantee of h from D to D.
Proposition 4.4. Let err(h, D) denote the error rate (expected 0-1 loss) of a classifier h on the distribation D. Let drv (rr DD. denote the total variation distance between two distributions and D. while D&L l denotes the KL-Divergence between them. Denote the Bayes optimal classifier on the ideal distribution D as h (and similarly the Bayes optimal classifier h. Then, we can bound the error rate and the Equal opportunity of h on the original distribution D as follows:
lerr(h, D) err(h, DY1 S (2DKL(D.)) and AEO D) s V8DKA (DIID).
The proof for Proposition 4.4 is given in Section B of the Appendix. The above bounds informs us that when the ideal distributions are close enough the true distribution, we do not lose much when going towards an ideal distribution.
5 Case Study on Gaussian Distributions
In this section, we adapt a stylized Gaussian setting from prior work (Definition 3.1 in (59, Section 5.3 in [4] to examine unfairness and Bayes optimal error before and after distributional interventions. We assume homoskedastic Gaussians within each group A a, i.e., doo = aias so that the Bayes classifier admits threshold form. This enables tractable analysis of interventions aimed at achieving a aaccuass-acuuracy trade-off. Further details are provided in Section the Appendix.
We evaluate four interventions: (a) the Bayes optimal classifier on the original distribution (no correction). (b) EF Affirmative: transforming the underprivileged group (A 0) via the univariate KL program in Corollary (4. (c) EF-All Subgroups: modifying all groups to minimize KL diver- gence to the original distribution under exact fairness constraints (Proposition 4.3) and (d) Mean Matching: aligning group means while minimizing KL divergence (Proposition B.6 in Section of the Appendix). Note that 'EF-All Subgroups' involves non-convex optimization, for which we apply line search to estimate the optimal mean-variance scaling factor 7. We report Demographic Parity (ADP), Equal Opportunity (AEO), KL and Jensen-Shannon divergence, as well as Bayes Error (BE) under each intervention. For the univariate Gaussian case, the Bayes-optimal thresholds are known in closed form (LemmalA.) in Section A of the Appendix), and allow precise computation of ADP and AEO via standard Gaussian CDFs.
We have already demonstrated the effect of different interventions for the case of high AEO in Figure T To cover another interesting case, we look at distribution where ADP is very high in Figure 2 Here, the affirmative action intervention transforms both the ander-privileged subgroups to high-variance ones, which results in a reduction of BE and ADP, but at the cost of high KL/JS- divergence with respect to the true distribution. However, the EF-All intervention simply tries to match the variances of uundr--priillee fairness and accuracy while staying close to the true distribution. Mean Matching very similar to EF Affirmative in this case and, as result, has relatively high KL/JS numbers. Due to a lack of
and privileged subgroups and, as a result, achieves perfect
7

---

1.6
DC-6.666 APP 6.56 1EO 0,00
DE: 0.00000 -SP-6.00 00, (FO: 0:20
16
05:0.00 00.0%,0000 0.00
1.0
35:0.00 650 15,,600:.00
0.22
0.1
0.
(5:
3.0
0.2
02
02
12
0.5
00
30
Original Distribution
(b) EF Affirmative
(c) EF-All Subgroups
(d) Mean Matching
Figure 2: Comparison different interventions when the ADP on the original distribution high. In this case, EF-All manages to stay close to the true distribution and achieves perfect fairness and error rate, while others deviate significantly.
space, show more results for different settings: (1) same but shifted group distributions (Figure 5). (2) the case of AEO being close to 0 (Figure 6), and (3) the case of different cost sensitive risk ( 3/4 on n1x, a)) (Figure 7. in Section of the Appendix.
6 Applications: Steering the Representations of an LLM
We now empirically demonstrate how our guarantees can improve representation and generation steering in LLMs active area with limited theoretical grounding [37] 67 79 68 Adopting the
setup of Singh et al. [67]. we first apply affine steering to pretrained LLM representations to reduce class-specifie TPR gaps the Bias in Bios dataset (27). We then use the framework of Zhao et al.
[79 to steer generations movic reviews, showing that our intervention improves expressed joy for the underperforming group
6.1 Reducing Per-class disparity in Multi-class Classification
In this experiment, we aim to steer data representations to reduce disparities between groups in a multi-class classification setting. We use the Bias in Bios dataset [27). which comprises web-soureed biographies labelled by profession and annotated for gender. The task is to predict the profession from the biography while ensuring fairness with respect to chosen metric. Our experimental setup closely follows that of Singh et al. [67]. who generate representations using the Llama-2 7b model 722 and propose method called MiMiC (Mean+Covariance Matching), which steers representations via least squares alignment of the first two moments. They measure the TPR-gap (Definition 2.1). [P] A) y Y y, A = 1] P(M(X, A)
defined TPR-gap,.hh
Y y, A OJ.
They demonstrate that MiMiC significantly reduces the TPR-gap on the Llama-2 7b embeddings of the Bios dataset. We adopt the same experimental pipeline as Singh et al. [66]; further details are provided in Section D.I of the Appendix. Our intervention, referred to as EF Affirmative in Figure 3, begins by estimating the first two moments of the representations from Singh et al.'s pipeline. We then apply our Affirmative Action framework to compute target moments corresponding to an ideal distribution. As in Singh al., we estimate affine transformation parameters that map the original moments to the target ones and apply this transformation to all data representations. We use multi-class generalization of our optimization program in Theorem and lay down the details of the program and intervention in Section D.I of the Appendix.
We additionally evaluate the LEACE transformation { which Singh et al. [66] used as a baseline. Figure reporeBreports reports the not-mean-squaal TPR-gap across classes for various methods. Our intervention consistently reduces TPR-gaps across all classes and, in many cases, outperforms both MiMiC and LEACE. This provides empirical support for our approach: actively searching for and aligning with an ideal distribution can meaningfully reduce group disparitics in representation lcarning.
6.2 Steering Activations for Joyful Generation
We steer LLM generation towards joyful movie reviews using the Gaussian Concept Steer (GCS) framework of Zhao et al. (79), which samples steering vectors from a Gaussian of N( Σ for concept e layer This improves robustness across models and datasets compared to using single vector. Concept vectors for joy are estimated from GPT-4o [40] outputs and used to steer a Llama-3

---

RMS TPR Gap Different Professions
0.35
Steering
Mcan Matching
0.30
MIMIC (Mean+Cow Matcaing)
LEACE
RMS TP3 Gap
0.25
Affirmative
0.20
0.15
0.10
0.05
0.00
tracher
professor
accountant
compaser
palater
surgeen
model
architect
pors train
hnt destgn
dontat
chirm
paralegral
di Doet
rappor
minnkkrr
soft, ong.
yoja tch paychologist
nurso
distitan
pastor
cometlla
Journallst
photographer physician
attorney
Profession
Figure 3: TPR-gap between Gender groups for all professions. All methods to steer feature rep- resentations achieve roughly the same accuracy (in the range of 0.77-0.79). Our intervention (EF Affirmative) is able to significantly reduce the TPR-gap gap for all professions. In many cases, it is even comparable better than previous interventions Belrose al. 8 Singh et al. [67.
8B model [36. At each layer I, the final token representation h is updated as h - (1-a)- + where a is the strength of the steering. Zhao et al. [79] experiment with values of a E (0.03, 0.08). We a 0.03 for our experiments. Following [7],, we evaluate the joyful tone using GPT-4 oracle with fixed decoding parameters. We apply the GCS framework steer movie review generation toward joy over anger, focusing on two groups: comedy and horror reviews. For each group, estimate Gaussian distributions of steering vectors for both directions (joyful angry and angry - joyffl), with the latter used for intervention. Full details are in Section D.2 of the Appendix.
While steering improves joyfulness overall, its effectiveness varies between groups To address this, we apply our EF Affirmative intervention to nudge the joyful vector for the horror group. Let N Nie, E% be the original distribution and (( (%) be the distribution obtained after applying EF Affirmative intervention (Theorem 4.1 We define i (1 a) vf + a oy where v N Σ and 8 (pL Σy). The final steering step updates the last-token representation layer 1 as h a) h a..
Figure 4 presents the simulation results. We report the change in joyfulness (denoted as A-Joyful) before and after steering for both the comedy and horror review groups. For the horror group, additionally apply our EF intervention, denoted as Steering+Ei to adjust the steering vectors. We vary the nudging parameter a, where al. [79]. As expected, moderate values of a improve joyfulness in the horror group, but excessive nudging distorts the steering vector, reducing its effectiveness.
0 corresponds to the original steering vectors from Zhao
7 Discussion and Future Work
We address fair classification by introducing the notion of ideal distributions-thoss that admit the Bayes-optimal classifier satisfying exact fairness. For common parametric families, we show that identi-
Joytul Score
fying such distributions reduces to a KL divergence minimisation problem, subject to fairness constraints on the Bayes-optimal classifier. We characterize conditions under which this problem is feasible and tractable. Through simulations, we demonstrate that the resulting interventions can steer the original dis- tribution towards both perfect fairness and Bayes- optimal accuracy while remaining close in distri- butional distance. Additionally, we show that our method extends to post- post-training interventions, effec-
Joyoyful Horror (Steering EF)
d-joytul Comedy
6-joytul Horror (only Steering)
0.0
0.2
0.4
0.6
0.8
1.0
Alpha
Figure 4: Change in Joyful score(A-Joyfi) before and after adjusting the steering. Our intervention (EE Affirmative") pushes up the effectiveness of steering the 'Horror' group, relative the 'Comedy' group.

---

tively steering learned representations toward desired behavioural outcomes.
We study the foundational problem of finding the nearest ideal distribution to given distribution. This has many potential applications, such as correcting training data bias, learning fair representations, steering intermediate representations for fair generation, etc. Fair pre-processing or reweighing closer our mathematical setup, and steering intermediate representations gives a compelling application for LLMs. Our KL optimization program can be used to measure training data bias and guide data collection policies in practice, especially when the models are trained for fairness, but the inherent bias in data is unresolved.
Our theoretical results are on the faimess of the Bayes optimal classifier and the corresponding optimization programs, which are tractable and mathematically easier to analyse for parametric families of distributions, e.g., multivariate Gaussians. Multivariate Gaussians may not always fit real-world data, but can be justifiably used to model concepts and internal representations [79 32). There can be scenarios and applications where parametric assumption may not fit well with the input data distribution, and hence, analyzing the Bayes optimal fair classifier will become highly non-trivial.
An important extension is to determine how to steer a given distribution within a fixed budget so that exact fairness constraints satisfied. This is relevant when deviations from the original distribution are costly or infcasible, and can inform data collection or active learning strategies for fair classification 33 42 3 In such cases, it may be necessary relax the fairness constraints and seek approximate solutions, prompting the need for efficient algorithms that operate under finite-sample settings. In finite-sample regime, we have high probability estimates of the moments and other distributtonn-deeeddent quantities, so our approach leads to approximate fairesss guarantees. To go beyond distributional assumptions, we can leverage previous work that maps given data distribution to tractable, parametric families using invertible transforms that preserve Bayes error, so our results become applicable (7T). We can also assume distributions with bounded moments instead of parametric families, and still bound the Bayes eπror and other relevant quantities [5 This requires a careful analysis and is certainly a very important future direction.
The framework of ideal distributions can also be applied to audit deployed models for fairness, particularly when the evaluation data may itself be biased 2 65 9 offering pathway toward more
robust fairness evaluation protocols. Prior work on fairness audit has mostly focused on auditing a given model instead of auditing training evaluation data. There is long line of work to demonstrate
that the fairness-accuracy trade-off disappears when data bias is accounted for 74 [3 30 51 64, 48).
The KL gap in our formulation can be used as metric of data bias, and Theorem 4.1 essentially gives an algorithmic recipe to find the minimally different distribution relative to the given one.
Our conditions can also be incorporated into optimization objectives to steer data generation toward ideal distributions. We demonstrate this by applying post-training interventions to steer LLM repre- sentations for multi-class fair classification and emotion control. More generally, our optimization framework and parametric conditions can be embedded into the training objectives of generative models. Modern generative approaches such Variational Autoencoders (45) and Normalizing Flows (57] often assume parametric latent distributions, such as mixtures of Gaussians. Embedding fairness-aware ideality conditions into these objectives enables the generation of fair and accurate data distributions that remain close to the original. This represents a promising direction for fair generative modeling, an area of growing theoretical and practical interest 776
70 69].
Our work improves theoretical understanding of data bias and proposes an optimization program for steering data distributions towards provably exact fairness guarantees. The real-world socictal biases in data and the fairness harms are complex and more nuanced, where our fairness interventions can provide good guiding principles but silver bullet to solve the real-world problems.
Broader Impact:
Acknowledgments: Mohit Sharma would like to thank Prof. Masashi Sugiyama and the Imperfect Information Team at RIKEN-AIP for hosting him while the author was submitting and rebutting this paper. The authors would also like to thank Prof. Manuj Mukherjee at IIIT Delhi for initial discussions around Dutta et al. (30) and the optimization programs. Rajiv Ratn Shah is partly supported by the Infosys Center for AI, the Center of Design and New Media, and the Center of Excellence in Hcalthcare at IIT Delhi.
10

---

References
[1] Alckh Agarwal, Alina Beygelzimer, Miroslav Dudik, John Langford, and Hanna Wallach. A reductions approach to fair classification. In International conference machine learning. pages 60-69. PMLR, 2018.
[2] Nil-Jana Akpinar, Manish Nagireddy, Logan Stapleton, Hao-Fei Cheng, Haiyi Zhu, Steven Wu, and Hoda Heidari. A sandbox tool to bias (stress)-test fairness algorithms. arXiv preprint arXiv:2204. 102333 2022.
[3] Pranjal Awasthi, Alex Beutel, Matthäus Kleindessner, Jamie Morgenstern, and Xuezhi Wang. Evaluating fairness of machine learning models under uncertain and incomplete information. In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency, pages 206-214, 2021.
[4] Chloé Bakalar, Renata Barreto, Stevie Bergman, Miranda Bogen, Bobbie Chern, Sam Corbett- Davies, Melissa Hall, Isabel Kloumann, Michelle Lam, Joaquin Quiñonero Candela, Manish Raghavan, Joshua Simons, Jonathan Tannen, Edmund Tong, Kate Vredenburgh, and Jiejing Zhao. Fairness on the ground: Applying algorithmic fairness approaches to production systems. CoRR, abs/2103.06172, 2021.
//tpxvaaaiiv.v///a//aa wwg/abs//////eeeee/oooe 06172
[5] Mislav Balunović, Anian Ruoss, and Martin Vechev, Fair normalizing flows. arXiv preprint arXiv:2106.05937, 2021.
[6] Solon Barocas, Moritz Hardt, and Arvind Narayanan. Fairness and Machine Learning: Limita- tions and Opportunities. fairmlbook.org, 2019. http::t:t///w..aaaaalbbooo/// www.mlaoo............ org
[7] Rachel K. E. Bellamy, Kuntal Dey, Michael Hind, Samuel C. Hoffman, Stephanie Houde, Kalapriya Kannan, Pranay Lohia, Jacquelyn Martino, Sameep Mehta, Aleksandra Mojsilovic, Seema Nagar, Karthikeyan Natesan Ramamurthy, John Richards, Diptikalyan Saha, Prasanna Sattigeri, Moninder Singh, Kush R. Varshney, and Yunfeng Zhang. AI Fairness 360: An extensible toolkit for detecting, understanding, and mitigating unwanted algorithmic bias, October 2018. URL httpts////axix http://aaxvvvoraraaaaa/ abs///1bs/1111//.//// 01943
[8] Nora Belrose, David Schneider-Joseph, Shauli Ravfogel, Ryan Cotterell, Edward Raff, and Stella Biderman. Leace: Perfect linear concept erasure in closed form. Advances in Neural Information Processing Systems, 36:66044-6606,, 2023.
[9] Karan Bhanot, Ioana Baldini, Dennis Wei, Jiaming Zeng, and Kristin Bennett. Stress-testing bias mitigation algorithms to understand fairness vulnerabilities. In Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society, pages 764-774, 2023.
[10] Sarah Bird, Miro Dudík, Richard Edgar, Brandon Horn, Roman Lutz, Vanessa Milan, Mehrnoosh Sameki, Hanna Wallach, and Kathleen Walker. Fairlearn: A toolkit for assessing and improving fairness AI. Technical Report MSR-TR-2020-32. Microsoft, May 2020. URL ma/
https://ake Backaa.aaa/a/ hte/ffarleaane www.reeeiaannppppeeeeeee whitepaper
[11) Sumon Biswas and Hridesh Rajan. Fair preprocessing: towards understanding compositional fairness of data transformers in machine learning pipeline. Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium the Foundations of Software Engineering, ESEC/FSE 2021, page 981-993, New York, NY, USA, 2021. A Association for Computing Machinery. ISBN 978 1450385626. 1145 3468264 3468536
doi: 10. 00.1145/3462264.44665333 URL
httpps://oi..xx:
[12] Christina Hastings Blow, Lijun Qian, Camille Gibson, Pamela Obiomon, and Xishuang Dong. Comprehensive validation reweighting samples for bias mitigation via aif360. Applied Sciences, 14(9), 2024. ISSN 2076-3417. doi: 10.3390/app14099822. URL httph:tt///// //www mdpi.com/ 341776/-331//4///822
[13] Avrim Blum and Kevin Stangl. Recovering from biased data: Can fairness constraints improve accuracy? In Symposium on Foundations of Responsible Computing (FORC), 2019.
[14] Avrim Blum, Princewill Okoroafor, Aadirupa Saha, and Kevin Stangl. On the vulnerability of fairness constrained learning malicious noise. arXiv preprint arXiv:2307.1189) 2023.

---

[15] Stephen Boyd. Convex optimization. Cambridge UP, 2004.
[16] Joy Buolamwini and Timnit Gebru. Gender shades: Intersectional accuracy disparities in commercial gender classification. In Conference on fairness, accountability and transparency, pages 77-91. PMLR, 2018.
[17] Toon Calders and Sicco Verwer. Three naive bayes approaches for discrimination-fie free classifi- cation. Data mining and knowledge discovery, 21:277-292. 2010.
[18]
Flavio Calmon, Dennis Wei, Bhanukiran Vinzamuri, Karthikeyan Natesan Ramamurthy, and Kush Varshney. Optimized pre-prrcsssigg for discrimination prevention. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, ed- itors, Advances Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL httpshhttss//ppooeeeeeeiiie neurdddddd neurips. cc/ wccppape__aeee www.fflllpppaeee /tt.es//ppperr/017///2ee pdf
[19] Clément I.. Canonne A short note an inequality between kl and tv, 2023. URL https
//arxiv. http:/..orrgab///22/// 07198
[20] I. Elisa Celis, Lingxiao Huang, Vijay Keswani, and Nisheeth K Vishnoi. Fair classification with noisy protected attributes: framework with provable guarantees. In International Conference on Machine Learning, pages 1349-1361. PMLR, 2021.
[21] Mattia Cerrato, Marius Köppel, Philipp Wolf, and Stefan Kramer. years of fair representations: Challenges and opportunities, 2024. URL UtR.httss://axxi org/ abs/ /ass//400 /207 008844
[22] Jiahao Chen, Nathan Kallus, Xiaojie Mao, Geoffry Svacha, and Madeleine Udell. Fairness under unawareness: Assessing disparity when protected class is unobserved. In Proceedings of the conference on fairesss accountability, and transparency, pages 339-348, 2019.
[23] Kristy Choi, Aditya Grover, Trisha Singh, Rui Shu, and Stefano Ermon. Fair generative modeling via weak supervision. In International Conference on Machine Learning, pages 1 1887-1898. PMLR, 2020.
[24] Evgenii Chzhen, Christophe Denis, Mohamed Hebiri, Luca Oneto, and Massimiliano Pontil. Leveraging labeled and unlabeled data for consistent fair binary classification. Advances in Neural Information Processing Systems, 32, 2019.
[25] Thomas M Cover. Elements of information theory. John Wiley & Sons, 1999.
[26] Bo Cowgill, Fabrizio Dell' Acqua, Samuel Deng, Daniel Hsu, Nakul Verma, and Augustin Chaintreau. Biased programmers? or biased data? field experiment in operationalizing ethics. In Proceedings of the 21 ACM Conference on Economics and Computation, EC '20, page 679 -81, New York, NY, USA, 2020. Association for Computing Machin- ery. ISBN 978 478145937..
doi: 10. (0.1145/3911403333999
URL wttph::t////ttt .ociii.on org/10
1 1145/ 1145/3391403 3399545
[27] Maria Dr-Arteaga, Alexey Romanov, Hanna Wallach, Jennifer Chayes, Christian Borgs, Alexan- dra Chouldechova, Sahin Geyik, Krishnaram Kenthapadi, and Adam Tauman Kalai. Bias in bios: A case study of semantic representation bias in a high-stakes setting. In proceedings of the Conference on Fairness, Accountability, and Transparency, pages 120-128, 2019.
[28] Lue Devroye, László Györfi, and Gábor Lugosi. The Bayes Error, pages 9-20. Springer, 1996. ISBN 978-1-4612-0771155 doi: 10. 00.00077978-14441-00711
[29] Michele Donini, Luca Oneto, Shai Ben-David, John S Shawe-Taylor, and Massimiliano Pon- til. Empirical risk minimization under fairness constraints. Advances in neural information processing systems, 31, 2018
[30] Sanghamitra Dutta, Dennis Wei, Hazar Yueksel, Yu Chen, Sijia Liu, and Kush Varshney. Is there trade-off between fairness and accuracy? perspective using mismatched hypothesis testing. In International conference machine learning, pages 2803-2813, 13. PMLR, 2020.
12

---

[31] Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Richard Zemel. Fairness through awareness. Proceedings of the 3rd Innovations Theoretical Computer Science Conference, ITCS *12, page 214-226, 2012. ISBN 9781450311151.
[32] Daniel Eftekhari and Vardan Papyan. On the importance of gaussianizing representations. arXiv preprini 250.:25005.00655. 2025.
[33] Charles Elkan. The foundations of cost-sensitive learning. In International joint conference artificial intelligence, volume 17, pages 973-978. Lawrence Erlbaum Associates Ltd, 2001.
[34] Sorelle A. Friedler, Carlos Scheidegger, and Suresh Yenkatasubrammnnian. The (im)possibility of fairness: different value systems require different mechanisms for fair decision making. Commun. ACM, 64(4): 143, March 2021. ISSN 0001-0782. doi: 10. 10.1145/3433949, URL
https:///oi..oo org/ 10. 1145/34335399
[35] Gene Golub. Some modified matrix eigenvalue problems. SIAM review, 15(2):318-3444 1973.
[36] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama
3 herd of models. arXiv preprint wrXiv:2407.21783, 2024.
[37] Chi Han, Jialiang Xu, Manling Li, Yi Fung, Chenkai Sun, Nan Jiang, Tarek Abdelzaher, and Heng Ji. Word embeddings are steers for language models. arXiv preprint wrXiv:2305.12795 2023.
[38] Moritz Hardt, Eric Price, and Nathan Srebro. Equality of opportunity in supervised learning. In Proceedings of the 30th International Conference on Neural Information Processing Systems, NIPS' 16, page 3323 3331, 2016. ISBN 9781510838819
[39] Kenneth Holstein, Jennifer Wortman Vaughan, Hal Daumé III, Miro Dudik, and Hanna Wallach. Improving faimess in machine learning systems: What do industry practitioners need? In Proceedings of the 2019 CHll conference on human factors in computing systems, pages 1-16, 2019.
[40] Aaron Hurst, Adam Lerer, Adam Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21226, 2024.
[41] Heinrich Jiang and Ofir Nachum. Identifying and correcting label bias machine learning. In International Conference Artificial Intelligence and Statistics, pages 702-712. PMLR, 2020.
[42] Eun Seo Jo and Timnit Gebru. Lessons from archives: Strategies for collecting sociocultural data in machine learning. In Proceedings of the 2020 conference on fairness, accountabiliry, and transparency, pages 306-316, 2020.
[43] Faisal Kamiran and Toon Calders. Data preprocessing techniques for classification without discrimination. Knowledge and information systems, 33(1):1-33, 2012.
[44] Toshihiro Kamishima, Shotaro Akaho, Hideki Asoh, and Jun Sakuma. Fairness-awar clas- sifier with prejudice remover regularizer. In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2012, Bristol, UK. September 24-28, 2012. Proceedings, Part II 23, pages 35-50. Springer, 2012.
[45] Diederik Kingma, Max Welling, et al. An introduction variational autoencoders. Founda- tions and Trends@ in Machine Learning, 12(4):307-39) 2019.
[46] Oluwasanmi O Koyejo, Nagarajan Natarajan, Pradeep K Ravikumar, and Inderjit S Dhillon. Consistent binary classification with generalized performance metrics. Advances in neural information processing systems, 27, 2014.
[47] Tosca Lechner, Shai Ben-David, Sushant Agarwal, and Nivasini Ananthakrisshan. Impossibility results for fair representations, 2021. URL https: //tp///aavvvv///////a//eaaeaaavv www.oogg///////e///o/o.o/eeeee// 2111/11177703333333
13

---

[48] Charlotte Leininger, Simon Rittel, and Ludwig Bothmann. Overcoming faimess trade-offs via pre-processing: causal perspective. arXiv preprint arXiv:2501.14710 2025.
[49] Liu. Zenan Li, Yuan Yao, Feng Xu, Xiaoxing Ma. Miao Xu, and Hanghang Tong. Fair representation learning: An alternative to mutual information. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD '22. page 1088-1097, New York, NY, USA, 2022. Association for Computing Machinery. ISBN 978 9781450393850. doi: URL https: //do: org/r0../... 10. 1141/1114456511 3534678 3539302
00.1145/354678835339999
[50] David Madras, Elliot Creager, Toniann Pitassi, and Richard Zemel. Learning adversarially fair and transferable representations. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning. volume 80 of Proceedings of Machine Learning Research, pages 3384-3393, PMLR. 10-15 Jul 2018. htttp::s///ooceeeee mlr. http://////mmmmmaaaaaa html
[51] Subha Maity, Debarghya Mukherjee, Mikhail Yurochkin, and Yuckai Sun. Does enforeing fairness mitigate biases caused by subpopulation shift? Advances in Neural Information Processing Systems, 34:25773-25784 2021.
[52] Sandra Gabriel Mayson. Bias in, bias out. Yale Law Journal, 2218, 2019. URL https org/ wwt/ooggg/////////////oo//////// pptpg/oo__ppg222mmmmmm pdf
/ // //ww.yye www.yyyllaawwoooooo
[53] Daniel McNamara, Cheng Soon Ong, and Robert C. Williamson. Costs and benefits of fair representation learning. In Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society, AIES '19, page 263-270, New York, NY, USA, 2019. Association for Computing Machinery. ISBN 470443333 doi: 011114//3306188331177773 URL https///:// org/ 10.1146/3306618 3317964
[54] Aditya Krishma Menon and Robert C Williamson. The cost of fairness in binary classification. In Conference on Fairness, accountability and transparency, pages 107-118. PMLR, 2018.
[55] Chaitanya Murti and Chiranjib Bhattacharyya. Discedit: Model editing by identifying discrim- inative components. Advances in Neural Information Processing Systems, 37:47261-47221
2024.
[56] Frank Nielsen. Generalized bhattacharyya and chernoff upper bounds on bayes error using quasi-arithmetic means. Pattern Recognition Letters, 42:25-34, 2014. ISSN 0167-8655 doi: http:://doi.or//1100001001
201... URL htt.hhtps:/// URI.httts://www.ee
http::ppps//w..ccccnnne........ www.ssscieneedrret...cmm com/
[57] George Papamakarios, Eric Nalisnick, Danilo Jimenez Rezende, Shakir Mohamed, and Balaji Lakshminarayanan. Normalizing flows for probabilistic modeling and inference. Journal of Machine Learning Research, 22(57): 2021.
[58] Kaare Brandt Petersen, Michael Syskind Pedersen, et al. The matrix cookbook. Technical University of Denmark, 7(15):510, 2008
[59] Emma Pierson, Sam Corbett-Davies, and Sharad Goel. Fast threshold tests for detecting discrimination. In Amos Storkey and Fernando Perez-Cruz, editors, Proceedings of the Twenty- First International Conference on Artificial Intelligence and Statistics, volume 84 of Proceedings of Machine Learning Research, pages 96-105, PMLR, 09-11 Apr 2018. URL https:// proceedings. mlr. paress/v44/piereee
html
[60] Drago Plečko and Nicolai Meinshausen. Fair data adaptation with quantile preservation. Journal of Machine Learning Research, 21(242):1-44, 2020. URL hittp://jmlz. wrtp/ppers/p/////
19-966.. html
[61] Drago Plečko, Nicolas Bennett, and Nicolai Meinshausen. fairadapt: Causal reasoning for fair data preprocessing. Journal of Statistical Software, 110(4))1-35, 2024. doi: 10. 18637/jss.v 10. i04. http:httt////// http://////w..ttttsssttttt jstatsoft. wwwg/riggnd...... httppp///sssstaaaaee//e/////////
14

---

[62] Ashesh Rambachan and Jonathan Roth. Bias In, Bias Out? Evaluating the Folk Wisdom. In Aaron Roth, editor, 1st Symposium on Foundations of Responsible Computing (FORC 2020), volume 156 of Leibniz International Proceedings in Informatics (L.PPcss) pages 6:1-6:15, Dagstuhl, Germany, 2020. Schloss Dagstuhl Leibniz-ZZentrum für Informatik. ISBN 978- 3-95977-142-9. doi: 10.42:00L.PPcssFORR 2020.6. URL https://drops: www.ddoos.dgssuuul...... dagstubl .d/ entitiees/oocment// 10. 4230/LIPIIs...../ 2020.6
[63] Clayton Scott. Calibrated asymmetric surrogate losses. Electronic Journal of Statistics, 6(none): 958 992, 2012. doi: 10. 1211///---- URL httts:://doi..oo
10t112114/112 12-44/12-BB EJS699
[64] Mohit Sharma and Amit Jayant Deshpande. How far can fairness constraints help recover from biased data? In Proceedings of the 41st International Conference on Machine Learning. ICML'24. JoLR.org. 2024.
[65] Mohit Sharma, Amit Deshpande, and Rajiv Ratn Shah. On testing and comparing fair classifiers under data bias. arXiv preprint UrXiv:2302.05906 2023.
[66] Shashank Singh and Justin Khim. Optimal binary classification beyond accuracy. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Informa- tion Processing Systems, 2022. htttttts:///peerertttee wwtperewwwwiiieeefoeeeeeeeeeeeee moet/forumiid=pmm@ Y8unkkJJ
[67] Shashwat Singh, Shauli Ravfogel, Jonathan Herzig, Roee Aharoni, Ryan Cotterell, and Ponnu- rangam Kumaraguru. Representation surgery: Theory and practice of affine steering. arXiv preprint arXiv:2402.09631, 2024.
[68] Daniel Tan, David Chanin, Aengus Lynch, Brooks Paige, Dimitrios Kanoulas, Adrià Garriga- Alonso, and Robert Kirk. Analysing the generalisation and reliability of steering vectors. Advances in Neural Information Processing Systems, 37:139179-139211 2024.
[69] Christopher Teo, Milad Abdollahzadeh, and Ngai-Man Man Cheung. On measuring fairness in generative models. Advances in Neural Information Processing Systems, 36, 2024.
[70] Christopher TH Teo, Milad Abdollahzadeh, and Ngai-Man Cheung. Fair generative models via transfer learning. In Proceedings of the AAAI conference artificial intelligence, volume 37, pages 1429-2437, 2023.
[71] Ryan Theisen, Huan Wang, Lav R Varshney, Caiming Xiong, and Richard Socher. Evaluating state-of-the-art classification models against bayes optimality. Advances in Neural Information Processing Systems, 01199367-3777 2021.
[72] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint 2tt.:2337.09888, 2023.
[73] Boris Van Breugel, Trent Kyono, Jeroen Berrevoets, and Mihaela Van der Schaar. Decaf: Generating fair synthetic data using causally-aware generative networks. Advances in Neural Information Processing Systems, 34:22221-22233 2021.
[74] Michael Wick, Jean- -Bptiist Tristan, et al. Unlocking fairnsss: a trade-off revisited. Advances in neural information processing systems, 32, 2019,
[75] Zikai Xiong, Niccolò Dalmasso, Alan Mishler, Vamsi K. Potluru, Tucker Balch, and Manuela Veloso. Fairwasp: fast and optimal fair wasserstein pre- prepprocessing. In Proceedings of the Thirty- Eighth AAAI Conference on Artificial Intelligence and Thirty-Sixti Conference on Innovative Applications of Artificial Intelligence and Fourteenth Symposium Educational Advances in Artificial Intelligence, AAATAAAAAAT2AAAATTTT AAAI Press, 2024. ISBN 978-1-57735- 887-9. doi: 10. 10.1609/aaai 030.9//2ai..881142955555 htRps:/ts:://dd org/ 10. 1609/ aaai. v38i14 29545
[76] Depeng Xu, Shuhan Yuan, Lu Zhang, and Xintao Wu. Fairgan: Fairness-aware generative adversarial networks. 20/8 IEEE international conference on big data (big data), pages 570-575. IEBE, 2018.
15

---

[77] Rich Zemel, Yu Wu, Kevin Swersky, Toni Pitassi, and Cynthia Dwork. Learning fair rep- resentations. In Sanjoy Dasgupta and David McAllester, editors, Proceedings of the 30th International Conference on Machine Learning, volume 28 of Proceedings of Machine Learn- ing Research, pages 325-333, Atlanta, Georgia, USA, 17-19 Jun 2013. PMLR. URL https://proceeedingg mlr weteasssvv//eeeeeell html,
[78] Xianli Zeng, Edgar Dobriban, and Guang Cheng. Fair bayes-optimal classifiers under predictive parity. Advances in Neural Information Processing Systems, 35:27692-2777 2022.
[79] Haiyan Zhao, Heng Zhao, Bo Shen, Ali Payani, Fan Yang, and Mengnan Du. Beyond single concept vector: Modeling concept subspace in Ilms with gaussian distribution. arXiv preprint arXiv:2410.00153, 2024.
A Proofs for Section 3: Ideal Distributions for Fair classification
We will require helper result about threshold classifiers to prove our next set of results.
Lemma A.1. Let 7( a) Pr Pr =1|X- = x, = a). qia Pr (Y = i = a) and Paa(a)
Pr (X- x Y a). a). Then the Bayes optimal classifier can be written as h (x,a) a)
I (log (z)
2 log 90a
pau (z)
qla
Proof. Let n a)
= 1 9aa Pr and (i)
Pr (r = Y --=iAA
We consider group-aware threshold classifiers on D of the form ha(xaa) = (()(x,a) 2 t), which can be equivalently written as
(1(x, [1)(2, a) 21)
(Pr (Y = 1
X = x, = 2t)
Pr (Y
X x A=
2
Pr 0 X x,
1-t
Pr 1,X =1
1
IV
Pr(Y = 0,X = = -
Pr (X x Y = 1,
Pr 1, A a)
Pr (X = x
Y = 00 Pr = 0, A = 1-t
Pia (z)
40a
-T
Poa (x)
1-1
aia
Pia (ia(x)
t
qua
log
log
log Pia
Poa (I)
well-known that the group-aware Bayes optimal classifier h* huja by setting 1/2, or equivalently.
-1(1110 [og Pia(x) (x)
(²) = hy/(1, =I a)
> log
90a
Qla
We now prove Proposition 3.2 from the main paper.
Proposition A.2. (Proposition 3.2 in the main text) Let (X, Y, A) denote the features, class label, and group membership, respectively, of a rundom data point from any duta distribution D with qsa = (Y = i, a), for E y and a E A. Let X = i = a ~ N( Eta) be multivariate Normal distributions with mean Pia E RRd and covariance matrix Eig E Rdddd for i y and a E A. If the means pia and the covariance matrices Eia satisfy
1-1 (pra M4aa) Eio' (Pia' Mjja)) and
1/2
1//2 1 >2/2 11/22-111 1/2 io!
qio Iia' Vi.j E V,a, A.
and
qia qia'
then the group-aware Bayes optimal classifier on D satisfies equal opportunity.
16

---

Proof. The Bayes optimal classifier for group A = in multi-class setting can be written down as maximum over posterior probabilities:
he (2r arg max Tly (2,(2,)), a); where my (2, a) Pr Pr y 2, a yey
We can say that hn (a a) = y, whenever the following happens:
he (Ir, a) arg max my a) = (0 1, =1 =1(10g Pyo (X)
[yl(x, (T(
2 loo
No
Vi EY)
yEy
Pio Pia(X)
Aga
Using the above simplification, the EO-fairness condition Pr ( (X, = Y = V, a) = Pr (h* (X, Y = (AA=aa) Vy E EV,, a, E A means Pr
log Pyo (X)
Tio
2 log
Vi E Y y, A =0
Pia (X)
Aya
= Pr log Pye (X)
qaal Viey Y = y, A --A=d))
2 log
Dia (X)
Yyal
Since X|Y = i, a N(uia Eia) are multivariate Normal distributions, their probability densities are
(=a(x) (2x)-
4/2 det ( -1/2 exp (-2(2 (x x msa) I (x (saa)
Now write
Pya (x)
log
Pia (1)
((x - -Hia) (ia))'Eij Din (x pia) - (x - Mya) Dyd Dydlx (x= - pya) + log det (Eao) log det Ega)
(I%// + pya Mia) Dio (Ill + Hya Uha) TT - log det 1/2 Ija you
1=112)
2
by substituting E1/22 r Mya where NN Taxd)
1 r 1/2/ 1/2 r + (Pya Mia) Σー I 11/² +
2 (Mya Mia) x (Hya pio)
2 ta
TT log det 1 ia 2//²2)
2
Let us denote the above expression as Eys (r). We can now write the group TPR as:
Pr
log
Pya (X) (X)
2 log
qia
Vi
E y
= y, A A= = Pr (R) 2 log
Gia
Vi E EV)
4ya
Aya
for R
NO, Idxa). Now if we have
Yya
9ya'
and
Iba
gia'
Eyo ( pia) = Ey (Myaa
Pear)
and
11 -
10 11/2
200" -ga' "a'
then the probability of the above event written terms R ~ NO, Taxa) becomes identical Va, a' A, i E V. Hence, the Bayes optimal classifier satisfies equal opportunity with these set of conditions.
Proposition A.3. (Proposition 3.3 in the main text) Let (X,Y, Y,Y.) A) denote the features, binary class label, and binary group membership, respectively, of a random data point from any data distribution D with qia Pr = A= a), for {0, 1) and a {0,1). and let X|Y i, N (Hpaa 22 univariate normal distributions, for i E {0, 1} and a E {0, 1). Then the distribution
D is ideal for equal opportunity (see Definition 3.1) if and only if
Po1
P11
μ100
P10
011
T10
910
911
010
001
000
goo
9o1
17

---

Proof. For any cost matrix C' E R the group-award classifier that minimizes its corresponding
cost-sensitive risk is given by I (17( x., a) 2 tc). for a threshold & (C10 - Cgo)/ Coo + Co1
en) E [0, 1]; see Equation (2) in 33 and 663 The distribution D is ideal for equal opportunity if
Pr (m{X, .A 2 Y = = = Pr A)) 2 t t Y = i, A = 1), for all thresholds E [0. 1]
and i E {0, 1]. Since the CDFs are identical, the random variables vxX,,) A) Y = i, A = 0 and
nXx A) Y i, A 1 must be identical. Note that
7(2, = Pr (Y = X = ==
-
Pr PY
-, E Pr Pr = A= Pr X = x Y= -
P1(Y=1,A=d)Pr(X
na (1)
Eizo qia (z)
(x Ma)) )2
na exp
20
(x Mio)
Ei Ta exp
20 ia
(x M1a) 2
(x M0o)2
+ log 11
exe
201a
20
Q0a
(Hia 10 sa
M1aa22 2
(Usa + 10 Moa) )2
log 11a
exp exp
20
20
Q0a
for f = 0
O0a Moa), Rlg
++ (Maa Moa) )2
20a 1aa
1 1 exp
-1) ,2
log
o%a
20%
Q1a
for i =
1 oia
p2 0la(non - MMia,, r + (Non m1aa² oa
01a
1 + exp
log
0ia
20%
Qia
If X- = i, A = a ~ N(maa 22 then X | Y = i ~ N (()) Thus, for (X, A) Y = i, A = 0 and nXX, A) Y= i, A = 1 to be identical, we must have
0
R² 000 - Mm0)
R ++ (n10 m20)3
+ log 100010
and
2 1
o
20%
010
R² 001(H11 - Mm)) R
(111 - Mo1)
901
+ log
2
on
20
011001
as identically distributed for R ~ NN 1). Similarly, must also have
0
R² 000(H00 - (10)
R+ (NOO - p10)2
+ log 100
and
2
000
2
20%
000
oi
R² 011(11 - M1))
R+ (HO1 - 111)
+ log 111011
2
dol
00
20
011001
as identically distributed for R ~ NO 1). Therefore, we must have
No1
P11
Mo0
1110
σ10
910
U1
and
and
T1
Goo
900
q01
In the other direction, it easier to prove that the above conditions imply the distribution to be ideal. It can be proved by simply backtracking the steps above.
D
18

---

B Proofs for Section 4
We first derive the KL divergence between two distributions, where each subgroup in the distribution follows multivariate normal distribution.
Lemma B.1. Let (X,Y, A) denote the features, binary class label, and binary group membership. respectively, of a random data point from any data distribution D with qıa Pr A a), for i y and A. Let X|Y i, A a N(pias Eia) be multivariate Normal distributions with mean pia E Ra and covariance matrix Eia E Rdxd. Let D denote a distribution obtained by keeping (Y, A) unchanged and only changing X|Y i, a to X i, A NN Eta)- Then,
DKL (DIID)
1
Σ
Dia (Pri - Mia) )² Exo (ihia - Hial
2
(i.a)
Σ
(tr
( -log log Ena)
aio
2
(1,0)
Proof.
DKL (DIID)
Pr x = x, ý = i,A=u) a
Σ Pr X = v Y = i,A a) a log
Pr (X Y, = i,A a)
(2)),a)
Σ
Pr ( À = a) Pr X = x y = ;;A==
(.,i.a)
Pr (v= A
Pr x= x ý = A =
log [ a (x=x Y Y
= Σ
Pr(Y=i,A = a) Pr [x=x| = iiA=
(0.1,a)
Pr i, A 0) Pr {x= x Y = i, 1,
log
Pr = =1,A = a) Pr
X -= Y = =1,1 =
Pr P { i,A=a
Σ Gio Σ Y log
Pr = x Y aa
(i.a)
Σ (1.0)
Via DKL (PiallPio
Pio denotes the distribution of X | Y = -i= a N(raa Eia) and Pro denotes the distribution of X y== = i, A a ~ N(iia ). Their probability densities are
(r)(x)) = (2xx) 4/2 del (Eto exp
(Ir Mio) ""1 mtsa)) and
(r)( = (2x) det (Sia) -1/2 exp
4/2
{xr -miaa ) 11-1100 (ssa))
19

---

respectively. Hence, the Kullback-Leible divergence between P.o and Piu can be written as
DKL
Psall P(o))
Pial (X)
E log
Y -1=,A==0
Pial (i)(
E (x viaa) " "mu 1 (X - Mia) -(X (Maa) ) T5-1 "xx - piia)
2
dot (oddctt(//250 101/25-11 11/2
| = i, 1.A=0 =
F. (x - mia) Dzu - [sa)) Y Y== =1, = a
log det [=
det((
1//25-1511/
= 2
2 2
-ia 2'a
using E |(X - Miia) EIna (X -X-1i (iia) Y = i, =
5-1 Lia = tr (Taxd) = d
E. (x - iMia + iiia- - uss) ( TT 1x fiia + Pia - M(a)
Y i, A
d + log (2)/
1/25--11!! -/0 L/ia
2
2
10
2 tr (ia))
Sia) + 1 + (Winn pio) T En (Miia pia) d
log det det( (2/1 2 1113)
The Kullback- Leibler divergence between D and D can now be written
DRL. (BIID)
Σ (2,a)
Yia DRL. (PiallPm))
+ 1 (Ria Miia) ( 2no (Pra - Mia)
d + log det (5/ 1-11-1111
1
Dia
tr
2i 2/00
2
2 2
(F,a)
1
Σ
qia (tr (IulSio) Fia) + (Nia Mia) E'E (itia pia) + log det det(t (1)/ 1/2 1111/-11111
-ia -ia
2
2
(3,a)
-
+
Σ
Yoa (Jtia - Usi)) )T "io (Nsza - Uia)) + 2
Σ
yia (ir [--1Smm
ia Sim) + log del EEia $t-1)
(i.a)
(K.a)
d
1
Σ
Isa (Fta Pia) ) T (Jta Mio) +
tr
D E log det( dogdeteDr
gia
2
(1,a)
(R.a)
Theorem B.2. (Theorem 4.1 in the main text) Let (X, Y, A) denote the features, binary class label, and binary group membership, respectively, of a random data point from any data distribution D with Pia = Pr(Y = A,A= a). for i {0, 1} and a E 1), such that quo q00 9ı1 O11/90 991 Let X|Y Eia) be multivariate Normal distributions, with mean p4a E Rd and covariance matrix Σia E Rdxd, i E {0,1) and a E {0, 1}. Let D denote a distribution obtained by keeping (Y, unchanged and only changing X = -, A = a. to X = A a ~ N(psa: Ega). Then in the case of Affirmative action (changing only t&o and Σ,o), we can efficiently minimize DKL (DIID) as a function of the variables ia and Egg subject to the constraints in Proposition 1.2, so that the Bayes optimal classifier on the optimal D is guaranteed to be EO EO-fair.
i, A = al=a~~((uu
Proof. Using Lemma B.I and Proposition 1.2, objective is to minimize
DKL. (1ID)
d 1
Σ
qia (Jria pia) End (Jta Mio)
Σ gia (Ir E (D Eia -log del( Se)) ((,a)
2
(i,a)
20

---

subject to the constraints
/ (N110 - Aoo) = E11 1-11//2 (M11 - P01)
and 200 -1
-11 -1
-10 =
-11
Suppose En and E,1 do not commute. The constraints can be equivalently rewritten as follows.
io Poo = 10 1102""11/2 - Po1) and Σn -0 0 16 -11 -11
-
$1 1/2 For any fixed positive semidefinite matrix I Rdxd, our optimization problem
Let I = ;0 11
can be divided into two separate parts that minimize
Σ (5.0)
qio Cirlara
Em (Pii (ia) subject to
io0 Ni
[ f001)
over Pia Rd, for i {0, 1), and minimize (after substituting D1/2 T
Σ 940 1=0
tr
D-1 +0
11122 10
2) )
log dot(Ew) (rE!!²) )
- + qa1 (tr Ei Em1
log det ( (In)Em1)
subject 1 21
$1/25511 201
symmetric, positive semidefinite matrix-valued variable E. E Rdxd for i E {0, 1). The first optimization in Pia is a constrained eigenvalue problem with linear constraints, i.e., minimize x" Ar b²6b subject x²' [35].
Let's consider the case of Affirmative Action, where we only change the means jh:o and the covariance matrices Eạ0 for the underprivileged group but keep those for the privileged group unchanged, i.e.,
Au = Ui and Σn Σn1 In that case, 51/2 TE and 51/2 = rΣ|| get fixed. By substituting
No0 Noo + [( Poi) = Poo [(1 (oi ) we only need to optimize
(Po0- - Eno (Āo0 Moo) + 9ı0 (No0 + [(111 - -01) 2 (Aoo+I + +I - -M01) - -110),
or equivalently (ignoring the terms independent of Poo).
P&o Eoo + Ejo) i0o 2 (Eoo Aoo + Σjo /10 Eio Ho1)) Aoo-
This is a convex objective in Poo because its Hessian is positive semidefinite, i.e., 900 900 + 11o > 0 [15]. By equating the gradient to zero, get the optimal solution for Aoo, and we denote by Moo (T). Thus, the optimal solutions (oo (io Moo (T), (T)(() for a fixed positive semidefinite
gaxd are given by
-1 (Eo0 /00 + ETO Mio T - moi)) and (T) (9002100' + 9100)T0))
voo(T) Eno + E1ooEi))
(2 40o + 2To P10 I - po1)) l' H1o)
E%o (T) (TE
(T) = (1221/3)2
By substituting these, when we look at the objective as a function of a positive semidef- inite matrix-valued variable 1, turns out be convex. This requires rewriting the expressions using the identities tr (AB) most importantly, tr (AXBX) tr
tr (BA), (AB) = det()), and and log det(AXXX))) for symmetric, positive semidefinite matrices A, B, X [58] The convexity of the objective in I' follows from the convexity of ir (AXBX) and log del (X for matrix-valued variable X. Finally, we can solve it efficiently to get the optimal I*
log det (A
Corollary B.3. (Corollary 4.2 in the main text) For the case where X A= (piay o2) are univariate normal distributions, for i, E (0, 1}, the optimal distribution D from Theorem 4.1, with 7 being function of the original distribution parameters, can be written down as:
Po0
(pol
M1))
1110
900
+ Q10
T === Ti iioo mio + 7 M11), and Niio
0
q00
910
2
0
21

---

Proof.
DRL (DI1D)
Pr X = 2, = i,
Σ Pr X = = Y = log Pr(X = x, = =i,A= a)
(x,),a)
Σ (w))a)
Pr Pr(v= i, Ā = a) Pr X
y = i, i, =a)
Pr y - a Pr X =
Y = =
log
Pr(Y = ,, P Pr = x
Y -ii,A=0
Σ
= i, a) Pr [ = x Y = =
(2,7,0)
Pr i,A Pr Pr log
( X= = x Y = -
Pr(Y=i,AA = a)
(X = x Y = i, i = a)
Pr 8 = Y = i, 1,1=
Σ die Σ (X =x Y= --iiiA ii log (f,0)
Pr xx=x Y i, 1-1
Dia DKI. (PullPPaa)
(i.a)
N(pra oz2n) and P:. denotes the distribution of A= = a ~ (issa ). Their probability densities are X Y = i
Pia denotes the distribution of X Y= A
1
(x Mia)
1 exp
(x pia)
Pio (x)
exp
and pia(x) 2ō.a 2T
20 2m
20 ia
27 2
in
respectively. Hence,
Pio (X)
DKL
Pia Pio
E log
Y = i,A=a
(X)
(X-
)
(X-
)2 12
Tia Tia
E
log
Y = i =
20 ?
2
E
x +
Nia
Pia
X +
1.3
log
Gia σia
Y = i, A =
20 v/
2a
0
20
2
1
(Pa
02, +
Pia
Pio
Pioa
2
+ log
Oia 0ia
20 10
282
72 ia
20
27
LG
su
10.
(Nia
Min)
2
0ig
+ log
Tia
20
20 ia
using E log X
Y = -, a = Pia and E (log x)2 - = a) = -122a + + Since we
only change group = 0, we want to minimize
DKL. (DII)) =
Σ
qan DKL.
(PiolIP50)
1=0
(f440
22
2
Σ
10
+ +log
2
20 10
0
7=0
22

---

as function of the variables P.o and ōs subject to the constraints
Moi
Poo
Pio
di
0o
and
and
osa 2 0, for all (i,a).
011
010
00l
000
Let's fix 7 Rgo and minimize
b
920
(JiD
(s0)
G0 log
20
20
1=0
as function of the variables Pia and o% subject to the following constraints
Puo
M10
T10
ōoo
and 0e 2 0, for all (i,a).
PoI
P11
01
6ot
The objective C. is convex and for fixed E Rso. the constraints
are linear in nıo and ō:o- Let's denote the optimal solution for a fixed 7 E R20 by (7) and (f)(),, for {0, 1). For fixed E Rzo. the above constraints fix (7) you for i E {0, and by plugging in Puo = AA10 + +N - M11), we only need to minimize the following convex, quadratic objective in single variable P10.
(#10 + 7HH1 - P11) - 2o0)²
(Ā10 - (10)²
minimize
900
2000
20
By equating the derivative zero, get the optimal solution
pio(?)
400
910
-1
M000
7Ppi m11)
+ 910
1100
000
o
-b
6²
and the optimal value at vio(7) is (The min of ax? bæ + e occurs at x
and has value e
2a
4a
Poo - YPoo 111)
(0(po1 M11)
Moo) 1
1
400
210
900
20
+ 910
20
2
fog +
210
000
c1
-1
900
y10
1oo400
((400 - (110) - 7(401 - 111))
2
000
2
o
1
000
-1
((100 - p10) - (Wo1 - m11))²²
2
900
Q10
By plugging in the optimal solution, the minimum value of L.y for a fixed E R2o is given by
(1))(T)
2
+ 010(7)"- log 0
2
20
20
(7)
1=0
-1
0000
+
(1100 - M11) - -(101 211))
2
900
910
7000
20 - 0.0
1
doo
010
+
400
20
+
9ı0
20
+ (gon + Q10) log 7
+
4oo log
+ qıo log
dol
0
-1
((100 - M10) - 7(p1 m2z))) +
9oo
2
2
910
2
00
1
000
T10
(goo + v10) log
+ 40o log
+ qıo
log
7
do1
011
This is convex objective in y (because the second derivative is non-negative) and by equating the derivative to zero, we have that the optimal 7- must satisfy
(Ho) P11) (7" (Moz
um1 (PoD (2u)) + 7² 900
900 + Q10 0.
a
+
2 0
+
910
10
7*
900
Q10
23

---

Tuu
Multiplying with y*
+
we can write it as a quadratic equation as follows.
400
910
4100 2
1000 2 2
(No1
mn)) 2 + σú + o +
900
2
01
+
9200
01
0 I0
-(M01 (11) p1o) (9oo + qıu)
0
900
910
The discriminant of the above quadratic polynomial is non-negative because the leading coefficient is positive and the constant term is negative. So this polynomial has two real roots. Moreover, since the constant term is negative, it cannot have both positive or both negative roots. Its only non-negative root the optimal solution y" E R20 we want.
(Nm1 111 111) (M10 110) + VA
where
2 (140 - x21) +
σ8 + of
+
Dio of +
200010
Qaod in 2 910070
00
A = (No1 M11)² (Po0 - m10)²
+ ((Moi m11)² + +o0 + o
2 Qi00
200
ouu
2
011
9100
2
00l
(000 + q10)
400
Q10
Proposition B.4. (Proof of Proposition 4.3 in the main text) Let (X, V.Y.A. A) denote the features, binary class label, and binary group membership, respectively, of random data point from any data distribution D with Qia = Pr (Y = A a), for i E {0, 1} and E {0, 1}. such that q10/ 400 911/q01, and let X|Y i, i {0.1) and a E {0, 1}. Let Î denote a distribution obtained by keeping (Y, A) unchanged and only changing X|Y =i a to X|Y
N(uaa oaa) be univariate normal distributions, for
(iras o2a)) Then minimizing DKL. (DIID)
A
as a function of the variables Pio and o.a subject to the constraints in Proposition 3.2 leads to non-convex program. Furthermore, let n- arg min L for some non-convex function of that 7E(0,00)
only dependent on the original distribution parameters. Then, all the new distribution parameters Pio and oia be expressed function of and the original distribution parameters pia and dia-
Proof. We consider the following optimization program
DRL (DIID)
Σ DKL (PiollPia)
Σ (1.a)
Gia
(ftia
202
pia)
0
202
o
+ log
tia
Tia
as function of the variables Jiia and oia subject to the constraints
Poi δn
7oo
δ11 ō0l
and
and
δia 2 0, for all (i,
ōuo
Let's fix E Rgo and minimize
Cy
Σ Dio
(Min Pia) )
02 o2
Tia log
20 ia
20?
0ia
(3.0)
function of the variables Piia and ota subject to the following constraints
Po1
P11
and
Tio 2 0, for all (i,a).
A00
pio
5oo
Now the objective E. is convex and for fixed 7 E Rzo, the constraints on are linear in jio and Pra-
Let's denote the optimal solution for fixed y R>0 by pha (7) and o (7), for i, E {0, 1}. To

---

find this, we can split the above objective into parts that can be optimized separately as follows.
(Pia- - Mio)
minimize
Σ
Gia
subject to
joi Aii = -(000 - m10), and
20%
(i.a)
- - o
minimize
Σ (F,a)
Gia
log
Cia
ō2 = To and oia 2 0, for all (2,a). subject to
20
Oia
For each i E {0, 13. by substituting ãμ oo we need optimize a function in only one variable T.o- The optimal solutions (7) turn out to be
%l(7) =
9:0
+
qa1
and
(n) =7
9í0
qu +
for i e (0,1),
4.0
917
lu7 2
%
% o
Now let's find the optimal solutions Hio (7). The gradient of the objective must be parallel to the linear constraint,
goo (7) -H0O)
-7,, 901 (Ao1 (7)
Mo1) X,
000 910 (+io(?) 110))
001
Q11 qu (1i1 (7) m11)
= yd,
of
for some À R. which gives
-7> T00
Po1 (7) x J01
Poo(7)
+ M40.
+ P01.
Y00
901
1i (1) = - d
Pio(n)
7>
+ Piu,
+ Mm1-
910
411
Noi
PI
Since pr (7) satisfies the constraint
7, we have
1400
0l
T11
+ M101
+
M11
7400 -110)
(po1 1111)
q01
911
= = and hence,
À
X 0
010
001
2
0
010
+ 1100
7X
1110
+
+ 72
900
Q10
901
911
quo
910
Thus, we can express pia (7) as
7H40 1110)
(A41
111)
Moo(?) (7) =
+ 1400
Ø01
2
01
000
010
2
400
+ 722
201
911
400
Q10
(b)())
0
7 (P60
+
of
m10)
+ +
(Mol
+
111)
20
Ø01
401
401
401
411
400
410
7140
m00) - (Mo1
111)
2
(io(7)
+ 1110
of
0
0
910
+
+ +
+
Q01
Q11
900
910
(P40
M11)
(Hon
M11)
pir())
+ p11
+
01
+ +
[
Q1
Ym
4o0
210
Thus, the optimal value of C, for a fixed
E Rgo is given by
C Σ Lio
(H(((() - -1sa)2 + )
0.
+ log
raa
20
20
ia (7)
(1.0)
25

---

Dividing the above expression into three parts, the first part evaluates to
(nta (7)
400
-012000
901
910
22
2 11 q1
Gia
20
2000
460
+
20
961
20
no
+
20 11
(f.a)
2222000
12081
7222090
00
+
2900
2q01
2910
2911
X
001
01
00o
T
+
2
2
901
911
900
910
2
(Pon M10)
(Mot
pmn)
c
o1
+
72
oi
+
011
+
72
0oo
+
iiu
401
Q11
400
410
Q01
q11
900
910
(7o00
(M10)=
(Jo1
(M11)
00
0
010
+ 72
+
9o1
911
900
910
The second part evaluates to
Σ
aio
0i
(7)
02
i0
qia
Tia (7)
(1,0)
20 i0
(d.a)
2
sa
4i0
4.0
411
Qa1
7.2 (q00
+ 411)
1
2
940
9:1
2
t=U
2
2
940
9:1
10
2
01
2
2
10
01
10
01
2
7
01
QE1
9i0
410
41
0
2 7
2
0
010
+ Q11
0
140
2
+
717
2
2 2
4:1
1
9:0
0 2
1
4.0
0
q11
0 7
2
0 7
Q:0
+ [ii
2
+
2 0
0
and the third part evaluates to
Σ (F,a)
gia log
Via
Q40
log
00
log
T
oi,(7)
T (7)
2
o (7)
140
210
2
+
1io log
0
2
0
o
log
2
4.0
+ 4a1
2
72 (g:0
+ qa)
1=0
2
440
50
400
+
2 10
920
411
0
911
41
log
log
2
9i0
2
9ı0
+
Ma
2
$1
9:1
0:0
+ 4i1
log
400
+
40
4:0
+
4i1
log
9:0
+ +1)
- 9:1 log
log T
2
41
2
9u1
0=
26

---

Putting it all together
(Mia(7) -Ma) Mia)
Tz (7)
Jia
C
Gia
20
+
20
log
Jia (7)
(1.0)
(7
1110) - (HO1 - 1(11))
+
lii
9:0
2 20
log
2 01
d1
%0
100
2
9h1
+
72
+
1=0
901
911
900
Q10
460 + qii
log
980
+ +1)
- 4/1 log 7 - Qa log 0i0
2
9:1
di
Minimizing L leads convex program. Since y is the ratio between variances of the new subgroup distribution, for a practical solution, can do line search y (0, B) for some B < 00.
Bound on Unfairness and Error Rate For completeness, we now derive upper bounds on the error rate and the unfairness gap ΣçO of the Bayes optimal classifier h D with respect the original distribution D. These bounds show that both the accuracy loss and the fairness gap depend only on the KL divergence between D and D. It also shows that the optimal value of our optimization problem can be used to approximately translate the accuracy guarantee of h from D to D.
Proposition B.5. (Proposition 4.4 in the main tex)) Let err(h, D) denote the error rate (expected 0-1 loss) of a classifier h on the distribution D. Let drv (D, D) denote the total variation distance between two distributions D and D, while DKL denotes the KL-Divergence between them. Denote the Bayes optimal classifier on the ideal distribution D as h (and similarly the Bayes optimal classifier h. Then, we can bound the error rate and Equal opportunity of h on the original distribution D as follows:
|err(h, D) err(h,DD) (2Dr).D,D) and AFo < (8DKI (δIID)
Proof. For the sake of this proof, assume countable data domain. Using the definition of the expected - 1 loss, we can write:
err(h, D) - err(h, D)
Σ (h(x,,a) # v) (p(x, -(2,
(x.).a)
$ 2drv (D D) S $ (D, D)
(Pinsker's Inequality 19
The first inequality follows from writing the error as expected 0-1 loss and using the definition of total variation distance. The last line follows from Pinsker's ' inequality [19]. We can similarly prove the other direction to obtain the first inequality. Similarly, for the true positive rate of group TPR, (h. D) - TPR. (h. D) < 2D (D.
We can also write for the other group A = a', TP. Đ) TPR., (h, D) S 2Drv (D, D). Adding both LHS and RHS and repeating the exercise in the other direction, noting that the TPR difference of h in D is zero (since in the ideal distribution have exact fairness), we can bound the absolute value of the TPR difference, which is our definition of Ago, we get:
Ako D) < VSDRa. (DIID)
Equalizing the first moment popular intervention in the fairness literature to equalize the first moment of the two sensitive groups or the mean outcomes of two groups, also known as the
27

---

We, therefore, also study an intervention where we only change the mean of the under-privileged group and try to match it with the mean of the privileged group. We can show that the resulting optimization program is convex. We leverage this intervention in Section 5 of the main text.
Calders- Verwer gap [7 44 22
Proposition B.6. (Affirmative Action by Equalizing First Moments) Let (X,Y, A) denote the features, binary class label, and binary group membership, respectively, of a random data point from any data distribution D with qia Pr (Y = i, A = a), for E {0,1) } and E (0, Let X = i, A = N (ira dw) be univariate Normal distribution, for i E {0, 1 and a E {0, 1.. Then in the case of Affirmative mean change, where we impose the following constraints:
910 Pno
Qoo Poo
Q11 M11
4o1 Mol
+
+
910 + 9oo
910 + 900
4ı
9o1
4n
+ g01
we can efficiently minimize DKL (DIID) as a function of the variables P:o and Σgo-
Proof. We are dealing with the following optimization problem:
DKL (DID) = Σ
Y:o DKL (ProlIP10)
1=0
Σ
(jii0 - 2=an)2
0
T +oog
4ı0
20
20
0
==0
as function of the variables jı:o and 0 subject to the constraints
910
900
911
901
1400-
P11
M01
@10
400
Q1o +
900
911 + 901
9i1 + 401
Since we are only changing the means and keeping the variances the same, the objective only depends on f.s. Furthermore, let K (910 + foo)/(euutuu/)) (911/111 + (01 so that
(īin - m2002
subject to jioo = K Āio0
C =
Q4o
20
200
1=0
Substituting the constraint on Ωo0 in the objective L gives us convex quadratic in M10. and the solution is obtained by setting the derivative to zero:
K
Ho0 660
K
M1g Gio
700
'110
4i0
A110
900
Aon
Mo1.
and
An A11-
900
910
9110
0.0
00
900
C Additional Figures for Section 5
In this section, we lay out additional plots from our Gaussian case study. We first describe the setup. We modify a stylized setting of Gaussian distributions from previous work (see Definition 3.1 in (59), Section 5.3 in (4 to investigate the unfairness and the Bayes optimal error on the original and ideal distributions obtained through various interventions. We fix qia E (0, 1) such that Q00 + Qi0 + 401 + Q11 = 1, and our data generation works as follows. We simulate a data with probability yia and X Y =, A
distribution where Y = A
is sampled from univariate Gaussian ((uaa of We choose homoskedastic Gaussians within each group A = ie., Oaa = 0ia. so the we can show the Bayes optimal classifier boundary as threshold. We choose different ia 's that cover ground truth distribution that can the entire spectrum of being ideal or close to ideal to very far, and then we apply different interventions to change all or some subset of μ4a 's and aia" 's to find the nearest ideal distribution in KL-divergence as given in Section 4 of the main text.
We first look a case where the subgroup distributions are the same shifted versions of each other in Figure 5 Note that all interventions, in this case, result in the same Bayes error (BE), but affirmative
28

---

1.4
DC-0..00 602-6.00 (E0 624
DE: 10.00.0 -5P.0.0000000 (FO 0:23
16
05:0.07 WWP.0PPP.0000000 6C0 0.5
1.0
25:2,07 -00 670:0.0
1L.04L
K-0-22
0.1
0.
(5:
3.-
0.-
0.2
02
02
12
05
00
30
(a) Original Distribution
(b) EF-Afmmaaiiv
(c) BF-AIl Subgroups
(d) Mcan Matching
Figure 5: Comparison of Different Interventions when the subgroup distributions are shifted version of each other. While all methods achieve the same Bayes Error, Affirmative action is able to bring down the Bayes Error and achieve exact faimesss.
1.5
02:0.00. ADP 0.20 ADOL D.02
DE: 0.000 A2P.0/PP.000 (00000.00.00 JEDI 0.23
1.0
22. 0.00 (2P: A00.0000........ 0.00. 0.53
1.0
22: 2.:22.000 00020.00000000
0.0.
0.5
0.0
0.0
3.0
0:300
booth for
0.4
0.2
02
02
02
0.5
08
30
Original Distribution
(b) EF-Affrmative
(c) EF-AIl Subgroups
(d) Mean Matching
Figure 6: Comparison of Different Interventions when the original distribution is already fair. In this case, EF-All ensures that it stays close to the true distribution, as no intervention as required, while others relatively deviate.
action brings the BE down with zero unfairness at the cost of incurring a deviation in terms of KL and IS divergence. However, in the next subplot, changing all four subgroups not only helps reduce the Bayes error and unfairness but also stays very close to the true distribution in the KL/JS sense. Matching the means also helps reduce the unfairness while staying close to the true distribution, but sub-optimal compared to the EF- Affirmatteev and EF-All interventions.
Next. we look at case where the Bayes optimal classifier is already fair (AEO is close to 0 while ADP=0) in Figure 6 The expected solution here should be that any intervention must leave the distribution as it is. EF-Affirmativ intervention keeps the unfairness and error rate numbers as it is, but deviates from the true distribution, as indicated by the KL/JS divergences. However, the EF-All intervention only makes major changes to variances and stays close to the true distribution. The Mean Matching intervention shifts both the under-privile privileri subgroups and strays away from the true distribution, as indicated by relatively high KIJJS values.
simulate the cost-sensitive risk for different cost matrix C' other than 0-1 loss by considering a threshold &c = 3/4 on (7 a) in Figure 7 The original distribution has high unfairness. EF-Affirmative intervention manages achieve almost perfect fairness and zero error rate, but incurs relatively high KL/IS numbers. However, once again, changing
Finally, in light of Proposition 3.3
0.25, BEOL D.54
0.00 DESI
1.0
0.01 O.O.. 0.3
2.0
0.1.8
081
11:0.
K0:0.31
0.35
0.2
0.0
15:4.:4
0.8
15: 0.54
0.0
S. 0.25
Yel
0.5
Seam
O.C-
0.6
3.6
0.4
7 2
(a) Original Distribution
(b) Af-Armmativv
(c) EF-AIl Subgroups
(d) Mean Matching
Figure 7: Comparison of Different Interventions when we use different threshold (3/4) than the Bayes optimal threshold (122). As derived in Proposition 3.3 the EF-Afirmative and EF-All interventions work with any threshold.
29

---

all four subgroups, results in a solution that is perfectly fair and accurate, with low KL/JS. Mean Matching is unable address the accuuryy tension all in this case and also manages to drift away from the true distribution, as indicated by non-zero KIJJS values.
D Details and Additional Results for Section 6
In this section, we lay down all the details for the experiments performed for LLM steering. The code to reproduce our results is provided here For the multi-class experiments, we use lot of helper functions from the code of Singh al. [67] For the emotion steering experiments, we reproduce the methodology from Zhao et al. 79 and provide the Jupyter notebook in our code.
D.1 Reducing Disparity in Multi-class classification
To apply our intervention for multi-class settings, first come up with version of Theorem 4.1 for multiple classes. We show this for univariate distribution, and for our intervention, we assume diagonal covariance. Since experiment setup only requires two groups for each class, show the effective constraints assuming two sensitive groups, but this methodology can be readily extended to handle countable number of groups as well. To make our program convex (and affirmative), fix class y y. We fix our class y" according to the following heuristic: y" arg min (,TP)()) y
xx and where h is the empirical risk minimizer the given data. This fixes ratio % 0
9y-1
0g
4y*u"
We write multi-class version of the optimization program in Theorem 4.1:
C, c- = E qia (Aiia - [iaa)
à o
dia
log
20
20
Ōia
(i,a)
as function of the variables Piia and oia subject to the following constraints
Pai
P+1
qs1
qsl
and oia 2 0, for all i E V.j E Mi).
P:0
1300
00
4:0
4jo
Just like in the proof of Theorem 4.1, the resulting program will result in separable objectives for class y and then in the underlying optimization variables Nya and ōya
Program for Nya:
(Ny0
22
(1y1 (g1)
9y0
qyl
subject to By1 Byy 70000 P0
20 00
20
Program for ya::
x2
82
oz
4y0
90
+ log
Tyo oyo
+ 9y1
yl
yl
+ log
Tyl 0yl
subject to ō,1 = =:ē0o
20
20 yl
where y is the class we fixed earlier and ? = To- following:
The solution for the following programs are the
Myo
79yt "1
J0
qua
Ty1
by
qy1 2 55
Pyo
and Byy
9y0
Ruo <go
Sy0 (Ma*)
7/ to +7
Hyl
90
Once we have the corrected distributions N(maa: Exe), we set up an affine intervention, following the design choices of Singh et al. [67. We assume affine relationship between the original
30

---

RMS TPR Gap Different Professions
0.35
Steering
Mean Malching
0.30
MIMIC (Me Matching)
LEACE
RMS TPR Gap
0.25
0.20
0.15
0.10
0.05
0.00
architect
nureo päysician
pastor
fimmaker
lnt dasun
woft ong.
attorney
pont
teacher
acrountant
compaser
painter
surgeon
dietitian
photographer
pors, train.
yoga, Inlt
model
comodlan
paralegal
payahologist
Journafint
chirn
dentat
profossor
rapper
Profession
Figure 8: TPR-gap between Gender groups for all professions. All methods to steer feature rep- resentations achieve roughly the same accuracy (in the range of 0.77-0.79). Our intervention (EF Affirmative) is able to significantly reduce the TPR-gap gap for all professions. In many cases, it is even comparable better than previous interventions Belrose al. [8) Singh et al. [67]
and transformed samples per subgroup: Y aya X + byan where Niigas Tya) and X
N (Mya: gya). Taking expectation both sides gives us: Pyya ayaHya + byas Jja d o yau and we get the following coefficients:
aya = = 5ym and bya -uya + dya ayu Mya-
We have two choices of the parameters corresponding to the positive and negative solutions. Since we are working with empirical estimates, we use the validation error to decide the best set of estimates. A detailed implementation is given in the code. To implement the conditions for 9ya, we use the reweighing scheme of Kamiran and Calders
43 The plot in the main text (Figure 3) assumes
4y0 4y for the corrected covariances. Figure 8 shows the plot with no such assumption for qyo and confirms with same trends as observed Figure
D.2 Steering activations for Joyful generation
Zhao al. 779 propose obtain distribution over the steering vectors for concept instead of a single steering vector. In this section, we lay out all our prompts and the design choices for the emotion steering pipeline.
We first generate training data for each concept (joyful, angry) each of the groups (horror, comedy), resulting in four subgroups. The following prompt was used to generate an initial set of data:
Prompt to generate 1000 Comedy movie reviews that are joyful.
Compose a concise 30-word movie review, assuming it is a comedy movie, that covers these four aspects: plot, sound and music, cultural impact, and emotional resonance. Choose a joyful tone for your review. For the plot, comment on its structure or originality. Regarding sound and music, mention how enhances the storytelling. For cultural impact, touch on any relevant social commentary. Finally, describe how the film resonates emotionally. Ensure your joyful ful tone consistent throughout the review. Please include emotions like "joyful" in these texts and generate 1000 samples.
We obtain the last token embeddings from each layer of Llama-3.1 8B model [36 for each of the samples. We now proceed towards obtaining the steering vectors. We want to obtain steering vector for each group. Treating angry reviews as an irrelevant sample for the 'joyful' concept, we assign y ( to angry samples and 1 to joyful samples. Because we want to estimate distribution over the steering vectors instead of a single vector, we sample 300 points with replacement and repeat this for 50 iterations. In each of these iterations, we train a Logistic Regression model to classify between
31

---

the relevant and the irrelevant samples. We get 50 weight vectors using this pipeline, and we use those to obtain the sample mean and covariance. We denote the resulting distribution for the steering vectors for layer / as NN I(a) where la denotes that this distribution represents the steering vector for joyful emotion for a group A (horror or comedy reviews). To apply our intervention later, we also obtain the steering vector in the other direction by flipping the relevance labels, i.e. joyful + angry, and we denote the resulting Gaussian distribution with N ((uba 2/a)
To perform steering, representation of layer I with strength a: h (1 ajhh + au-- To measure the performance of steering, we ask the Llama model to generate angry reviews using the following prompt:
sample steering vector of NV Eja) and add it the last token
Prompt to evaluate the joyfulness of generated reviews (for group horror).
System Prompt:
You are well informed movie critic.
User Prompt: Compose concise 10-word movie review, assuming is horror movie. The tone of the review must be angry. The review should be within 10 words. Generate 100 samples. Please make sure that you do not repeat the same review and keep each review under 10 words. Also, make sure that the reviews are coherent and not repetitive.
We first measure the joyfulness score of the generated reviews without adding any steering vectors.
Just like Zhao et al. [79], we use the following prompt to ask the GPT 4. model rate the joyfulness of the text:
32

---

Prompt to generate angry 10-word reviews for the horror genre.
System Prompt: You are deterministic evaluation agent trained assess text samples using two strictly defined criteria: (1) emotional tone (joyful score) and (2) linguistic structure (chaotic score).
For each imput, your task is to produce exactly two integer scores between 0 and 10. These must follow the user's 's scoring rubric precisely. The first score reflects emotional positivity or dark exuberance without considering sentence structure. The second score reflects linguistic disorder or repetition, without considering sentiment.
You must produce output in CSV format, strictly matching the following structure: "sample", joyful score, chaotic score
Do not include any additional content, commentary, variation. Your outputs must be fully deterministic: identical input always produces identical output.
Your responses should match the tone and evaluation logic demonstrated in the provided reference examples.
You are an automated evaluator tasked with rating horror movie reviews using two fixed criteria: **joyful expression and **textual coherence and novelty**. For each review, you must generate exactly two integer scores as described below:
User Prompt:
### ##1 Joyful Score**
Rate from 0 to 10 based strictly emotional tone:
* **0** not joyful at (strongly negative, angry, or emotionally flat tone)
** ****** extremely joyful, enthusiastic, or darkly exuberant in tone
Intermediate values reflect mix of enthusiasm and criticism
Focus solely affect and sentiment; ignore linguistic structure for this score
### **2. Chaotic **ore**
Rate from to 10 based strictly linguistic organisation: **0** = fully coherent, fluent, and structurally sound
** 100** highly chaotic, incoherent, repetitive, nonsensical
Intermediate values reflect partial disorganisation, verbosity, or broken phrasing
Focus only syntax, flow, and novelty of expression; ignore emotional content
### **Rfference Examples**
** Joyful and Coherent Horror Reviews:** **
"Blood-soaked fun ensues in this delightfully terrifying slasher film."
"Chilling thrills abound this creepy haunted mansion tale."
"Jump scares galore in this electrifying horror comedy gem."
"Unetttlig unease fills this unnerving psychological horror masterpiece."
"Bone-chilling chills chill to the bone in this one."
** Anggr and Coherent Horror Reviews:
* "Abysmal plot twists ruined what could've been a decent film."
"Mind-numbing terror fails to deliver in this lazy horror."
"Weak jump scares can't 't save this trainwreck disaster."
"Poor production values ruin what little suspense exists." "Frustratingly predictable, making it boring and unscary too."
####
**Output *
For each sample, return one line in strict CSV format: "sample"..jon joyful score, chaotic score
**Example 0u
sample, joyful score, chaotic score
"This horror film was painfully dull and predictable.", Joyful_score____ chaotic_scooeee _core_____o_______
"Terrifying, stylish, and packed with chilling moments!", joyful score_2 2, chaotic_scom score
Do noo include explanations, commentary, or additional formatting.
* Output must be **fully deterministic**; the same input must always yield the same scores. Begin processing the dataset now. Here is the batch of review samples:
33

---

A few notes on evaluation are in order. Zhao et al. 779 report both joyfulness and coherence scores for the generated text. However, we observed that coherence scores were all over the place and did not make sense. Second, Zhao et al. evaluate using the GPT--4o model, whereas we used the GPT 4.1 model since we observed that the joyful scores corroborated more with the qualitative inspection of the generated samples.
Following the above pipeline, observe increase joyfulness scores of the generated reviews by Llama model after steering. However, since the effectiveness of joyful steering was not the same for the horror and comedy movie review generations, we apply our affirmative intervention (Theorem 4.1), assuming that the horror and comedy movie reviews define two groups. Let the modified steering vector be denoted by o N "1,). where the new gaussian distribution is obtained after applying the affirmative action intervention from Theorem 4. assuming horror group is the under- privileged group.
However, simply replacing of will not work. We demonstrate that empirically in the main text, where in Figure 4,
corresponds to using ot instead of vn But we can always use oه to nudge the existing steering vector vy in the right direction. To do that, we modify the steering vector and the representation h with the following rule: h = (1 a)h' + a((1 a')o ati)), where a controls the strength of mixing the old and new steering vectors. In Figure we show that for small values of o, the steering vector indeed starts performing better in steering the reviews of the horror group towards a more joyful tone.
34

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We believe our contributions and scope of work is accurately reflected in the abstract and the introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of paper in Section 7
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
35

---

Justification: All theorems and supporting Lemmas are proved in the supplementary material and all assumptions are mentioned the theorem statements and the proof.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Our experiments use the setups used Singh et [67 and Zhao al. (79). and we provide anonymized Jupyter notebooks to reproduce all the results included in our paper and all the details of our experiment pipeline the supplementary material.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
36

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: We provide anonymized Jupyter notebooks to reproduce our experiments, on top of the codebase used by Singh et al. [67]
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cd
CeeiatccguudessCddeuubiiii sow.:suuuubmssooooooooooooo for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
httppullliucudeess//eppppee /ttpppccc//ala///ggiiia/aa/a///a hat////ooSSSSiiaaeeeeeeee wwriiisssiiniiiinin for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [No]
Justification: Our Jupyter notebooks and supplementary material provide all the details for our experiments. For LLM generation experiments, we also include all the prompts required to generate the data. We also provide the file containing all the prompts used. For the multi-class debiasing experiments, 667 upon request, and hence we cannot include that in our codebase. Those files can be requested from Singh al. directly.
used the representations provided by Singh et al.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: We report results over deterministic generations from LLMs and deterministic evaluation using GPT-4o model. We provide e all the prompts and evaluation code. However, due to the unavailability compute, for the first experiment of TPR-gap reduction, we were only able to fit few inference and generation cycles from these LLMss.
Guidelines:
37

---

The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi-
dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative етror rates).
If error bars are reported in tables or plots. The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We mention the compute resources used for experiments in the supplemen- tary material.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics /ttttt/tttp/////ppppp...ooooppp
Answer: [Yes]
Justification:
The paper conforms with the NeurIPS code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [Yes]
38

---

Justification:
We discuss the broader impacts of our work in the Discussion section (Section
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation. generating fake profiles, surveillance), fairness considerations (c.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We do not release any such data or models.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We use publicly available codebases and assets. Data representations were requested from Singh et al. [67] and they are credited for this in the paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
39

---

The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperssi thoode com/ com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's ' creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release any new assets.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the datasct/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve such aspects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does note involve working with human subjects.
Guidelines:
40

---

The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [Yes]
Justification: We show the applications of our theorems on the LLM steering problem. While our problem is motivated from a point of view of distribution steering, steering LLM distributions is very relevant and important application for us.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https:// /ttpp//ss....ooooeeeeee/e/////// for what should or should be described.
41


Reviews:
[{"review_id": "jw0rsqO88c", "reviewer": "Reviewer_1dpz", "summary": "The paper proposes the concept of 'ideal distribution', defined as a distribution on which any trained model that minimizes loss will have exact fairness. The paper then provides theoretical analyses of tools to find the 'nearest' ideal distribution given a real-world distribution, as well as several practical approximations. Overall, the setup aims to show how a given input distribution of data can be 'steered' towards an ideal distribution such that the model trained on this new distribution is fair. Finally, it shows the benefits of their technique by steering LLM representations, achieving good fairness scores.", "questions": "One big 'discussion' missing in the paper is the gap between being able to steer underlying distributions and concerns of fairness in a real-world task. This includes both explicitly acknowledging that the distribution needs to change for training and inference, and discussing the impact of access to finite data instead of distributions (expanded further in the 'Limitations' section).\n\nCorrect me if I've misunderstood the paper, but given that the underlying distribution itself is steered, this technique is much closer to fair representation learning than any 'fairness pre-processing'. Consider the perspective of how a practitioner would take advantage of this technique. When using fair pre-processing techniques, like reweighing, massaging, sampling, etc., these techniques are only applied during training to learn a fair classifier, which can then be applied directly to the original distribution during inference. But similar to fair representation learning, the method proposed in this paper also changes the distribution itself, and its 'steering' is something that'll need to be applied not just to the training set but also to the input during inference.\n\nThis is a vital clarification, in my opinion. One, I suggest the authors improve the intutions provided in the introduction by not comparing their technique to pre-processing and instead of representation learning (or please correct me in the rebuttal if I'm wrong). And two, I suggest the authors provide a better discussion of how a practitioner can take advantage of this technique in classical settings (different from representation steering in LLMs). No additional experiments needed, but a discussion of how this technique fits within the broader landscape of fairness is important.", "limitations": "One limitation of the work is the missing link between distribution-level analysis and real-world applications, where one has access to only a finite amount of data. \n\nAlthough misaligned distributions between the minority and the majority play an important role in the bias encoded in the ML model, this is not the only cause of bias. When there is a lack of enough data from the minority, even approximating the underlying distribution is error-prone, which can further exacerbate fairness concerns (and their variance).\n\nThe paper mostly focuses on distribution-level analysis, assuming access to the underlying distribution and directly steering this distribution. While they do provide empirical results with real-world datasets, a deeper analysis of this missing link and a better discussion of what's lost when you only have access to finite data would further strengthen the paper.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Quality: The paper appears technically sound to me. Most claims are well supported, and the methods used are appropriate.\n\nClarity: The paper is clearly written and easy to follow. My main concerns in clarity come from a lack of appropriate framing of the work in the context of existing fairness literature and a missing discussion of the same. Please see 'Questions' and 'Limitations' for more details.\n\nSignificance: I believe the results are impactful for the community, and I can imagine useful future work building on the results presented here. It takes a unique approach to fairness, and further exploration of the concept of 'ideal distributions' can be interesting for the field.\n\nOriginality: To my knowledge, the perspective of 'ideal distributions' and steering the underlying distribution to achieve fairness is novel. However, clearly there are connections to existing literature, something that I believe can benefit from better framing (see Questions and Limitations section for details).", "was_revised": true, "final_justification": "I continue to maintain my positive score.", "rebuttal": "Thank you for carefully reading our paper and providing valuable feedback. We address your questions regarding our work in the context of existing fair classification works below. We will certainly include this discussion in future versions of this manuscript.\n\n1. **Relation to fair representation learning and benefit to practitioners**: We study the foundational problem of finding the nearest ideal distribution to a given distribution. This has many potential applications, such as correcting training data bias, learning fair representations, steering intermediate representations for fair generation, etc. Fair pre-processing or reweighing is closer to our mathematical setup, and steering intermediate representations gives a compelling application for LLMs. Our KL optimization program can be used to measure training data bias and guide data collection policies in practice, especially when the models are trained for fairness, but the inherent bias in data is unresolved. Prior to our work, the popular strategies to correct data bias were reweighing, fair pre-processing, and collecting more data from underrepresented groups.\n\n2. **Going beyond Distribution-level analysis and working with finite samples**: This is certainly an important future direction for us. A distributional-level analysis allows us to express the Bayes error and true positive rates (and analogously false positive rates) as functions of the distribution parameters. These are generally intractable without any distributional assumptions. In a finite-sample regime, we have high probability estimates of the moments and other distribution-dependent quantities, so our approach leads to approximate fairness guarantees. Since the scope of our paper is exact fairness, we did not include this further analysis. To go beyond distributional assumptions, we can leverage previous work that maps given data distribution to tractable, parametric families using invertible transforms that preserve Bayes error, so our results become applicable [1]. We can also assume distributions with bounded moments instead of parametric families, and still bound the Bayes error and other relevant quantities [2]. This requires a careful analysis, and is certainly a very important future direction for us.\n\nWe greatly appreciate your positive feedback and request that you champion our paper for acceptance.\n\n[1] Theisen, Ryan, et al. \"Evaluating state-of-the-art classification models against bayes optimality.\" Advances in Neural Information Processing Systems 34 (2021): 9367-9377.\n\n[2] Murti, Chaitanya, and Chiranjib Bhattacharyya. \"DisCEdit: Model Editing by Identifying Discriminative Components.\" Advances in Neural Information Processing Systems 37 (2024): 47261-47296."}, {"review_id": "RfrGF6TBVv", "reviewer": "Reviewer_KFDe", "summary": "This work introduces a methodology to steer a data distribution towards an ideal one, formalised as a data distribution on which a Bayes-optimal classifier satisfies exact group fairness (e.g., no DP or EO difference). The framework optimises for the closest ideal distribution via minimal KL divergence, pushing the data (or latent representations) towards it. The paper provides the conditions under which a data distribution is ideal, especially for common parametric families (Gaussian, log-normal). The authors further provide efficient solutions for interventions: an “affirmative action” intervention, adjusting the under-privileged group only), and changing all the subgroups. \nResults on simulations show perfect fairness at low KL cost, and some real-world applications show bias reduction on multi-class classification by steering LLM representations, without sacrificing accuracy. \nThe main contributions are:\n- The definition and formalization of ideal distributions without a fairness-utility trade-off\n- The formulation of the problem of finding the nearest ideal distribution as a KL divergence minimization problem, with some efficient solutions through particular interventions\n- Empirical results showing bias reduction in synthetic and real-world cases, while preserving accuracy.", "questions": "- This work is limited to common parametric families. What kind of applications still work under this assumption, and what others do not? A couple of examples would help. Although the checklist mentions that limitations are discussed in Section 7, this is not clearly stated (on the other hand, the future work is well discussed).\n- The authors mention the framework of ideal distributions can also be applied to audit deployed model for fairness, particularly when the evaluation data may itself be biased. Beyond the steering showed, how can this be achieved in practice? Use the KL gap as a proxy for what’s needed to debias the model? It's not trivial to understand from the text.\n- In the future work the authors mention normalising flows or mixture models as possibly integrated. Would this be a relaxation of the parametric condition (through intermediate transformations or mapping), or would this have more profound implications?\n- It’s not clear if the framework would generalise beyond one binary sensitive attribute.", "limitations": "The authors could improve their limitations by explicating stating them as such: the parametric assumptions, for instance. The practical guidance is not discussed as well. For example, larger KL gaps should be harder to overcome, but what would be the practical consequences, limits, and constraints?", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths\n- Although the high-level idea of searching for an unbiased distribution has been explored theoretically before, with the existence of such distributions being proven, this work has a significant contribution by bridging a solid theoretical foundation on how to achieve such a distribution with provable fairness guarantees (even if constrained to some parametric families), and their practical application.\n- The paper has a strong theoretical foundation, from the formalisation of the ideal distribution (for any cost-sensitive Bayes-optimal classifier). Several theorems and propositions provide a framework to understand the necessary and sufficient conditions for the “ideal” property (in Gaussian conditions). The optimization formulation is also well defined. Although recognised as a non-convex problem, the authors provide solutions by restricting to some parametric families. The proposed interventions are also well supported, and it is shown that the fairness/accuracy bounds depend only on the KL divergence.\n- The paper demonstrates the framework’s potential in different and versatile scenarios, from synthetic data, real-world bias reduction (Bias-in-Bios dataset), and generative text-sentiment steering (for “joy”). The results are consistent and show the fairness improvements with negligible accuracy drops (or even improvements).\n\nWeaknesses\n- Not an actual weakness, but some limitations (expanded below) ar ethe parametric assumptions and the application to Bayes-optimal classifiers\n- The paper is dense in some parts, and there are some notation and clarity issues. A thorough revision is suggested to improve clarity and correctness at this level, but here are some catches:\n   - Typos: D’ vs D tilde, BR where BE should be (Fig 1), a “tildeh” for h tilde, etc.)\n   - Abstract: initial sentence is a strong statement since other fairness angles (other than group fairness) can also be applied; and the final sentences seem repetitive\n   - The optimization is for an ideal distribution, but some text parts give space to there being more of such distributions, although it is not clearly discussed.\n   - In Definition 2.1 regarding DP the positive rate is mentioned. Shouldn’t it be the positive rate difference?\n   - It’s not really clear in Section 6.2 how the simulation results are obtained exactly.\n- Some details are not clear on the optimization procedure in practice, and for the representation steering as well (e.g., is the downstream classifier retrained or is it inference-time like the post-training intervention?). Also, the alpha value influence (e.g., it is said that excessive nudging “distorts the steering vector” but the potential reason is not discussed).", "was_revised": true, "final_justification": "I appreciate the authors' effort in addressing my questions and concerns. There still seem to be some important open questions to be addressed in future work. Nonetheless, this is a relevant contribution to the topic, theoretically grounded but hinting at some practical applications. I'll keep my positive score.", "rebuttal": "Thank you for carefully reading our paper and providing valuable feedback. We address your concerns below.\n\n1. **This work is limited to common parametric families. What kind of applications still work under this assumption, and what others do not? A couple of examples would help. Although the checklist mentions that limitations are discussed in Section 7, this is not clearly stated (on the other hand, the future work is well discussed).**: Our theoretical results are on the fairness of the Bayes optimal classifier and the corresponding optimization programs, which are tractable and mathematically easier to analyse for parametric families of distributions, e.g., multivariate Gaussians. Multivariate Gaussians may not always fit real-world data but can be justifiably used to model concepts and internal representations in machine learning as shown in prior work [1,2]. There can be scenarios and applications where a parametric assumption may not fit well with the input data distribution, and hence, analyzing the Bayes optimal fair classifier will become highly non-trivial. We will discuss these limitations more elaborately in future versions of the manuscript. Thank you for this feedback.\n\n2. **The authors mention the framework of ideal distributions can also be applied to audit deployed model for fairness, particularly when the evaluation data may itself be biased. Beyond the steering showed, how can this be achieved in practice? Use the KL gap as a proxy for what’s needed to debias the model? It's not trivial to understand from the text.**: Prior work on fairness audit has mostly focused on auditing a given model instead of auditing training or evaluation data. There is a long line of work to demonstrate that the fairness-accuracy trade-off disappears when data bias is accounted for [4,5,6,7,8,9]. The KL gap in our formulation can be used as a metric of data bias and our Theorem 4.1 essentially gives an algorithmic recipe to minimally change a given data distribution to bridge this gap. \n\n3. **In the future work the authors mention normalising flows or mixture models as possibly integrated. Would this be a relaxation .... or would this have more profound implications?**: Thisen et al. [5] showed that the Bayes error is invariant to invertible transforms, like the ones used with Normalizing flows. Furthermore, they show how to obtain an entire family of distributions using a single family of flows. The ideas presented in this paper can be used to construct invertible maps to a desired parametric family, apply our ideas of searching for ideal distributions within the available family of distributions, and then use the map back using the invertible transformation. This is our plan for future work.\n\n4. **It’s not clear if the framework .... one binary sensitive attribute.**: Proposition 3.2 is presented for a multi-class, multiple attribute, and multivariate Gaussian setting. To show a stronger necessary and sufficient set of results, we make use of the binary label and sensitive attribute setup for Proposition 3.3. Furthermore, even for the Affirmative action intervention in Theorem 4.1, we can write down a multi-class and multiple sensitive attributes version. In fact, that is what we used to set up experiments for Section 6.1 (Multi-class debiasing) and the details of this is described in Section 4.1 of the supplementary material. One thing to note is that Affirmative action needs a non-trivial definition for situations beyond binary labels and binary sensitive attributes, since we don't have the notion of a favourable group and a positive outcome anymore. That is why we can either define Affirmative action class-wise, assuming the most \"accurate\" group to be favourable, or we can do it the other way around, where we first find the most favourable group and then find ideal distributions relative to that fixed group. This is what we did for the results in Section 6.1, where we worked with a multi-class, binary attribute setup, but the same setup can be perfectly used for multiple sensitive attributes.\n\nBelow, we comment on some of the points raised in the Weaknesses section.\n\n1. **The paper is dense in some .... at this level**: We will carefully review all notations and make the exposition easier in the next version of this manuscript. More discussion about the experiments in Section 6.2 is included in Section 4.2 of the Supplementary material. We will add more details in the main text in future revisions. \n\n2. **Abstract: initial sentence is a strong .... seem repetitive**: The scope of this work is exact group fairness, and hence we wanted to be very precise in the abstract. We can refine the abstract to highlight this. Regarding the repetitiveness, we wanted to emphasize in the first sentence that we want to improve the representations used for downstream classification. In the second sentence, we wanted to express that we are also able to steer the internal representations of an LLM so that we can improve generation across groups in LLMs. We will make these clarifications in future versions of this manuscript.\n\n3. **The optimization is for an ideal distribution, but some text parts give space to there being more of such distributions, although it is not clearly discussed.**: Our aim in Section 4 is to propose an optimization program to obtain an ideal distribution where the Bayes optimal classifier is also fair. The objective is to find the optimal ideal distribution that minimizes the KL divergence relative to the given distribution. \n\n4. **In Definition 2.1 .... positive rate difference?**: Thank you for pointing that out. We will fix that typo in future versions of the manuscript. \n\n5. **It’s not really clear in Section 6.2 how the simulation results are obtained exactly.**: Due to space constraints, we have laid out the details of the experiments in Section 6.2 in Section 4.2 of the supplementary material, along with all the prompts and a flow of the pipeline. To summarize, we observe a disparity in performance between groups when we measure the effectiveness of steering generation of joyful responses, using the method a recent work on modeling concepts using gaussian distributions [1]. We therefore applied our Affirmative intervention from Theorem 4.1 to obtain the set of ideal distributions for the concepts such that both groups in the data get steered effectively during inference. We use the obtained ideal distribution to nudge the optimal steering vector towards better generation for both groups.\n\n6. **Some details are not clear on the optimization procedure in practice..... intervention?.**: For Section 6.1, the downstream classifier is not retrained. For the setup in Section 6.1, we are steering the distribution of representations towards an ideal distribution, and then training a classifier without any constraints. As indicated in Figure 3, this improves the fairness across multiple professions and for some professions, even matches and beats the Affine Steering method from Singh et al. [10].\n\n7. **Also, the alpha value influence ... not discussed.** As stated in the paper, for the experiments in Section 6.2, alpha is required because after our intervention, we get the ideal subgroup distributions where the tradeoff between utility and fairness is alleviated. This notion of utility is related to, but different from, accuracy, for which we have ideal distribution guarantees in Theorem 4.1. Therefore, the newly obtained steering vectors can be drastically different than the optimal steering vectors (as demonstrated in Figure 4, where a high value of alpha closer to 1 puts more weight on the affirmative action vector and therefore cannot steer towards joyful generation). But we can use our obtained ideal distribution as a direction to nudge the optimal steering vector. As indicated by Figure 4, this successfully steers the text generation towards joyful sentiment for the underperforming group. We have laid out more details of the steering experiment in Section 4.2 of the supplementary material, and if you recommend, we can also add a pseudocode of the algorithm for more clarity in future revisions of the manuscript.\n\nWe sincerely thank you for reading our paper carefully. We request that you kindly revisit our paper with our detailed explanations above and upgrade your rating.\n\n[1] Zhao, Haiyan, et al. \"Beyond single concept vector: Modeling concept subspace in llms with gaussian distribution.\" arXiv preprint arXiv:2410.00153 (2024)\n\n[2] Eftekhari, Daniel, and Vardan Papyan. \"On the Importance of Gaussianizing Representations.\" arXiv preprint arXiv:2505.00685 (2025).\n\n[3] Theisen, Ryan, et al. \"Evaluating state-of-the-art classification models against bayes optimality.\" Advances in Neural Information Processing Systems 34 (2021): 9367-9377.\n\n[4] Wick, Michael, and Jean-Baptiste Tristan. \"Unlocking fairness: a trade-off revisited.\" Advances in neural information processing systems 32 (2019).\n\n[5] Blum, Avrim, and Kevin Stangl. \"Recovering from biased data: Can fairness constraints improve accuracy?.\" arXiv preprint arXiv:1912.01094 (2019).\n\n[6] Dutta, Sanghamitra, et al. \"Is there a trade-off between fairness and accuracy? a perspective using mismatched hypothesis testing.\" International conference on machine learning. PMLR, 2020.\n\n[7] Maity, Subha, et al. \"Does enforcing fairness mitigate biases caused by subpopulation shift?.\" Advances in Neural Information Processing Systems 34 (2021): 25773-25784.\n\n[8] Sharma, Mohit, and Amit Deshpande. \"How far can fairness constraints help recover from biased data?.\" arXiv preprint arXiv:2312.10396 (2023).\n\n[9] Leininger, Charlotte, Simon Rittel, and Ludwig Bothmann. \"Overcoming Fairness Trade-offs via Pre-processing: A Causal Perspective.\" arXiv preprint arXiv:2501.14710 (2025).\n\n[10] Singh, Shashwat, et al. \"Representation surgery: Theory and practice of affine steering.\" arXiv preprint arXiv:2402.09631 (2024)."}, {"review_id": "fwXRUQv4gT", "reviewer": "Reviewer_c89k", "summary": "This paper first explores an ideal distribution under which group-aware classifiers can achieve absolute fairness metrics, along with the corresponding conditions required to reach such a distribution. The authors then propose a method that solves a KL divergence problem to approximate this ideal distribution. The results show that both the “Affirmative” approach and altering all subgroups lead to significant fairness improvements, while also enhancing classification performance. Based on this theory, the authors test LLMs on two tasks to validate their practical effectiveness in real-world scenarios.", "questions": "1. On the definition of the ideal distribution, I don't understand why DP can also be incorporated. As the optimal classifier and DP can be in conflict.\n\n2. How is it possible to make the prediction both fair and accurate? Does it mean the previous Pareto frontier works [1, 2] are not meaningful? \n\n\n[1] Kim, Joon Sik, Jiahao Chen, and Ameet Talwalkar. \"Fact: A diagnostic for group fairness trade-offs.\" International Conference on Machine Learning. PMLR, 2020.\n\n[2] Xian, Ruicheng, Lang Yin, and Han Zhao. \"Fair and optimal classification via post-processing.\" International conference on machine learning. PMLR, 2023.", "limitations": "Yes", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n**1.**  The author finds the condition for ideal distribution.\n\n**2.** It is theoretically guaranteed to achieve EO fairness. \n\nWeakness: I have read this paper at least three times, below are the weaknesses I found. \n\n**1** Although this paper is theoretically based, its presentation lacks rigor. For example, in line 122, it introduces $h(\\cdot)$ without a definition. Line 148 should be argmin $L$, not the probability. Line 127, the second Probability expression, etc., makes it very hard to check the correctness. \n\n**2.** I also found this theory's condition is relatively strong, as the condition introduced in line 171, I don't believe real-world data can fit such a condition.\n\n**3.** It also assumes univariate Gaussion on condition $X$ in Proposition 3.3, Theroem 4.1. It is an oversimplification of the data distribution.\n\n**4.** The theoretical part is based on a Bayesian optimal classifier. I don’t quite understand the experiment, which involves an LLM-based representation steering task.\n\n**5** Did the author assume that Test and Training are IID?", "was_revised": true, "final_justification": "After the rebuttal, I believe some of my questions have been well addressed. I actually don't quite get the constraint of this theoretical work, i.e. there must be some constraint on the optimization procedure. But from my perspective, it has a relatively free assumption about data distribution in the discussion for fairness and performance. Therefore, I still lean towards a borderline score.", "rebuttal": "Thank you for carefully reading our paper and providing your valuable feedback. We address your concerns below.\n\n1. **On the definition of the ideal distribution, I don't understand why DP can also be incorporated. As the optimal classifier and DP can be in conflict.**: We clarify the definition of ideal distribution and correct the typos you pointed out in lines 127 and 148. \nDefinition 2.1 for Demographic Parity (line 122) and lines 126-127 should read as: $$\\Delta_{DP, y}(h, D) = \\max_{a, a' \\in \\mathcal{A}} |Pr(h(X, A)=y|A=a) - Pr(h(X, A)=y|A=a')|.$$ \nWe can explicitly define $l_C$ in line 136 as: Let $C \\in \\mathbb{R}^{|\\mathcal{Y}| \\times |\\mathcal{Y}|}$ be a matrix that defines the cost-sensitive loss function $l_{C}$ of predicting label $i$ when the true label is $y$ as $l_C(h(x,a), y) = c_{h(x, a) y}$, the $(h(x, a), y)$-th entry of $C$. The cost-sensitive risk of a classifier $h$ is then given by $E_{D}[l_{C} (h(X,A), Y)]$.\nFinally, Definition 3.1 should be: Let $D$ be a distribution over $\\mathcal{X} \\times \\mathcal{A} \\times \\mathcal{Y}$. Let $\\mathcal{H}$ be a hypothesis class of group-aware classifiers $h: \\mathcal{X} \\times \\mathcal{A} \\rightarrow \\mathcal{Y}$, and let $h^* = \\underset{h \\in \\mathcal{H}}{\\arg\\min}~ E_{D}[l_{C}(h(X,A), Y)]$, where $(X,A,Y) \\sim D$. We call $D$ an ideal distribution if $\\Delta_{DP, y} ( h^{\\*} , D)=0$, for all $y \\in \\mathcal{Y}$ (similarly, $\\Delta_{EO, y}(h^{\\*}, D)=0$, for all $y \\in \\mathcal{Y}~$).\nPropositions 3.3 and 3.2 show conditions under which many univariate and multivariate Gaussians provably give ideal distributions for Equal Opportunity (i.e., equal TPRs across groups). The exact same idea works for equality of FPRs and gives equality of positive rates across groups, or equivalently, Demographic Parity (DP).\n\n2. **How is it possible to make the prediction both fair and accurate?**: This observation is not new, as several prior works demonstrate that the fairness-accuracy trade-off disappears when data bias is accounted for [3,4,5,6,7,8]. Our Theorem 4.1 essentially gives an algorithmic recipe to minimally change a given data distribution and provably achieve this.\n\n3. **Does it mean the previous Pareto frontier works [1, 2] are not meaningful?**: Pareto-frontier or fairness-accuracy trade-offs are meaningful when the underlying data distribution is fixed. [1,2] maximize accuracy subject to various fairness constraints to explore this trade-off, but do not change the data distribution.\n\nWe will now address some of the points raised in the Weakness Section.\n\n1. **Presentation lacks rigor. For example, line 122, 148, 127 make it very hard to check the correctness.**: Thank you for pointing out these typos. We will correct all of them and have explained our corrections above in our response to your question 1. Our proofs are correct, and I hope our explanation helps you verify the same.\n\n2. **I don't believe real-world data can fit such a condition [as in line 171].**: We do not model real-world data using Gaussians. Our empirical work applies Gaussian modeling for concepts and internal representations where it is justified by previous work [9,10]. The condition in line 171 can be simplified further to f-divergence as follows. If $\\Sigma_{ia} = \\Sigma_a$ (same group covariance for all classes), then our condition is equivalent to having the same f-divergence between any two class-conditional distributions across different groups. In other words, $\\Delta_{\\Sigma_{a}}^{2}(\\mu_{ia}, \\mu_{ja})=  \\Delta_{\\Sigma_ {a'}}^{2}(\\mu_{ia'}, \\mu_{ja'})$, $\\forall i,j \\in \\mathcal{Y}, a,a' \\in \\mathcal{A}$; see f-divergence formula (P1) near the end of page 2 in arxiv: 2204.10952. f-divergence is commonly used in fair classification and generation.\n\n3. **It also assumes univariate Gaussian on condition in Proposition 3.3, Theorem 4.1. It is an oversimplification of the data distribution.**  Theorem 4.1 and Proposition 3.2 are stated for multivariate Gaussian distributions. Proposition 3.3 and Corollary 4.2 for univariate Gaussians are only to help the reader as special cases.\n\n4. **The theoretical part is based on a Bayesian optimal classifier. I don’t quite understand the experiment, which involves an LLM-based representation steering task.**: Our theoretical results are on the fairness of the Bayes optimal classifier, which is easier to mathematically analyse for parametric families of distributions, e.g., multivariate Gaussians. Multivariate Gaussians may not always fit real-world data, but can be justifiably used to model concepts and internal representations in machine learning, as shown in prior work [9,10]. Hence, we felt that LLM-based representation-steering makes a compelling application for our technique.\n\n5. **Did the author assume that Test and Training are IID?**: Yes. It is a standard assumption in Machine Learning.\n\nWe sincerely thank you for reading our paper carefully. We request you to kindly revisit our paper with our detailed explanations above and upgrade your rating.\n\n[1] Kim, Joon Sik, Jiahao Chen, and Ameet Talwalkar. \"Fact: A diagnostic for group fairness trade-offs.\" International Conference on Machine Learning. PMLR, 2020.\n\n[2] Xian, Ruicheng, Lang Yin, and Han Zhao. \"Fair and optimal classification via post-processing.\" International conference on machine learning. PMLR, 2023.\n\n[3] Wick, Michael, and Jean-Baptiste Tristan. \"Unlocking fairness: a trade-off revisited.\" Advances in neural information processing systems 32 (2019).\n\n[4] Blum, Avrim, and Kevin Stangl. \"Recovering from biased data: Can fairness constraints improve accuracy?.\" arXiv preprint arXiv:1912.01094 (2019).\n\n[5] Dutta, Sanghamitra, et al. \"Is there a trade-off between fairness and accuracy? a perspective using mismatched hypothesis testing.\" International conference on machine learning. PMLR, 2020.\n\n[6] Maity, Subha, et al. \"Does enforcing fairness mitigate biases caused by subpopulation shift?.\" Advances in Neural Information Processing Systems 34 (2021): 25773-25784.\n\n[7] Sharma, Mohit, and Amit Deshpande. \"How far can fairness constraints help recover from biased data?.\" arXiv preprint arXiv:2312.10396 (2023).\n\n[8] Leininger, Charlotte, Simon Rittel, and Ludwig Bothmann. \"Overcoming Fairness Trade-offs via Pre-processing: A Causal Perspective.\" arXiv preprint arXiv:2501.14710 (2025).\n\n[9] Zhao, Haiyan, et al. \"Beyond single concept vector: Modeling concept subspace in llms with gaussian distribution.\" arXiv preprint arXiv:2410.00153 (2024)\n\n[10] Eftekhari, Daniel, and Vardan Papyan. \"On the Importance of Gaussianizing Representations.\" arXiv preprint arXiv:2505.00685 (2025)."}, {"review_id": "oPbS49IgF2", "reviewer": "Reviewer_YLZ4", "summary": "This paper proposed a preprocessing method to achieve fair classification by steering the feature distribution into an ideal distribution where all cost-sensitive Bayes optimal classifiers will ensure equal opportunity (EO) (also other metrics under a univariate Gaussian setting). The paper first defined the ideal distribution and characterized its property when the distribution is multivariate/univariate Gaussian. Then it formulated the feature steering as an optimization problem to minimize the KL divergence between the target distribution and the ideal distribution $\\tilde{D}$. Since the KL divergence constraint is convex and finding $\\tilde{D}$ can be tractable for certain distribution families, we can solve for $D$ in practice. Numerical experiments demonstrated the effectness of the method.", "questions": "Could you provide the trade-off results on accuracy?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "> Strengths\n\n- The problem formulation is clear and well-motivated.\n- The theoretical results are plausible.\n- The LLM experiments are interestiing.\n- The writing is clear.\n\n> Weaknesses\n\n- Most theoretical results of finding $\\tilde{D}$ in the main paper are based on Gaussian distribution. And the conditions are only sufficient for multivariate Gaussian. Also, the discussed classifiers are Bayes optimal ones. I understand it is somehow standard in theoretical papers, but not sure of the practical applicability. Is it possible to discuss more general distributions?\n\n- Theorem 4.1 and Prop 4.3 demonstrate how we can find $D$, but the discussions on accuracy guarantee are deferred to Appendix (prop 2.5). However, this trade-off can be very important while understanding whether the preprocessing methods should be used in practice. I suggest discussing this more in the main paper.\n\n- Similarly, in the experiment results of section 6, it would be better to show the accuracy comparison (though it is mentioned that accuracy of EF  Affirmative is at the same level of other methods).\n\n\n> Minor: I think the literature on long-term fairness can also be related to this work. For example, \"steering\" and long-term fairness in performative prediction happens naturally when agents respond to the decision-maker [1,2]. It would be interesting to consider whether your preprocessing method can also benefit long-term fairness.\n\n[1] Hardt M, Jagadeesan M, Mendler-Dünner C. Performative power[J]. Advances in Neural Information Processing Systems, 2022, 35: 22969-22981.\n\n[2] Jin, Kun, et al. \"Addressing Polarization and Unfairness in Performative Prediction.\" arXiv preprint arXiv:2406.16756 (2024).", "was_revised": true, "final_justification": "I encourage the authors to add the accuracy results and the discussions on accuracy trade-offs.\n\nCurrently, I will retain my score and see whether there are some other points raised in reviewer AC discussion phase.", "rebuttal": "Thank you for your careful reading and valuable feedback. We answer your questions below.\n\n1. **Could you provide the trade-off results on accuracy?**: On page 12 of the supplementary material, Proposition 2.5 shows how to bound the change in accuracy in terms of the KL divergence between the original distribution and the ideal distribution. Interestingly, this change can be a drop in accuracy in the worst case, but could also be a gain in the best case, unlike the accuracy traded off for fairness when we do not change the underlying data distribution.\n\nWe also comment on the some points raised in the Weakness section below:\n\n1. **Most theoretical results of finding in the main paper are based on Gaussian distribution. And the conditions are only sufficient for multivariate Gaussian. Also, the discussed classifiers are Bayes optimal ones. I understand it is somehow standard in theoretical papers, but not sure of the practical applicability. Is it possible to discuss more general distributions?**: You are correct in pointing that out. The fairness of the Bayes optimal classifier, and the corresponding optimization programs to search for the ideal distributions, are easier to mathematically analyze for parametric families of distributions, e.g., multivariate Gaussians. Multivariate Gaussians may not always fit real-world data but can be justifiably used to model concepts and internal representations in machine learning, as shown in prior work [3,4]. For future work, we can move towards general distributions by considering a class of distributions with bounded moments [5] or working with invertible transformations to parametric families [6]. \n\n2. **Theorem 4.1 and Prop 4.3 demonstrate how we can find $D$, but the discussions on accuracy guarantee are deferred to Appendix (prop 2.5). However, this trade-off can be very important while understanding whether the preprocessing methods should be used in practice. I suggest discussing this more in the main paper.**: We will definitely include it in the main manuscript in further revisions of the paper and discuss the implications of this result.\n\n3. **Similarly, in the experiment results of section 6, it would be better to show the accuracy comparison (though it is mentioned that accuracy of EF Affirmative is at the same level of other methods).** Please find the accuracy numbers for all the methods in Section 6.1, where we reduced the disparity in multi-class classification -> Before steering: 0.789, Mean Matching: 0.788, MiMiC (Mean+Cov Matching): 0.793, LEACE: 0.788, EF Affirmative: 0.779. We will also include them in the main manuscript in further revisions.\n\n4. **Minor: I think the literature on long-term fairness can also be related to this work. For example, \"steering\" and long-term fairness in performative prediction happens naturally when agents respond to the decision-maker [1,2]. It would be interesting to consider whether your preprocessing method can also benefit long-term fairness.**: This is certainly a very interesting direction, and we will definitely comment on this as a potential future work in further revisions. Thank you so much for suggesting this direction.\n\nWe greatly appreciate your positive feedback and request you to champion our paper for acceptance.\n\n[1] Hardt M, Jagadeesan M, Mendler-Dünner C. Performative power[J]. Advances in Neural Information Processing Systems, 2022, 35: 22969-22981.\n\n[2] Jin, Kun, et al. \"Addressing Polarization and Unfairness in Performative Prediction.\" arXiv preprint arXiv:2406.16756 (2024).\n\n[3] Zhao, Haiyan, et al. \"Beyond single concept vector: Modeling concept subspace in llms with gaussian distribution.\" arXiv preprint arXiv:2410.00153 (2024)\n\n[4] Eftekhari, Daniel, and Vardan Papyan. \"On the Importance of Gaussianizing Representations.\" arXiv preprint arXiv:2505.00685 (2025).\n\n[5] Murti, Chaitanya, and Chiranjib Bhattacharyya. \"DisCEdit: Model Editing by Identifying Discriminative Components.\" Advances in Neural Information Processing Systems 37 (2024): 47261-47296.\n\n[6] Theisen, Ryan, et al. \"Evaluating state-of-the-art classification models against bayes optimality.\" Advances in Neural Information Processing Systems 34 (2021): 9367-9377."}]

Author rebuttal:

Meta-review: The authors study how to steer biased data distributions toward their closest "ideal" distribution, where "ideal" means that the distribution does not have a fairness-accuracy tradeoff. They present theoretical results for parametric families and supplement with empirical results for multi-class classification and emotion steering in LLM outputs. 

Strengths: 
- The paper is clearly written and well-motivated. 
- The authors introduce "ideal distributions" -- something that could be useful much more broadly in fair ML.
- The authors effectively rebut many questions/limitations raised by reviewers during the review process.

Weaknesses:
- The results only apply to certain parametric families.
- The concept of a "closest ideal distribution" is insufficiently motivated. In particular, what if the "closest ideal distribution" actually has quite different downstream performance than the original one? Reviewer 1dpz brings up this point; changing the input data will lead to other downstream changes.
Decision: Accept (poster)