On Fairness of Task Arithmetic: The Role of Task Vectors
Authors: Laura Gomezjurado Gonzalez
Hiroki Naganuma
Kotaro Yoshida
Takafumi Horie
Yuji Naraki
Ryotaro Shimizu
Venue: ICLR 2026 Poster (2026)
Keywords: Fairness
Model Editing
Task Arithmetic
Primary area: alignment, fairness, safety, privacy, and societal considerations
TL;DR: We analyze fairness implications of task arithmetic in model editing to guide responsible practices.

Abstract: Model editing techniques, particularly task arithmetic with task vectors, offer an efficient alternative to full fine-tuning by enabling direct parameter updates through simple arithmetic operations. While this approach promises substantial computational savings, its impact on fairness has remained largely unexplored---despite growing concern over biased outcomes in high-stakes applications such as hate speech detection. In this work, we present the first systematic study of group fairness in task arithmetic within this binary text and image classification regime, comparing it against full fine-tuning (FFT) and Low-Rank Adaptation (LoRA). We evaluate across multiple language models and datasets using standard group fairness metrics, including Demographic Parity and Equalized Odds. Our analysis shows that task vectors can be tuned to achieve competitive accuracy while reducing disparities, and that merging subgroup-specific task vectors provides a practical mechanism for steering fairness outcomes. We further provide a theoretical bound linking task vector scaling to fairness metrics, offering insight into the observed trade-offs. Together, these findings establish task arithmetic not only as a cost-efficient editing method but also as a fairness-aware alternative to existing adaptation techniques, within the standard group-fair classification setting, laying the groundwork for responsible deployment of large language models.

Published as a conference paper at ICLR 2026
ON FAIRNESS OF TASK ARITHMETIC: THE ROLE OF TASK VECTORS
Laura Gomezjurado Gonzalez 1- Hiroki Naganuma 2.g
Kotaro Y
Yuji Naraki
Takafumi Horie
Ryotaro Shimizu?
Stanford University Mila Université de Montréal Instittuee of Science Tokyo
IIndepeneent Researcher
"Kyoto University 7ZOZO Research
ABSTRACT
Model editing techniques, particularly task arithmetic with task vectors, offer efficient alternative to full fine-tuning by enabling direct parameter updates through simple arithmetic operations. While this approach promises substan- tial computational savings, its impact faimess has remained largely unex- plored-despite despite growing concern over biased outcomes in high-stakes applica- tions such as hate speech detection. In this work, we present the first system- atic study of group fairness task arithmetic within this binary text and image classification regime, comparing it against full fine-tuning (FFT) and Low-Rank Adaptation (LoRA). We evaluate across multiple language models and datasets using standard group fairness metries, including Demographic Parity and Equal- ized Odds. Our analysis shows that task vectors can be tuned to achieve com- petitive accuracy while reducing disparities, and that merging subgroup-specific task vectors provides practical mechanism for steering fairesss outcomes. We further provide a theoretical bound linking task vector scaling to fairness met- rics, offering insight into the observed trade-offs. Together, these findings es- tablish task arithmetic not only as a cost-efficient editing method but also as a aaire alternative to existing adaptation techniques, within the standard group-fair classification setting, laying the groundwork for responsible deploy- ment of large language models. Our code is available https: / / thub. com/ Bom/Lew Con//auraGomer httat//aaaaeeeetttteeeteetttrrrr
INTRODUCTION
As large language models (LLMM) are deployed across increasingly diverse applications, efficient techniques for adapting them to specific tasks have become essential. While model distillation and compact architectures reduce computational demands (Sanh et al., 2019b; Jiao al., 2020; Ture et al., 2020; Abdin et al., 2024), task-specific fine-tuning (FFT) remains resource-intensive. This has motivated parameter-efficient fine-tuning (PEFT) methods such adapters and Low-Rank Adaptation (LoRA) (Houlsby et al., 2019; Hu et al., 2022; Ben Zaken et al., 2022; Dettmers et al., 2023), which update only small fraction of parameters.
LoRA exemplifies this trade-off: it preserves most of the pretrained weights while reducing train- ing costs. However, PEFT methods do not resolve deeper concerns. In high-stakes domains with imbalanced data-Such as toxicity or hate-speech detection-they can maintain or even amplify biases (Ding al., 2024; Sap et al., 2019), raising concerns about fairness.
A promising alternative is task arithmetic with task vectors (Ilharco et al., 2023; Zhang et al., 2024; Yoshida et al., 2025b; Yoshikawa al., 2025; Yoshida al., 2025a). A task vector is defined the difference between fine-tuned model and its base counterpart. By adding, subtracting, or scaling such vectors, one can directly edit model behavior without gradient- retraining. This approach
Conrespnninng author Ipgomez @ stanford.ed
I "oorespnniing author naganuma. hiroki hatauuma.hiroki@e @ quebec.
1 CConesponding author yoshida.k k.0253 @ issstt.cc.jp These authors contributed equally this work.

---

Published as a conference paper at ICLR 2026
offers (i) computational efficiency, (ii) fine line-grained control over transferred capabilities, and (iii) enhanced interpretability when task vectors are associated with specific subgroups (Cerrato et al., 2022). Yet its faimess implications remain poorly understood. For example, enhancing performance on one demographic subgroup may inadvertently degrade outcomes for another, and the trade-offs with standard metrics such Demographic Parity (DPD) Equalized Odds (EOD) remain unclear.
To address this gap, we conduct the first systematic study of faimess in task arithmetic. We com- pare task vector editing against both FFT and LoRA, and further investigate whether injecting subgroup-specific task vectors into an FFT model provides additional control over fairness out- comes, In the NLP domain, our experiments focus on speech detection with LLaMA-7B (Tou- vron et al., 2023) and toxicity detection on Civil Comments (Borkan et al., 2019) using DistilBERT and Qwen2.5-0.58 (Qwen Team, 2025). In the computer-vision domain, we evaluate age classifi- cation on UTKFaae (Zhang et al., 2017) using ViT-Base/16 (Dosovitskiy et al., 2021). Across both domains and model architectures, we observe consistent dirness-utilinn trade-offs.
Our contributions are as follows:
Comprehensive evaluation: We compare FFT, LoRA, task vector editing, and hybrid approach that injects task vectors into FFT, analyzing their impact on fairness metrics and predictive performance (Figure 1).
Fairness through scaling: We show that adjusting task vector coefficients can substantially improve faimnsss while maintaining accuracy (Figure 2).
Subgroup-sensitive editing: We demonstrate that merging task vectors from underrepre- sented subgroups allows targeted fairness adjustments with negligible accuracy loss (Fig- ures 3a, 3b, 4a).
Theoretical grounding: We derive upper bound linking task vector scaling DPD and EOD, providing a principled explanation for the observed fairness hirness-accuracy accuracy trade-offs (Section 5.1 and Appendix C).
Throughout, the prediction task remains binary, but the fairness structure is not: we evaluate dispar- ities across multiple demographic groups, following widely used friunefairness benchmarks (Hardt et al., 2016; Agarwal et al., 2018; Zafar et al., 2017; Kearns et al., 2018; Das et al., 2024; Dutt al., 2023; Sukumaran et al., 2024). Taken together, our results provide empirical foundation for understanding -airness-accuracy trade-offs in task arithmetic within the standard group-fair binary- classification regime, while pointing toward extensions to more complex tasks (e.g., multi-labe or generative settings) as future work.
2 PRELIMINARIES
In this section, we first provide an overview of the fundamental concept of task vectors and the procedure known as task arithmetic, which applies these vectors to edit model behavior. We then introduce methods for merging multiple task vectors into single model.
Task arithmetic. A task vector is defined the difference in model parameters between fine- tuned model on a given task and the original base model. Formally, if Obuse are the pre-trained weights and 0mn the weights after fine-tuning task, then the task vector is: A Glak O (Ilharco et al., 2023).
This vector represents direction in weight space such that moving the base model's weights by A0 steers the model to perform well on that task. In other words, adding AO to Opase yields a model with improved performance the target task, without any additional training. Once computed, task vectors can be manipulated through simple arithmetic operations to edit model behavior directly in weight space (Ilharco et al., 2023; Ortiz-Jimenez et al., 2024). Key operations include:
Addition: Given two task vectors AdA and Ad (for tasks A and B), their sum can be applied to the base model (Frase + AP A0g) to produce model that exhibits improved performance on both tasks A and B (Ilharco et al., 2023). This task addition effectively combines knowledge from multiple tasks into one model.
2

---

Published as a conference paper at ICLR 2026
EDO
samn wwwpeeeelte
(a) Gender-base demographic subgroups
1 111
000
(b) Race-based demographic subgroups
Figure I: LoRA and FFT vs. Task addition with the optimal coefficient for the training accuracy (A = 0.8 for gender setting and X 0.5 for race setting) on group-wise accuracy, demographic parity difference (DPD, lower is fairer), and equalized odds difference (EOD, lower is fairer). Error bars denote the standard error across three seeds. Columns: group-wise accuracy, DPD, EOD. No consistent pattern emerges indicating that task addition systematically degrades subgroup fairness relative to LoRA FFT. While some subgroups show improvements or comparable results under task addition, others exhibit small declines.
Negation: Using the negative of a task vector, ~20, one can subtract a task's s influence. For example, applying Orase A (or equivalently Orase + (-A0x)) yields model with reduced performance on task A effectivvely unlearning or forgetting it-while preserving other behaviors (Ilhareo et al., 2023). This use- ful for removing undesirable skills or biases.
Scalar scaling: Multiplying a task vector by scalar À adjusts the strength of the edit. For example, using Orose + AÂ0 allows partial (0 < < 1) or amplified (A 1) application of a task's 's effect. This scaling provides fine- fine-grained control over how strongly the task knowledge is injected into the model.
Merging task vectors. Because task vectors live in a common weight space, they can be merged by linear combination. Let Og denote a base model and let (A0,) A be task vectors. generic merged model takes the form
mergged 0g + A,A0,
1=1
where the coefficients >; govern the relative contribution of each task (Zhang et al., 2024). Prior work explores learning these coefficients with, for example, Bayesian optimization or multi- objective search over À = (A1,...,Ak) but such methods introduce additional computational cost and stochasticity.
In this work we deliberately adopt the simpler and widely used single-scalar parameterization, in which global coefficient À scales the entire task vector update. Concretely, the models we evaluate can be written as
0(A) 0, A10,
(2)
where A a fixed task vector (or sum of task vectors), and the same scalar applied uniformly across all inputs. demographic subgroups, and faimess metrics. This single global coefficient mir- rors how task-arithmetic toolkits are exposed in practice, and provides a one-dimensional, easily interpretable control knob for tracing fairness-accuracy trade-offs. We treat À hyperparameter and select it by grid search over a fixed set of candidate values on a held-out validation set, using a common objective that balances accuracy and group-fairmnn faimess measures. The same validation proto- col is used to tune the corresponding regularization weights for our SFT. LoRA, and aast-arithmetic models, ensuring that all methods compared under matched cameess-accuran trade-offs.
3

---

Published as a conference paper at ICLR 2026
3 RELATED WORK
This section outlines how FFT, LoRA, task merging, and fairness are related. Further discussion of task-arithmetic efficiency, interpretability, and fairness in VLMs appears in Appendix A.
FFT and LoRA under fairness constraints. Parameter-efficien methods such as LoRA (Hu al., 2022) address computational bottlenecks by training only a small subset of parameters, yet they do not inherently resolve fairness concerns. Empirical studies show that LoRA's impact on sub- group performance mixed: in some settings it achieves results comparable to full fine-tuning (Ding et al., 2024), while in others it preserves undesirable biases or fails to mitigate toxic behaviors (Das et al., 2024). These divergent outcomes depend on factors such as the rank the LoRA updates, the base model's pre-existing biases, and the distributional properties of the fine-tuning data (Das et al., 2024). Recent work begins to incorporate faimess objectives directly into PEFT. Wang and Demberg (Wang& & Demberg, 2024) introduce multi-objective method that jointly targets accuracy and reduced stereotypical bias, and Sukumaran et al. (Sukumaran et al., 2024) propose FairLLRR. A, which embeds fairness-driven structure into low-rank updates for vision models. These efforts show that fairness-aware PEFT is feasible but often requires custom objectives and careful tuning.
Merging tasks and fairness composition. Despite the potential efficiency gains and inter- pretability offered by task arithmetic, the merging of task vectors for multiple groups can trigger new challenges. For instance, simply summing vectors may lead to "negative transfer," where up- dates beneficial to one subgroup degrade performance for another (Ding et al., 2023; Yu et al., 2020). In highly imbalanced settings, merging models through supervised fine-tuning can also dispropor- tionately favor majority groups while disadvantaging minorities (Cross et al., 2024).
Additionally, prior work shows that fairness guarantees often do not compose: even individual components satisfy group or individual faimess in isolation, composing them can break those guar- antees (Dwork& & Ilvento, 2018). This motivates our focus on post-hoc fask-arithmetii edits: adding scaling subgroup task vectors can be viewed composing behaviors, and interactions among subgroup-speeiffc task vectors can produce unpredictable shifts in metrics like Demographic Parity and Equalized Odds (Gohar al., 2023). Consequently, identifying effective ways to adjust task vectors- suchrsssuchhhe as through scalar scaling-remains a key step toward fairness-aware model editing. This work aims fill that gap by systematically evaluating how these operations influence both fairness and overall model accuracy.
In parallel, multi-task faimness methods such as Multi-Task- Aware Fairness (Wang al., 2021), Learning-to Teachh Teach Fairness- MTL (Roy & Ntoutsi, 2022), and FairBranch (Roy et al., 2024) manage fairness- fairness-accuracy trade-offs during training. Our study complements these by asking: when we edit models after training via task vectors, can simple controls (e.g., X-scaling) recover fairer behavior without retraining?
Group fairness metrics in binary text classification. Research on group fairness in text classi- fication predominantly adopts the binary prediction setting (e.g., toxic vs. non-toxic) with multiple demographic groups (Hardt et al., 2016; Agarwal et al., 2018: Zafar et al., 2017; Kearns al., 2018; Das et al., 2024; Dutt et al., 2023; Sukumaran et al., 2024). Within this framework, two metrics have emerged as canonical due to their interpretability and broad adoption: Demographic Parity Differ- ence (DPD), which measures disparities in positive prediction rates, and Equalized Odds Difference (BOD), which measures disparities in true- and false-positive rates (Hardt et al., 2016; Feldman et al., 2015; Kennedy et al., 2020a). These metrics form the standard baseline in faimess auditing toolkits (Bellamy al., 2018: Fairleam contributors, 2025) and underpin much of the fair-PEFT literature (Fraenkel, 2020; Pitoura, 2019; Quan et al., 2023; Ding et al., 2024). Their centrality motivated by theory: for calibrated, score-based classifiers with group-agnostic thresholds, many group-faimeet desiderata collapse to constraints on selection-rate and error-rate disparities. and clas- sical impossibility results show that calibration, selection parity, and error parity cannot be simul- taneously satisfied when base rates differ (Kleinberg et al., 2017). Consequently, DPD and EOD reveal the essential structure of fairness taimess-accuuacy trade-offs without invoking additional structural or counterfactual assumptions.
Our evaluation follows this binary prediction framework but focuses on a multi-group setting. We report both macro-averaged and worst-group versions of DPD and EOD. This choice allows to
4

---

Published as a conference paper at ICLR 2026
capture disparities across the full distribution of subpopulations, where faimess accuracy tensions are often most pronounced. Fairnsss metrics for multiclass, multi-label, or generative models remain active research areas, and no widely accepted standard analogous to DPD/EOD currently exists. Given this landscape, the binary multi-group regime offers a well-defined and empirically robust foundation for systematic faimess analysis. Formal metric definitions appear in Appendix B, and implementation details are provided in $4.1.
Gender Subgroups
Race Subgroups
4 EXPERIMENTAL SETUP
Men
817
Asian
311
Non-hinary
114
Black
1,007
4.1 CONFIGURATION
Trans men
178
Latinx
368
Trans unspecified
173
Native American
153
Building on the experimental framework established by Ding et (2024), we adopted their evaluation and experimental procedure to assess the fairness implications of LoRA in comparison to FFT. In our how
Trans women
148
Middle Eastern
493
Women Other
2,057
Pacific Islander
138
59
White
580
Other
302
work, we extend this analysis by focusing task arithmetic compares to both LoRA and FFT in terms of fairness and performance. The detailed ex- perimental setup is provided in Appendix D.
Total
3,546
Total
3,352
Table Berkeley D-Lab Hate Speech data statistics in the gender and race subgroups.
Datasets. We build on the Berkeley D-Lab Hate Speech dataset introduced by Kennedy et al. (2020a) and adapted by Ding et al. (2024). Our version contains 6,898 tweet-length snippets an- notated for hate speech and two sensitive attributes, Race and Gender, each with fine-grained sub- groups (e.g., Women, Non-binary, Men for Gender; see Table 1). We treat hate speech detection as binary classification and use the subgroup annotations (e.g., gender woman, race Asian) to com- pute subgroup-level performance and fairness metrics. To test generalization beyond hate speech, we also evaluate on the Civil Comments dataset (Borkan et al., 2019), large-scale toxicity cor- pus with sensitive-attribur labels. We binarize toxicity using a threshold of 0.5, treating comments above this value as positive ('flagged")), and assess fairness across Gender and Race subgroups. For the vision domain, we follow Ding al. (2024) and use the UTKFace face image dataset (Zhang et al., 2017), which provides age (in years), gender, and coarse-grainee race labels. We convert into a binary classisiiiiinnnonn task by thresholding age at 30 years into "younger" (<30) vs. "older" (>30), with the older group as the positive class. We treat race and gender as sensitive attributes and evaluate subgroup performance across the standard UTKFace race categories (White, Black, Asian, Indian, Others) and gender categories (Male, Female), mirroring the bogroup-based analysis in Ding et al. (2024).
Evaluation metrics and fairness scope. Our faimness analysis follows the standard group-fairness framework for supervised classification, where two criteria form the canonical basis for auditing disparate impact across demographic groups. DPD captures disparities in selection rates, and EOD captures disparities in class-conditional error rates (TPR/FPR). These metries widely used, the- oretically grounded, and highly interpretable: together they summarize the allocation and error-rate harms that motivate much of the fairness literature. This makes the binary text and image classifica- tion setting-whir selection and error rates are well-defined-al the natural domain in which to study fairness effects of task vector editing. For each protected attribute (e.g., gender, race/ethnicity), we compute subgroup-resolved DPD, EOD, and accuracy. We report per-subgroup values, macro- averages, and worst-group results. These choices mirror established practice and enable direct com- parison to prior TEFT-fairnes evaluations discussed in $3. Formal definitions and computation details appear in Appendix B.
4.2 PROTOCOL
We evaluate our methods main generative base model, LLaMA2-7B (Touvron et al., 2023)', and two compact baselines for CivilComments toxicity, DistilBERT (Sanh et al., 2019a)² and Qwen2.55-
!I..MA 2 is licensed under the LLAMA Community License, Copyright (c) Meta Platforms, Inc. All Rights Reserved. See https: /a meta . com/1 1 1ama/1 cense.
DistiBBERT is released under the Apache 2.0 License. See https: / //gf www..th..u.......eeeeee cww.huggi//hcccccco iggaceegggfa/e/ trans former rs/bl ob/ma // CENNS..
5

---

Published as a conference paper at ICLR 2026
0.5B (Qwen Team, 2025)². For the vision experiments on UTKFace, vitskiy et al., 2021)*. a 12-layer Vision Transformer with 768- -18ddmmensionll hidden size and 16x16 patch embeddings. We select ViT-Base/16 it represents a standard, high-capacity encoder chitecture commonly used in fairness studies (Ding et al., 2024) and provides a strong counterpart to text-based models, enabling cross-domain comparisons. Our faimess evaluations focus on two sensitive attributes: gender and race across both datasets, computing subgroup accuracy, DPD, and EOD, These selections span distinct architectures (decoder-only vs, encoder-only), parameter scales, tasks, and label taxonomies.
ViT ViT-BBse/16 (Doso-
For FFT, the pretrained model was fine-tuned on the combined training data from all subgroups of the target attribute (gender or race). Evaluation was then performed on the test data from each corre- sponding subgroup, enabling fine-grained assessment of both performance and fairness. For LoRA, we followed the same training and evaluation procedure as FFT. The rank of LoRA's adaptation modules was set 8, following Ding al. (2024).
For task arithmetic, we applied compositional fine-tuning approach. The training data was par- titioned by subgroup (gender or race), and FFT was applied separately to each subgroup's data to produce fine-tuned models 0,. From these, we computed task vectors A0, relative to the base model. These vectors were then merged using the approach described in Eq. (1), with a single, uniform scal- ing coefficient À applied to all vectors. À served as the sole hyperparameter in the merging process and was tuned on the training data. The evaluation metrics were computed in the same manner as for FFT and LoRA.
Task vector coefficient adjustment. Building on the task vector merging framework introduced in Eq. (1), we further explore the impact of the scaling coefficient À faimesss outcomes. Specifically, we vary the uniform task vector coefficient across broad range (from 0.0 to 1.0 with 0. intervals) and evaluate how this adjustment influences subgroup-level fairness metries, including accuracy, DPD, and EOD.
Impact of worst- perrst-prforming subgroup task vectors on fairness and performance. To investi- gate whether incorporating task vectors from underperforming subgroups can improve faimess with- out sacrificing overall performance, we first identified the perest-pefformin subgroups within each attribute based on the average of DPD and EOD under the FFT setting. We excluded the "others" group from this analysis as it does not reflect the characteristics of any specific subgroup. This selce- tion was informed by both our experimental results and those reported in Ding et al. (2024), which showed consistent patterns. For gender, the porf--performnn subgroups were men and women; for race, they were Asian and Native American. We constructed a model variant by injecting worst-perfomne pernsteerfoning subgroup task vector into the base fine-tuned model:
Onew = 0srr + X( X(Oworst-pette perst--peroonning subgroup 0g).
where \ controls the strength of the task vector injection. We varied from 0.0 to 1.0 at 0.2 intervals to analyze the effect of this targeted addition subgroup fairness metries and overall accuracy.
5 RESULTS
5.1 THEORETICAL INTUITION.
To situate the subsequent analysis, we first provide a high-level rationale for why task vector scaling interacts systematically with group-wise faimess metrics. Scaling the task vectors modifies the model parameters directions associated with each subgroup, and the resulting model (()) can be viewed as perturbation of the balanced reference point 0 obtained at X, 1 for each g. Since group disparitics are minimized at this balanced model, the behavior of fairness metrics is largely governed by the magnitude of the deviation 0()) A and by the norms of the subgroup-specific task vectors that induce this deviation. For convenience, we write X = (Ag)s= g=1 for the vector of subgroup-specific scaling coefficients, with the balanced setting corresponding to Xg 1 for all g.
'See the Qwen2.5-0.5B model card and license details at https: //wggii acc..c...... co/ wtwcc.coo/wee///// /wen/ Qwen2.5-1 5..0 5H.
'Original VIT implementation is released under the Apache 2.0 License. See https: //t///gtttt tittbb.bcm/ google- research /vi sion siionttraa formr..

---

Ag) Published as a conference paper at ICLR 2026
0.12
0.8
0.07
Accuracy
EOD
0.10
0.7
0.05
Full
0.08
0.6
LoRA
0.04
Task Addition
0.06
Figure 2: Varying the task arithmetic coefficient and comparing against FFT (dashed baseline blue line) and LoRA (orange dashed) for macro-averaged accuracy (left), demographic parity difference (DPD, center), and equalized odds difference (EOD, right) on the gender subset. Higher accuracy is better; lower DPD/EOD indicate improved group fairnsss. For X 2 0.3, task addition maintains competitive accuracy while typically lowering DPD/EOD relative to both baselines.
We complement our empirical findings with an analytical upper bound that links task vector scal- ing to fairness metrics. We make several assumptions for the analysis: [A1] the prediction score (a()) is L-Lipschitz in 0 and group-wise calibrated to its binarized prediction (see Appendix C), [A2] a task vector for each group g (i.e. 20,) is obtained under an identical optimization protocol, [A3] the scaling coefficients satisfy the normalization Eg Ag = G, [A4] the group distributions differ only through the sensitive attribute so that the balanced model 0 0g + à L0, satisfies DPD 0) 0. [A5] near the binarization threshold, both pa (X) and pa(X) X) have uniformly bounded conditonnii densities, i.e., for some By for each true label y and sufficiently small y.
A0,
Assumptions [A1] and [AS] impose mild regularity and calibration conditions on the prediction scores, whereas [A2]A44] correspond comparatively natural requirements on the training setup. Under those assumptions, the DPD and EOD satisfy the following inequalities:
Theorem (informal), Consider the merged model ((A) 0o + En Ag A0g. Then DPD and EOD satisfy:
DPD(0(A)) S UA) = 2L 2222211 AAg - 11110 for Lipschitz constant L,
9
ROD(O(A)) S (OD(O) + (Bo (.)
A full derivation and tighter constants are provided in Appendix C (Proposition 1,2).
These upper bounds explain the empirical trends observed Figure 2. Both DPD and EOD share a common upper-hound term (A), which monotonically decreases as X (= A1
approaches 1 for each g. Notably, the bound (PD(((A)) < (()) together with U(A) + 0 as A + 1 implies that deviations of À exert pronounced influence on DPD. Bo and B1 are class- specific constants that upper-bound the score density near the decision threshold for y=0 and y=1, respectively. Intuitively, this bound shows that deviations of the scaling coefficient À from the balanced setting (A₁ ... λg = 1) enlarge disparities in proportion to the norms of subgroup task vectors.
5.2 EMPIRICAL RESULTS OVERVIEW.
Key takeaway. Across all datasets and architectures we study, task arithmetic exposes two com- plementary mechanisms for navigating Muinesss-accural trade-offs. First, sweeping a single scalar coefficient À over uniformly merged subgroup task vectors yields a smooth fairness-utility fron- tier: over a broad range of X, task addition matches FFT/LoRA (or SFT/LoRA) accuracy while substantially reducing DPD and EOD on Berkeley D-Lab hate speech and Civil Comments, and in our additional vision experiments ViT-Base. Second, injecting individual subgroup task vectors into an FFT model (Section 5.4) pro- duces structured, sabbroup-dependee shifts in faimess: some subgroup vectors (e.g., Men, Asian) move the model along favorable fairness-an accurray frontiers, while others (e.g.. Native American) can worsen DPD/EOD when accuracy remains stable. This behavior is consistent with the
UTKFace, across LLaMA-2, DistilBERT, Qwen-2.5, and
7

---

Published as a conference paper at ICLR 2026
group- weighted ERM view developed in Section 5.1 and Appendix C.2, where sensitivity to À scales with || 1 Together, these observations show that task arithmetic provides both a global con- trol knob (a uniform A) and subgroup-targee targeted knobs (the A0₂) that not exposed by standard FFTYLoRA training. The empirical trends and their first-order mechanism also align with prior lit- erature: our macro-averaged accuracy, DPD, and EOD findings for FFT and LoRA consistent with (Ding et al., 2024).
Figures la and 1b instantiate this story for hate-speech detection on Berkeley D-Lab with LLaMA-2 by comparing FFT, LoRA, and task addition across gender and race subgroups. For task addition, we selected À= 0.8 for gender and X 0.5 for race, as these achieved the highest average training accuracy across three random seeds within the tested range À E [0.0, 1.0]. These visualizations provide a direct comparison of subgroup- wise model behavior. From the subgroup-level bar plots in Figure 1, we observe that accuracy remains consistently high and comparable across all three adapta- tion methods, regardless of subgroup. On Civil Comments, on both DistiBBERT and Qwen-2.5, task addition similarly reduces group disparities while keeping accuracy competitive (see Appendix F and Table for full confidence intervals and results).
We also observe that, relative to FFT, task addition improves faimess in five of seven gender sub- groups and in three of cight race subgroups. with no single method dominating across all groups. The subgroup-dependent nature of these effects motivates treating À deliberate tuning knob and inspecting subgroup behavior explicitly. As shown in Appendix C.2, task addition realizes a group- weighte ERM in the linearized model: concretely, 8(2) 0 + Eg the one-step minimizer of first-order surrogate where subgroup g is re- re-weighted by Xg. This ex- plains the smooth fairness-utility frontier traced by sweeping A, and Theorem 5.1 predicts larger parity swings for groups with larger || ||02| The observed curves in Fig. align with those predic- tions without further assumptions.
g >g A0g coincides with
5.3 CONTROLLING ACCURACY AND FAIRNESS METRICS THROUGH LAMBDA.
Figure illustrates the overall performance of FFT, LoRA, and task arithmetic as the scaling coeffi cients for task addition vary from 0.0 to 1.0. We observe how varying the trit-erit coefficient impacts macro-averagee
accuracy (left). demographic parity difference (DPD, center), and equal- ized odds difference (EOD, right) on gender subset of the data. As À increases from 0.0 to 0.2, we observe a peak in accuracy, but this configuration yields higher DPD and EOD, indicating re- duced fairness. Beyond À = 0.3, accuracy remains competitive compared to FFT and LoRA, while both DPD and EOD progressively decline, suggesting that fairness improves without severely com- promising performance. Notably, these task addition curves stay consistently lower than FFT and LoRA in terms of DPD and EOD at higher À values. Overall, this ablation could indicate that tuning X provides practical mechanism for balancing accuracy and faimess objectives, offering guide- lines for practitioners who wish to fine-tune fairness outcomes while maintaining strong predictive performance.
5.4 SUBGROUP-TARGETS VECTORS: GAINS WITH TRADE-OFFS
To further analyze the effects of subgroup-specift task composition, Figure 3a-3b illustrate heatmaps where the y-axis lists each method or configuration under evaluation: FFT as baseline, followed by task arithmetic with varying scaling coefficients (0.0 to 1.0 with 0.2 intervals). The x-axis represents the subgroups- (e.g., Women, Trans, etc. for Gender). Each cell shows the cor- responding performance metric (e.g., mmacro-averaged accuracy, DPD, or EOD for a given method on specific subgroup. For these experiments, we added the task vector of the perft-pero subgroups (Women and Men for the gender dataset subset, and Asian, and Native American for the race dataset subset) to the FFT model, as explained earlier.
We generally observe that increasing the scaling coefficient À tends to improve overall accuracy, consistent with the trends observed in Figure 2. However, effects are not uniform across all sub- groups. In the race-based plots, for example, the Asian subgroup consistently achieves the highest accuracy and lowest DPD/EOD BaiEEODD-ihhiihhtining a recurring tradeoff where performance gains for one group may exacerbate disparities for others. When the Women task vector is added (Figure 3b), accuracy improves for the Trans Women subgroups. However, faimesss metrics for subgroups such as Men tend to worsen the scaling coefficient À increases.
8

---

Published as a conference paper at ICLR 2026
Accuracy
DPD
DDD
2,700
F47
1:11
3:M
0.22
2.84
P+1
0.01
0:00%
2:344
2:30
0.155
0.01
5:T1
6.21
01447
0:335
Jm
02
0.2
080
900A
RALS
A399
4:24
6:04
2.250
3/4
2,83
2.125
2.78
0.0g
-,100
2.78
2.2
0.04
2,575
There Hen
Theis Mespeccked
Thank Wiattee
-274
Mann Binary
Trens den
Brong Wereet
Man Wilurr
Thand Hem
Theis seeppcctee
Slas Woreen
0.00
(a) When Men task vector added to the FFT model on the gender subset.
0.12
Accuracy
DDD
EDD
2,200
F-1
1+1
9.377
Jm
0.2
2,00
2.250
9.125
2.78
0.05
3.76
0.04
Am
5.74
3.6
-50775
Siddyy
TieO Men
Thenk Ueage Undpectlod
Tlank Oittea
Wixtoo
Maan Blaas
Stace Sernes
Wommun
0.02
Thenk sespperteed
Nem
Winmon
Trand
(b) When Women task vector added to the FFT model on the gender subset.
Figure 3: Heatmaps of Accuracy (left), DPD (center), and EOD (right) for Men (top) and Women (bottom) subgroups under the baseline FFT model (A 0.0) and with increasing À values from 0.2 1.0 in 0.2 increments. The task vector for Men was added on the gender subset (top), and the task vector for Women was added on the gender subset (bottom). Darker cells indicate higher values on each metric's scale; for DPD/EOD, lower values are better.
In Figure 3a, injecting the Men task vector improves performance for some subgroups, yet Women consistently show lower accuracy and do not see consistent fairness improvements at higher X. Some groups (e.g.. Other, Trans Men, Trans Women) begin with relatively poor fairesss under FFT and show partial improvements with task vector addition. Still, these improvements are not uni- versal-for example, the Other subgroup often retains high EOD values regardless of X. Likewise, Native American accuracy remains mostly unchanged across A. while fairness metrics can deterio- rate when injecting task vectors for other groups To visualize these results in more detail, Figure 4a shows macro-averaged accuracy, DPD, and EOD for the Men task vector added to the FFT model. The plots illustrate how varying the scaling coefficient À impacts overall performance and fairness, highlighting the effects of subgroup-speeiffe task injection. We can observe in Figure 4a that inject- ing the Men task vector into the FFT model results in a slight accuracy gain and a clear monotonic decrease in both DPD and EOD À increases-indication a favorable and consistent improvement in fairness on the gender subset.
However, Figure 4b and the additional plots in Figures 10 and 11 in Appendix E.2 show more varied patterns as seen on Figures 3a and 3b. When injecting the Native American task vector (Figure 11), accuracy remains stable while fairness seems to decrease (increased DPD and EOD). Asian (Figure 10) shows the same behavior as injecting the Men task vector (Figure 4a), positive increase fairness metrics as À increases. These results show that injecting task vectors shifts fairness and performance in group-specific manner, tracing clear fairess heterogeneity is expected: per $5.2 and Theorem 5.1, sensitivity scales with |A0z/2. Practically, task vector merging thus offers cuubrruup-coddiioon or hurt which groups provides a new actionable design consideration that SFT/LoRA do not expose, and that hasn't I been explored in previous task arithmetic literature.
frontier. This
control knob: identifying which Ad, help

---

Published as a conference paper at ICLR 2026
0.840
0.07
0.12
Accuracy
0.B35
0.11
090
0.06
EOD
0.830
0.10
Full
0.825
0.05
0.09
LoRA
Foll-Worst
0.04
0.08
(a) When Men task vector added to the FFT model the gender subset.
0.B4
0.07
0.12
Acouracy
0.83
DPD
0.08
00D
0.10
0.82
Full
0.05
0.09
LoRA
0.81
Full-Worst
0.08
(b) When Women task vector added to the FFT model on the gender subset.
Figure 4: Impact of injecting the Men (a) and Women (b) subgroup task vectors into the FFT model on the gender data subset. The plot illustrates how scaling coefficient reduces DPD and BOD, outperforming the baseline FFT (blue dashed) and LoRA (orange dashed), with negligible impact macro-averts avero-averaged accuracy.
CONCLUSION AND LIMITATIONS
Conclusion. In this study, investigated how task arithmetic with task vectors affects group fair- ness, in comparison to conventional FFT and LoRA methods. We conducted experiments to assess how adding a fairness-oriented task vector impacts prediction accuracy and fairness metries, includ- ing DPD and EOD across multiple subgroups. The results indicate that, with appropriate settings of the scalar coefficient \, task arithmetic can improve DPD and EOD without significantly compro- mising overall model accuracy. Notably, low to moderate values of À often reduced prediction bias in minority groups compared to FFT and LoRA, pattern we observe across three datasets (hate speech, toxicity, and age detection) and four model families (LLaMA-2, DistilBERT, Qwen-2.5, ViT-Base).
Furthermore, the arithmiitmettic formulation exposes single interpretable parameter X, which al- lows practitioners to trace -peeefc changes along a fairness- accurray trade-off curve. This transparency makes it easier to diagnose when edits disproportionately benefit or harm particular groups
Limitations. Despite these promising results, several limitations remain. First, experiments focus on the open-weight 0.5-7B regime (DistilBERT, Qwen-0.5BB LLaMA-2-7B, ViT-Base), where task arithmetic is practically deployable and comparable to prior work. This setting gives full weight access to construct subgroup task vectors and compute exact DPD/EOD, We do not evaluate against proprietary frontier models; prior work shows that prompt- based debiasing can offer partial gains but is brittle and does not provide principled control over group-level disparities (Ma et al., 2023; Furniturewala al., 2024; Chu et al., 2024). Extending our framework larger or API-only models is an important direction for future work. Second, we study simple scaling scheme based on a shared global coefficient A. This keeps the intervention interpretable but cannot capture richer multidimensional faimesss constraints. More expressive parameterizations (e.g.. multiple coefficients or optimization over combinations of X, represent natural extensions). Finally, our evaluation covers limited set of classification tasks with single sensitive attributes. The effectiveness of task arith- metic may depend on dataset and subgroup structure, so assessing generalization to broader domains and intersectional settings remains important.
10

---

Published as a conference paper at ICLR 2026
Reproducibility statement. We provide code " configs, and scripts to reproduce all experiments, including data preprocessing, training, and evaluation. All datasets and base models used are open- sourcc/publicly available; we include scripts to fetch the exact versions. Exact hyperparameters, model identifiers, and implementation details are documented the appendix, along with seeds and hardware/software specs. Results reported over multiple runs, and we provide instructions to regenerate all figures and tables from logged outputs.
Acknowledgments Our deepest gratitude goes out to the anonymous reviewers whose invaluable insights substantially enhanced the quality of this manuscript. This work was supported by RBC Borealis through the RBC Borealis AI Global Fellowship Award, which was awarded to Hiroki Na- ganuma. We are also profoundly grateful to The Masason Foundation for their generous support in providing computational resources and for fostering an environment that encourages deep research collaboration. The computation resources were further supported by "TSUBAME Encouragement Program for Young/Female Users" of Center for Information Infrastructure at Institute of Science Tokyo and by "Joint Usage/Research Center for Interdisciplinary Large-scale Information Infras- tructures" in Japan.
REFERENCES
Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari. Jianmin Bao, Harkirat Behl, et al. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404. 142 2024.
Alekh Agarwal, Alina Beygelzimer, Miroslav Dudik, John Langford, and Hanna Wallach. A reduc- tions approach to fair classification. In Jennifer Dy and Andreas Krause (eds.). Proceedings of the 35th International Conference Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 60-69. PMLR, 10-15 Jul 2018.
Alekh Agarwal, Miroslav Dudik, and Zhiwei Steven Wu. Fair regression: Quantitative definitions and reduction-based algorithms. Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Pro- ceedings of the 36th International Conference on Machine Learning. volume 97 of Proceedings of Machine Learning Research, pp. 120-129. PMLR, 09-15 Jun 2019.
Junaid Ali, Matthaus Kleindessner, Florian Wenzel, Kailash Budhathoki, Volkan Cevher, and Chris
Russell. Evaluating the fairness of discriminative foundation models in computer vision. In Pro- ceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society, AIES '23, pp. 809-833, New York, NY, USA, 2023. Association for Computing Machinery. ISBN 9798400702310, doi: 10.1145/360021 3.064720. URL https: doi. wtt.:....1..........1...1.......
114. /600425333111111 36047200
Rachel K. F. Bellamy, Kuntal Dey, Michael Hind, Samuel C. Hoffman, Stephanie Houde, Kalapriya Kannan, Pranay Lohia, Jacquelyn Martino, Sameep Mchta, Aleksandra Mojsilović, Scema Nagar, Karthikeyan Natesan Ramamurthy, John Richards, Diptikalyan Saha, Prasanna Sattigeri, Monin- der Singh, Kush R. Varshney, and Yunfeng Zhang. AI Fairness 360: An extensible toolkit for detecting, understanding, and mitigating unwanted algorithmic bias. arXiv preprint, 2018. URL pestps:/aareen otg/oogg//ss... 810 01943.
Elad Ben Zaken, Shauli Ravfogel, and Yoav Goldberg. Bitfit: for transformer-based masked language-modee models.. Findings the Association for Computational Linguistics: ACL 2022, pp. 1-9, 2022.
Simple parameter-effiient fine-tuning
Daniel Borkan, Lucas Dixon, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. Civil com- ments: A new public text data set of annotated online comments. In Proceedings of the First Workshop on Natural Language Processing for Internet Freedom (NLP4IF): Censorship. Disin- formation, and Propaganda, 2019. URL httph: /ttp//aaxivv./. http://wwwww....aaaa./.o//// http://aa//1119.0.553. 045611
Mattia Cerrato, Marius Köppel, Alexander Segner, and Stefan Kramer. Fair interpretable learning via correction vectors. arXiv preprint arX.v:2201.66344442 06344333.3 2022. doi: 10.48550/ar1iv.2201 063433.
URL https::////. pettps://arxivt
*Our cnde is available at https: /ttp//gitttttaaa http://///aaaaeeeeerrrrrooo////e vector_6eploy
11

---

Published as a conference paper at ICLR 2026
Zhibo Chu, Zichong Wang. and Wenbin Zhang. Fairness in large language models: A taxonomic survey. arXiv preprint arXin:2404.01334 2024. doi: 00.48550/aXx..22444.011333 URL https: / llwaaii..wg/ab////eeeaaeaaa /ww.////pp/......o////o//./ 01349. Version 2, 19 Dec 2024.
James L. Cross, Michael A. Choma, and John A. Onofrey, Bias in medical ai: Implications for clinical decision-making. PLOS Digital Health, 3(10))0000065, Nov 2024. doi: 10.1371/journal. pdig.0000 0000651. Published on 7 Nov 2024.
Saswat Das, Marco Romanelli, Cuong Tran, Zarreen Reza, Bhavya Kailkhura, and Ferdinando Fioretto. Low-rank finetuning for Ilms: A fairesss perspective. arXiv preprint arXiv:2405. 18572, 2024. doi: 10.48550/arxiv.4444 240.. URL https:: /tttpps///rxii.../ httpaaaaaw...oraaaaraaaaaaaaas Submitted on 28 May 2024.
wwt./bog/////oo./..oooo 2405 18572.
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. In Advances in Neural Information Processing Systems (NeurIPS), 2023.
Chuntao Ding, Zhichao Lu, Shangguang Wang, Ran Cheng, and Vishnu Naresh Boddeti. Mitigating task interference in multi-task learning via explicit task routing with non-leamable primitives. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), URL https: http///axvvvv.././e org/ abs 2308 02066.
2023. doi: 10.48550/arX1.23388 02066. CVPR 2023.
Zhoujie Ding, Ken Ziyu Liu, Pura Peetathawatchai, Berivan Isik, and Sanmi Koyejo. On fairness of low-rank adaptation of large models. arXiv preprint 2tti::2005.15122 2024. doi: 10.48550/ arXiv.2405.1 17512. URL. https:: /a marxii.oommm www.iivorrg/aaio abaaa///222 2405. 17512. COLM 2024 camera ready; version 2 revised 18 Sep 2024.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image worth 16x16 words: Transformers for image recog- nition at scale. In International Conference on Learning Representations (ICLR), 2021. arXiv:2010.11922
Raman Dutt, Ondrej Bohdal, Sotirios A. Tsaftaris, and Timothy Hospedales. FairTune: Opti- mizing parameter-efficient fine-tuning for fairness in medical image analysis. arXiv preprint 10000533 2023.
Cynthia Dwork and Christina Ilvento. Group fairness under composition. In FAT/ML Workshop, 2018. URL https: fttm/llwllllll...m///./../.m pegg/media/eee //g.///naaa/cccouutt///rrouou fairness.___ under_ccomposition..ee Workshop paper.
Fairlearn contributors. Reductions Faiream 13.10.0..0 13..0.ev0 documentatin,i,,,nn, 2025. URL https:// fairlearn.
html. Accessed:
March 16, 2025.
Michael Feldman, Sorelle A. Friedler, John Moeller, Carlos Scheidegger, and Suresh Venkata- subramanian. Certifying and removing disparate impact. In Proceedings of the 2/st ACM SIGKDD International Conference Knowledge Discovery and Data Mining (KDD), pp. 259- 268. ACM, 2015. doi: 10.1145/2783258.278888 1145/2783258. 2773311..
URL https:/ /tl///dl./// acm.
Aaron Fraenkel. Fairness and Algorithmic Decision Making. UC San Diego, 2020. URL https: / //afraenkel. www.llgithbb..... io/
html. Lecture Notes for UCSD
course DSC 167.
Shaz Furniturewala, Surgan Jandial, Abhinav Java, Pragyan Banerjee, Simra Shahid, Sumit Bhatia, and Kokil Jaidka. "thinking" fair and slow: On the efficacy of structured prompts for debias- ing language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 213-227, Miami, Florida, USA, 2024. Association for Computational Linguistics. doi: 10. 18.18853/v 120224.mmllpmmin..t. URL https: /ates://cclaaae //aclanthooogy /rr/ 2024 emnlp-maing 13/.
12

---

Published as a conference paper at ICLR 2026
Antonio Andrea Gargiulo, Donato Crisostomi, Maria Sofia Bucarelli, Simone Scardapane, Fabrizio Silvestri, and Emanuele Rodolà. Reducing task interference in model merging. arXiv preprint arXiv:2412.00081, 2025. doi: 10448500arXx..2412.00088 URL https: ptttps://arxiv..o /arxiv.orgg/
abs/2412. 00081.
CVPR 2025 paper.
Usman Gohar, Sumon Biswas, and Hridesh Rajan. Towards understanding faimess and its com- position in ensemble machine learning. In Proceedings of the 45th International Conference on Software Engineering (ICSE), 2023, doi: 10:48550/arxxw:2222.44553 URL https: / //aaxiv.orrg//////..vv......o..a abs/ 2212 04593. Accepted at ICSE 2023.
Moritz Hardt, Eric Price, and Nathan Srebro. Equality of opportunity in supervised learning. In Advances in Neural Information Processing Systems 29 (NeurlPS), pp. 3323-3331 2016. URL
02413.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, An- drea Gesmundo, Mohammad Attariyan, and Sylvain Gelly. Parameter-efffcient transfer learning for nlp. International Conference on Machine Learning, pp. 2790-2799. PMLR, 2019.
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shawn Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Confer- ence on Learning Representations (ICLR), 2022. URL https: /ttp///avvvv/////////////////aav org/ abs/ 09685.
Gabriel Ilhareo, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. In International Conference on Learning Representations (ICLR), 2023. Published as a conference paper at ICLR 2023.
Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. Tinybert: Distilling bert for natural language understanding. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 4163-4174, 2020.
Michael Kearns, Seth Neel, Aaron Roth, and Zhiwei Steven Wu. Preventing faimess gerryman- dering: Auditing and learning for subgroup fairness. Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings Machine Learning Research, pp- 2564-2572. PMLR, 2018
Brendan Kennedy, Xisen Jin, Aida Mostafazadeh Davani, Morteza Dehghani, and Xiang Ren. Con- textualizing hate speech classifiers with post-hoc explanation. In Proceedings of the 58th Anmual Meeting of the Association for Computational Linguistics, pp. 5435 5442, Online, 2020a. As- URL https:
sociation for Computational Linguistics. doi: 10. 1021 //aclanthology wrgg 2020 acl-ma i n. 483/.
Chris J Kennedy, Geoff Bacon, Alexander Sahn, and Claudia von Vacano. Constructing interval variables via faceted rasch measurement and multitask deep learning: a hate speech application. arXiv preprini (XXi::0009.102777 2020b.
Jon Kleinberg, Sendhil Mullainathan, and Manish Raghavan. Inherent trade-offs in the fair de- termination of risk scores. 8th Innovations Theoretical Computer Science Conference (ITCS), volume 67 of Leibniz International Proceedings in Informatics (L.PIcs), pp. 43:1- 43:23. Schloss Dagstuhl-Leibnizz ZZetrru für Informatik, 2017. doi: 10.4230/LIPIcs.TTC 2017.43. URL https: drops.
4230/
LIPICS ITCS. 2017.43.....
Hongkang Li, Yihua Zhang, Shuai Zhang, Meng Wang, Sijia Liu, and Pin-Yu Chen. When is task vector provably effective for model editing? generalization analysis of nonlinear transformers. doi: A44.5500arxiv.2544... arxiv. www.orrgaaaoo abs 2504 1 10957. Published as conference paper at ICLR 2025.
arXiv preprint arXiv:2504. 10957, 2025.
URL https://
Mingxin Li, Zhijie Nie, Yanzhao Zhang, Dingkun Long, Richong Zhang, and Pengjun Xie. Improv- ing general text embedding model: Tackling task conflict and data imbalance through model merging. arXiv preprint vrXiv:2410.150354 2024. URL https: //axivvv...// org/ abs/ 241 0.10..2 15035. 19 Oct 2024.
13

---

Published as a conference paper at ICLR 2026
Wei Ma, Jieyu Zhao, Emily Sheng, Xuezhi Li, Ananya Kumar, al. Fairness prrness--uiiedd few-shot prompting for large language models. In Advances Neural Information Processing Systems, 2023.
Guillermo Ortiz-Jimenez, Alessandro Favero, and Pascal Frossard. Task arithmetic the tangent space: Improved editing of pre-trained models. arXiv preprint, 2024.
Evaggelia Pitoura. Towards diversity-aware, fair, and unbiased data management. In ISIP 2019, Heraklion, Greece, May 9 2019.
Tangkun Quan, Fei Zhu, Quan Liu, and Fanzhang Li. Learning fair representations for accuracy parity. Engineering Applications of Artificial Intelligence, 119:105819, 2023. doi: 10.1016/) engappai.2023.1066
Qwen Team. Qwen2.5-0.5b. https: ottp:///oo...t//////////////../a Model card: accessed 2025-09-19.
http://aa.....ee/////////eeee22a 0 5B, 2025.
Arjun Roy and Eirini Ntoutsi. Learning to teach fairness-aware deep multi-task learning. In Machine Learning and Knowledge Discovery in Databases. ECML PKDD 2022, volume 13715 of Lecture Notes in Computer Science, pp. 710-726. Springer, 2022. doi: 10.10079988
URL httph:ttp////.....m happs://azxive /ttpxaiiivi.////e///ee////eeevva org/ abs /2t. 08403.
Arjun Roy, Christos Koutlis, Symeon Papadopoulos, and Eirini Ntoutsi. Fairbranch: Mitigating bias transfer in fair multi-task learning. In Proceedings of the 2024 International Joint Conference on Neural Networks (IJCNN). IEEE, 2024. doi: 10. 109/IINN6609999.222222 10651026111111 URL
https: / /ttp//aaxvvv.////////ee//..aa..t org/ abs 2310 13746. Also available as arXiv:2310.13746.
Zahraa Al Sahili, loannis Patras, and Matthew Purver. Data matters most: Auditing social bias in contrastive vision-language models. Transactions on Machine Learning Research, 2025. ISSN
2835-8856. URL httttt://pppeeeeeteeee http///pppeeeeeeeeeeeeee net
wtwp:e/ewwww..eeooonnnn......ee. Eorumr id- 3vF2
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, distilled version of bert: smaller, faster, cheaper and lighter. In Proceedings of the 5th Workshop on Energy Efficient Machine Learning and Cognitive Computing (EMC2) NeurIPS 2019, 2019a. URL https: /ttp///aavvv.//aa///ai/eeee..aal org/ org/abs/////ooooooooogggss/.sssr 1910. 01108.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv: 1910::19900,08,, 2019b.
Maarten Sap. Dallas Card, Saadia Gabriel, Yejin Choi, and Noah A. Smith. The risk of racial bias in hate speech detection. In Proceedings of the 57th Annual Meeting of the Association for Com- putational Linguistics, pp. 1668-1678 Florence, Italy, 2019. Association for Computational Lin- wwwaaaaccttctttnorrnoeceeee
guistics. doi: 10.18653/v1/P19- 163. URL
1116..
Rohan Sukumaran, Aarash Feizi, Adriana Romero-Sorian, and Golnoosh Farnadi. Fairlora: Un- packing bias mitigation in vision models with fairness-driven low-rank adaptation. arXiv preprint arXiv:2410.177 17358, 2024. 2ass/2410.17355
doi: 10448500axXi.2410..775 URL https: http///axvvvv///////////
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Ar- mand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation lan- guage models. arXiv preprini arXiv:2302. 2023. URL htttt:://axii..oeeett http///wxx.og////eeee//////o abo/ 2302.13971.
Catalin Ture. Ming Wei Chang. Kenton Lee, and Kristina Toutanova. Well-read students learn better: On the importance of pre-training compact models. In Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 3592-3601, 2020.
Shiyue Wang and Vera Demberg. A parameter-efficient multi-objective approach to mitigate stereo- typical bias language models. In Proceedings of the Sth Workshop on Gender Bias in Natural Language Processing (GeBNLP 2024), 2024.
14

---

Published as a conference paper at ICLR 2026
Yuyan Wang, Xuezhi Wang, Alex Beutel, Flavien Prost, Jilin Chen, and Ed H. Chi. Understanding and improving fairness accuraac trade-offs multi-task learning. In Proceedings of the 27th ACM SIGKDD Conference Knowledge Discovery and Data Mining, pp. 1748-1757. ACM, 2021. doi: 00.1145/3447548.446333 3467336.. Uttpttss: /tt.xiiivvv//////////e/////.aevv org/ abs 2106. 02705.
Xuyang Wu, Yuan Wang, Hsin-Tai Wu, Zhiqiang Tao, and Yi Fang. Evaluating fairness in large vision-language models across diverse demographic attributes and prompts. Chris- tos Christoddoulopoulos. Tanmoy Chakraborty, Carolyn Rose, and Violet Peng (eds.). Findings of the Association for Computational Linguistics: EMNLP 2025, pp. 22973-22991, Suzhou, China, November 2025. Association for Computational Linguistics. ISBN 979- 979-8-89176-335-7 doi: 10.18653/v1/20255 hind//////./....in-----aaa Sindings-emin 1251/.
URL https: htttttt//ccaatttaeeee http///lllantcol..orgr//////////
Kotaro Yoshida, Yuji Naraki, Takafumi Horie, Ryotaro Shimizu, and Hiroki Naganuma. Dis- tac: Conditioning task vectors via distillation for robust model merging. arXiv preprint arXiv:2508.01145, 2025a.
Kotaro Yoshida. Yuji Naraki, Takafumi Horie, Ryosuke Yamaki, Ryotaro Shimizu, Yuki Saito, Julian McAuley, and Hiroki Naganuma. Mastering task arithmetic: jp as a key indicator for weight dis- entanglement. The Thirteenth International Conference on Learning Representations, 2025b.
wwwptttpss//pppeeeeeeeeeeeeeeeee wet.erreeiieeeeeeeeeeeeee forum? 1id:1dd=lwW VwWi VwW162bxs.
copenreeiee..ne//formmm
Yuya Yoshikawa, Ryotaro Shimizu, Takahiro Kawashima, and Yuki Saito. Transferring visual ex- plainability of self-explaining models through task arithmetic. arXiv preprint arXiv:2507.043880 2025.
Tianhe Yu. Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learming. In Advances in Neural Information Processing Systems, volume 33, 2020. URL https: http::///ppoccec........oo 3fe78a8aca
neurips. hcc/pappe/2020//EE00 .df.
fdd F3
Muhammad Bilal Zafar, Isabel Valera, Manuel Gomez Rodriguez, and Krishna P. Gummadi. Fair- ness beyond disparate treatment & disparate impact: Learning classification without disparate mistreatment. In Proceedings of the 26th International World Wide Web Conference, 2017.
Frederic Z. Zhang, Paul Albert, Cristian Rodriguez-Opazo, Anton den Hengel, and Ehsan Ab- basnejad. Knowledge composition using task vectors with learned anisotropic scaling. In Ad- vances Neural Information Processing Systems, 2024. doi: 10.48550//xx...247...2222 https: http://axxvv.v/../ wttg//bs////t/ 02880 Accepted to NeurIPS 2024.
URL
Zhifei Zhang, Yang Song, and Hairong Qi. Utkface dataset. https: /ts.aasuaggggii..ao github. htt.tioubooo/o.. UTKFace/, 2017.
15

---

Published as a conference paper at ICLR 2026
APPENDIX
A EXTENDED RELATED WORKS
Task arithmetic: efficiency and interpretability. Task vectors offer computationally efficient framework for editing and analyzing model behavior. Once a task vector is computed-nament the weight difference between base model and its fine-tuned variant (Ilharco et al., 2023; Zhang et al., 2024; Yoshida et al., 2025b) additional training data or retraining is required to transfer remove task-specific capabilities. By treating each fine-tuning update direction in weight space, practitioners can combine or negate these updates through simple addition or subtraction (Ilharco al., 2023). This modularity not only reduces computational overhead but also enhances interpretability by isolating the contribution of each task. Beyond modularity, task arithmetic can reveal valuable information about how and where model adapts new tasks. Li al. (2024) show a near-linear relationship between data size and the norm of task vector, suggesting that over- represented tasks can dominate weight space shifts in multi-task settings. addition, the orientation of task vectors can indicate synergies or conflicts among tasks (Li et al., 2025), and decomposing these vectors by layer can pinpoint which parts of the model are most affected (Zhang et al., 2024; Gargiulo et al., 2025). Hence, task vectors offer a promising lens for diagnosing training dynamics and identifying potential biases.
Vision-Language Models and Demographic Bias Audits. Audits of vision-language systems consistently report demographic biases zero-shot classification, retrieval, and captioning, along with mixed effectiveness of post-hoc debiasing methods (Ali al., 2023). Large studies of CLIP-style models show that scaling data or parameters does not reliably reduce these disparities, which vary with dataset composition and training regimes (Sahili al., 2025). Even current vi- sion-language models exhibit measurable group-level gaps under standard prompting (Wu et al., 2025).
B FAIRNESS METRICS
B.1 DEMOGRAPHIC PARITY DIFFERENCE (DPD) (AGARWAT. ET AL., 2018; 2019)
DPD measures how varied the model's rate of positive predictions across attributes. This metric calculated follows:
MoeD
= A= -r()- P-U0 1
where A is the sensitive attributes, f(X) is the prediction from the models, and X is the feature vector. The larger the DPD, the greater the difference in prediction outcomes across attributes. indicating greater unfairness in the model predictions.
B.2 EQUALIZED ODDS DIFFERENCE (EOD) (DING ET AL., 2024)
EOD is a metric that measures whether the model exhibits similar predictive performance in terms of true and false positives, regardless of the attribute.
Meod max (MTp, Mrp)
(3)
Here, letting denote the true label, Mrp and MFP are defined as follows:
MIP Pr[f(() = Y 4= -- -r((()))
0/.
Mp 1 1t-p://--110--AA---PPP00--------
16

---

Published as a conference paper at ICLR 2026
B.3 ACCURACY PARITY
Accuracy parity refers to the expectation that a classifier achieves comparable accuracy across differ- sensitive attribute groups. Formally, accuracy parity is satisfied when the probability of correct classification is equal across groups, i.e.,
B(Y =
Y 1
This notion of fairness ensures that all subgroups receive equally reliable predictions, and is partic- ularly relevant in applications where consistent model performance across demographics is critical. Unlike statistical parity or equal opportunity, accuracy parity focuses equal overall correctness rather than specific error types or outcome rates (Quan al., 2023).
We observed high degree of accuracy parity in both gender and race settings, as the accuracy differences between subgroups are negligible, indicating that the model performs consistently across all groups
C UPPER BOUNDS ON DPD AND EOD FOR OPTIMAL TASK-VECTOR SCALING
C.1 NOTATION AND ASSUMPTIONS
A1 Smooth and calibrated scores. Soft scores (2) are L-Lipschitz in the parameters, i.e., [po(x) - Per (x) s 0 0'112 for all x.
Hard predictions are obtained by thresholding group-agnostic score,
fo(x) (x) 2 t).
Moreover, soft scores and hard predictions agree in expectation at the group level: for every group 9.
EE (X) A= Pr(fo(X) = 1|A=g)
A2 Task vectors. For each group (I,..., G), A0g 08² learning rate and schedule.
Oo is obtained with the same
A3 Scaling coefficients. Coefficients obey Eg \g G.
A4 Balanced data assumption. The joint distribution satisfies D = U, D, where all Dy share the same conditional distribution except for the sensitive attribute label.
A5 Local uniform margin condition near the threshold. For each class label y {0, 1), there exists a constant Bg > 0 and a neighborhood. N (() of parameter space such that, for all E (0),, all groups g, and all sufficiently small 0 < 7 S Yo.
Pr (lpo(X) - 4 S Y = y, A = g) < B,7, - Pr - In words: around the decision threshold, the class-conditional score densities of both 0 and 0 are uniformly bounded by B, in neighborhood of 0. This is the standard "low density around the decision boundary" or "margin" assumption, here required to hold uniformly in local neighborhood.
A g) Byy-
The merged model is
0(2) Op Σn A0g
By [A1], the soft scores and hard predictions have matching expectations, so DPD computed from Pe coincides with DPD based on the classifier fo in Appendix B. In particular,
DPD(#) ED, [po] F., [Po]
17

---

Published as a conference paper at ICLR 2026
C.2 TASK ADDITION AND WEIGHTED ERM
Lemma (First-order link). Let E(0;x) be the training loss. For any non-negative {
0(2) R arg min Σ Xg REWD, [e(00;
Vol(On; x (0 0n)]-
That is, task addition gives the first-order solution of group weigh- ERM.
Proof. Insert the linear Taylor expansion of & at Og and minimise the resulting quadratic form; the solution is exactly ((A).
Implication. Hence, deviations 1>g - alter the group weights and therefore directly pushes DPD upward, as made explicit in Proposition I below.
C.3 DPD UPPER BOUND
Proposition (DPD bound), Under Assumptions Al-A4.
DPD(O(A)) $ 2L Σ D4g 11 A8112.
Proof. Define 0 = 00+ & Σ A8g. Assumption A4 gives DPD(#) 0. Define h(*) == Po(P) (x)
Pa(x). Then DPD(O(A)) = |E, [h] - ED, [k]]. Triangle and Jensen incquality yield < 2L [(0(A)
0 21... Finally, ((A) E, (Ag 1)20, and the triangle inequality give the stated bound.
C.4 POINTWISE SYMMETRIC BOUND VIA A RAMP FUNCTION
Lemma 2.
(1po(x) 2 1p (x) 2 1
< 110 0/12 1(lp(()) - 7) 11 S ST).
Proof. Define the ramp function Φe, R + [0, by
u SSt-%
(t - 7)
Oen (u)
t t-ク < <ttt
(5)
2
This function is -Lisschpt and satisfies the following inequalities: [1fu 2 t (2n 5 I[ S 7) For arbitrary 0, 0", applying the triangle inequality gives
(p) t} 1 (x) 2 t + +
(6)
where
Ti (p) 2 t] - (po
(7)
T2 (po()) or, (po (x))
(8)
T3 Oor, (por(x)) 1 \p (x) 2 t||-
(9)
By the ramp construction,
Ti < 1 (1) + s rt, By the Lipschitzness of Φi,7 and the soft scores,
Ts $ 11pp (z) - - 5})
(10)
I 100 012-
T2 (po(z)) po (x:)
(11)
7
Thus obtain the stated equation.
18

---

Published as a conference paper at ICLR 2026
C.5 EOD UPPER BOUND
Proposition 2 (EOD bound). Under Assumptions A1-A5,
(O) & + +4/()) + (A),
where (X) is the upper bound of EOD:
U(x) 21.
Σ A2g 11122
Proof.
EOD(O) max max = v.A= 9l Pr(fo(X)= = v.A 07 sE{0.1) s.
(12)
= 11
Prifo = 1 = = -
y, y-A=9) 9)
(13)
Prifa = 1 - y. - 97
Pr
+ Pr(/a = 1 -v 96 P v. 97
Put
(0)
(7) (7) Pr(pps - S V =v g)
(14)
m ( (7) Pr((pa(X)- 7 y =v.A g).
(15)
By Lemma
= y, = g] Pr(fo 1 | = y y 91 s - 011 + =m/ (7) + m
(0)
(7).
8.9
Now take the maximum over g' and E {0,1). 1}. By the definition of EOD, we
2T
EOD(O) < (0
||0 0|2 +
(ml (v) + + (7))
(16)
?
VE (0,1)
From assumption A5,
(m) (1) + m (x)) - 2(Bo + Bih.
(17)
Substituting this into equation 16 yields
2L
EOD(O) < (OD(O) + 0 0 + 4(Bo + B1)y.
(18)
7
Then minimize the upper bound over γ 0. By the fithhetic-geomettie mean inequality,
F(y) = 2L ||0 0|| + 4(B + B1); is minimized when 2 B,) ||0 0112:
Fin) = 4 2L + Bi) 10 0/2-
(19)
19

---

speech Published as a conference paper at ICLR 2026
Then, the triangle inequality gives the stated bound:
0)D & BOD + 2L + B1) 10 - 8
(20)
= EOD 0 + 4/ (Bo + BB) U()).
(21)
D EXPERIMENTAL DETAILS
D.I COMPUTATIONAL RESOURCES AND SOFTWARE ENVIRONMENT
Hardware and Software: All experiments presented in this study were performed using compu- tational resources equipped with two NVIDIA H100 GPUs. The experiments leveraged a GPU environment consisting of CUDA 12.1.0, cuDNN 9.0.0, and NCCL 2.20.5.
The experiments were conducted using Python 3.9.18, incorporating several essential Python li- braries specifically optimized for deep learning tasks. The primary libraries included PyTorch (ver- sion 2.6.0), transformers (version 4.49.0), tokenizers (version 0.21.1) I), DeepSpeed (version 0.16.4) and Accelerate (version 1.5.2).
The training experiments utilized the DeepSpeed framework with the following key configurations: a gradient accumulation step of 4, optimizer offloaded to the CPU, zero redundancy optimizer at stage 2 (ZeRO-2), and mixed precision training employing FP16 and BF16 for enhanced perfor- mance and memory efficiency. All experiments were conducted with a total computational cost of approximately 30 GPU-hours.
Protocol: We fine-tuned models based on the Llama-7B (Touvron et al., 2023) architecture obtained via HuggingFace repositories. Each model was trained for 4 epochs, employing a cosine learning rate scheduler with a learning rate of x 10-5 a warm-up ratio of 0.01, and a weight decay of 0.001. Training utilized a per-device batch size of 2, with effective batch size of 16 achieved through gradient accumulation. Reproducibility was ensured by setting random seed of 13, 14, 15 across all experiments.
For Qwen2.5 experiments, models were trained for epochs using a learning rate of 2x100 a batch size of 16, and sample fraction of 25% of the Civil Comments dataset. DistilBERT experiments utilized 2 epochs with a learning rate of 1 x 10-5 a batch size of 16, and the full dataset (100% sample fraction). Both architectures employed weight decay of 0.01 and evaluation/save strategies set to" "ppoch"" with early stopping enabled.
For Low-Rank Adaptation (LoRA) experiments were conducted with a rank (lora_r) of 8, scaling factor (lora_alpha) of 16, and no dropout.
D.2 DATASET
We use the Berkeley D-Lab hatespeech detection dataset (Kennedy et al., 2020b) for our experi- ments.
The dataset is divided into subgroups based on the following attributes: Race or Ethnicity, Religion, National Origin or Citizenship Status, Gender Identity, Sexual Orientation, Age, and Disability Status. In our study, we use some of these subgroups to evaluate fairness.
Following Das al. (2024), we binarize the hate speech score associated with each review using threshold 0.5 to determine whether the review constitutes hate speech. When multiple annotations exist for the same instance, we obtain one human annotation to avoid duplication.
E ADDITIONAL RESULTS
Here, we present results focusing on diverse subgroups, which we could not include in the main paper due to space constraints.
20

---

Published as a conference paper at ICLR 2026
E.1 COMPARISON OF FFT, LORA, AND TASK ARITHMETIC
3230
Accuracy
3.8
119
933
375
Additon
Ld64
fedt Additon
Lonk
bus Aatiion
Figure 5: Boxplots of group-wise accuracy, demographic parity difference (DPD), and equalized odds difference (EOD) for FFT, LoRA, and task addition with coefficient (A 0.8) evaluated on the gender subset of the data. Higher accuracy is desirable, whereas lower DPD and EOD values indicate improved fairness. Boxplots show medians, interquartile ranges, and variability (with standard eπror across three seeds). While accuracy is similar across methods, Task Addition generally yields lower DPD and EOD medians than FFT and LoRA, suggesting better balance between performance and fairness, though overlapping distributions imply these differences are not uniformly significant.
412
Accuracy
3.85
134
0.2
334
Test Additton
Text Addictin
Yost Asdotton
Figure 6: Boxplots of group-wise accuracy, demographic parity difference (DPD), and equalized odds difference (EOD) for -FF,, LoRA, and Task Addition with optimal coefficient (A 0.5) evallaaeed on the race subset of the data. Higher accuracy is desirable, whereas lower DPD and EOD values indicate improved fairness. Boxplots show medians, interquartile ranges, and variability (with standard error across three seeds).
Figure 7 illustrates the overall performance of FFT, LoRA, and task arithmetic as the scaling for task arithmetic vary from 0.0 to 1.0. Trends observed reinforeed results the gender subset Figure 2. Overall, À provides practical mechanism for balancing accuracy and fairesss objectives, and similarly there is peak fairness).
0.2 for highest accuracy, and higher DPD and BOD (less
0.8
0.035
Accuracy
0.7
DPD
0.030
EOD
0.08
Full
0.05
LoRA
0.025
Task Addition
0.04
Figure 7: On a race-focused subset, we vary task arithmetic's coefficient À and compare against FFT (purple dashed) and LoRA (orange dashed). The plots show group-wise accuracy (left), de- mographic parity difference (DPD, center), and equalized odds difference (EOD, right). Higher accuracy better, while lower DPD and EOD indicate improved fairness. As À changes, task arith- metic remains competitive in accuracy and can reduce faimess gaps relative to the baselines.
21

---

Published as a conference paper at ICLR 2026
Accuracy
DPD
EOD
FF7
FPT
3.50
-0300
Jm 02
Am 0.2
0:0
SAI
0.51
0.32
Am 0.2
1:33
SUM
5.00
0.04
5.000
Am 6
300
3122
0.6
6:00
0.03
0.6
GGR
12
6.000
2.84
Am1
3.2
1.0
RU4
Aztion
Lasgect
Nartie demertCan
Aesiall Bosioon
Wiide
Actten
Nostve Amneican
Medsis Cestoo
Puodic stander
Wribe
Oboer
Actan
Lacket
Netion Ammeticnn
Metico Kaudero
Peofic odeecten
Sende
-0.04
Figure 8: The task vector corresponding to Asian was added to the FFT model on the race data subset. Heatmap of Accuracy (left), DPD (center), and EOD (right) under the baseline (FFT) and increasing À values (0.2 to 1.0). Darker cells indicate higher values in each metric's s scale; for DPD/EOD, lower is better.
E.2 SEBGROUP-SPECCIEE TASK ADDITION TO FFT
We include additional heatmaps that visualize subgroup-wise performance across FFT and varying scaling coefficients for the FFT model injected with worst-performing subgroup. These supple- mentary plots, which follow the same setup described earlier, are consistent with the trends observed in Figures 3a-3b.
In both gender and race subgroup experiments, increasing the scaling coefficient À generally leads to improved macro-averaged accuracy. However, its impact on fairness metrics-DPD and EOD-is less predictable and varies across subgroups. For instance, some subgroups benefit from improved fairness as their corresponding task vectors are added, while others experience increased disparity, even if accuracy remains stable or improves.
This nuanced behavior reflects a broader pattern: gains in performance for certain subgroups can sometimes come at the expense of fairness for others. Injecting task vectors from worst porst-performig subgroups does not consistently reduce disparities and, in some cases, can amplify them.
0.07
Accuracy
DPD
0.05
EOD
0.14
FTT
0.044
8:057
FFT
1:44
6009
6015
0314
0009
0.03
FT7
1//4
0.12
8/00
6435
2004
0405
0.12
3.58
4
VAL
5,00
0.4
A312
KCN
A00
0.04
An 5.4
KNN
630
3.00
6250
12.7
COZY
0209
515
25.4
0.02
DEV
5006
0.
3.8
0.06
1m 1.0
U8i4 Heddle Cantorss
J 3,2
D00
00:0
0012
0.02
Ja 2.0
6.00
BUW
6.000 0414
Black
Madive Amettraas
Laiine
Pactfic Staander
Rtack
Madive Ammost
secidi Contor
Pactffrr salander
Cothur
0.02
Bludi
Maxtre Anserrouu
Landon
sodate Gestion
Pactfir oodender
Wileo
Other
00044
Figure 9: The task vector corresponding to Native American was added to the FFT model on the race data subset. Heatmap of Accuracy (left). DPD (center), and EOD (right) under the baseline (FFT) and increasing À values (0.2 to 1.0). Darker cells indicate higher values in each metric's scale; for DPD/EOD, lower is better.
Figures 10-11 and 4a-4b present additional results for the Full and Worst configuration, in which task vectors from the worst-pert corst-performing subgroups (Native American, Asian, Men, and Women) are added to the FFT model. These plots show macro-averaged accuracy, DPD, and EOD function of the scaling coefficient X.
Across these figures, observe mixed effects: while accuracy generally remains stable im- proves slightly, fairness outcomes vary by subgroup. In Figure 11, DPD and EOD worsen despite minimal accuracy changes. Meanwhile, Figure 4b reveals stable performance with minor fairness
22

---

Published as a conference paper at ICLR 2026
0.845
0.035
Accuracy
0.B40
DPD
10D
0.07
0.030
Full
0.830
0.06
LoRA
Woll-Worst
0.825
Figure 10: Effect of adding the Asian task vector to the FFT model on the race subset. Accuracy keeps competitive with increasing X, and both DPD and EOD decrease consistently.
improvements, though gains not consistent across metrics. These results further emphasize that task vector injection alone does not ensure universal faimess improvements and often introduces subgroup-specific trade-offs.
0.85
0.075
0.035
Accuracy
0.84
0.070
0.030
0.065
Full
0.83
0.025
0.060
Foll-Worst
LoRA
Figure 11: Results of injecting the Native American task vector into the FFT model. Accuracy shows minimal change across A, while DPD and EOD increase (worsen fairness).
F ADDITIONAL EXPERIMENTS ON CIVIL COMMENTS
Protocol uncertainty. Unless noted, we follow the LLaMA-2 setup (Section 4.2): SFT and LoRA (r=8) to obtain subgroup- specific models, compute task vectors w.r.t. the pretrained base, and merge with uniform scalar X. We sweep À on the validation split (maximize overall accuracy) and evaluate on the test split. Uncertainty is 95% stratified bootstrap over the test set (2,000 resamples, preserving group x label frequencies). When multiple seeds are used, we pool predictions before resampling, For accuracy, additionally report Wilson Cls when relevant.
At glance. On Civil Comments with DistilBERT (67M), task addition maintains accuracy within 1.0.6-..lpp of SFT/LoRA while reducing fairness gaps: for gender, DPD drops by s=41-54% and EOD by <34-47%; for race, DPD drops by x-11-8 and EOD by =58-73% (midpoint com- parisons). These patterns align with LLaMA-2 on the Berkeley D-Lab dataset (Table 5). As a complementary arshttrchie check, 2.50--2.500.5B gender exhibits the same qualitative X-controlled trade-off, improving substantially over LoRA with competitive accuracy.
F.1 CIVIL COMMENTS GENDER
Notes. Relative to LoRA, Qwen-2.5 2.5-0.5B task addition halves DPD/EOD (~54-56%% while re- gaining ~3.3pp accuracy; relative SFT, accuracy lower and fairness is mixed (DPD comparable; EOD higher). DistilBERT shows consistent reductions in DPD/EOD with S 1pp accuracy cost.
F.2 CIVIL COMMENTS RACE
Discussion. Together with LLaMA-2 on Berkeley D-Lab (Table 5), these experiments indicate that the A-controlled tainees-utility trade-off extends across architectures and datasets: task addition typically preserves accuracy within Ip while materially reducing worst-case DPD/EOD.
23

---

Published as a conference paper at ICLR 2026
Table Civil Comments (Gender). Headline metrics (Accuracy worst-case DPD worst-case EOD 4). Entries are 95% Cls from stratified bootstrap
Model/Method
Accuracy
Worst-DPD
Worst-EOD
DistilBERT SFT
0,9457-0.9476
0.0887-0.1100
0.6157-0.6433
DistilBERT LoRA
0.9447-0.9453
0.0735-0.0812
0.5024-0.5084
DistiBBERT
Task Addition
0.9395'
0.0454'
0.3358'
Qwen-2.5-0,5B
SFT
0.884-0.88 00886
0.093-0.119
0.060-0.084
Qwen-2.5-0.5B LoRA
0.774-0.790
0.210-0.251
0.232-0.362
Qwen-2.5-0.5B
Task Addition
0.810-08820
0.100-0.103
0.130-0.143
Point estimates.
Table 3: Civil Comments (Race). Headline metrics (Accuracy t. worst-case DPD L. worst-case
EOD 4). Models evaluated for this attribute are shown. CIs are 95% stratified bootstrap; indicates point estimates.
Model/Method
Accuracy
Worst-DPD
Worst-EOD
DistilBERT SFT
0.9467-0.9473
0.0987-0.0995
1.2568-0.3544
DistilBERT LoRA
0.9446-0.9453
0.1360-0. 1425
0.4649-0.4895
DistiBBER Task Addition
0.9362
0.0580
0.1289
G ADDITIONAL EXPERIMENTS ON UTKFACE (VIT-BASE/16)
Protocol & uncertainty. We follow the same task vector protocol as in the NLP experiments: for each sensitive attribute (race or gender), obtain subgroup-speciffic SFT and LoRA (r=8) models, compute task vectors relative to the ViT-Base/16 pretrained backbone, and merge them using a uniform scalar A. Images are preprocessed following standard VIT practice (resize to 224 patchify to 16 x 16, normalize using ImageNet statistics). The UTKFaae age label is binarized following prior work: ages below threshold form the "younger" class and ages above form the "older" class. We sweep À over the validation split (maximizing overall accuracy) and evaluate on the held-out test split. Uncertainty reflects 95% stratified bootstrap (2,000 resamples preserving group x label frequencies).
At a glance. Across both race and gender, task addition again provides a favorable fairness-utility compromise, closely mirroring the patterns observed in NLP.
For race, SFT achieves the highest accuracy (0.8463-0.8599) but also the largest gaps (worst-DPD 0.3374-0. 0.3881 worst-EOD 0.1815...2693). LoRA substantially reduces these disparities (DPD 0.2298-0. 02229--027789; EOD 0. (670-0.2238) but at the cost of sharp accuracy drop (0.6389-0.6572), Task addition recovers much this lost performance (0. 00.7227-07788. while keeping Worst-DPD/EOD close to LoRA and far below SFT (DPD 0.2390- 0.2924; EOD 0.1700-0.2337 producing a more balanced trade-off.
For gender, we see the same trend. SFT again yields the best accuracy (0.8466-0.8600) but rela- tively large gaps (DPD 0.2513-0.2872; EOD 14446-..1135)., LoRA lowers DPD slightly but wors- EOD and reduces accuracy substantially (0.6389-0.65)) 0..672).. Task addition strikes the strongest accuracy improves meaningfully over LoRA (0. (0.7777777384 while worst-DPD and worst- EOD are simultaneously reduced (DPD 0. 0.2080; EOD 0.0931-0.1379).
balance:
Overall, UTKFace confirms that task addition reproduces the same fairness fairness-utility utility behavior ob- served LLaMA-2 and Civil Comments conments-closing large portion of LoRA's accuracy gap while preserving most its fairness gains. This suggests the effeet is consistent across domains (vision text), model classes (VIT transformers), and sensitive attributes (race gender).
24

---

Published as a conference paper at ICLR 2026
Table 4: UTKFaae (Race). Binary age classification with ViT-Base/16. Accuracy 1, worst-case
DPD 4, worst-case EOD 4. Entries are 95% bootstrap Cls.
Method
Accuracy
Worst-DPD
Worst-EOD
SFT
0.8463-0.8599
0.3374-0.388
0.1815-0.2693
LoRA
0.6389-0.6572
0.2298-0.2789
0.1670-0,2238
Task Addition
0.7227-0.7384
0.2390-0.2924
0.1700-0.2337
Model
Race (95% CT)
Accuracy
Worst DPD
Worst FOD
LLaMA2-7B
SFT
0.7901-0.9039
0.0000-0.0345
0,000-000730
LoRA
0.7599-0.9143
0.0000-0.0459
0.0000-0, 1087
Task addition
0.7972-0.8724
0.0000-0.0265
0.0000-0.1308
DistilBERT
SFT
0.9467-0.944 0.9473
0.0987-0.099 0.0995
0.2568-0.35 00,544
LoRA
09946-0.9453
0.1360-0.144 1.12
1.4649-0.4899
Task addition
0.9362
0.0580
0.1289
Model
Gender (95% CI)
Accuracy
Worst DPD
Worst EOD
LLaMA2-7B
SFT
0.7914-0.8491
0.0621-0.1125
0.0000-0.177
LoRA
0.8031-0,882,
0.0535-0.0596
0.0105-0,0906
Task addition
0.8031-088823
0.0259-0.0943
0.0000-0.0858
DiRT
SFT
0.9457-0.9476
0.0887-0.1 11887-0.1101
0.6157-0.6433
LoRA
0.9447 0,9453
0.5024-0.55
Task addition
0.9395
0.0454
0.3358
Qwen 2.5-0.5B
SFT
0.884 0.886
0.093 0.119
0.060 0.084
Qwen- 2.5-0.5B
LoRA
0,774 0,790
0.210-0.251
0,232-0.362
Qwen-2.5-0.5B
Task addition
0,810-0.820
0.100-0.103
0.130-0.143
Table 95% confidence intervals. Models evaluated for each attribute are shown: LLaMA2-7B on Berkeley D-Lab; DistilBERT and Qwen-2.5-0.5B on Civil Comments (Qwen-2.5 for gender). Task addition maintains accuracy while showing competitive or improved fairness compared to SFT and LoRA.
H
USE OF LARGE LANGUAGE MODELS (LLMs)
Scope of assistance. For polishing grammar, wording, concision, and transitions in the abstract, introduction, and discussion. Light edits on figure/table captions and section headings. And style normalization, enforcing consistent terminology and tense across sections. No ideas, claims, analy- ses, datasets, model architectures, experiments, or results originated from an LLM.
Models and interface. Edits were produced with state-of-the-art LLMs (e.g., ChaGGPT/GPT-clas models) via standard chat interface. To preserve anonymity, no identifying information (author names, affiliations, URLs) included in prompts. For data privacy, proprictary data, code, or non-public results were provided. We avoided uploading full drafts and removed any metadata that could compromise double-blind review.
Prompts and examples. Typical prompts included: "Please copyedit the following paragraph for clarity and brevity without changing technical meaning." and "Standardize terminology (task vectors, task arithmetic) and flag any ambiguous phrasing." The models were instructed not to add facts alter technical content.
25


Reviews:
[{"review_id": "wPXYD16vYI", "reviewer": "Reviewer_8Kur", "summary": "This paper presents the first systematic study of the fairness implications of task arithmetic—a parameter-efficient model editing technique that applies task vectors (parameter differences between fine-tuned and base models) via arithmetic operations. The authors compare task arithmetic against Full Fine-Tuning (FFT) and Low-Rank Adaptation (LoRA) across multiple models and datasets, using group fairness metrics such as Demographic Parity Difference (DPD) and Equalized Odds Difference (EOD). Key contributions include: (1) demonstrating that task arithmetic can achieve competitive accuracy while reducing fairness disparities, especially when scaling coefficients $\\lambda$ are tuned; (2) showing that merging subgroup-specific task vectors enables targeted fairness control; and (3) providing a theoretical bound linking $\\lambda$ to fairness metrics. The work establishes task arithmetic as a fairness-aware and efficient alternative to existing adaptation methods.", "questions": "1. The discussion of the scalar coefficient $\\lambda$ appears limited, particularly when complex tasks require optimization across multiple coefficients.\n\n2. Please clearly define the scope of the research. If the study is focused solely on the fairness of binary text classification tasks, then using the broad research category of \"fairness\" seems overly expansive. If the claim is a general investigation into fairness, experimental results on other types of tasks need to be supplemented.\n\n3. What is the advantage of your method-edited 7B model, in terms of its overall performance (considering both accuracy and fairness), compared to a larger-scale base model that relies solely on well-crafted prompts for zero-shot/few-shot inference? If a significant advantage cannot be demonstrated, the practical significance of the proposed method would be considerably diminished.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The paper presents the first systematic comparative analysis of fairness in task arithmetic, filling a critical gap in understanding the societal implications of this efficient model editing paradigm.\n\n2.The scaling coefficient technique is demonstrated to be theoretically grounded and empirically viable.", "weaknesses": "1. The core methodological approach relies on manually tuning a unified scalar coefficient, to balance accuracy and fairness. However, for complex real-world scenarios involving multidimensional fairness constraints—such as simultaneous optimization across gender, race, and age—this single-scalar control mechanism appears overly simplistic and lacks scalability. Manually identifying the optimal $\\lambda$ configuration for every possible subgroup combination is inefficient and impractical.\n\n2. The proposed method is presented as a general approach for improving fairness. However, its empirical validation is confined exclusively to binary classification tasks. The scope of algorithmic fairness extends far beyond this, encompassing more complex tasks such as text generation, dialogue, and translation. The present experimental design does not demonstrate the method's effectiveness for these broader and often more relevant fairness challenges.\n\n3. Performance validation was conducted only on models with a maximum of 7B parameters. Given the significant differences in emergent capabilities observed across model scales, these experiments are insufficient to establish the applicability and effectiveness of the proposed method for larger-scale models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eoeK8pz09q", "reviewer": "Reviewer_cnud", "summary": "This paper presents the systematic investigation of fairness in task arithmetic, offering a comprehensive comparison between task-vector editing, full fine-tuning (FFT), and Low-Rank Adaptation (LoRA), while further examining whether the integration of subgroup-specific task vectors into an FFT model can enhance fairness control. By evaluating multiple language models and datasets through standard group fairness metrics, Demographic Parity Difference (DPD) and Equalized Odds Difference (EOD), the study demonstrates that adjusting task-vector scaling coefficients can substantially improve fairness outcomes without compromising predictive accuracy. Furthermore, the merging of task vectors derived from underrepresented subgroups enables targeted fairness adjustments with minimal performance degradation. The authors also derive a theoretical upper bound linking task-vector scaling to demographic parity difference, thereby providing a principled explanation for the observed fairness–accuracy trade-offs. Collectively, this analysis positions task arithmetic as a cost-efficient, interpretable, and fairness-aware alternative to existing model adaptation techniques.", "questions": "1. How well would the proposed approach generalize beyond binary classification tasks such as hate speech and toxicity detection?\n2. Why does the theoretical analysis focus solely on DPD while omitting a formal justification for EOD, which is equally emphasized in the experiments?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1.\tIt presents a comprehensive evaluation comparing full fine-tuning (FFT), Low-Rank Adaptation (LoRA), task-vector editing, and a hybrid approach that injects task vectors into FFT, systematically analyzing their effects on fairness metrics and predictive performance.\n2.\tIt demonstrates that fairness can be achieved through task-vector scaling, showing that adjusting scaling coefficients effectively improves fairness while maintaining model accuracy.\n3.\tIt integrates task vectors from underrepresented subgroups, facilitating targeted fairness adjustments while maintaining minimal degradation in model performance.\n4.\tIt provides an analytical  upper bound which links task-vector scaling to demographic parity difference, offering a principled explanation for the trade-off between fairness and accuracy.", "weaknesses": "1.  The experimental scope is limited to binary classification on two related tasks (hate speech/toxicity detection) with specific demographic annotations, raising questions about generalizability to other NLP tasks.\n2. The theoretical bound in Proposition 1 only addresses DPD, leaving EOD, which is equally emphasized empirically, without theoretical grounding.\n3. The related work section does not discuss several recent and relevant fairness studies.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IxHuG9uIZq", "reviewer": "Reviewer_RrTq", "summary": "This paper investigates the fairness implications of task arithmetic–based model editing, where task vectors are manipulated through simple arithmetic operations as a lightweight alternative to full fine-tuning. The authors benchmark task vectors against Full Fine-Tuning (FFT) and LoRA across multiple language models and datasets, using standard group fairness metrics such as demographic parity and equalized odds. Experimental results show that task vectors can maintain competitive accuracy while reducing group disparities, and combining subgroup-specific task vectors enables controllable fairness adjustments. The paper further provides a theoretical bound linking task-vector scaling to fairness metrics. Overall, the work positions task arithmetic as a cost-efficient yet fairness-aware model adaptation strategy.", "questions": "See weaknesses", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. Analyze the impact of task vector on fairness is interesting and seems novel.\n2. Well-structured paper and easy to understand", "weaknesses": "1. The model used in this paper (Llama2-7b, DistilBERT, Qwen2.5-0.5B) seems out of date\n2. The informal theory is a little hard to understand. \n3. The observation in 5.3 and 5.4 seems to be useful, but we need a more general or interesting conclusion in 5.3, which is a section called empirical results overview.\n4. As a benchmark, it should be more comprehensive, such as include more baseline, dataset, base model.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)