Fair Decision Utility in Human-AI Collaboration: Interpretable Confidence Adjustment for Humans with Cognitive Disparities
Authors: Jiashi Gao
Kexin Liu
Xinwei Guo
Junlei Zhou
Jiaxin Zhang
Xiangyu Zhao
Guanhua Chen
Xin Yao
Xuetao Wei
Venue: ICLR 2026 Poster (2026)
Keywords: Fairness
AI-assisted decision making
Primary area: alignment, fairness, safety, privacy, and societal considerations

Abstract: In AI-assisted decision-making, human decision-makers finalize decisions by taking into account both their human confidence and AI confidence regarding specific outcomes.  In practice, they often exhibit heterogeneous cognitive capacities, causing their confidence to deviate, sometimes significantly, from the actual label likelihood. We theoretically demonstrate that existing AI confidence adjustment objectives, such as *calibration* and *human-alignment*, are insufficient to ensure fair utility across groups of decision-makers with varying cognitive capacities. Such unfairness may raise concerns about social welfare and may erode human trust in AI systems.
To address this issue, we introduce a new concept in AI confidence adjustment: *inter-group-alignment*. By theoretically bounding the utility disparity between human decision-maker groups as a function of  *human-alignment* level and *inter-group-alignment* level, we establish an interpretable fairness-aware objective for AI confidence adjustment. Our analysis suggests that achieving utility fairness in AI-assisted decision-making requires both *human-alignment* and *inter-group-alignment*. Building on these objectives, we propose a multicalibration-based AI confidence adjustment approach tailored to scenarios involving human decision-makers with heterogeneous cognitive capacities. We further provide theoretical justification showing that our method constitutes a sufficient condition for achieving both *human-alignment* and *inter-group-alignment*.
We validate our theoretical findings through extensive experiments on four real-world tasks. The results demonstrate that AI confidence adjusted toward both *human-alignment* and *inter-group-alignment* significantly improves utility fairness across human decision-maker groups, without sacrificing overall utility.
*The implementation code is available at* https://github.com/WEILaboratory/AI-Ethics-Safety-PaperCode/tree/main/Fair_HAI (ICLR2026).

Published as a conference paper at ICLR 2026
FAIR DECISION UTILITY IN HUMAN-AI COLLABO- RATION: INTERPRETABLE CONFIDENCE ADJUSTMENT FOR HUMANS WITH COGNITIVE DISPARITIES
Jiashi Gao"
Kexin Liu'
Xinwei Guo
Junlei Zhou'
Jiaxin Zhang!
Xiangyu Zhao²
Guanhua Chen
Xin Y
Xuetao Weil*
1 Southern University of Science and Technology City University of Hong Kong
Lingnan University
ABSTRACT
In Al-assisted decision-making human decision-makes finalize decisions by tak- ing into account both their human confidence and Al confidence regarding spe- cific outcomes. In practice, they often exhibit heterogeneous cognitive capacities, causing their confidence to deviate, sometimes significantly, from the actual label likelihood. We theoretically demonstrate that existing Al confidence adjustment objectives, such calibration and human- malmanalgggmmett,..mmmmmm utility across groups of decision-makers with varying cognitive capacities. Such unfairness may raise concerns about social welfare and may erode human trust in AI systems. To address this issue, we introduce a new concept in Al confidence adjustment: inter-group-ce goo---raiuammeennn By theoretically bounding the utility disparity between human decision-man maker groups as a function of human-ali aaman--lignment level and gropp-rrou-lllnnmentt level, we establish an interpretable lairness-aware ob- jective for Al confidence adjustment. Our analysis suggests that achieving utility fairness in Al-assisted decision- maiiing requires both human-ale anlgamal and inter- group-alignment. Building on these objectives, we propose multicalibration- based Al confidence adjustment approach tailored to scenarios involving human decision-make with heterogencous cognitive capacities. We further provide theoretical justification showing that our method constitutes a sufficient condi- tion for achieving both human--alinmmet and inter-group-alinmment date our theoretical findings through extensive experiments on four real-world tasks. The results demonstrate that AI confidence adjusted toward both human- alignment and inter- enter-group-aligmmees significantly improves utility fairness across human decision-make groups, without sacrificing overall utility. The implemen- tation code is available at https://github.cmm wtapttthubboowWlleee...e WElLaboratory/ Al-Ethics- SathicsSSaftt... PPaperCode//ree/mmai//iiii HAI (ICLR2026)
insufficient to ensure fair
We vali-
INTRODUCTION
In recent years, artificial intelligence (AI) has been increasingly leveraged assist human in
decision-making across various domains. For example in typical binary classification tasks, AI
systems have been developed to support clinicians in medical diagnosis (Rajpurkar et al.) 2020
Wysocki et al. 2023), aid financial institutions in credit risk assessment (Bussmann et al. (021)
and assist legal professionals in bail or sentencing judgments (Dement Inglis] 2024] Grgić-Hlača
et al. 2019). However, AI systems trained on datasets inherently exhibit uncertainties and remain far from achieving accurate predictions approach under uncertainty considerations is to have Al systems provide confidence ranging from to 1 for their outputs, which enables human decision-maken to better interpret and utilize the Al's assistance in their decision-making processes (Bhatt et al. 2021 [Steyvers & Kumar 2024 Ma et al. 2023). Consequently, the Al-assisted decision-making process is structured as follows: human decision-maker integrate their own confidence on specific labels with Al's make final decisions.
every case (Prabhudesai et al., 2023); a recommended
"Corresponding author; weixt sustech.edu. eddu.en

---

Published as a conference paper at ICLR 2026
Ideally, human decision- maakrss place greater reliance AI when its confidence is higher and rely more on their own judgment when the Al's confidence is lower.
Existing research in Al-assisted Decision-making has primarily focused on enhancing final decision- making's 's utility (i.c., accuracy in classification tasks or prediction errors in regression tasks). Early studies suggested that AI confidence should be well-calibrated estimates of the probability that the predicted label matches the truth label (Pakdaman Naeini et al., 2015: Yin et al. 2019) Zhhng al. 2020). However, VVdrahalli et al.) (2022b) experimentally demonstrated that in certain sce- narios, explicitly uncalibrated Al advice led to substantially higher decision utility compared to well-calibrated advice above. Subsequently, Corvell Benz & Rodriguez (2023) provided a detailed theoretical analysis, demonstrating that rational decision becision-makers make optimal final decisions when Al confidence exhibits a natural alignment with human decision makers' confidence in their own predictions, referred to as human-aligeen Human-alignment.
Human decision-maken makers, shaped by different historical and social contexts, exhibit varying cogni- tive capacities on decision-making tasks, causing their confidence to deviate by different magni- tudes from the actual likelihood of the labels. We demonstrate that existing Al confidence cali- bration mechanisms fail to provide fair decision utility across diverse groups of human decision- makers, who are differentiated by their cognitive capacities. Such unfairness may undermine human Decision-makers trust and willingness engage with AI. Furthermore, it may exacerbate existing disparities in access to information and knowledge. For example, in Al-assisted medical diagnosis, unequal AI support might widen the diagnostic error gap between less experienced doctors (who typ- ically have lower cognitive capacities) and experts (who typically have higher cognitive capacities). Such Al-assisted decision-making system not only limits opportunities for skill improvement among less experienced professionals but also risks reinforcing the Matthew Effect (Merton 1968)-a so- cioeconomic phenomenon whereby disparities in access to resources (e.g., education, economics, or information) become self-reinforcing over time, leaving already disadvantaged groups even further marginalized. Therefore, in this work, we aim to mitigate the utility disparities in Al-assisted decision-making that arise from human decision-makee makers' heterogeneous cognitive capacities.
Our contributions. To the best of our knowledge, this is the first work to identify and address the utility faimess issue affecting human maaess in Al-assisted decision-making and we make the detailed contribution as follows: ! We identify a previously unexplored source of utility unfair- ness in human-Al AI collaboration, which arises from human decision-maker with heterogeneous cog- nitive capacities. We theoretically demonstrate that existing AI confidence adjustment objectives, including calibration and human-aligument, are insufficient to guarantee fair utility for such hetero- geneous human decision-makers. @ We introduce a novel Al confidence adjustment objective, inter- group-alignment, and derive theoretical upper bound on utility disparity that is determined by the degrees of both inter-group-ee aliup-aaiggment and how AI confidence should be configured to achieve fair decision utility. 3 We provide a theoretically grounded unification of the dual objectives, inter- inter-group-aee caoup-aligmment and human- aaiman-alggmmet, under a multicalibration- based AI confidence adjustment approach. This approach offers a practical solution for achieving both utility fairness and optimal utility among human pacision-makens with heteroge- neous cognitive capacities in Al-assisted decision-making on four real-world Al Al-assisted decision-making tasks involving actual human decision-makers. The results validate our theoretical findings and demonstrate that adjusting Al confidence toward both groop-aalggnnnnee and human-ale aaimann-llgmmnttt substantially improves utility faimess across groups of human decision- makers, without compromising overall utility.
This provides actionable insight into
We conduct extensive experiments
PRELIMINARY
We focus on binary decision-makinn scenarios to investigate the presence of unfair utility among hu- man decision-male maker groups with diverse cognitive capacities in existing AI confidence calibrations and to explore potential solutions. Such scenarios are common in real-world applications, including loan approvals, disease diagnoses, and job assignments. We define the attributes used to group the human
makers as S E S. The variable S can represent any human characteristic that may contribute disparities in cognitive capacities for given task, including direct features (e.g., level of education) or indirect features (c.g., gender, race), which may have historically resulted in unequal access to knowledge resources. Let In : X [0, 1] denote the human decision-makert confidence
2

---

Published as a conference paper at ICLR 2026
function for positive outcomes. Initially, the human decision-maker maker observes sample with features
I E X' and assigns a confidence h (x) E H. Subsequently, the Al model (i.e., a classifier) provides its confidence value fz s), where represents specifie value of the variable
S, indicating the group to which the human decision-maker belongs. Here, fA 2 + [0, 1] de- notes the Al's confidence function for positive outcomes, with 2 =[X, H, S. Finally, the human decision-maker makes a binary decision '"" based on the probability P(T = 1) T Th,, a) E {0, 1}. Specifically, T if P(T = 1) 2 0.5, and '"' = otherwise. Here, π II(H, denotes the decision-making policy. Upon making a decision, the under the truth label Y E {0. 1].
maker receives utility u(T, Y) E IR
Utility. A natural setting for a utility function, consistent with most real-world scenarios, assigns higher utility to cases where the final decision, aligns with the ground truth label, Y. compared SioloiiegCCovlloooo
to cases where T and Y diverge. Following Corvelo Benz & Rodriguez (2023), the utility function (T, Y) satisfies the properties as follows:
u11, 1) u11, (11,) 1) > 1),
u(1,0), u(0, (0,
(1)
Cognitive disparity. Assume there are [S] distinct human decision- makers groups categorized by a sensitive attribute E S. To enable statistical quantification, we measure the cognition disparity (CD) between the i-th and j-th human decision-makers groups by calculating the likelihood dispar- ities of the truth labels between different groups, despite human decision-maker maakess having identical confidence h (which reflects the different cognitive capacities to estimate the likelihood of the truth label correctly) as follows,
-Dii)))
1|2 Zn...) P(Y 1/2 E Zn,sy),
where 2,,,, ((x, h, 8) (x) h, 8;} represents the subset of decisions made by human decision-makers in group 9, who have a confidence level of h. Human decision-maken are referred as cognitive-hettrogeeeeos if Ji E {1,..., [8]]. CD(i.) # 0.
Utility disparity. We aim to achieve equal utility across different human decision-maker groups despite disparities in their cognitive capacities. Specifically, for any given Al-provided confidence and human confidence h. which result in human mackiss having identical final decision probabilities P(T = 1), the utility disparity (UD) across the human-decision maker groups is ex- pected to approach zero as follows:
UD = ( Σ E. (u(T, E Za.x.- - E. [u[T, Y)aa E Enn,]] 1)//)//
SS)]
- 0.
(3)
(.JE Juj
Specifically, in scenarios involving binary sensitive attributes, the Eq. (3 can be simplified to:
UD E., [u(T, Y)1/a(=) a, 2 E, [u(T, Y) = a, E Za.ol 0. (4)
Monotone. When the human decision-maker act rationally, increasing human decision-maker makers confidence h and AI confidence raises the probability of human decision- makers making positive final decisions. Regarding this assumption, we provide detailed discussion of its rationale and scope Appendix[A.]] verify that Assumption 2. holds broadly real-world Al-assisted decision- making contexts, and validate the robustness of the subsequent method under certain violations of the monotonicity assumption.
Assumption 2.1. (Monotone Decision Policy in Al-Assisted Decision Making) Assume that human decision-makers are rational. The decision policy is monotone, meaning that for any Al confidence ai and 02, and any human decision-maken confidence in and 12, if a1 S 02 and hi < h2, then,
P(T = a1) < P(T = 1[h2,
(5)
3 CAN FAIR UTILITY BE ACHIEVED IN AI-ASSISTED DECISION-MAKING?
3.1 LIMITATION OF By CALIIAIBRATION AND Q HAA-HHMAN--LIGNNEA
When the AI model produce confidence estimates that accurately represent the distribution of truth labels, it achieves perfect calibration Pakdaman Naeini et al. (2015); Yin et al. (2019): Zhang et al. (020). We adopt the statistical notion of ay calibbrato introduced by Hebert-Johnson et al. Hebert- Johnson et al. (2018), which transitions from approximate calibration to perfect calibration by ad- justing the hyperparameter o, from 1 to 0.
3

---

Published as a conference paper at ICLR 2026
Definition 3.1. (a, Calibbaaatti An Al system with confidence function fa 2 + [0, where
2 = {x, S) satisfies ag-cabbbrttion with respect to 2 if there exists 2 C with 3 2
(1 a:y) |2|, such that for any Al confidence
[0, 1]. it holds that:
P(Y = 1 fa(z) = a, E 2¹) al S Oty
(6)
The Definition 3.1 bounds the proportion of samples where the difference between the Al confi- dence and the positive label likelihood exceeds ag to be less than ay. When the AI confidence is perfectly calibrated (a'y - 0). it implies that, for the entire sample space Z. the Al confidence fA aligns exactly with the likelihood of the positive label. Based this definition, we demonstrate the limitation of calibration in achieving fair utility, as in Theorem 3.2
Theorem 3.2. (Utility disparity under calibration (Proof in Appendix A.3 Given an Al-assisted decision-makinn with utility function u(T, Y') in Eq. (7) and the human decision-maker with any monotone Al-assisted decision policy π E [H(H, A). Even when the Al confidence function f is
perfectly calibrated, the cognitive disparity in Eq. 2 is non-zero, then there exists Al confidence a and human confidence such that the utility disparity is also non-zero.
EET (u(T, ()) = 0, E 2n,i] - ET (u(T, ()Lx(()))))))) a, Zno]] #0.
argued that ay-calibration fails ensure optimal utility for monotone policies and proposed human- sriman-aligmment as new confidence alignment objective. They demonstrated that a perfectly human-aligned confidence function guarantees the existence of monotone policy π achieving optimal utility. Inspired by the superior performance of human- alignment, we further analyze its resulting utility disparity.
Corvelo Benz & Rodrigue7 2023)
Definition 3.3. (a), HHuman-alignmenn An AI system with confidence function fA 2 - [0, 1] where Z = =(X, H. S}, satisfies dy.-alignment with respect to human decision-maker confi- dence function JH X [0, 1] if, for any h H. there exists Z/ Zn with Z, {(x, h, s) E 21,fa
h} and 22 2 (1 22) |zl,, such that, for any 0 S S 02 S 1 and
S h1 S he S 1, holds that,
P(Y
f2(2) = 01 2%,,) - P( = = 1 fA())) 20 2 S ah.
(8)
For in and h2 satisfying the monotonicity condition ha < 12, the above definition bounds the violation of monotonicity in the positive label introduced by SA to at most o2/2 /2 over the sample spaces Zh, and Zhz- However, even if the Al confidence function fA is perfectly buman-aligned (an, + 0), the monotonic decision policy π still be suboptimal in terms of fair utility.
Theorem 3.4. (Utility disparity under human-alignment (Proof in Appendix nitive disparity in Eq. (2) is non-zero, there exist (infinitely many) Al-assisted decision-making processes with utility function u(T, Y) in Eq. | and the human decision-make with any monotone Al-assisted decision policy π E II(H, (HH,AA) A), such that even the Al confidence function f is perfectly aligned with the human's ss the Al-assisted decision-makinng still fails to achieve optimal utility fair- ness. Specifically.
A.4 When the cog-
Ex [T, Y)1Sa(=) 0, 2x,1] E. (T, (T,Y)) = =a, Zn.o|
(9)
E. Y)ff(() = = Zn,1l - E, [u(T, = =a 2n.ol
where,
x²- arg min =EII(H(.))
ES Y)JAaa) a, 2,,1] 1 E, (UTTYY)(A(3)) a, Znoo| (10)
We analyze that the cause of failure in fairness arises from human-alle -lligmmett without considering differences in the correctness of human decksion-makers' confidence due to heterogeneous cognitive capacities. This discrepancy results in differing levels of human- auman-alignment between groups. For groups with weaker alignment, this can lead to utility disadvantages.
is a common occurrence in practice for there to be non-zero cognitive disparity among human decision-makers. Consider, for instance, disease diagnosis scenario involving two groups of human
4

---

Published as a conference paper at ICLR 2026
decision-makers: experts (S 1) and general practitioners (S 0) working under Al assistance. Suppose both groups diagnose patients as having the disease with human decision-maker confi- dence level of h = 0.9. Due to differences in cognitive capacities, there may be a disparity in the true probability that the patients actually have the disease: P(Y 1|2 E 20.9,) > P(Y = 1|2 E 20,9,)). Based on Theorems 3.2 and 3.4 both calibrated and human-aligned Al confidence cannot guarantee utility fairness in such scenarios. In the following sections, we propose a novel alignment objective to address this issue.
3.2 INTER-GROUP WEE-GGRRRUUUUAAAAA ALIGNMENT AND UTILITY DISPARITY UPPER BOUND
Given the limitations of existing calibration methods in ensuring optimal fair utility, we introduce the core concept of inter-grouph anter-group-alignnn gropp-aliggamennt in Definition 3.5 Building on this foundation, we give an upper bound on utility disparity of any Al-assisted decision making process, as in Theorem 3.6
Definition 3.5. per--nerrgrou--liinnmee
An Al system with a confidence function SA 2 - [o, 1] where Z [X, H.S.. satisfies ag- wyrriterrgroup-alignmen
if, for any h E H. there ex- h and |2/1 2 (1-a,/2) 12 Let Zh,
ists Z/ C Zn with Zh {(x, h) [ (x)
{1x, h, s) 2 S s}. the AI confidence 0g inter-group-alignn alignment if,
P(Y = fA() = a, 2,,) - - 1 a, 2/,0) ag.
(1I)
given identical AI confidence and human decision-maker confidence h, hu- man decision- decision-makers makers will exhibit the same probability of making the final decision, P(T = 1). The definition of p--ntter-gruupaaiignmett constrains the distribution of positive label Y 1 to be statistically equal across different human decision-make, groups when 0g - 0. This alignment en- sures that human decision-makes within each group achieve statistically similar utilities for making correct decisions.
Based on Eq.
Theorem 3.6. (Utility disparity upper bound under 0g Og.-human ha-human--ligmment alignment and 0gg ig-inter-group alignment (Proof in Appendix A.. For a given Al-assisted recision-making process with a utility function u(T,Y Y) satisfying Eq. @ and the human with any monotone Al-assisted decision policy t E IHH, A), if the Al confulence Junction Sa on oug-humnn--e humann-alignneen and satisfies 0g ty--inte-gguwn group- alignment, then the utility disparity is bounded by,
E. [u(T, = = 2n,1] E. (T ())f(()) =a,2E Zn,ol
(12)
ahh (Ø (1- ) (3ar -a²))
(u(1,1) (001)-ww00,11 u(1, =())))) u
Theorem 3.6 provides tight upper bound on the utility disparity, which is constrained by both the human-aligument level oy and neee-group-aligmment level og. This offers valuable insights into the fairness-aware AI confidence alignment objectives, which seek to align the Al confidence as closely as possible with human decision-maker decision-makers from different groups, with identical confidence h, receive statistically similar pos- itive label distributions when provided with the same Al confidence a. Based on this, we will next outline a calibration framework to achieving the dual alignment objectives simultaneously.
while simultancously ensuring that human
4 AI CONFIDENCE MULTICALIBRATION FOR SIMULTANEOUS IMPROVEMENT OF UTILITY AND FAIRNESS
Corollary 4.1. For any Al-assisted decision-making with a utility function u(T, Y) in Eq. 1 and is an cun-umman-ee ea--humnn-aiigmment alignment, the upper bound of utility disparity across different human decision-maker groups is minimized when the decision function fA satisfies perfect aater-grup-aalgnmmen
The above corollary holds as (3a, - ag)) 2 for all ag E [0. 1]. Consequently, under any human- alignment level ans the utility disparity upper bound in Theorem 3.6 is minimized when Gg 0. We can further refine the conditions under which the Al-assisted decision-makin process provides both optimal utility and fair utility across heterogeneous human decision-maker groups follows:
5

---

Published as a conference paper at ICLR 2026
Corollary 4.2. For Al-assisted decision-maiee maaing processes with utility function u(T, Y) satisfying Eq. 77. fA achieves both perfectly human- aamman-lgggmmntttt and perfectly inter-group group-alignment, there exist monotone Al-assisted decision policy E II(H, A) that simultaneously attains optimal overall utility and fair utility among heterogeneous human decision-make groups.
Based on the utility disparity upper bound established in Theorem 3.6 In the following, we demon- through multical-
strate how to simultaneously achieve human-alignmeni and snee-group-aligee ibration, thereby ensuring that Al-assisted decision-makinn provides fair utility while guaranteeing optimal utility for all human decision-makers
Method 4.3. (Al confidence a-multicalibration accounting for cognitive disparities) Suppose hu- man decision-makers can be partitioned into N groups, with group membership values {s: N determined by human-sensitive attributes related to cognitive capacity. For each group 8ú, let the collection of subsets as C, {Zh... ) kew An AI confidence function fA is said to satisfy a- multicalibration accounting for cognitive disparitie if it satisfies a-calibration (as defined in Defini- tion 3.1 with respect to Zn.s, E C, for E NN.
We refer to Method 4.3 cognition-aware multicalibration, to distinguish it from the multicali- bration that does not account for cognitive disparities Corvelo Benz & Rodriguez (2023), defined as follows:
Method 4.4. (Al confidence a-multicalibration (ignoring cognitive disparities)) Let C {2x) heh be a collection of subsets over the domain Z. An AI confidence function fA Z - [0, 1] is said to
satisfy a-multicalibration it satisfies a-calibration (as defined in Definition 3.1 3.1 with respect to all 2,, EC
Theorem 4.5. Suppose fa(z) is 0z se/2-coonitionnawarre multicalibrated with respect to the collection C (Z,,. JNE.. Then, fA satisfy both a-human--lignmeen and a-inter- hintte-group-alignneee (see proof in Appendix A.6V.
To reduce the computational burden of adjusting confidence over continuous intervals, we imple- ment Method 4.3 Jusing X-discretization Hebert-Johnson et al. (2018); the full algorithm is presented Specifically, X-discretization partitions the confidence interval [0, of SA into [1/A] bins of width X, with centers at A { 3A
in Algorithm
2 ..... 1 }}. Confidence adjustment is then applied to the sample set within each bin. According to Hebert-Johnson et al. (2018), the resulting discretized confidence function can achieve (ā + X) J-multicalibratin. In light of Theorem 4.5) to ensure fA satisfies at least nhuman-alignmen and a-inter- cunter-group-aaae group aliggmmn,,,, suffices to ensure
A S a/2
EXPERIMENTS
5.1 SETTINGS
Dataset. We utilize a publicly available dataset for human-Al interactions across 4 tasks Vodrahalli et al.] (2022a). In each task, human decision-maker first provide their confidence (used to struct JT). After receiving Al advice (used to construct fa), participants update their final decision confidence (used
construct 7 TTh, a)). The tasks span different data modalities (visual, text, and tabular). ! In the Art (Image) task, participants determine the art period of a painting from two options. @ In the Cities (Image) task, participants are asked determine the originating city of an image from binary choice. 3 In the Sarcasm (Text) task, participants determine if Reddit text snippet contains sareasm. + In the Census (Tabular) task. participants assess whether an individ- ual earns at least $50,000 annually based their demographic information. In our experiments, human decision which includes individuals with at least a Master's degree, and Group S = 1, consists of those with a degree lower than a Master's. s. The data were preprocessed to exclude samples with missing information and confounding factors (see Appendix A.7.Ip. resulting in final dataset of 14, .999 Al-assisted decision Dacision-making records from 469 participants overall.
are categorized into two groups by their "education" levels: Group
O,
Baselines. ① No Adjust, where the original confidence values inherent in the dataset are retained without any modification; @ Cognition-unawart Multicalibration, where the Al confidence is adjusted using multicalibration ignoring cognitive disparities (see Method

---

Published as a conference paper at ICLR 2026
Algorithm 1: Cognition-Aware Al Confidence Multicalibration
Input: Calibration tolerance a, Al confidence hin width
Output: Calibrated Al confidence function fa
Define the attributes Si,- S., related to cognitive capacity to group human decision-makenn let
($1,..., 8n) E Si X ... x S. denote group membership
Construct C- (2
= (ar hr (x) h, = s}} for all E H. E S1 x S,
repeat
foreach Z E do for = to 11/\] do
Define 2% {z E n, fa(2) E [A[a] 2/2, A(a] 1/2))
if P(z E 2 < āλ P(z E 2h.s) then
continue
if [[fA(z) E 2
P(Y 11| E 2%,a) > ā then
Update fa(z) < f=(=) + (P(Y= = 2% - -[[f(=)) 2g,,l) for all z E 2%
until 2h., updated; for = to [1/A] do
foreach s E S1 x .... x Sn do
Let 2 = (= [== ht) 1 s= (a(=) E [A[a] 1/2, A4d x/2))
Set [=(= [=)a(=) E 2 for all 2
return /A
Hyperparameters. We configure the hyperparameters as follows: = 0.0001 and À 0.125, ensuring that the level of cognition-aware multicalibration is approximately 0.125. We report the impact of hyperparameters a and À on alignment in Appendix A.8.3] showing that smaller ā + values lead to improved human-aligament and titerggroup-aligmment
Decision policy function. Since the dataset only provides uncalibrated confidences, we evaluate Al-assisted decision performance after calibration by learning the decision policy π (h,,a) using multi-layer perceptron (MLP) classifier with one hidden layer of 20 nodes and ReLU activation.
Experimental setup. @ Alignment quantification: We evaluate how effectively Cognitiio--wware Multicalibration achieves both mimman-aligmmet and inter- inter- group-aligmmmm. @ Al-assisted deci- sion utility: We assess whether Cognition-awart aware Multicalibration improves both expected utility and utility fairness of the final decision policy. 3 Ablation study: We compare the expected util- ity and disparity of human-only decisions T (h) and Al-only decisions I (a) to understand how Cognition-aware Multicalibration contributes to fairness across diverse decision-maker groups. @Additional experiment: Due to page limitations, we provide additional experimental results in Appendix A.8 including quantitative ablation study results (Appendix II..II, group-level accuracy distributions under fairness considerations (Appendix(A.3) A.8.2), the impact of discretization parameters on calibration performance (Appendix A.8.3), generalization to multiple groups (Appendix A.8.4), computational complexity analysis (Appendix (A.8.5), and multi-class classification scenarios (Ap- pendix pe..6. further validating the effectiveness and robustness of the proposed method.
Evaluation metric. We use accuracy to evaluate the decision utility and, naturally, evaluate the fair utility to diverse human decision-maker groups by measuring accuracy disparities follows,
Disp E [ Y)|s
E [1(T Y) 0]
(13)
In terms of measuring human- human--ligumentt alignment, Corvelo Benz & Rodriguez (2023) proposed two discrete metrics: the expected alignment error (EAE) and the maximum alignment error (MAE). The dis- cretization details are in the Appendix A.7.2 The limitation of metrics based discretization lies in the finite number of discrete intervals, which may fail to capture alignment across the entire continu- ous confidence space accurately. However, significant variations in the metrics can reflect variations in alignment more accurately. In such cases, the influence of unmeasured alignment on the overall results is reduced.
EAE max (0, NV Eisosey (x 2 21) - P (y 2
(14)
MAE max max. 1p(Y=1|== 2!) P
(15)
7

---

Published as a conference paper at ICLR 2026
Table I: Alignment quantification results (Bold: best result; Black: mean; Blue: standard deviation). Lower values indicate better alignment for all metrics.
No Adjust
Method
4.4
Method
4.3
Exp.
EAB
MAB
EIAE
MIAE|
EAE
MAE
EIAE
MIAE
EAB
MAE
EIAE
MIAE
0.0010 0.1167 0.0525 0.4286 0.0009 0.0494 0.0345 0.2291 0.0025 0.0675 0.0209 0.1360
1
(6.5E-19) (1.5E-16) (1.28-16) 16.11 (6.9E-18) IE-17) (5.5E-17) (1.7-18) (9.7E-17) (2.7E-17) (2.5E-16)
0.0047 0.2239 0.1094 0.4085 0.0003
0.0179 0.0145 0.2609 0.0007 0.0493 0.0031 0.1132
2
(6.9E-18) (3.3E-16) (199-16) (7.5-16)),.58-19) (2.5E-17) (2.6E-17) (3,3E-16) (L.1E-19) (2.7E-17) (3.9E-18) (2.3E-16)
0.0008 0.0830 0,1180 0.5702 0.0006
0.0576 0.0784 0,5702 0.0013 0.0989 0.0063 0.0458
3
(4.3 (1.2E-16) (1.9E.16) (1.1F-1)
(6.2F.17) (6.9E-17) (L.IE. (1.SE.18) (2.4E-16) (1.6F.7) (7.6F.17)
0.0091 0.2985 0.0794 0.3601 0.0002 0.0195 0.0328 0.5275 0.0005 0.0410 0.0072 0.1062
4
(1.72-18) (4.4E-16) (1.25-16) (1.1--199) (2.7B-17) (3.4E-17) (1.3B-15) (3.2E-19) (4.2E-17) (7.88-18) (6.9E-17)
Table 2: T-tests on the performance of cognition unnaaaee multicalibration and cognition-awars multicalibration under 100 repeated experiments.
Utility
Utility Disparity
Exp.
p-value
p-value
3.018
0.003
9,486
0.000
2
0.345
0.731
15.484
0.000
3
-12.187
0.000
3.186
0.002
4
4.839
0.000
27.556
0.000
We develop two metrics for measuring tiver-rroup-aaggmmmnn the expected ainiaget error (EIAE) and the maximum inter- snver-group-daee aligommen error (MIAE).
EIAE 3.
Eds P 1 - -P (r-11 2 =
(16)
MIAE max {I((Y=1)=E2))
(
2% 1}
(17)
where Z% and Z contain samples from the groups S = 0 and S = 1, respectively, located in the (i,j)-th cell of the grid formed by the discretization human confidence and AI confidence.
5.2 RESULTS
The key takeaway from the experimental results presented in Table i the proposed Cognition- aware Multicalibration's effectiveness in maintaining competitive human-alignment while sig- nificantly improving inter-group-alignmene
In the Cities and Centus tasks, both Method 4.4 and Method 4.3 significantly reduce EAE and MAE compared to the No Adjust case, indicating better alignment with human decision-maker"s confidence. the Art and Sarcasm tasks, while Cognition-awalt -wware Multicalibration exhibits slightly higher EAE and MAE values, the differences re- main marginal with magnitudes of 10-3 and 10 respectively. More importantly, when evaluating inter-group aaligenn using EIAE and MIAE metrics, Cognition-aveen aware Multicalibrattio consis- tently achieves the best performance across all four tasks, substantially reducing both EIAE and MIAE compared to both the No Adjust baseline and Cognitiion-nawaal Multicalibration.
Figure and Table 2 yield three key takeaways: @ Cognittionunaware Multicalibration fails to improve fairness and sometimes worsens it, as in task where utility disparity exceeds the No Ad- just baseline; @ Both calibration methods consistently improve decision utility over No Adjust across all tasks, with T-tests showing comparable performance between them; @ Cognition-awwaee Multicalibration consistently achieves the lowest utility disparity across all tasks, substantially reducing group disparities compared to both baselines. Using an MLP-basee decision model, report the distribution of final decisions over 100 trials with random seeds (0- 99). The No Adjust case shows variance, its decisions are fixed by the dataset.
8

---

Published as a conference paper at ICLR 2026
Al-assisted decision making PV7 1) mth,a)
Al-assisted decision making P(T m1h,a))
0,050
0.82
Utility Dispartty
0.025
Utiiky
0.000
0.78
No Adjsst
-0.050
Cognition award Nuttoaliration
0.74
2
3
Experiment
Experiment
Figure Statistics of utility and utility disparity over 100 experiments, where the final decision P(T = 1) x x(h, a) is made by human with Al assistance. The Al confidence is adjusted by No Adjust, Cognition-unenn unaware Multicalibration and Cognition- Multicalibration, respectively. Higher utility and lower utility disparity indicate better performance.
Human-ony deciaion PV- x0h)
Human-anty decision PJT n/h1
Al-enty deciiion PIT njo)
Al-only decision PL7 13
0.04
04-
Ubilty Disparity
0.
0.5
0.09
04
022
0.2
-0.04
Experiment
Experiment
Experiment
Experiment
Figure 2: The utility and utility disparity where the final decision is made by human-only P(T 1) (h) or Al-only P(T =1) =(a).
In Figure provide the utility and utility disparity when final decisions are made solely by human decision-makers and AI independently. The key observation is that Cognition-aware Multi- calibration adjusts the AI's confidence mitigate utility disparity caused by human-only de- cisions, either by reducing the disparity or by offsetting disparity with the opposite sign. This capability is absent in Dognition-unawann Multicalibration, which in some cases (e.g., the Sarcasm task) can even worsen utility disparity compared to human only decisions.
RELATED WORK
AI confidence helps decision-makers calibrate their trust in the Al and appropriately apply AI knowl- edge to make final decisions, especially in cases where the Al model is likely perform poorly. To enhance human decision-maker's comprehension of Al prediction uncertainty, Al model confi- dence is primarily calibrated to reflect the probabilities of classification correctness Hebert-Johnson et al. (2018): Guo et al.) (017);; /Zhao et al.] (202T). However, experimental evidence by Vodra- halli et al. (2022b) indicated that AI models, when perceived as more confident than they actually are are-rather than being well-calibrated--cre enhance the accuracy of final decisions made by human decision-makers after considering Al advice. The work most closely related to ours is Corvelo Benz & Rodriguez (2023), which conducted systematic theoretical analysis of scenarios where well- calibrated AT confidence may lead to suboptimal utility for rational decision-makers. They also introduced the concept of AI confidence fuman-alignment, enabling rational decision-makers to achieve optimal utility. Previous works in Al-assisted decision making assume that human decision- makers are homogeneous, overlooking heterogeneity |Rambachan (2024); Rambachan et al. (2022); De-Artcaga et al. (202T) in their cognitive capacities. When AI assistance is used as new informa- tion resource, failing to account for cognitive heterogeneity can exacerbate utility disparities among decision-makers, exacerbating societal inequities. Distinguished from Corvelo Benz & Rodriguez (002)), our work addresses a novel and previously unexplored dimension of Al-assisted decision- making: ensuring equitable utility for human decision-maken with varying levels of cognitive abil- ity. Furthermore, we contribute to a solid theoretical framework for analyzing and mitigating fair- ness issues in Al-assisted decision-make makinng. Different from the commonly used algorithmic fairness in the fair machine learning area, which mainly aims to ensure unbiased outcomes for individuals being decided upon (e.g., patients), focus is on ensuring fairness for the decision-man makirs (e.g..

---

Published as a conference paper at ICLR 2026
doctors). Pleiss al. (2017) investigated the compatibility between calibration and equalized odds. Beyond differences in application contexts- -heee their focus is on predictor scenarios rather than work also targets distinct faimess concept. This faimesss cept shares some similarities with accuracy disparity in centralized model training Chi et al. (2021) and egalitarian fairness in decentralized learning DDonahue & Kleinberg (2023), but it requires fun- damentally different solving methods tailored to the context of Al-assisted decision-making
Al-assisted decision- making
CONCLUSIONS AND LIMITATIONS
In this work, we systematically analyzed the issue of unfair utility that arises when human decision- makers with heterogeneous cognitive capacities engage in Al-assisted decision-making. We identi- fied that rational decision-maker may not achieve equal utility under existing Al confidence cali- bration methods. To address this issue, we introduced novel AI confidence adjustment objective, inter-gru greer-rrouulllgmmettt which, when combined with stman-alignment, establishes an upper bound on utility disparities. Building this foundation, we proposed cognition-awes multicalibration, which ensures interpretable utility fairness and optimal overall utility, as it serves as a sufficient condition for simultaneously achieving both human alignment and mnter-group-aaignmmntt Experi- ments conducted on real datasets thoroughly evaluated the effectiveness of our proposed AI confi- dence adjustment approach providing optimal and fair utility across human decision-maiee makers with heterogeneous cognitive capacities. Our study offered a new perspective on fairness concerns in Al-assisted decision-maie making. As the first work in this area, we acknowledged that future research can explore additional fairness metries, live user studies, and dynamic human behavioral evolution in long-term interactions. Due space limitations, these aspects are discussed Appendix A..
ACKNOWLEDGMENTS
This work was supported in part by Major Program of Guangdong Province under Grant 2021QN02X166. and in part by the National Natural Science Foundation of China (Project No. 72031003) Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the funding parties.
REFERENCES
Umang Bhatt, Javier Antorán, Yunfeng Zhang, Q. Vera Liao, Prasanna Sattigeri, Riccardo Fogliato, Gabrielle Melançon, Ranganath Krishnan, Jason Stanley, Omesh Tickoo, Lama Nachman, Rumi Chunara, Madhulika Srikumar, Adrian Weller, and Alice Xiang. Uncertainty as a form of transparency: Measuring, communicating, and using uncertainty. In Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 401-413 ACM Press, 2021.
Niklas Bussmann, Paolo Giudici, Dimitri Marinelli, and Jochen Papenbrock. Explainable machine learning in credit risk management. Computational Economics, 57(1):203-216, 2021.
P.C. Cacciabue, F. Decortis, B. Drozdowicz, M. Masson, and J.-P. Nordvik. Cosimo: A cognitive simulation model of human decision making and behavior in accident management of complex plants. IEEE Transactions on Systems, Man, and Cybernetics, 22(5): 1992.
Jianfeng Chi, Yuan Tian, Geoffrey J. Gordon, and Han Zhao. Understanding and mitigating accu- racy disparity in regression. In Proceedings of the 38th International Conference Machine Learning, volume 139, pp. 1866-1876. PMLR, 2021.
Nina Corvelo Benz and Manuel Rodriguez. Human-aligned calibration for ai-assisted decision mak- ing. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), (eds.))Adie Advances in Neural Information Processing Systems, volume 36, pp. 14609-14636. Curran Associates, Inc., 2023.
Maria De-Arteaga, Vincent Jeanselme, Artur Dubrawski, and Alexandra Chouldechova. Leveraging expert consistency to improve algorithmic decision support. arXiv preprint arXiv:2101.09648, 2021.
10

---

Published as a conference paper at ICLR 2026
Carl Dement and Melissa Inglis. Artificial intelligence-assisee assistee criminal justice reporting: An ploratory study of benefits, concerns, and future directions. Criminology Criminal Justice, 2024.
Kate Donahue and Jon Kleinberg. Fairness in model-sharing games. In Proceedings of the ACM Web Conference 2023, pp. 3775-3783. ACM Press, 2023.
E Bruce Goldstein. Cognitive psychology: Connecting mind, research, and everyday experience. Cengage learning Stamford, CT, 2015.
Nina Grgié-Hlača, Christoph Engel, and Krishma P. Gummadi. Human decision making with ma- chine assistance: An experiment on bailing and jailing. Proc. ACM Hum.- -Comput Interact. (CSCW), 2019.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. In Proceedings of the 34th International Conference on Machine Learning, volume 70, pp. 1321-1330. PMLR, 2017.
Ursula Hebert-Johnson, Michael Kim, Omer Reingold, and Guy Rothblum. Multicalibration: Cal- ibration for the (Computationally-idntiii Conference on Machine Learning, volume 80, pp. 1939-1948. PMLR, 2018
masses. In Proceedings of the 35th International
Faisal Kamiran, Indré Žliobaité, and Toon Calders. Quantifying explainable discrimination and removing illegal discrimination in automated decision making. Knowledge and information sys- tems, 35()):613-64,, 2013.
Shuai Ma, Ying Lei, Xinru Wang, Chengho Zheng, Chuhan Shi, Ming Yin, and Xiaojuan Ma. Who should trust: Ai myself? leveraging human and ai correctness likelihood to promote appropriate trust in ai-assisted decision-making, Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems. ACM Press, 2023.
Ninareh Mchrabi, Fred Morstatter, Nripsuta Saxena, Kristina Lerman, and Aram Galstyan. A survey on bias and fairness in machine learning. ACM Comput. Surv., 54(6), 2021.
Robert K. Merton. The matthew effect in science: The reward and communication systems of science are considered. Science, 159(3810)556-66 63, January 1968.
Mahdi Pakdaman Nacini, Gregory Cooper, and Milos Hauskrecht. Obtaining well calibrated prob- abilities using bayesian binning. Proceedings of the AAAI Conference on Artificial Intelligence, 29(1), 2015.
Geoff Pleiss, Manish Raghavan, Felix Wu, Jon Kleinberg, and Kilian Q Weinberger. On fairness and calibration. In Advances in Neural Information Processing Systems, volume 30. Curran As- sociates, Inc., 2017.
Snehal Prabhudesai, Leyao Yang. Sumit Asthana, Xun Huan, Q. Vera Liao, and Nikola Banovic. Un- derstanding uncertainty: How lay decision-makers perceive and interpret uncertainty in human-ai decision making. Proceedings of the 28th International Conference on Intelligent User Inter- faces, pp. 379-396. ACM Press, 2023.
Pranav Rajpurkar, Chloe O'Connell, Amit Schechter, Nishit Asnani, Jason Li, Amirhossein Kiani, Robyn L Ball, Marc Mendelson, Gary Maartens, Daniël van Hoving, et al. Chexaid: deep learning assistance for physician diagnosis of tuberculosis using chest x-rays in patients with hiv. NPJ digital medicine, 3(1):115 15, 2020.
Ashesh Rambachan. Identifying prediction mistakes in observational data. The Quarterly Journal of Economics, 139(3):1665-177) 2024.
Ashesh Rambachan, Amanda Coston, and Edward Kennedy. Robust design and evaluation of pre- dictive algorithms under unobserved confounding. arXiv preprint arXiv:22/2.09844 2022.
Ryan Rifkin and Aldebaro Klautau. defense of one-vs-all classification. Journal of machine
learning research, 101-111 2004.
11

---

Published as a conference paper at ICLR 2026
Salvatore Ruggieri, Jose M Alvarez, Andrea Pugnana, Franco Turini, et al. Can trust fair-ai? In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 15421-15430, 2023.
Mark Steyvers and Aakriti Kumar. Three challenges for ai-assisted decision-making. Perspectives on Psychological Science, 9(5):722-7344 2024.
Kailas Vodrahalli, Roxana Daneshjou, Tobias Gerstenberg, and James Zou. Do humans trust advice more if it comes from ai? analysis of human-ai interactions. In Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 763-777. ACM Press, 2022a.
Kailas Vodrahalli, Tobias Gerstenberg, and James Y Zou. Uncalibrated models can improve human- ai collaboration. In Advances in Neural Information Processing Systems, volume 35, pp- 4004- 4016. Curran Associates, Inc., 2022b.
Heather Burrell Ward, Frances R Levin, and Shelly Greenfield. Disparities in gender and race among shysician-scientisst call action and strategic recommendations, Academic Medicine, 97(4):487-411, 2022.
Oskar Wysocki, Jessica Katharine Davies, Markel Vigo, Anne Caroline Armstrong, Dónal Landers, Rebecca Lee, and André Freitas. Assessing the communication gap between ai models and health- care professionals: Explainabilityy, utility and trust in ai-driven clinical decision-making. Artificial Intelligence, 316, 2023.
Ming Yin, Jennifer Wortman Vaughan, and Hanna Wallach. Understanding the effect of accuracy on trust in machine learning models. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, pp. 1-12. ACM Press, 2019.
Chiara Zappalà, Luca Gallo, Jan Bachmann, Federico Battiston, and Fariba Karimi. Gen- der disparities in the dissemination and acquisition of scientific knowledge. arXiv preprint arXiv:2407.1 17441, 2024.
Yunfeng Zhang, Q Vera Liao, and Rachel K. Bellamy. Effect of confidence and explanation on accuracy and trust calibration in ai-assisted decision making. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency, pp. 295-305. ACM Press, 2020.
Shengjia Zhao, Michael Kim, Roshni Sahoo, Tengyu Ma, and Stefano Ermon. Calibrating predic- tions to decisions: A novel approach to multi-class calibration. In Advances in Neural Information Processing Systems, volume 34, pp. 22313-22324. Curran Associates, Inc., 2021.
A TECHNICAL APPENDICES
A.1 RATIONALE AND SCOPE OF ASSUMPTION 2.1
Assumption 2.1 describes the behavior of rational decision-makers who aim maximize decision utility: they increase their likelihood of making positive decision T = 1 when both their own confidence h and AI confidence increase. While Assumption 2.1 could be violated in atypi- cal scenarios-such as AI aversion, where decision-makerg systematically decrease their decision likelihood as Al confidence increases, or random errors where decision-maken accidentally make incorrect dociisiossssue cases represent departures from rational meliit-mmximizing behavior and are outside our current scope.
Assumption 2.1 violation rate evaluation. To empirically assess the prevalence of rational be- havior in real-world settings, we evaluated Assumption 2.1 across all four real-world Al-assisted decision-making tasks by computing pairwise violation rates. Specifically, for all pairs of instances (i,j) where h; < h, and a, S a,,, we checked whether P(T: = 1) > P(T; = 1) (a monotonicity violation). Table 3 reports the results. The violation rates are consistently low across all datasets. hollds generally in real-world Al-assisted
This provides strong empirical support that Assumption holds decision-making contexts.
Performance under increased violation rate, To validate robustness under monotonicity viola- tions, sampled instances set with increased Assumption 2.1 violating rates. The violation rates
12

---

Published as a conference paper at ICLR 2026
Table 3: Assumption 2.1
violation rates across 4 tasks.
Exp.
Art
Cities
Sarcasm
Census
Violation Rate
0.53%
0.41 %
0.52%
0.20%
Table 4: Assumption
2.1
violation rates across tasks in sampled set.
Exp.
Art
Cities
Sarcasm
Census
Violation Rate
4.7%
2.7%
1.9%
4.9%
Performance comparisons on this "noisy" dataset are presented Figure and Table Results show that Cognition-awee aware Multicalibration maintains competitive utility compared to Cognition-tmawary Multicalibration while significantly improving fairesss by reducing utility disparities.
after sampling are shown in Table
4
A.2 PRE-LEMMAS
Lemma A.1. If the utility function u satisfies Eq. T and the distribution of Y 1 satisfies
P(Y 1|s = 1) > P 1|S = 0). then a trivial policy that always decides T = will consistently result a positive utility disparity, while a trivial policy that always decides T 0 will consistently result in negative utility disparity.
EYeP (u(1Y)|s 1] EvEP (){S, - 0] 0,
(18)
Eyep [u( Y) = - (0)5, 0] S.
(19)
Lemma A.2. If the utility function satisfies Eq. 1 and the distribution of Y satisfies P(Y 1|s 1) P(Y = 1|5 = 0), then a trivial policy x that always decides will consistently result in negative utility disparity, while trivial policy that always decides T will consistently result in a positive utility disparity.
Eyer [u(1, Y)JS 1] - Eyer ()|S = 0] < 0,
(20)
Eyep [u, Y) = 1] - Eyer (u(O = 0] 2 0.
(21)
Proof.
For Lemina A.I: As u(1, 1) > u(1, 0) and u0, 2 u(0, 1). when P(Y = 1|$ 1) >
P(Y 1|5 = 0), have
Eyer
1] - 0
= P(Y = =1|$ = -1 (11 = 1|$ = 1)) =(1,0)
(22)
-P(Y =1S (11 P(Y 1|5 = 0)) u(1
= =(P(Y = 1|SS- = - P = 1|$ = 0)) (u(1, (00.1)-u((110))) =((,0) > 0.
Eyer [u(0, - Eyer
= =P(Y = =1[S = 1) +10 P(Y = 1|$ = =1) -(0,0)
(23)
-P(Y =1|S u(0, P(Y 1/$ = =0) (0,
= -PYY
= -
= -1 P 1 = 0)) (u(0, 1) u00 s 0.
For Lemma A.2: Similarly, when P(Y = 1 = 1) < P(Y = 1 = 0). we have,
Eyep (u1)) = 1] - Ever [u(1,)]]5 = 0]
(24)
=P((Y = 1|S 1) P(Y 1|5 00) (uu(,, 1) v(1, 0)) 0.
Euer|u(0, 1] Eyer (
(25)
(P(YY 118 1) P(Y 1|5 = 0)) (u(0, 1) u(0,0) > 0.
13

---

Published as a conference paper at ICLR 2026
Table 5: T-tests on the performance of cognition-unawarr multicalibration and cognition-aware
multicalibration under 100 repeated experiments.
Utility
Utility Disparity
Exp.
t
p-value
p-value
-5.350
0.000
16.789
0.000
2
-3,571
0.002
6,949
0,000
3
-19.342
0.000
55,487
0,000
4
1.782
0.092
1.010
0.326
Alassisted decision making P(T m(h.a)
Al-assisted decision making P(7 1) m(h,
0,75
Unilty Disparty
0.70
0.65
0.00
No
Cognition unaware Multcolbratoon
-0.05
Cogation-wers Multcaibmtion
0.56
Experiment
Experiment
Performance with increased monotonicity violation rates. Grouped by "education": Statistics of utility and utility disparity over 100 experiments, where the final decision P(T 1) (h, a) is made by human with Al assistance. The Al confidence is adjusted by No Adjust, Cognition-unann cennition-unawaee unaware Multicalibration, and Cognition-awan Multicalibration, respectively.
Figure 3:
A.3 PROOF OF THEOREM 3.2
Theorem 3.2 (Utility disparity under calibration) Given an Al-assisted decision-making with utility function w(T, Y)) in Eq. ① and the human decision decision-makers makers with any monotone Al-assisted decision policy π 11(H, A). Even when the Al confidence function fA is perfectly calibrated, if the cognitive disparity in Eq. is non-zero, then there exists AI confidence and human confidence such that the utility disparity is also non-zero.
1x [u(T, ()) = a, Zh,1]
(26)
E, (()))))) a,2 Znol #
Proof.
According the law of total expectation, and the final decision P(T = =) (h, a)
independent of the sensitive attribute S (consistent with reality that different human decision-makes make decisions based solely their own confidence and those of the Corvelo Benz, & Rodriguez (203 ), the expected utility disparity can be formulated as follows:
BR YY.ff))), a, Zn,1] Exx [u[T, ())
E Z
= = = 1[fa(2) = =a,z Zx.1) P(Y = (1) = a, E Zn.o)l
(1((1 - u(1, =((0 + u).,
([TT]]ff(]]) = a, E Zn) -(u(0,0) u(0, 1)) (27)
To prove the above equation, let's first look at Ex [u(T, ()) a, Zn.1)
Ex ()) = a, 2 E 2
E E) = a, E Zn,1]
PT (2) a, Zn) [1 (T 1ff a, 2,)
+
= 0, E Zn.1]
PP.(T 11]f(>> a, 2,)
[[[[ a, Z
[E,, -azE ZZ.1]]
E ())f(()) a, 2
14

---

Published as a conference paper at ICLR 2026
Similarty, E
a, Zn,u] can be formulated followed.
E [u(T, = a, E 24.0]
PP = ==] = a, 2 2n,) [B ())[a(=) = a, 2 Zn,o|
- [m(0 ()LA(() = a, = [n,o]]
Blu0 (=) = a, E Zn,ol
Therefore, this equation [E, [u(T, ()) a, 2n,1] Ex (I(TT a, Zn,o]| can be expanded into the following form.
E [u(T, fx) a, E 2n,1] Ex [u f2) a, E Zn,ool
= - 1[fa(2) = 2n,1) - PPYY 1[fA(2) = a, 2n,o)l
+) +000,) u() [C[T=1lfa(C = a, E Bx.) [u(0, 0) u(0, 1)]l
Based on the law of total probability, if the cognitive disparity in Eq. 22 is non-zero, there exists Al confilence a such that,
P(Y 1[fa(=)===)))1)))))))(()))) = a Zn.i) - P(Y = 1[fa(=) = =a Zn,o) +
(28)
Let
Q = 0=][u(1,1) - -((,0) + 00)
= o0, 2n) [u(0 0) u(0, 1)]l
~ ~10, max (m(0.0) u00 1), u1, (1,,0))
Given that the decision policy π T(h a) is monotone, it holds that, for any given Al confidence
w(0, 0) - u000 1)
3h (T= a0= Zx) u(1, u(1 u(0, u(0.0)
Q +
(29)
Therefore, there exists Al confidence a and human confidence h such that,
|E (Y)))((())) = = E 2n.1] - E. ()(T(a) = a, E Zn.oll # 0.
Thus, Theorem 3.2 is proven.
A.4 PROOF OF THEOREM 3.4
Theorem 3.4 (Utility disparity under human-alignment) When the cognitive disparity in Eq. 2 is non-zero, there exist (infinitely many) Al-assisted decision-making processes with utility function uTT, Y) in Eq. (I) and the human decision-make, with any monotone Al-assisted decision policy E 11(H A), such that even the AI confidence function fA is perfectly aligned with the human's s, the Al-assisted decision-maken still fails to achieve optimal utility fairness. Specifically.
E. [T, Y)ff(()) a. Z,.a] E, (T, ff((())))))))))))))))))))))))) a. Zn,oll = a, Zn,1l - Ex ())A) =
(30)
Ex [u(T,Y))fa(=)=
= a, E Zn,ull-
where,
arg min [EE [u(T, ())ff(z)))))))))))))))))))())) = - [u(T. Y)LA(()) - E Zn.o]l =EU(HAA)
(31)
We first define a, which represents the smallest Al system's ''s confidence value for given confidence level h, such that,
Proof.
à minnfa = P = - Zn.o) > 0
a,
(32)
We demonstrate through the following four cases that there are infinitely many Al-assisted decision- making processes where, despite the Al confidence being numan-alignan the Al-assisted system fails achieve optimal utility disparity.
15

---

Published as a conference paper at ICLR 2026
Case 1. For any confidence [h1. a1] with an 01, according Eq. 132 it holds that,
P(Y=1) (A(z) = , Bny,1) - P(Y [ a1,= = E 2by,0) S O.
(33)
Furthermore, there exists another (h2, 02), where 02 max (22; 01) and h h such that,
P(Y = P f = 02, E 2ha,1) - P(Y = fa(z) - 02 E Bhz,.) > O.
(34)
In the case where fA is au, -dl-gnnnmment with respect to fH. according Definition 3.3 for any h H. there exists 2/, C 2n with 2n = {(x, h, (x) (x) = h) C 2 and 2/ 2 (1 22) 1 such that,
P(Y = Mar,= = E 2 P(Y = 1/22 E 2/22) a < ax max(0, (0) Based on the law of total probability, the Eq. (35) can be expanded as follows:
(35)
P(Y = 1 E 2/,,,) P(S 1|,1, 2%1y)
+ P(Y - 1/01, = E m) P(S 0/01; 2/ny)
(36)
P(Y 1|a2, E k,g,1) P(S 1|02, 2 k/2
P(Y 1|22, = 2/22,) P(S 0/a2, 2/2,) + +aa
We can quantify the utility disparity gap under different confidence settings when the decision-maker consistently chooses as follows:
E () h - (a) Zhy,ol)
E [u(1, 12, - [u(,,) (),1,)))222 a2, E Zhrool
(37)
(u(t,))-u(L.0)) A1.
where,
A P(Y = 1/a1, 2 E Z/ P(Y 1|01, 2),,)) P(Y = l|a2, 2/n.1) + P(Y =1|a2, Z/2,0)-
Similarly, can define the utility disparity gap under different confidence setlings when the 0,
decision maker consistently chooses T
(E [u(0, Y) 1a,, 2/,,1] - E [u(0, Y) E 2hy,ol) (E [u ))),, E Zhg.1] -E [u(0, l2, =E Zhhzol)
(38)
(u(0, 1) - uu0,0)) A..
As P(Y 1 fA(2) 01,2 2,,,1) P(Y fA(2)) =0,2 = E 2n,,0) S O, according to
Lemma A.2 we have:
E ()) = 2 2 E (M(1 = a1, Zh,,o]
(39)
< 505B ()) .1,2 E Zn,a) E. () = .1,2 Zng.ol
Combining Eqs.
38 and 39 it holds that,
[1, Y) a2 2 E 2ha,1l E [1(11, (1,,Y))a22 2ha.ol)
EE [u(0, () E 2/,,1] - [u(00, a2, = E Z/h.ol]
(40)
< (u(0, -0.0)))-00000 - u(1, 1) + uu(, A..
As P(Y = 1 12,22 2 E Bhg,1) 1) - P(Y a2, E Zhn > 0, it holds that,
E [1([1 (a2,, Zn..1] E [u( Y) aa, E Znzol
(41)
> 0 > 1E (0, Y))a2, 2 E Zh - E [u(0, Y) |a), E Zhz,ol.
16

---

Published as a conference paper at ICLR 2026
Based on Eq.
41,, the upper bound of the utility disparity of policy π is,
0 <
Ex [u(T, la,, 2 E Zhg,1l - Ex [u(T, |22 2 E Zh
E (u(1,Y)|a2
2 E [u(1 a2, 2 E o1)
(E (u(0,)) 22, E k E [u(0, Y) 02, 02, E Bho,
(1 - a) 2
(E (a(1,Y)|as E 1/ - E ())2,)) E 1h2,ol) )
(42)
(E [u 2) E Ehn.1] - E [u(0, Y) a2, E Zhgool))
a
2
(E [u(L, a2) E Zhza \ 1/g,1] - ()(,
aa),, E Ziz \ (hg.0))
(E [u(0, Y) = E Zhn,1 1/2,1] - -E [w(,, a) = E Zaz.0 Zhg.ol))
For z E Zrv2 Thng as
111, 0) S E [111, Y)
0z; 2 E Ehg
1%,1 S u11 1),
(43)
u 1) S E
[u(0, Y)
022 2 E Bhz
1%22 S u(0,0), 0).
we have:
0 < E [u(1,)) 02 E Znz.1 \ 3/2.11 - EE uu(1 Y)
02, E Enz,0 \ 1/2.0| S su(1,
-o(11111) 0);
(44)
u(0, 1) - u((,0)= S E () | 122 E Zno,1 \ 1/c,1] [u(o,y) 1) 12, E Bhoo0 \ 2/2.0/ S 0.
(45)
Then, Eq.
(2 can be reorganized as follows:
- | [u Y) E 2h2,1l Ex [u(T, ()
laz E Zh2
a) ) (u(1.0) 0) uuL, 1) u(0. + u(0, 1)) A
(46)
a u[(1, 1) - ull,0) u(0, 1) u()
2
Based on Eq. and it follows that:
4 0.
(47)
Therefore, the optimal utility disparity is achieved when:
A 0
(48)
When fA is perfectly aligned with fH (a* S a = 0) and Eq. many cases that
48 does not hold, there are infinitely
Ex (T, (T),, E B - E, (),T))) 2 E Bhg,oll
(49)
Ex [M(T, Y) 2 E Zny,1] Ex [u Y) 2 E Zhg
Case 2. For any confidence [bi, ai) with a u, according Eq. 332 it holds that,
P(Y = 1 01,= = E Bmn.)) - P(Y= =1 01, E 2m,,o) > O. Furthermore, there exists another [22, where 02 > 02 > ai and h2 > hi such that,
(50)
P(Y =1 1 02, Zhn.1) - P(Y 02 2 E Zho S 0.
(51)
According to Lemma A.2 we have:
E [u(1, Y)a,,, = E Zhy,1] E[u(1,Y)|a, E Zn.o] E [u(0, Y),,,, E Zh..1l- -
(52)
E Zn.o]. Y )a) E(((O,
(0(
17

---

Published as a conference paper at ICLR 2026
Combining Eqs.
52
38 and 399. it holds that,
[0, Y) a2, 2 2/on1l - E [u(0, a2
a2; E 2/220l)
[ [u(, 02, E 2/ - -E [u(1, a2, 2 E Zhg.0l
(53)
x (u(1,1) - u(1,0) - u0, 1) + =(0,0)) A2.
where,
2 P(Y = E 1/ P(Y 1|11, E 0))
- PPY 1/22 2 E 2/2,,) P(Y 1|02, 2 Z/1,,)
As P(Y = 1 02,2 11,,,) - P(Y
02, E 0 S 0, it holds that, E (1, Y) a2, 2 E Tn2,1l - E [u(1, Y) a2, E oh
(54)
s 0 < E [u(0, a) 2 Zhn,1 - EE [u(0, Y a22 o
Based on Eq.
54), the upper bound of the utility disparity of policy π is,
0 ~
|Ex [u(T, la,, = E 1ng,1] - Ex [u(T, a2, 2 E Znz.oll
(E [u(0, |2,; Iha,1l 18 [u(0, la2, E Oh2
02, = E Bng,o)) (E [u(1, Y) (),,))))a))) 12,2 E kb E [u(1, a2,
(1 (1- )
(B [u(0, Y)|a2, E 2/ - E ()(0,)) E ohn))))
(55)
(E ( 2/n,1] - [u(1, a2, Zhg,o)) 1)
(IE [u(0, Y)) E Zhz.l \ 1/ - E [u(0, a) E Znz \ Zha,ol)
EE [ul,, a22, 2 E Zh2.1 1 EE () E Zhz.0 Zha.ol))
For Sha Eyer according to Eq. (43 it holds that,,
(1100) - u(1, 1) < E Y) | 1222 Zng.1 [/g.1] - R (u(, az, E 2ng.0 \
2/2.0 SO
(56)
0 < E () a2 2 E Zbz,1 3/a.1 E
(( |222 E Zna,0 \ |hh.o| su(0,0) u(0, 1).
(57)
Then, Eq. 53 can be reorganized as follows:
x Ex [u[T, Y)),2, 2 E 1 E, [u[T, Y) a22 E Zhzolll
< (1 a) ) (u(1, - u(1,0) u(0, 1) + =(0, 0)) A2 u(0, 2 (u(1, - -(1,0))u00,
(58)
a
u 1) + (00 0))
Based on Eq.
50 and 57) it follows that:
A2 2
(59)
Therefore, the optimal utility disparity is achieved when:
A₂ 0+.
(60)
When fA is perfectly aligned with fH (a" S a 0) and Eq. 160 does not hold, there are infinitely many cases that
Ex [T,, Y) Zng,1] Ex [u(T, Y) Ja2, 2 Znz,o]l
(61)
[E,- [u(T,)))as Y) 2 E 2ng - Ex- [u(T, Y) E Zhz,oll.
18

---

Published as a conference paper at ICLR 2026
Case 3. For any confidence [A1 a1) with an 01, according Eq. (32 it holds that,
P(Y = 1 01, BmI,)) - P(Y = |1, E Zn, S O. Furthermore, there exists another [2), where 02 02 > ai and h2 hi such that,
(62)
P(Y 02 1hg,1) P(Y 1 02, E Zhz.0) S 0.
(63)
When the decision- maker consistently chooses T holds that:
E la,, 2 m - E [u(1, 1a,,, E 2/,,0)) E [u(L, Y |), E 2/,,,] - E a2, Z/a,o))
(64)
(u(1,1) - u(1, Az.
where,
A P(Y = ||a1, E 2/ P(Y = 1a11 Z/1,0)
P l|a2, 2hn,,) - P(Y 1|22, 2 E 2/g.a)-
Similarly, when the decision-maker consistently chooses T 0, it holds that,
1/1,1] E [u( l), o)1,))) + [u Y) 02, E 1/2,1] [u(0, Y) a2, E 2/2.0l) a2,
( [00,, Y) =
(65)
(u(0.1) 1) - u(0, 0)) A3.
According to Corollary
A.2 we have:
E [u(1, (1, lai, E - E Zn
(66)
EE (0, (ala = 1E (on0,,))aa:: lai, Zho,ol
Combining Eqs. 166
64 and 655 it holds that,
(E ((, Y) a22 1/n,,] - E [00 1u2, E Eh2.ol)
(E [u(1, |), E h/.,]] - E (2[u1, 2hn,ol)
(67)
$ (u(0,) 1) - (0, 0) - uu1, + u1,, As.
As P(Y 1 |1,22,2 1 - P(Y
a2, E hh 0) I 0, it holds that, E [u(, Y) |a2, Zhg.il - [u(11 a2, E Zh
(68)
- 0 < E [u Y))22, 2 E I - E [u(0, l) E Zh
Based on Eq.
68 the upper bound of the utility disparity of policy π is,
0
<
Fx [u(T, Y) E Bhn,1] - Ex [u(T, () E Bhz.oll
< E ((0,Y))la.
2hn.1l [u(0, Y) a2, E Zhzool) 1))2,,
(E [u(1,Y) a2,2 E Bho.1l E [u(1, Y) a2, 2 E olo
(1
E [u(0, Y))a2, 1/ E (u(0,Y) Zhg,0l )
(69)
( (u(1, Y))a2, 1/ -E [u(1, a2, Zho.ol)
(IE) (u, Y))a2, 2 E Zha.1 \ Z/n,1] - [u(0, a2,, E Znaoo \ Zha,o))
E [u(1, |a2, & E Zny,1 1h -E [u(1, laz, E Zhz.0 Zhg.0l))
For z E Zho \ Zhn according to Eq. 143 it holds that,
u) - u(1, 1) < E ju(L, X) a22 E Zhzal \ Zha.1 [u(,,Y) Y) a2 2 E Zr \ Zhy.0l 50.
(70)
19

---

Published as a conference paper at ICLR 2026
0 S E ((0,)) 2 E Zha,1 \ Tha.1] - E (0 ag, E Zhz \ 2/a.0] S su(0,0) -0
(71)
Then, Eq.
(69 can be reorganized as follows:
0 s T [u(T, a2, 2 E 1/2,1] Ex [u(T, |a,, 2 E Zh2,oll
< (1 2 (u(1, - u(1, =,0, + m(0 1)) As
(72)
a
(m(1, 1) - (( 1) + ( ().
2
Based on Eq. 62) and 633) it follows that:
A3 S 0.
(73)
Therefore, the optimal utility disparity is achieved when:
A3 o-.
(74)
When fA is perfectly aligned with fH (a" S a 0) and Eq. 74 does not hold, there are infinitely many cases that
|& [T, 2a2, 3 - Ex [u( Y) a2,, Zh2.oll
(75)
E [u(T, 102, 2 E Zhz,1l - Ex- [U( a2 a2, 2 E Zhz,
Case 4. For any confidence (hi, ai) with a > ã₁ according to Eq. 32 it holds that, P(Y =1 a1, 2), - P(Y
a1, E 2n, .u) O. Furthermore, there exists another [h2, 02), where 02 > 01 > 02 and h2 > hy such that,
(76)
P(Y- = 1 02, E 1 - P(Y 022 E 202 > O.
(77)
According to Lemma
A.2 we have:
E [u( 1a,, 1 (El,)_E(((1,y)) a1, E Zn, ol [u(00, Y) a1, 2 E Zn1,ol-
(78)
> E (0, ())a1, 2 E Zn,,al - E -
Combining Eqs. 78
64 and 65 it holds that,
(E [1(1, Y) a2, 1/n,1] - E [u(1, Y) l02, E Eh2,,0)
(E [u(0, Y) = E 1/ E. [m( Y) la2; = E 2hy,ol)
(79)
< (u(1, 1) - u(1, -((0 1) + #0, 0)) A..
where,
A P(Y = 1|a1, 2 E 2/1, P(Y = 1aa1 2 Zh1,0)
+ P(Y 1|02, = 2/ P(Y 1|02, 2 Zhg.o)-
As P(Y =1 ||02, = E 1) - P(Y = 1/,2 E h22,0) > 0, it holds that,
E (0. Y) la2, x Zn2.11 - [u Y) |) E Zozool
(80)
W 0 < E [u(1, Y))a2, 1 En,,il - E [u(1, V) E Zhg.ol-
Based on Eq.
80) the upper bound of the utility disparity of policy n is,
0 <
Ex [u(T, Y)2,,, E 1hn,1] - Ez [[[T,,TTT Y)),,, E Znozull
S EE (u(1,Y)las: V))a2, 2 E 2, E [u(1, ((11,))ae (a2, E ob)
(E [u(0, Y) 22, E 2
E [u(0, Y))a2, 2 E Zng,ol)
(1- (1-2) ))
(E [u(1, Y))a2, 2 E 1h E [u(1,) E Zh3.0))
(81)
(E [u(0, V))a2, E 1 E. [u(0, Y) a22 = 2hy.ol)
EE)) [u(1, Y))az, E Bho.a \ 1/22.1 - E [u(1, Y) laz 2 E Zho.o \ 2h2o0)
E [u(0, Y))a2, E Engal
1//o.]] - E [u(0, Y 1022 2 E Znz.0 2hh,0l))
20

---

Published as a conference paper at ICLR 2026
For 2/12 \ 2/g according Eq. (43 it holds that,
0 s B u(l, a22 2 E Zha.1 h/2.1 - EE [1l,))) a2, E Bhz,0 \ [/2.0 S su(1, 1) =u(1 0);
(82)
u(0, 1) w(0,0) < E () 02, E Zha.1 \ Zha.1| [u(o,)) r) a2, Zhano \ Zh2.0/ S 50.
(83)
Then, Eq. can be reorganized as follows:
$ [ [u 1a2, 2 E 2h2,1] Ex [u(T. Y) a2, E Zhz w(0 1) =(0,))) 0)) A4
a) 2 (u(1, 1 u(1,
(84)
a
(m(1, 1) - 0),
((0.)))) 0))
2
Based on Eq. 76 and 77 it follows that
A >
(85)
Therefore, the optimal utility disparity achieved when:
A = 0+.
(86)
When fA is perfectly aligned with fH (a* S a = 0) and Eq. 86 does not hold, there are infinitely many cases that
Er [u(T,Y)Ja,, E 2ha,1l Ex [u Ya2, E E Zh2.0l1
(87)
> Ex- [u(T, aa2, z E 2ha,1] - Ex- [u(T, Y) 2 E 0h2.ol1.
Based on the above proof, we have demonstrated the existence of scenarios in which, even when f' perfectly aligned with fH. any monotone policy π leads to a suboptimal utility disparity, thereby supporting Theorem 3.4
A.5 PROOF OF THEOREM 3.6
Theorem 3.6 (Utility disparity upper bound under o-human-alignmen) For given Al-assisted decision-makinn process with utility function u(T, satisfying Eq. {1) and the human decision- maker with any monotone Al-assisted decision policy æ E [(H, A), if the AI confidence func- tion fA is ag cu-annman-llign and satisfies a, wyy-inter-rrou-alinnnee then the utility disparity is bounded by,
ER ()) = =a 2 E. [u(T, ()]fa(=)=))))) = =0, Bn.oll ahh
< (u(1,1))w(0,1)) 100,1)-=(1,,) -(1
( (0, ((.)))
2 (1- 0th 2 (3eg az))
(88)
Proof. Given Cgg-inter-grout grwpgggiiiaaannnnttfooeee for any two confidence levels (hn, an) and (12, with 0g 01 and h2 > h the following conditions hold for all z E 2%, and 2 E 2hn respectively:
-ag S P(Y = 1|a1,2 E 2 - P(Y = 1|a1, 2 2%h,,) S ag.
(89)
ag S P(Y 1|a2, 2 E 2% P(Y =1|a2, 2 2hg,0) S SOgg
(90)
where 2i,, C Zh, and Zh, Zio with | 2 (1-6) a2) - -2) |Z and 2%% 2 (1 -1-00 ag 22) Zrnz
(1-ag/2)-
Referring to Case 1 in Appendix A.4 confidence levels (h2, a2):
we have the following utility disparity upper bound for Al
< E [T, la2, E 1ng,1] Ex [u(T, 1) |22 E Zng,oll
< [1 - a ( (u(1, u(1, 1) (0, + (0, A
(91)
a
- u<0, (0,
2
21

---

Published as a conference paper at ICLR 2026
where,
A = P(Y = 1|a1, = E 2/,,,) - P(Y = 1/0,, E (0,))
PPY = 1|22,2 2/n,1) + P(Y 1/ E 2/2,0)
Using the conditions given by Eqs. 189 and (90), we have:
22g S 2 20kg
(92)
where,
2 P(Y 1|01, E E Zhn n 221 P(Y 1|a,, 2 2h,,00
,0 n
(%
-P(Y 1 a,, 2 1/2g'1 (2g,11) + P(Y 1|02, E 2h2,0 2/120))
For zorze E 2/ \ 2% it holds that,
- Az 2.
(93)
where,
A P(Y 1|01, 2 1h,,1 1%,,1) P(Y 1|a1, 2 E 2 2h1,0)
- P(Y =1|az, 2 E 2/2211 \ 2%2,1) + P(Y =1|u2, 2 E 2hh,00 \ 2)2
Incorporating these conditions into the utility disparity upper bound in Eq.
99 get:
0 $ Ex ()),,)))22, = E 2,,,a] - E, [T, Y) 2 E Bng.all
< (u(1, 1) u(1, 0) -00 1) + +(),0)) ((1
04
(11-2)
Ag'
20g
+
ag
2)
+ ah
2
2
2
2
= () 1) u(1 m(0 0)) ( + 1
0/h
(3ag
2
(94)
This bound can be similarly derived for Cases 2 in Appendix A.4 yielding a consistent utility disparity bound as stated in Theorem 3.6
A.6 PROOF OF THEOREM 4.5
Theorem 4.5 Suppose f2(=) is az /222ccogntioon-aaarr multicalibrated with respect to the collection C- = (Zh.s IEEH Then, fA satisfy both a-human-alignnen and o-inter-group-aligneess
Proof. According to the definition of cognition-aware multicalibration in Method 4.3 when hu-
man decision-maker are partitioned into two groups with group membership values = and
1, and given fixed human confidence level h E H. there exists subset Z/. Zn. with
1z () Zz.l such that, for any Al confidence 01,h1 E [0, 11, it holds that,
|P(Y (2) 01,2 2/n,1) a2l
az 1
(95)
|P(Y = 11]AA(;) a11z 1/,,0)) -
a1 5 2
(96)
From the given inequalities, we have:
P(Y fl(((2) a1,2 2/n,1) 01
01
+
a
(97)
2
2
P(Y - =1,2 Z), J01
a 2
2 01
+
(98)
Then, it's natural that SA satisfied o-inter-group--llgmmmnn as,
P(Y 1 fa(=) = 01, 2/,,)) P(Y = 1 f() 01, E 2/1,0) Sa.
(99)
As for any human decision-maker confidence 0 S hi S hr2 S 1 and 0 S S 02 S 1, it holds that,
P(Y = 11|A((<)>> -2, 02; 1/y,1)
02 2 02 2
(100)
P(Y = fa(z) = =2z E Z/2,0) E
02 a
02
+ a
(101)
2
2
22

---

Published as a conference paper at ICLR 2026
For P(S = ||, E 2/,) P(S 0/a, E 2/) it holds that,
P(Y 1 2,,) P(Y 2 2/n)
P(Y 1| E m) P(S 1|01, = E 2%,)
+PY 1/11 () P(S 0/ 2/,)
-P(Y 1 1 2 1/ PPS 1|02, 2/22)
P(Y 1 a022 2 E 2/z,)) P(S 0/02, 2 )
P(Y = 1|a,, E 1) PPS = 1| 2 6 2
+ PPY = 1 2 E () (1- (1- - ||,, E 2),))
(102)
PP(Y = 1 1022 2 E 1/,1)) -P((S = 1| 2/12)
P(Y 1 1 2 2/2,0) -(1 (1- P(S || 2/1z))
P(S 1a1, %%1y)= -aa + 2 + (1- P(S 1|a1 2/n)) (a1 + a
P(S 102, E 2/2y) (02 _ (1 - P = =1/a2, E 2/2) (02 2
2
=a + a1 02 <
Consequently, fA satisfies a-human--lignmeet and o-inter-group-aiigmmen mennae meanwhile.
A.7 EXPERIMENT SETTINGS
A.7.1 DATASET PROCESSING
Following the data processing Corvelo Benz & Rodriguez (2023), we transform the original dataset's confidence values from scale of [-1, 1J (0, 1] ensure consistency with our human-Al inter- active model (Section?) 2). In the original dataset, predictions by participants are from different but overlapping sets of countries across tasks, who are told the Al advice has different accuracy. Thus, to control for these confounding factors, we focus exclusively on participants from the United States who are informed that the Al advice had an 80% accuracy. Furthermore, we use education level as a sensitive attribute, recommended factor that influence cognitive capacities of human decision- makers across different tasks but should be treated with equal utility Al assistance for social good Zappalà et al. 2024): ) Ward et al. (2022)
A.7.2 DISCRETIZATION
Following the consistent experimental settings outlined in Corvelo Benz & Rodriguez (2023), we provide detailed description of the discretization process used in our experiments: the human con- fidence h is discretized into 3 bins per task, {H1, H2. Ha), corresponding to low, medium, and high confidence levels, respectively. The bin boundaries are set such that each bin contains ap- proximately equal probability mass, with the bin values assigned as the average confidence within each bin. The AI's confidence a are divided into uniformly sized bins per task with centred value given by A A 3A ...... } J where À 1/8. The above process discretizes the continu- ous confidence space H A into grid of 3 x [1/A] cells. The (i, grid cell contains the samples Z/- {(x, h, Zh Hi, f=(2) [АÍ]] X/2, Afj + X/2)}. Furthermore, based on the value of the sensitive attribute E S. the samples in z can be further divided into z!
{z(, 21,0} where z ((x, h, E Zhh H4, f2( [Аß] 2/2, + A/2), 1} and 2/0 = {(x, s) E 2//- E H4, fx(2) E [Аb]] - X/2, AbJ + X/2), = 0}.
A.8 ADDITIONAL EXPERIMENTS
A.8.1 ADDITIONAL. QUANTITATIVE ABLATION RESULTS
Tables and provide detailed quantitative results corresponding to the visualizations in Figures
and 2 The key takeaways are:
23

---

Published as a conference paper at ICLR 2026
Table 6: Decision Utility: Ablation quantification results (Black: mean; Blue: standard deviation).
No Adjust
Method 4.4
Method 4.3
Exp.
Al-only human-only Al-assisted AI-only human-only Al Al-assisted/ Alassisted/Al-onil Al-only human-only Al-assisted
0,7843
0.6601
0.7570
0.8163
0.6601
0,8068
0.8221
0.6601
0.8045
(1.3--5)
(0)
(1.1e-16
(1.3c-15)
(0)
(0.006)
(7.7c-16)
(0)
(0.005)
0,8082
0.7366
0.7755
0.8163
0.6601
0.8068
0.8586
0.7366
0.8253
(5.5e- 16)
(3.3e-16)
(5.5e-16)
(1.6e-15)
(3.3e-16)
(0.004)
(3.3e-16)
(3.3e-16)
(0.005)
0.7596
0.7293
0.7792
0.8008
0.7293
0.7973
0,8089
0.7293
0.8025
3
(3.3e-16)
(0)
(1.5e-15)
(1.1e-16)
(0)
(0.003)
(5.5e-16)
(0)
(0,003)
0.7596
0.7293
0.7792
0.8052
0.7103
0.7845
0.7967
0.7103
0.7817
(1.1e-16)
(1.2e-15)
(3.3e-16)
(6.6e-16)
(1.2e-15)
(0.004)
(8.8e-16)
(1.2e-15)
(0.004)
Table 7: Decision Utility Disparity: Ablation quantification results (Black: mean; Blue: standard deviation).
No Adjust
Method 4.4
Method 4.3
Exp. AAl-nlyy human-only Al-assisted Al-only human-only Al Al-assisted Al-only human-only Al-assisted
0.0426
0.0441
0.0420
0.0362
0.0441
0.0467
0.0304
0.0441
0.0382
(9.0e-17)
(5.5e-17)
(9.0c-17)
(0)
(3.5c-17)
(0.0038)
(6.2o-17)
(5.5e-17)
(0:0081)
-0,0688
-0,0507
-0,0595
-0,0226
-0,0507
-0,0148
0.0185
-0.0507
0.0014
2
(8.3e-17)
(2.7e-17)
(9.0e-17)
(3.4e-18)
(2.7e-17)
(0.0037)
(2.4e-17)
(2.7e-17)
(0.0072)
-0.0374
0.0410
0.0203
-0.0347
0.0410
-0.0075
-0.0136
0.0410
-0.0047
(3.4e-17)
(4.le-17)
(1.0e-17)
(4.le-17)
(4.le-17)
(0.0024)
(8.2e-18)
(4.1e-17)
(0.0056)
0.0049
0.0164
0.0222
0.0077
0.0164
0,0314
-0.0352
0,0164
0.0089
4
(4.3e-18)
(1.76-17)
(1.7e-17)
(1.1e-17)
(1.7e-17)
(0.0058)
(2.7e-17)
(1.76-17)
(0.0060)
Decision utility improvement: Both Cognition-umaware Multicalibration (Method 4.4 and
Cognition-aware Multicalibratiin (Method 4.3) enhance AI-only decision utility through AI confi- dence adjustment. Given that human-only decision utility remains fixed, this improvement in AI- only utility translates to higher overall Al-assisted decision performance, providing more substantial assistance to human decision-maker makers across all groups.
Decision utility disparity reduction: Compared both the No Adjust and Coguition- iagnition-inaware unaware Multicalibrution, Cognition-aware Multicalibration effectively reduces reverses decision utility disparities across different human decision-make groups in Al-only settings, ultimately achieving substantial fairness improvements in decision utility under Al assistance.
A.8.2 GROUP-LEVEL. ACCURACY DISTRIBUTION ANALYSIS
Figure shows the instances' decision accuracy distributions across different decision-maker groups. The results demonstrate that Cognition-aware Multicalibration achieves fairness by de- livering better instance outcomes for the disadvantaged group (Group 5=1 with lower "education" level) without negatively impacting the advantaged group (Group 5-0).
A.8.3 IMPACT OF DISCRETIZATION PARAMETERS
Figure presents the evaluation of EAE and EIAE metrics across tasks after performing cognition-aware multicalibration with varying hyperparameters ã ~ [0.0001, 0.01, 0.1] and X ~ 0.12,, 0.2]. The results indicate a general trend where smaller values of results closer to the left or bottom of the plots) lead to lower EAE and EIAE values. We focus on the general trend as we have claimed that discrete evaluation metrics like EAE and EIAE provide a more accurate reflection of alignment when there are significant changes and may fail capture alignment across the entire continuous confidence space fully. Nevertheless, the general trend strongly supports the notion that decreasing
(reflected in
leads to improved human-alignment and inter- gropgrrrap-llimmme..
24

---

Published as a conference paper at ICLR 2026
All Decision Makers
Group S-0
Group S-1
Dipartus Muticaaion
Dapiis vouee Nctoliadia
Dentel
%
Figure 4: Instance accuracy distributions across all decision makers, Group S=0, and Group S=1 (lower "education" level) on the Cities task.
0.412
00003
0.010
0.00
3.014
12013
2.011
02011
6.008
0000
1.000
AM00
2208
2012
1202
0001
0.00
4400
AM00
5000
kwkk
0.00
-.40
5.001
0.20
81
Figure 5: Visualization of EAE and EIAE metrics following cognition-aware multicalibration, with parameters ā [0.0001, 0.01, 0.1] and [0.1, 0.125, 0.2].
A.8.4 GENERALIZATION TO MULTIPLE GROUPS
In this experiment, introduce "gender" male" as Group 5 = 0 and "Male" as Group S = 1). We first validate the effectiveness of our method under the new demographic feature "gender". The experimental results are presented in Fig- ures 6 and 7 We further conduct experiments considering both "gender" and "education" as demo- human
additional demographic feature for grouping ("Fe-
maker groups: (0. 0), S (0, 1). S = (1, and S = (1, 1). The utility disparity is measured using the standard deviation of utility distributions across these subgroups:
graphic features, resulting
Disp
- (0. E Y) == (0, 1)),
(103)
(1,)) S (1,1))).
The results are presented in Figures g and
Across these experiments, where human decision-make makers are grouped by either single or multiple demographic features, the results align with expectations and demonstrate the following key take- aways:
1. Effectiveness in improving faimess across different settings: Across both single-grous group and multi- multi-group
-goup settings, Cognition Cognition-aware aware Multicalibration consistently outperforms both No adjust and Cognition-unaware Multicalibration in terms of fairesss. Notably, it avoids the fairness deterioration observed in Dognitiion-nawwrr Multicalibration (i.e., tasks 1, 3 and 4 in Figure 6 task 3 in Figure
2. Comparable utility performance: Cognition- aweee Multicalibration achieves utility per- formance comparable Cognition-unaware Mulicalibration. For example, in Figure
25

---

Published as a conference paper at ICLR 2026
al-astsee decision making PIT= 1) ntn,
Al-assisted decision making PIT= mth,
0.04
Utility Disparity
0.82
Usilty
0.80
0.78
Ne
0.76
-0.02
Muticallrrtion
Cogitton-ware Muttccaitratoon
0.74
Experiment
Experiment
Figure 6: Grouped by "gender": Statistics of utility and utility disparity over 100 experiments, where the final decision P(T 1) (hh, a) made by human with Al assistance. The Al confidence is adjusted by No Adjust, Cognition-umavver unaware Multicalibration and Cognition- aware Multicalibration, respectively.
Human-only decision nih)
Humarronly
Alonnyy PIT wh mta)
Al-only
0.10
34
0.8
Digattam Me locion
0.84
0.05
Unily
Unitly
0.4
0.00
0.2
8.0
0.0
06
Figure 7: Grouped by "gender": The utility and utility disparity where the final decision P(T
1) x (h) is made by human-only or Al-only, respectively.
Al-assisted decision making PIT m(h,
Al-assisted decision making P(T n(h,
0.82
0.04
Ubility Disparity
Fogntiomrrroeee Cogntiom-eerr Mirtca bridion
0.03
Unilty
0.78
0.02
0.76
0.01
0.00
Experiment
Experiment
Figure 8: Grouped by both "gender" and "education": Statistics of utility and utility disparity 100 experiments, where the final decision P(T = 1) T(h, a) is made by human with AI as- sistance. The AI confidence is adjusted by No Adjust, Cognition-unawal Multicalibration and Cognition-aware Multicalibration, respectively.
Human-onty decision PC7 = n(A)
Human-onty decision PVT n0))
Al-onty decision PIT 1] nia?
Al-onty decision PLT (Ea)
0.8
No-ddat
Oogrroane Viline
0.6
0.-
Experiment
Experiment
Experinent
Expertment
Figure 9: Grouped by both "gender" and "education": The utility and utility disparity where the final decision P(T = 1) = x <hh is made by human-only or Al-only, respectively.
Cognition-awann aware Multicalibration demonstrates similar utility performance compared to Cognition- Multicalibration. In tasks 2, and 4 of Figures Cognition-awer Mul- ticalibration shows superior utility performance. Although there is a slight accuracy drop in task 1, with the 25th percentile decreasing by 0.9% compared to Cognition-unawee unaware Mul-
26

---

Published as a conference paper at ICLR 2026
Table 8: Runtime comparison (in seconds).
Method 4.4
Method
4. subgroups]
Method +4 subgroups
Exp.
Overall
OOverall Group Group 1 |Overall Group Group
Group 2 Group 3
1.397
1.714
0.682
1.032
1.050
0.272
0,427
0.207
0.144
2
1,372
1,480
1.065
0.415
1.131
0.273
0.295
0.300
0.262
3
1.092
1.246
0.958
0.288
0.714
0.036
0.195
0.329
0.152
4
0.502
1.062
0.921
0.140
0.806
0.087
0.094
0.299
0.324
ticalibration, this trade-off considered acceptable given the significant gains in fairness across groups.
3. Adjusting Al confidence to mitigate utility disparities: Cognition-aware Multicalibration effectively adjusts Al confidence reduce or reverse utility disparities in Al-only deci- sions, compensating for disparities observed in human-only decisions. This behavior re- mains consistent across different group settings (Figures 7 andd
A.8.5 COMPUTATIONAL COMPLEXITY ANALYSIS
For each sensitive group E S, the Method 4.3 iteratively updates AI confidence values within each grid cell (i.j. s) until convergence, where E
NA} denotes the index of AI bins, j E (1,..., Nu) denotes the index of human bins, NA = [1/\] denotes the number of AT confidence bins, N₂ denotes the number of human confidence bins, and Ni.j.s denotes the num- ber of instances in cell (i.j,s] s). For each cell (i.j.)), s), the Method 4.3 update the Al confidence for ni.j.e instances until the tolerance threshold ā is met. By the convergence guarantee of X- discretization (Hebert-Johnas
et al. 2018). the algorithm terminates in most ( (ā A)) rounds, where each round processes all ni.j.s- Therefore, the computation complexity of the computation in each cell (i.j. s) is O(n....//
X)). The total time complexity over all cells is ONN/(C [ A)). as Eiggs Mii..s Nolt. where Nall is the total number of instances. Specifically in Table 8 we com- pare the runtime of Method 4.4 with Method 4.3 using 2 subgroups and 4 subgroups, respectively. The experimental results reveal no significant correlation between overall runtime and the number of subgroups. Instead, runtime variation is primarily driven by differences in per-group computa- tion time. This variation arises mainly from group-specific data distributions that affect the initial miscalibration and result in different convergence times within each subgroup.
A.8.6 MULTI-CLASS CLASSIFICATION
Our current theoretical results focus on binary classification, binary decision tasks encompass wide range Al-assisted decision-making scenarios, including loan approval, medical diagnosis, and hiring decisions. In this part, we discuss potential ways to extend our framework for providing fair and high-utility Al-assisted decision- eeeision-mmainggeee in multi-class settings.
For the multi-classificatii utility function defined in Equation (104). we employ Os-AVl (OVA)
decomposition to reduce the multi-class problem to binary subproblems. The proofs presented in our work are modular and can be directly applied.
" (T, (T, Y) {
i YY,
(104)
where r- < r+. For a K-class classification problem with labels Y E (0.1...., ......,, K - 1}, we decompose it into K mutually exclusive binary subproblems: (1) "Is the label 0?" (K) "Is the label K 1?". Each binary subproblem is associated with a binary label (k) {0, 1}. where (k) I(yall k). The labels satisfy mutual exclusivity: for any instance i, exactly one binary label is positive across all subproblems, K (k) 1, thus, (k) = 1 yall k, we have.
(k) 1/h(() a (k) P(Yall k/h (0) ..... (a (K-1))
(105)
For each binary subproblem, the decision policy is denoted as P(T(M) 11/00) (.)
(k) (k()) (K)).. When the Al confidence is calibrated to perfect human-alignment such that
27

---

Published as a conference paper at ICLR 2026
Table 9:
Performance evaluation of multi-class decision-making on the Art task (Black: mean;
Blue: standard deviation).
No adjust
Method 4.4
Method 4.3
Classes
Utility Utility Disparity DisplrityUttliity Utiliyy Disparity Utility Utility Disparity
3 0.749
0,049
0.758
0.052
0.775
0.027
(0.010)
(0.005)
(0.006)
(0.005)
(0.005)
(0.005)
0.460
0.035
0.495
0.035
0.505
0.026
5
(0.004)
(0.002)
(0.002)
(0.005)
(0.003)
(0.003)
x (1) (h()), (A)) is monotonically related to P(Y 1|h (l))), the multi-class optimal deci- sion is:
T arg max (N) (H(*,, (k) arg max P(Y. k/h (0) ((-1)), (a (K-)).
(106)
Equation (06) is Bayes optimal: selects the class with the highest posterior probability. Con- sequently, optimal utility in each binary subproblem guarantees optimal multi-class utility (Rifkin Klautau 2004) when the argmax rule is applied with perfect human-alignment cal- ibrated confidences.
Similarily, when each binary subproblem achieves optimal fairness, that is, minimize utility disparity UD2 0 via perfect nner-ggrup-allinmmett, the utility disparity in multi-clasification problem is bounded by:
[T, Yau)|s 1] E[u Yau)|s = =O]
K-
(107)
P(T =k) [[, / k. 1] E|u(T, y = k, = =O]]-
k=0
Since (()) = I(Y = k) and (T, ) = l represents the utility when class & is selected (not necessarily correctly classified), it holds that: R[[(TT, )TT = k] B(((((()) (())) Therefore, Equation (177) can further bounded as:
K-1
B[u Yau) = 1] E|u(T, Yan) s= = Oll -
Σ
P(T = k:) .UDD UDr
(108)
k=0
where UD2 denotes the utility disparity when class is selected:
UD = (()(T<(A, (+)) = 1] (ju(T((),
0j
(109)
The above process establishes that fairness each binary subproblem guarantees fairness in the multi-class setting for prcision-makery with heterogeneous cognitive capacities.
Table 9 presents a performance comparison of three methods for 3-class and 5-class classification on the Art task. The experimental results demonstrate that Cognition-aware Multicalibration effec- tively improves both overall utility and utility fairness across decision- maker groups
A.9 SCOPE AND FUTURE WORK
A.9.1 FAIRNESS METRICS
In this work, we use utility disparity for measuring fair utility, aligning with the Doccisionmmaeet primary goal of making more accurate decisions with AI assistance (Steyvers & Kumar 2024). The concept of utility disparity is also evident in centralized learning (Chi et al.] (002) and decentral- ized learning (Donahue & Kleinberg] 2023). While other fairness metrics, such as demographic parity and equalized odds (Mehrabi et al. 2021), emphasize the equality positive or true posi- tive outcomes, these metrics primarily relate individuals being judged (e.g., patients) and diverge (e.g., doctors). Nonetheless, exploring diverse
from the fair utility objective for faimess concepts remains interesting avenue for future work, particularly in understanding the underlying motivations of fairness principles beyond utility disparity the context of Al-assisted decision-making
28

---

Published as a conference paper at ICLR 2026
A.9.2 ARCHIVAL DATASET USAGE
Our evaluation employs archival human-Al interaction datasets, a methodological choice grounded in cognitive science showing that humans maintain consistent decision-makinn strategies when addressing similar problems or within short-term interactions (Goldstein, 2015 Cacciabue et al.
1992]. and following established precedent in the human-Al collaboration area, where archival datasets serve as the standard for algorithmic validation (Vodrahalli et al. 2022b Corvelo Benz & Rodriguez 2023).
As the first work addressing utility fairness for human decision-makers with cognitive dispari- ties, the archival-dataset approach is sufficient for establishing and validating our core theoretical and methodological contributions: @ identifying utility unfairness in Al-assisted decision-making among cognitively heterogeneous human decision-makers, developing cognition-aware multi- calibration grounded in the theoretically-fouded objectives of inter-group-aligmeent and human- aligument, and @ demonstrating substantial improvements in decision utility fairness compared to baselines while maintaining comparable overall utilitty.
While live user studies would introduce confounding factors arising from subjective human be- cavior--callenges that exceed the scope of our fairness-focueed investigation, we view carefully controlled live studies as a valuable future direction to investigate the impact of dynamic human be- havioral adaptation on Al-assisted algorithmic foundations established in this work.
performance, building upon the theoretical and
A.9.3 CONNECTION TO GAMB-THEORETH PERSPECTIVES
In this section, we discuss the relationship between our proposed framework and game-theoretic solution concepts, their applicability differences, and potential for integration.
Stackelberg equilibrium and strategic interaction. Our work addresses faimess issue in AI- as- sisted decision-making by adjusting the Al confidence towards theoretically-foune foundedd alignment ob- jectives. This represents an Al-side intervention approach. In contrast, the Stackelberg game model characterizes Al-assisted decision-making as a sequential strategic interaction where the Al system (leader) first commits to a policy, and human decision- makers (followers) subsequently respond by optimizing their own strategies given the Al's policy. Under this game-theoretic perspective, the focus shifts to strategically influencing human behavior. These represent two fundamentally different problem-solving paradigms, distinguished by their assumptions about human adaptabil- ity. Our work establishes a solid theoretical foundation for utility fairness in Al-assisted decision- making, grounded in humans employing consistent decision-maken making strategies. This is supported by cognitive science showing that humans typically maintain consistent strategies when addressing similar problems or within short-term interactions (Goldstein) 2015 Cacciabue et al. 1992). Game- theoretic approaches assume humans can and will strategically adapt their behavior response to Al policies. Modeling such strategic adaptation and the resulting equilibrium dynamics presents additional complexities that exceed the scope of ate choice between these different problem-solving routes depends on the application context and are willing and able to modify their strategies.
harness--ocusee investigation. The appropri-
whether
While our current work focuses on stable human behavior (Goldstein) 2015 Cacciabue et al. 1992, note that longer-term interactions involving repeated Al assistance may exhibit dynamic behav- ioral evolution. Investigating such scenarios where humans strategically adapt over time represents a valuable future research direction. A theoretic perspective could be beneficially integrated with our proposed Al-side alignment framework to account for dynamic changes in human decision- making behavior.
Shapley value. The Shapley value is a solution concept in cooperative game theory that allocates the total payoff generated by a coalition fairly among its members in proportion to their marginal contributions. While the Shapley value has been successfully applied to fairness problems in ma- chine learning contexts, it does not naturally apply to our Al-assisted decision decision-making setting due to fundamental structural differences. Al-assisted decision-makin involves collective payoff requiring allocation. Instead, each decision-maan maker operates independently on their own instances, deriving utility exclusively from their individual decisions. Moreover, the decision utility is inher-
29

---

Published as a conference paper at ICLR 2026
ently non-transferabl,, and the performance of one decision Decision-maker cannot be redistributed to benefit or harm others.
A.9.4 YULE'S EFFECT IN SUBGROUPS
As highlighted by Ruggieri al. (2023), achieving fairness at the group level does not necessar- ily guarantee fairness at finer granularities phenomenon known as the Yude's 's effect. This phe- nomenon arises because, beyond the directly observable grouping attributes, such as education level adopted, there may exist additional confounding variables that contribute to within-group hetero- geneity. As is difficult to determine whether feature a confounder or a mediator, Kamiran et al. (2013) proposed clustering strata into manageable number of groups to control for confounding factors. Such subgroup identification and stratification methods are orthogonal to our work and can be naturally integrated: framework provides the calibration objectives for Al confidence and determines how AI confidence is adjusted within groups, while stratification methods refine the hu- man decision- maker grouping by accounting for additional confounding variables. As demonstrated in our experiments (Appendix (.8.4), when we partition users into multiple subgroups, Cognition- aware Multicalibration continues to effectively reduce decision utility disparities within each sub- group. This makes the combination feasible and promising for future work, to ensure that fairness improvements do not cause utility drops for specific sub-populations within the broader cognitive capacity groups we define.
30


Reviews:
[{"review_id": "UScMvjebEp", "reviewer": "Reviewer_wqhf", "summary": "This paper studies fairness in AI-assisted decision-making, focusing on situations where humans have different cognitive capacities. The authors show that existing approaches, such as calibration and human-alignment, do not necessarily ensure utility fairness across different groups of human decision-makers.\n\nThey introduce a new concept called inter-group-alignment, which ensures that human groups with similar confidence levels receive comparable decision utility when supported by the AI. The paper provides solid theoretical analysis, deriving a clear upper bound on utility disparity as a function of both human-alignment and inter-group-alignment.\n\nTo achieve this dual goal in practice, they propose a Cognition-aware Multicalibration algorithm that adjusts AI confidences accordingly. Experiments on four real-world datasets (Art, Cities, Sarcasm, and Census) confirm that this method significantly reduces fairness gaps between human groups while maintaining overall performance.\n\nThe theoretical formulation, clarity of objectives, and validation through experiments make this a valuable contribution to the fairness and human-AI collaboration literature.", "questions": "Could you provide a clear quantitative table comparing human-only, AI-only, and human-AI decisions across all datasets to make the ablation more explicit?\n\nHow sensitive are the results to the parameters $\\tilde{\\alpha}$ and $\\lambda$? Would adaptive tuning or data-driven adjustment improve generalization?\n\nIf the human decision policy is not strictly monotonic (for example, due to inconsistent trust or cognitive biases), how does that affect your theoretical bounds or empirical fairness outcomes?\n\nCould the dual-alignment framework extend naturally to multi-class or regression tasks?\n\nHave you considered a Stackelberg or cooperative-game interpretation of your framework, where the AI strategically adjusts its outputs to optimize a fairness-aware social welfare objective? Are they feasible to perform and related at all?\n\nDo you think modifying AI confidence distributions could unintentionally change how different human groups rely on the AI (either over-trusting or disengaging)?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "The introduction of inter-group-alignment provides a fresh and rigorous way to think about fairness in human-AI collaboration.\n\nThe paper is theoretically complete, with well-defined assumptions and proofs that connect intuitively to fairness.\n\nThe proposed Cognition-aware Multicalibration method is interpretable, practical, and mathematically justified.\n\nThe empirical validation is strong: four datasets, clear fairness and utility metrics, and consistent results showing improvements.\n\nThe work bridges human and algorithmic fairness, showing how cognitive differences among people can be formally addressed rather than ignored.", "weaknesses": "Monotonicity Assumption (Assumption 2.1):\nThis assumption states that humans act rationally, increasing their probability of making a positive decision as either their own or the AI’s confidence increases. While this is reasonable for theoretical proofs, it may not always hold in practice.\nReal humans are often non-monotonic in their decision behavior. For instance, they may overtrust or undertrust AI advice due to biases, cognitive fatigue, or misunderstanding of confidence. In such cases, increasing the AI’s confidence may not increase the chance of a positive decision, which violates monotonicity.\n\nThe authors could strengthen the paper by (a) discussing how this assumption might break in real-world conditions, (b) analyzing the theoretical impact if monotonicity is only approximately satisfied, and (c) outlining how the proposed method could still remain effective. Even a small simulation with “noisy” or biased human decision policies would illustrate robustness and make the work more realistic and influential.\n\nAblation Study Clarity:\nThe paper does include human-only, AI-only, and human-AI results, but these are only mentioned in Figure 2 and not summarized in a table. It would be much clearer to show numerical comparisons between these three conditions across datasets. This would highlight how the collaboration truly benefits fairness and overall utility.\n\nHyperparameter Sensitivity:\nThe appendix covers sensitivity analyses for parameters like $\\tilde{\\alpha}$ and $\\lambda$, but the main text should summarize them briefly. Readers need to know how robust the fairness improvements are if these parameters change. A short paragraph or figure in the main paper would make the method easier to trust and reproduce.\n\nGame-Theoretic Perspective:\nThe problem naturally relates to concepts like Stackelberg games (AI as leader, human as follower) or Shapley-value-based fairness allocation. A short paragraph connecting the proposed framework to these ideas would situate the work more clearly within the broader literature on incentive alignment and cooperative fairness.\n\nBehavioral Impact of AI Adjustments:\nSince the AI modifies its confidence outputs to improve fairness, it would be interesting to consider whether this adjustment might influence how humans behave over time. For example, could one group become overly reliant on AI due to consistently higher confidence signals? Even a brief reflection on this would enhance the paper’s real-world applicability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sFVTTxvuAe", "reviewer": "Reviewer_Wqv4", "summary": "The paper demonstrates that existing AI confidence adjustment objectives, such as calibration and human alignment, may lead to utility disparities across groups of decision-makers with varying cognitive capacities.  Moreover, the paper shows that the utility disparity between human decision-maker groups is bounded by a function of human-alignment level and inter-group-alignment level. Hence, the authors propose  a multicalibration-based AI confidence adjustment approach to mitigate this concern, validating their approach over four different datasets.", "questions": "I would like the authors to discuss my highlighted shortcomings.\n\nMoreover, I wonder if the authors could detail some settings where $P(Y|S=1)$ is different from $P(Y|S=0)$? My intuition is that, for some reason, the instances that are considered by one group can be sampled from different parts of the feature space, but I want to be sure about it.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The main strengths are:\n\n1. The paper covers an overlooked problem in the human-AI alignment literature, as it focuses on fairness in human-AI collaboration\n2. The proposed multicalibration approach is principled and based on the proposed theory\n3. Overall, the paper's goal is well-motivated and interesting", "weaknesses": "The main shortcomings of the paper are:\n\n*Significance* While I appreciate the theoretical results, I think this new paradigm comes with some implementation choices that are not straightforward. For instance, in real-life scenarios, evaluating cognitive disparities might be even unethical, so such information might never be disclosed. Moreover, it is also quite challenging to assess if improving the fairness for decision-maker groups leads to better outcomes for the observed instances; hence, I think the authors should discuss these possible limitations.\n\n*Assumptions Discussion* I think the authors should discuss Assumption 2.1 better, as understanding when such an assumption might fail is a key factor for the proposed method.\n\n*Empirical Evaluation:* I think some of the results are not very clear. For instance, I would suggest that the authors reformat Table 1, as it does not report standard deviations and incorrectly displays the best entries in bold.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hHS0t5r8sd", "reviewer": "Reviewer_T784", "summary": "The paper tackles fairness in human–AI collaboration from the angle of ensuring that AI assistance provides equitable benefits to human decision-makers who possess heterogeneous cognitive capacities. The authors argue that existing alignment notions such as calibration and \"human-alignment\" are insufficient for guaranteeing fairness *across* decision-maker groups. They introduce the concept of \"inter-group-alignment\" to capture disparities in decision utility between subpopulations of humans. The paper provides a formal theoretical analysis linking human alignment and inter-group alignment. Building on this insight, the authors propose \"cognition-aware multicalibration\", and prove that it serves as a sufficient condition for simultaneously achieving both alignment objectives.\nThe claims are validated through experiments on four human-AI decision-making tasks. The results demonstrate that the proposed method successfully reduces utility disparities across groups while maintaining overall performance.", "questions": "I found the paper to be insightful and well-executed. The weaknesses section already incorporate most questions, primarily aimed at better understanding the practical implications and boundaries of the proposed framework. One additional question pertains to incorporating broader notions of utility within proposed framework. The paper defines utility as decision accuracy. In human-AI collaboration, however, overall utility can also encompass factors like decision time, cognitive load, or the user's trust and self-efficacy. How do you see the concepts of human-alignment and inter-group-alignment extending to these more multi-faceted, human-centric notions of utility?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. **Novel and Interesting Perspective:** The paper shifts fairness focus from data subjects to the decision-makers themselves. It's an underexplored and insightful perspective as it illuminates how AI systems can inadvertently widen performance gaps between varied decision-makers (e.g., experts and novices).\n2. **Elegant Theoretical Framework with Principled Operationalization:** The derivation of a utility disparity bound provides an interpretable bridge between fairness notions, offering practitioners a clear diagnostic for assessing group-level disparities. Cognition-aware multicalibration operationalizes the theory in a computationally practical way.  \n3. **Empirical Credibility:** Validation on multiple real-world tasks demonstrates both robustness and practical feasibility. Additional experiments in appendix demonstrate thoughtfulness and rigor.", "weaknesses": "1. **Scalability Challenges with Multi-Attribute Grouping:** The proposed framework requires partitioning the data and calibrating for each subgroup. While the experiments show this is feasible for a small number of groups, the computational and data requirements would grow exponentially with the number of sensitive attributes. A deeper discussion of the practical limits (e.g., the number of subgroups that can be realistically handled) would be beneficial.\n\n2. **Rational Decision-Making and Static Setting:** The theoretical claims hinge on the assumption that humans follow a rational, monotone decision policy (Assumption 2.1). However, a large body of literature shows that human behavior often deviates from pure rationality, exhibiting various cognitive biases. The framework does not currently account for such suboptimal but more realistic decision policies. The analysis assumes that the cognitive capacities and decision policies of the human groups are static. In many real-world collaborations, humans learn and adapt their strategies based on repeated AI interactions. The current framework does not address these dynamic or online contexts, where calibration needs might evolve over time.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KyHQjiXlLY", "reviewer": "Reviewer_MdV1", "summary": "In this paper, the authors explore how humans with different characteristics may derive unfair utility from the same advice. They introduce the new concept of inter-group alignment and propose an interpretable, fairness-aware objective for AI confidence adjustment. Their analysis shows that achieving utility fairness in AI-assisted decision making requires both human alignment and inter-group alignment. Building on these objectives, they present a multicalibration-based AI confidence adjustment approach tailored to scenarios involving decision-makers with heterogeneous cognitive capacities. Evaluations on four datasets with real human-behavior data demonstrate that the proposed method significantly improves utility fairness across groups without sacrificing overall utility.", "questions": "See above weakness.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is clear, well organised, and adds several fresh ideas to human-AI interaction research. It first defines human-alignment and the new inter-group alignment, proves why both matter, and then shows an elegant multicalibration fix to improve the fairness utility across the group.\n\n2. The experiment design and evaluation is thorough to justify the main claim of this paper.\n\n3. The authors publicly release their code, making replication and future extensions straightforward for other researchers.", "weaknesses": "1. The framework assumes people can report confidence on a common, well-calibrated scale, yet prior work shows self-reported confidence is often noisy and inconsistent across individuals.\n\n2. All experiments rely on archival human-AI datasets; the paper does not test how live users react to the adjusted confidences or how their perceptions and strategies might change.\n\n3. The evaluation covers only binary decision tasks. Extending the theory and method to multi-class settings would broaden its practical reach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)