The Value of Information in Human-AI Decision-making
Authors: Ziyang Guo
Yifan Wu
Jason Hartline
Jessica Hullman
Venue: ICLR 2026 Poster (2026)
Keywords: Explanation
Human-AI complementarity
Decision theory
Primary area: interpretability and explainable AI
TL;DR: We emphasize the value of complementary information in AI-assisted human decision making.

Abstract: Multiple agents are increasingly combined to make decisions with the expectation of achieving *complementary performance*, where the decisions they make together outperform those made individually. 
   However, knowing how to improve the performance of collaborating agents requires knowing what information and strategies each agent employs. 
   With a focus on human-AI pairings, we contribute a decision-theoretic framework for characterizing the value of information.
   By defining complementary information, our approach identifies opportunities for agents to better exploit available information in AI-assisted decision workflows.
   We present a novel explanation technique (ILIV-SHAP) that adapts SHAP explanations to highlight human-complementing information.
   We validate the effectiveness of our framework and ILIV-SHAP through a study of human-AI decision-making, and demonstrate the framework on examples from chest X-ray diagnosis and deepfake detection.
   We find that presenting ILIV-SHAP with AI predictions leads to reliably greater reductions in error over non-AI assisted decisions more 
   than vanilla SHAP.

Published as a conference paper at ICLR 2026
THE VALUE OF INFORMATION IN HUMAN-AI DECISION-MAKING
Ziyang Guo
Yifan Wu
Northwestern University
Microsoft Research
z yang- guw@northwesteem.eeee
fan f www..78gm3mmmm
Jason Hartline
Jessica Hullman
Northwestern University
Northwestern University
hartlIng@hotthhsserrr..cc
jhul www.@morrhwwstteen.c...
ABSTRACT
Multiple agents are increasingly combined to make decisions with the expecta- tion of achieving complementary performance, where the decisions they make together outperform those made individually. However, knowing how to improve the performance of collaborating agents requires knowing what information and strategies each agent employs. With a focus on human-Al pairings, we contribute a framework for characterizing the value of information. By defin- ing complementary information, our approach identifies opportunities for agents to better exploit available information in Al-assisted decision workflows. We present novel explanation technique (ILIV-SAP) that adapts SHAP explanations highlight human-commplmenntin information. We validate the effectiveness our framework and ILIV-SHAP through study of human-Al decision-male demonstrate the framework on examples from chest X-ray diagnosis and deepfake detection. We find that presenting ILIV-SHAP with Al predictions leads reliably greater reductions error over non-Al assisted decisions more than vanilla SHAPT
and
INTRODUCTION
As the performance of artificial intelligence (AI) models continues to improve across domains, workflows in which human experts and AI models are paired for decision-making are sought in medicine, finance, and law, among others. Statistical models can often exceed the accuracy of human experts on average (Agisdöttir et al., 20066; Grove et al., 2000 Meehl 1954). However, whenever humans have access additional information over an AI model, there is potential to achieve complementary performance by pairing the two, i.c., better performance than either the human or Al alone. For example, a physician may have access information that not captured in structured health records (Alur et al. 2024).
Many empirical studies, however, have found that human-Al teams underperform the AI alone (Buçinca et al.) 2020 Bussone al. 20155 Green & Chen 2019; Jacobs et al. [021 Lai & Tan 2019: Vaccaro Waldo 20199] Kononenko 2001). Two sources ambiguity complicate such results. One concerns the role of measurement: performance is often scored against post-hoc decision accuracy (Passi & Vorvoreanu] (022) rather than accounting for the best achievable performance given information available at the time of the decision (Kleinberg et al. 2015 Guo et al., 2024 Raamaccann 2024). Additionally, it often remains unclear how agents differed in their information access use, making it difficult to design interventions to improve these aspects.
Imagine one could identify information complementarities that can be exploited, such as when one of the agents has access to information contained in the other's judgments, has fully integrated contextually-availabe information (e.g., instance features) in their judgments. This would motivate interventions to improve decision-makkee For example, if we can identify how much complementary
'The code to calculate the main quantities in our framework and reproduce the experimental results is available phttps:///gttiiie wwwpchwww.o....aaaammameeeeeeeae wtt.uo/ziyagg////eeeeeee on_ fn

---

Published as a conference paper at ICLR 2026
information Al predictions provide over human judgments, can use this knowledge to guide model selection or motivate further data collection to improve the model. Conversely, finding evidence that model predictions contain relevatt information that humans do not exploit can motivate the design of explanations communicating complementary information.
We contribute a
framework for characterizing the value of information available in an Al-assisted decision workflow. In our framework, information is considered valuable to a decision-maker to the extent that it is possible, in theory, incorporate it into their decisions to improve performance. Specifically, our approach analyzes the expected marginal payoff gain from best case (Bayes rational) use of additional information over best case use of the information already encoded in agent decisions. The rational framework allows us to upperbound the expected payoff that is achievable by any strategies in the same experiment, and identifies sub-optimality in agent use of information by comparing to rational behavior. The upper bound our framework estimates holds regardless of the human's decision makiing process, which may deviate from rationality, given specified decision problem. Further, our methods can be used even when the decision problem definition is ambiguous, by using a robustness analysis over all possible proper scoring rules to identify the upper bound of performance in the worst case.
We introduce two metries for evaluating information value in human-Al collaboration. The first the value of a new piece of information
global comman-complementay information to an agent over all of its possible realizations among all instances. The second-instance-leve human-complemmentary information calue--dentifiess opportunities for decision-makerr to better use instance-level information such as specific AI predictions. Applying the second metric, we derive new explanation technique (ILIV-SHAP) that reveals how data features influence the value of complementary information for an individual prediction.
To evaluate these tools, we contribute the results of crowdsourced between-subjects experiment in which humans make decisions with and without Al models with varying human-complemen information and different explanation approaches. We find that an AI model with more human- complementary information leads to greater improvements in human-Al team performance over the numan-alone baseline. We also find that adding an ILIV-SHAP explanation to a traditional SHAP explanation leads to greater improvements in human-Al performance over the human-alone baseline than only the SHAP explanation or no explanation. We demonstrate further uses the framework in real-world
tasks, including chest X-rav diagnosis (Rajpurkar et al. 2018 Johnson et al. 2019) and deepfake detection (Dolhansky et al. 2020: Groh et al. 2022
2 RELATED WORK
Human-Al complementarity. Many empirical studies of human-Al collaboration focus on AI- assisted human decision -making for legal, ethical, or safety reasons (Bo al. 2021 Boskemper et al., 2022 Bondi et al.] [022]] (Schmmeer et al., 2022). However, a recent meta-analysis by Vaccaro et al. (2024) finds that, on average, human-Al teams perform worse than the better of the two agents alone. In response. growing body of work seeks to evaluate and enhance complementarity in human-Al systems (Bansal et al. 2021b 2019 2021a Wilder et al. 2021; Rastogi et al., 2023 Mozannar ct al. 2024b). The present work differs from much of these prior works by approac human-Al AI complementarity from the perspective of information value and use, including asking whether the
human and AI decisions provide additional information that is not used by the other.
Evaluation of human partsion-making with machine learning. Our work contributes methods for evaluating the decisions of human-Al Al teams (Kleinberg et al., 2015, 2018 Lakkaraju et al. 2017
Mullainathan & Obermeyer 2022] Rambachan 2024] Guo et al. 22044; Ben-Michael al. 2024
Shreekumar 2025). Kleinberg et al. (2015) proposed that evaluations of human-Al collaboration
should be based on the information that is available at the time of the decision. According to this view, our work defines Bayesian
benchmarks similar to several prior
works (Guo et al., 2024 Wu et al. 2023] Agrawal et al.) 2020; Fudenberg et al. 2022). Closest to work, Guo et al. (2024) model the expected performance of rational Bayesian agent faced with deciding between the human and AI recommendations as the thcoretical upper bound on the expected
We also include an observational study on the dataset from Vodrahalli et al. 2022) which identified the Al model with more numan-complementtrr information helps human-Al Al teams achieve greater improvements in performance the human-alone baseline. See Appendix [
2

---

Published as a conference paper at ICLR 2026
performance of any human-Al team. This benchmark provides a basis for identifying exploitable information within a decision problem.
Complementarity by design. Some approaches focus on automating the decision pipeline by explicitly incorporating human expertise developing machine learning models or human-Al
collaboration pipeline, such as by learning to defer (Mozannar el al., 2024a; Madras et al. 2018
Raghu et al., 2019 Keswani et al. 2022] 202T] Okati et al., 2 Chen al., 2022). Corvelo Benz & Rodriguez (2023) propose multicalibration over human and Al model confidence information to
guarantee the existence of an optimal monotonic decision rule. Other approaches exploit information asymmetry (i.e., cases where humans have additional contextual knowledge) and offer principled methods with provable guarantees to improve team decision performance (Straitouri et al.) [023]
De Toni et al. 2024 Arnaiz-Rodrigus incorporate human decisions into machine learning algorithms when the state is indistinguishable
et al. 2025). Alur al. (2024) propose a framework to
from the algorithm alone but be discriminated by humans. Our work also concerns information
asymmetry, but provides an interpretable analytical framework for quantifying the information value of all available signals and agent decisions in human-Al decision tasks, enabling the design of information-based interventions.
FRAMEWORK
Our framework takes as input decision problem associated with an information model, including decisions from one or more agents. It ouiputs the value of information of available signals to the agents, conditioning on the existing information in their decisions. Understanding the possible value of signal to a decision-maker requires defining the best attainable decision performance with that signal. We therefore adopt a Bayesian decision theoretic framework.
Our framework provides two separate functions to quantify the value of information: one globally across the data-generating process, and one locally in a realization.
Decision Problem. A decision problem consists of three key elements. We illustrate with an example of weather decision.
A payoff-relevant state from a space L2. For example, E S -0, 1] {no rain, rain). A decision d from the decision space D characterizing the decision-maker (DM))s choice. For example, d E D = {0, 1] = {not take umbrella, take umbrella).
A payoff function S D x Ω - R, used to assess the quality of a decision given a realization
of the state. For example, S(d = 0, = 0) 0, S(d 0,w 1) = 100, S(d 1, 0) S(d = 1, = 1) = 0, which punishes the DM for wrongly taking or not taking the umbrella.
Information Model. We cast the information available decisions, as a set of signals defined within an information model. Following the definition of an information model in Blackwell et al. (195I), the information model can be represented by data-generating model with a set of signals.
DM,including any available agent
Signals. There are n "basic signals" represented as random variables E,,... En, from the signal spaces Σ E,,.... Σn. Basic signals represent information available decision- as they decide, e.g., Σj = An for probabilistic prediction of raining, Eg fcloudy, not cloudy]. Eg (0 {0.... 100} for temperature in Celsius, Σ, D for human decisions D on taking umbrella not, etc. The decision-maker observes signal, which is a subset of the basic signals, 2 (P The combination of two signals V12 and Vz takes the set union V = V1 U V22 In the standard human-Al decision workflow where a human makes an independent judgment before consulting the AI, all basic signals are (a. DH, D features of the instance, the human's 's initial decisions and the AT's predictions.
Data-generating process. Decision benchmarks are defined relative to a specific data-generating process, a joint distribution e A x... x Σ, 52) over the basic signals and the payoff-relevant state. π can be viewed as the combination of two distributions: the prior distribution of the state Pr[w] and the eegnal-generating distribution Pr[u|w] defining the conditional distribution of signals. To represent the hest-attainable performance of observing a subset V of the n basic signals, we use the Bayesian posterior belief upon receiving signal V
π T(w Pr[w|o] TT,,w)
(T))
3

---

Published as a conference paper at ICLR 2026
where T(e,w) w) denotes the marginal probability of the signal realized to be and the state being with expectation over other signals, and x a(w) denotes the prior Pr[w]. Throughout the paper, we use capital letters to denote random variable of signal, e.g., V, and use little letters to denote realization of signal, e.g., V.
Information value. Our framework quantifies the value of information in signal V as the expected payoff improvement of an idealized agent who has access V in addition to some baseline information set. This corresponds to rational Bayesian DM who knows the prior probability of the state and conditional distribution of signals (i.e., the data-generating process), observes a signal realization, updates their prior to arrive at posterior beliefs, and then chooses decision to maximize their expected payoff given their posterior belief. Formally, given a decision task with payoff function S and an information model n, the rational DM's expected payoff given (set of) signal(s) V is
R(V) == [( (t),
where d'(() V D denotes the decision rule adopted by the rational DM.
d' (v) = arg max
[ w)]
(2)
dED
We further characterize the maximum expected payoff that can be achieved with no information. This can be used as the baseline to quantify the information value of V as the payoff improvement of V over. We use to represent a null signal and R(0) represents the expected payoff of a Bayesian rational DM who only uses the prior distribution to make decisions. In this case, the Bayesian rational DM takes the best fixed action under the prior, and their expected payoff is:
R(0)= == max ((d,
dED
Definition 3.1. Given a decision task with payoff function S and an information model n, the information value of V is defined as
IV(V)= = R(V) ((0)
The full information value in human-Al decision task is the information value of the set of all basic signals. For example, in the weather decision example, (V 22, Va, E4}) represents the full information value of the probabilistic prediction of rain, the cloudiness, the temperature, and the human decisions. This defines the upper bound of the information value of any signal, including the agent-complementtry information value and instance-level ggent-complemenntar information value that we will define in the following sections.
3.1
INFORMATION VALUE
With the above definitions, it is possible to measure the best case additional value that new signals can provide over the information already captured by an agent's decisions. Here, agent may refer to human, an Al system, or a human-Al team. The intuition behind our approach is that any information that is used by decision-maker Definition 3.2 captures how much complementary information value is offered by a signal over the agent decisions in expectation over the data-generating process. For example, in the weather decision task, this indicates how much the payoff from the peciiiionmmaers choice of whether to take the umbrella or not can be further improved by incorporating the temperature in their decision rule.
should eventually reveal itself through variation in their decisions.
Definition 3.2. Given a decision task with payoff function S and an information model æ, we define the agent-complementtrr information value (ACIV) of V on agent decisions D as
ACIV(V; D²) = R(D* VV) R(D*)
(5)
If the ACIV of a signal V is small relative to the baseline (3). this means either that the information
value of V to the decision problem is low (e.g., is not correlated with w), or that the agent has already exploited the information in V (e.g., the agent relies on V or equivalent information to make their decisions such that their decisions correlate with w the same way as V correlates with w). If, however, the ACIV of V is large relative to the baseline, then at least in theory, the agent can improve their payoff by incorporating V/ their decision making.
Furthermore, ACIV can reveal complementary information between different types of agents. For instance, if we view AI predictions as V and treat available human decisions as the agent signal D*, a large ACIV indicates that Al predictions add considerable value beyond what humans alone achieve. In the reverse scenario, if human decisions serve as V and AI predictions are Dd, we can measure how much humans can contribute over the information captured in the Al predictions.
4

---

Published as a conference paper at ICLR 2026
Of course, we unlikely observe identical realizations of the signals in continuous-valuce or high-dimensional data, such as images or text. A natural relaxation is to consider the realizations that sufficiently "similar" for the performance of the -ecksion-maker. Algorithm learns the posterior distribution of the payoff-relevan state from the signal and then uses that prediction as the probability distribution of the payoff-relevant state to choose the optimal action. This approach can be readily extended to the case where the signal is more complex, such as human decisions in the form of freeffom text (e.g., radiology reports) explanations of the AI predictions the form of images (e.g., saliency map).
Algorithm A method calculate the ACIV of V.
1: Input: Observed realizations {u,, di, wy 1- a predictive algorithm A, payoff function S, decision space D, and state space 52.
2: Output: AVIV(V; D
3: à + A((u; d wi)
4: a + A((fd, wi Wim
5: for i = to do
6: P: + d()
7: d t arg MDX E (S(,
8:
J + (d
9:
d. + arg max B (
10:
+ s(d;,w;) - S(d; wi)
end for
12: return Σ 8i
3.2 INSTANCE-LEVEL AGENT-COMPLEMENTAAR INFORMATION VALUE
ACIV quantifies the value of the decision-relevant information in signal V across the distribution of all possible realizations defined by the data gentrrannn model. To provide analogous instance-level quantification of information value, we define Instance-Level agent-complementtryy Information Value (IIIV) to quantify the value of the decision-relevant information encoded by single realization of the signal. This finer-grained view makes it possible to analyze how much an agent can benefit in theory from better incorporating instance-level information in their decision. For example, in the weather decision task, this indicates how much the payoff of the decision-maker can be improved by knowing the temperature is 21ºC
Given realization of signal gain from the access to v on the instances where is realized over the existing information encoded in agent decisions. Intuitively, this captures how much "room" there is for specific signal to be better used. Formally, given a decision task with payoff function S and information model æ, the expected payoff of the rational DM given signal V v' on instances with the signal realization V U is
{o),. (o).....,0 Jx ), we want know the maximum expected payoff
r (c') = E (w) (v)),
where d' (o") is the Bayesian optimal decision on receiving e' as defined in Equation 2 Note that we allow d' * v, i.e. the signal v' observed by the rational DM can be different from the actual realization v. The expected payoff of rational DM who is misinformed is guaranteed to be lower than is correctly informed, i.e., r" (0())) S (w) for any v'. This notion of flexibility allows us to consider the information value of counterfactual realization of signal. For example, in the weather decision task, this notation is able to describe how much payoff can change when the decision maker is misinformed that the temperature is 18 when the temperature is actually 21 C. We use this in designing the explanation of ILIV-SHAP in the next section.
P "Note that the algorithm (á, a²) should be cross-validated to avoid overfitting to the observed data and also be evaluated for calibration error since the rational DM treats it as the Bayesian posterior. We provide sensitivity analysis in Appendix where we compare the ACTV estimated under the rational helief estimator using linear regression (I.R), gradient boosting methods (GBM), and neural networks (NN).
5

---

Published as a conference paper at ICLR 2026
If we consider the agent decisions in addition to the realization v, the rational DM's expected payoff on instances where V can be written as
r" (v'; D*) (o)) wlle) ( S5 (dd"(v''' Ud"),w)
Definition 3.3. Given a decision task with payoff function S and an information model æ, we define the instance-level agent-complemenntar information value (ILIV) of signal realization V v on instances where V as:
ILIV (d'; D*) = : (v): r" (W;
(8)
where ("(0; D represents the expected rational payoff instances of V where the rational DM only knows the agent decisions without knowing any realizations of V. ILIV maximizes when the signal does misinform, i.e., ILIV" (v; DP) 2 ILIV" D*), for v' E.
INFORMATION BASED EXPLANATION
We define an information based explanation (ILIV-SHAP) to communicate where the Al prediction offers complementary information over the agent decisions. Traditional saliency-based explanations communicate the average contribution of each feature to a prediction over the baseline (prior) prediction, while ILIV-SHAP communicates the average contribution of each feature the agent- complementary information value contained in the prediction. Algorithm estimates the ILIV given of observed signals, agent decisions, and states.
Specifically, suppose a model f that, for example, takes as input m features and outputs real number. Given an instance ; Zm), the importance of one feature x; the model output f(x is encoded by the expected difference of model outputs when 2; is marginalized out. This is quantified by (x) (X) x-).. where X-, denotes all features except X,, Considering the interaction between features, SHAP (Lundberg & Leel (017) uses the Shapley value to quantify importance scores averaged on different combinations of features:
|x'][(m - xx| - 1)! [9)(x') (x((x(
Σ x'
where gy (x') denotes the expected output conditioned on x', i.c., E(f(X) = x for any x' C x where x' is the features that are not marginalized out.
Definition 4.1 (ILIV-SHAP). Given model f and data features x = (I1,..., Fm). ) the importance score of the i-th feature by ILIV-SHAP
x -x'\ 1)! m!
|x'I
!LLIV (f.x) = Σ
(g D*) (gg D
XCC
where ILIV (x); D*) denotes a counterfactual evaluation ILIV, which quantifies the expected payoff gain from additionally knowing gy (x') over 1 on the instances where the actual prediction is fx))
ILIV-SHAP shares the same properties as SHAP (Lundberg Lee 2017) by similarly constructing the importance scores. For example, ILIV-SHAP satisfies the critical efficiency axiom, i.e., the sum of the importance scores equals the information value of the model output, and the symmetry axiom, i.., the importance scores are the same for any two features that contribute the same amount to the information value of the model output.
Sample-hased methods for approximating SHAP values can also be applied to ILIV-SHAP, such as permutation sampling (Strumbelj & Kononenko 2010). Kernel SHAP (Lundberg & Lee 2017). and Partition SHAP (Lundberg et (2018). Because ILIV is expected be non-decreasing in the number of features included in g (x') "x''=more more features mean more ciermation--aaplee. methods can achieve better stability across the permutations on ILIV-SHAP than on SHAP.
5 EXPERIMENT
We use a preregistered between-subjects online experiment to answer two questions:
WWe note that IH.V does not resolve the foundational issues of SHAP (Huang & Marques-Silva 2024 but just as deranntration how the information value such II.VV he used in explanations.

---

Published as a conference paper at ICLR 2026
Feature name
Value
Impact on Al prediction
Value
Impact prediction
Feature Y
100
-$6.2K
Year Built
1982
-$11.5K
of Fireplaces
-$13.3K
Above Grade Living
Area
Built
1900
-$7.5K
of Firaplaces
0
-$10.3K
Car Capacity Garage
-$1.1K
Capacity Garage
2.0
Feature X
S80
+94.8K
Feature x
1400
50K
Above Grade Living
16276q
+51K
Area
Feature Y
100
S0K
Model prediction: $134.9K
Model prediction: $112.9K
Your provious guess: $160K
Your provious guess: $160K
Higlighted features are where the model gets significant
Highlighted features are whore the model gets significant
emount humon- complementory information.
amount of humon complementory information
(a) ATl with ILIV-SHAP and SHAP explanation.
(b) A12 with ILIV-SHAP and SHAP explanation.
The screenshot of the interface for ILIV-SHAP and SHAP explanations. The third column the SHAP value and the highlights are based on the ILIV-SHAP value. Because A12 does not have access to Feature X/Y, both their ILIV-SHAP and SHAP values are zero for A12.
Figure 1:
1. Can ACTV identify which AT model will result in the best human-Al decision-making?
2. Does ILIV-SHAP improve human-Al AI decision decision-making over SHAP alone?
Our experiment assigns participants to one of two Al models with varying ex-ante ACTV (All=high ACIV, Al2=low ACIV), and one of three explanation conditions { (ILIV-SHAP and SHAP, SHAP, No Explanation). We hypothesize that the higher ACIV model will lead to better human-Al decisions, and that presenting ILIV-SHAP explanations for this model will lead to better human-Al decisions than the other explanation types.
Task and data. We study Al-assisted house price prediction, following other recent crowdsourced al. (202T); Holstein eet al.] (2023), ). This task does not require any domain knowledge, making it suitable for a broad participant population. Additionally, the task is complex enough to benefit from Al assistance. We use the Ames, Iowa Housing Dataset (De Cock 2011).
studies (e.g.. Chiang & Yin (2021); Hemmer et al. (2022); (PoursazziSangdet
Participants and procedure. We recruited 421 US- participants via the crowdsourcing platform Prolific. Each participant was randomly assigned to 12 houses out of 733 in the Ames lowa Housing Dataset. Each completed a sequence of 24 trials, where two decisions were elicited for each house: one without Al assistance and one with. In the first 12 trials, participants were asked to predict the house price of the assigned house using six features of the house, including year built, the living area above ground in square feet, the size of garage in car capacity, the number of fireplaces, year remodeled, and ratings of the house's material and finish. We intentionally reduced the interpretability of the year remodeled and material and finish by relabeling them as "Feature X" and "Feature Y" and rescaling their values. Therefore, the Al model that takes these two features as input is expected to have complementary information over human participants. In the second 12 trials, participants were asked to revise their guesses in the first round after seeing the Al's prediction and explanation. We incentivized participants with a base reward of 3.00 USD and a bonus based on the mean squared error (MSE) between their guesses and the true house price in the 24 trials: bonus = 3.00
MSE
3x
AI models and explanations. Participants were assigned one of six experimental conditions resulting from crossing the two Al models with the three explanation types. We designed All and A12 to have varying potential to complement humans: All was trained with the six features and the true house price, with noise added to the six features (which is intended to lower the accuracy of All to be comparable with A12). On the other hand, A12 was trained with the four interpeatteeee features and the true house price, such that there would be less potential for the predictions to add complementary information over human judgments. We used the same training data for both AI models and ensured that the two AI models achieved similar performance (Figure 2 left), so that model accuracy differences would not confound our results. In the SHAP explanation, we used the SHAP values of the six features to explain the Al's prediction. To generate the ILIV-SHAP explanations, we used independent human decisions we collected on Prolific for the dataset using the same task and features prior to running the official study. We present the exact IHAV values
7

---

Published as a conference paper at ICLR 2026
Reduction in APE
10.0%
6.94% [6.5%, 7.38%]
7.5%
5.31%
5.88% [5.47%, 6.26%)
(5.71%,
6.22%
5.96% [5.5%, 6.42%%
6.42%
15.9%, 6.76%)
6.33%
[5.99%, 6.. 6.24%
6.24%
5.96%
14.8%, 5.83%]
5.0%
2.5%
A11
A12
0.0%
-LIV-SHAP
SHAP
None
Collapsing all explanations
Figure 2: Reduction the human-Al team's 's absolute percentage error (APE) with 95% confidence
intervals according to posterior predictions of our regression model.
Table I: The two Al models that are used in the experiment. All has access to all the features, while AI2 has access only the human- inttaannnterrretable features.
Input features
MAPE
R squared
ACIV (in MSE)
ACIV (in MAPE)
ATI
All features
14.30%
0.81
6.5
10
4.61%
A12
Human-interpretable features
14.51%
0.81
3.7 x 10
2.00%
of the six features, which ranked and highlighted the features where the human- numan-complemenntry information (ILIV) over the AI's prediction exceeds threshold See Appendix the screenshots of the explanations and instructions.
Evaluation metries. We report the ACIV of the AI models in mean squared error (MSE 6 and mean absolute percentage errоr (MAPE). To measure the human alone and human-AI team performance, we fit preregistered Bayesian hierarchical regression model with weakly informed priors to the percentage error (PE) between the human's prediction and the true house price, PE w) //w using R's brms package (Bürkner 2017).
PE ~ poudent_t((,,00
μ1 Al explanation + round participant _dd log(a) ) = Al explanation + round (1 participant_ii id)
Gamma (210
where AI and explanation are indicators of the experimental conditions, round is an indicator of whether the trial is in the first or second round, and participant_ sippaaattiddd is a unique identifier for each participant. We evaluate human-Al team performance by the reduction in absolute per- centage eтror (APE) over human-alone predictions in the first round, i.e., Reduction in APE E|PE]| round = 1] E| round 2). where the expectation is taken over the Bayesian posterior distribution of the above Bayesian model.
Results. T the expected ranking of models by ACIV. AI1, which had access to the human--ninterppetablle features, achieves higher ACIV than A12, which only had access the four human-interprettblle features. Note the similar predictive performance of the two Als.
The posterior predictions of human-AI team performance from regression model also align with expectations based on the complementary information each model provides (Figure far
We choose the threshold to highlight the features that can lead to least $0.25 boost the bonus to participants, which is translated into 7.5x 10 MSE as the threshold for ILIV-SHAP values. We included MSE since it is proper scoring rule while MAPE is not. htttss://sspreet cted. .ereedorrg/.. org/ fm6 wtw.mm.rrr...... pdd
8

---

Published as a conference paper at ICLR 2026
ADRVD
wiyC
De Tootree
Nut Informationn
Full Follfooomation
Du Mikeen hcs
on Nedeee
Dudko
Dags
(a) Chest X-ray diagnosis results.
(b) Deepfake detection results
Figure 3: Distributions show information values. For a), we plot ACIV of radiologist decisions, different choices of AI models and signals that combine Al predictions and radiologist decisions. For b), we plot the information value of the combination of video-level features and agent decisions (including human decisions, AI predictions, and human-AI teams' decisions). Full information represents all the information available to the human decision- the radiology images for the chest X-ray diagnosis task and the seven video-level features, human decisions, and Al predictions for the deepfake detection task. See Appendix forr the sensitivity analysis of Algorithm
right). Collapsing across explanation conditions, All (higher ACIV) reduces APE by more than A12 (6. 26.2%% 5. 55.99%,6655 6.5%) vs 5.96%(..68%, 6.24%] respectively). Access to the ILIV-SHAP + SHAP explanation results in greater reduction in APE for All than A12 (6.94% (6.94%/6.00%%, (6.50%, 7.38%] vs 5.31% (4.80%%..... 5.83%) respectively
Figure 2 also illustrates the effect of the different explanation conditions on human-AI team per- formance. When the AI has sufficient complementing information (AII), the ILIV-SHAP + SHAP explanation more effectively reduces APE for the human-AI decisions than the SHAP explanation and baseline (no explanation) (6. (6.94%/6.50% 7.38%] for ILIV-SHAP, 5.88%[ 47%, 6.28%] for SHAP, and 5. 5.96%[5.50%%) 6.42%] for no explanation).
6 DEMONSTRATIONS
6.1 CHEST X-RAY DIAGNOSIS
We study a chest X-ray diagnosis task. As agent decisions, D and DH, we consider five predictive models fine-tuned on the MIMIC-CXR database (Johnson et al. 2019) and radiologists textual reports recorded in MIMIC-XXR. We use five pretrained image models with the same choices in Irvin al. 2019). We train the models on 12,228 radiographs. and validate on 6,115 randomly sampled radiographs. For the payoff-relevad state, w E Ω = {0, 1}. we used results from two types of blood tests ('NT-proBNP' and 'troponin' cut by age-specific thresholds). The decision task formalized as prediction problem with D AΩ and S(d,
- (d - w)²
Can the AI models complement human judgment? Figure Ba shows all the models offer comple- mentary information value to the radiologists' reports (green distributions improve over the purple distribution), and in the other direction, the radiologists' reports offer complementary information value to the models (green distributions improve over the orange distributions).
Which AI model offers the most decision-relever information over human judgments? Fig- ure 3a shows that Vision Transfrrerrr contains slightly higher information value than the other models, and Inception v3 contains slightly lower information value than the other models. We further assess superiority over the other AI models across many possible payoff functions to test if there is a Blackwell ordering of models in Appendix H and Figure/6)
the robustness of
As shown in this demonstration, doctors may use framework to learn how much complementary information value the Al models offer over their decisions, and which model offers the most.
6.2 DEEPFAKE DETECTION
We study a deepfake detection task (Dolhansky et al.) 2020 Groh et al. 2022 We select the model in the Deepfake Detection Challenge (Zhang et al. 2016). with estimated 65% accuracy on holdout data. We use the human decisions and the human- Al team's 's decisions from Groh et al. (2022), who collected judgments on the videos from n=5.524 participants recruited on Prolitic. We use the Brier score as the payoff function, with the binary payoff-related related state: E {0, 1 (genuine, fake). This
*See Appendix P for the significant results

---

Published as a conference paper at ICLR 2026
choice differs from Groh et al. (2022) 's of mean absolute error, but again we prefer the quadratic score because it is a proper scoring rule where truthfully reporting beliefs maximizes the score. [Groh et al. (2022) manually label seven video-level features, which use as binary indicators in place of Algorithm| Din in light of the high dimensional signals: graininess, blurriness, darkness, presence of flickering face, presence of two people, presence of a floating distraction, and the presence of an individual with dark skin We show the ACIVs of the seven features over the decisions, the AI predictions, and the human-Al decisions in Figure 3b
How much decision-relevant information do agents' decisions offer? We first compare the information value of the AI predictions to that of the human decisions. Figure 3b shows that AT predictions provide about 65% of the total possible information value over the no-information baseline, while human decisions only provide about 15%. We next consider the human-Al decisions. Given that the AI predictions contain a significant portion of the total possible information value, we might hope that when participants have access to the AI predictions, their performance will be close to the full information baseline. However, the information value of the human-AI decisions only achieves a small proportion of the total possible information value (30%).
How much additional decision- rrelevnt information do the available features offer over agents' decisions? To understand what information might improve human decisions, we assess the ACTVs of different video-level features over different agents. As shown on the fifth row in Figure 3b] the presence of flickering face offers larger ACIV over human decisions than over AI predictions, meaning that human decisions could improve by a greater amount if they were to incorporate this information. Meanwhile, as shown on the fourth row in Figure 3b] the presence of an individual with dark skin offers larger ACIV over Al predictions than over human decisions, suggesting that humans make greater use of this information. This suggests that the Al and human rely on differing information to make their initial predictions 10
DISCUSSION AND LIMITATIONS
Our -theoreiccc framework quantifies the information value of signals available in a human-AI decision setting over the information value of agent decisions. Importantly, the basis of our frame- work in Bayesian decision theory does not require that actual (c.g., human) decision- Bayesian rational decision-making
achieve
Rather, it provides a theoretical basis support comparisons to human behavior to drive learning (see, e.g.. (Guo et al., 2024] Hullman & Gelman 2021 Wu et al.)
2023). This theoretical basis is necessary establish well-defined benchmarks. We experimentally demonstrate the power of the framework for analyzing and improving human-Al decision-making, Our proposed ILIV-SHAP explanatxiiiiiii explanation improves performance over an existing sfuee-o-thhe-ee expla- nation strategy. This suggests that communicating the value of information that an Al model used to human facing a decision problem improves the efficacy of explanations over simply explaining how the AI arrived at its prediction. Our work demonstrates new direction in explanation design grounded in decision theory (e.g.. Huulmmn et al. (205))
Our work provides a promising methodological framework for an emerging research agenda around optimally combining agents' information for decisions (e.g.. Alur et al. (2023) ). While we focused on human-Al decisions, the framework can be applied to any combination of Al human agent judgments. Though most of our definitions and analysis are focused on decision tasks with single well-defined payoff function, the framework tasks with an ambiguous set of payoff functions or totally unidentifiable payoff functions. We present robustness analysis framework in Appendix B to calculate the Blackwell ordering of models over proper scoring rules. A demonstration of the robustness analysis is shown in Appendix H
be readily extended to more complex decision
The experiment is designed to provide complementarity by construction in order to function proof of concept. We believe future work is needed to evaluate the impact of ILIV-SHAP in real deployment of human-AI decision-makin workflows, such where the human Al might have additional private information over each other.
9The "dark-skin individual" label reflects a subjective visual attribute. It is used here only for robustness analysis-not not for causal interpretation or demographic inference. For space constraints, we refer to the full descriptions and results in Appendices G and
10

---

Published as a conference paper at ICLR 2026
8 REPRODUCIBILITY STATEMENT
The main results of our paper are to provide a framework guiding the analysis of complementary information in human-Al Decision-making We provide a Python library supplementary material to calculate the quantities defined in our framework.
We provide the data and code (which are all put in separate Jupyter notebooks) reproduce all the results in our paper, including the demonstrations and the empirical study in the main text, and the observational study and robustness analysis the Appendix.
ACKNOWLEDGEMENTS
Thanks to Advik Shreekumar for his help on using MIMIC-CXR and MIMIC-IV datasets. Thanks to the ICLR reviewers for helpful suggestions (specifically, to include sensitivity analysis and cognitive load).
REFERENCES
Stefanía Eggisóóttir, Michael J White, Paul M Spengler, Alan S Maugherman, Linda A Anderson, Robert S Cook, Cassandra Nichols, Georgios K Lampropoulos, Blain S Walker, Genna Cohen, et al. The meta-analysis of clinical judgment project: Fifty-six years of accumulated research clinical versus statistical prediction. The Counseling Psychologist, 34(3):341-388 2006.
Mayank Agrawal, Joshua C Peterson, and Thomas L Griffiths. Scaling up psychology via scientific regret minimization. Proceedings of the National Academy of Sciences, 117(16)8855-8835, 2020.
Rohan Alur, Loren Laine, Darrick Li, Manish Raghavan, Devavrat Shah, and Dennis Shung. Auditing for human expertise. Advances in Neural Information Processing Systems, 36.79439-79468, 2023.
Rohan Alur, Manish Raghavan, and Devavrat Shah. Distinguishing the indistinguishable: Human expertise in algorithmic prediction. arXiv preprint arXiv:2402.00793, 2024.
Adrian Ardaiz-Rodriguez, Nina Corvelo Benz, Suhas Thejaswi, Nuria Oliver, and Manuel Gomez-Rodriguez Towards human-ai complementarity in matching tasks. arXiv preprint arXiv:2508. 13285, 2025.
Gagan Bansal, Besmira Nushi, Fce Kamar, Daniel S Weld, Walter Lasecki, and Eric Horvitz. Updates in human-ai teams: Understanding and addressing the perforaancecompattiilty tradcoff. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 2429-2437, 2019.
Gagan Bansal, Besmira Nushi, Ece Kamar, Eric Horvitz, and Daniel S Weld. Is the most accurate ai the best teammate? optimizing ai for teamwork. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 11405-11414, 2021a.
Gagan Bansal, Tongshuang Wu, Joyce Zhou, Raymond Fok, Besmira Nushi, Ece Kamar, Marco Tulio Ribeiro, and Daniel Weld. Does the whole exceed its parts? the effect of ai explanations on complementary team performance. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. CHI '21, New York, NY, USA, 2021b. Association for Computing doi: 11144/341174.34457111 URL htt.sttps::/// //tp:::://d..... /rg/ 10. 1 45/ 764 3445717
Machinery. ISBN 9781450380966.
Eli Ben-Michael, D James Greiner, Melody Huang, Kosuke Imai, Zhichao Jiang, and Sooahn Shin. Does ai help humans make better decisions? a statistical evaluation framework for experimental and observational studies. arXiv, 2403:v3, 2024.
David Blackwell et al. Comparison of experiments. In Proceedings the second Berkeley symposium on mathematical statistics and probability, volume 1, pp. 26, 1951.
Zi-Hao Bo, Hui Qiao, Chong Tian, Yuchen Guo, Wuchao Li. Tiantian Liang, Dongxue Li. Dan Liao, Xianchun Zeng, Leilei Mei, et al. Toward human intervention- clinical diagnosis of intracranial aneurysm via deep neural network. Patterns, 2(2), 2021.

---

Published as a conference paper at ICLR 2026
Elizabeth Bondi, Raphael Koster, Hannah Sheahan, Martin Chadwick, Yoram Bachrach, Taylan Cemgil, Ulrich Paquet, and Krishnamurthy Dvijotham. Role of human-ai interaction in selective prediction. In Proceedings of the AAAI Conference Artificial Intelligence, volume 36, pp. 5286-5294, 2022.
Melanie M Boskemper, Megan L Bartlett, and Jason S McCarley. Measuring the efficiency of automation-aided performance simulated baggage screening task. Human factors, 64(6): 945-961, 2022.
Zana Buçinca, Phoebe Lin, Krzysztof Z. Gajos, and Elena L. Glassman. Proxy tasks and subjective measures can be misleading in evaluating explainable ai systems. In Proceedings of the 25th International Conference on Intelligent User Interfaces, IUI '20, pp. 454-464, New York, NY. USA. 2020. Association for Computing Machinery. ISBN 978 9781450317 14503711866 doi: 10.1145/3377325.
3377498. Jtthtss::://dd
org 1145/3377325 3t 3377498
Paul-Christian Bürkner. brms: An package for bayesian multilevel models using stan. Journal of statistical software, 80:1-28, 2017. 2017.
Adrian Bussone, Simone Stumpf, and Dympna O'Sullivan. The role of explanations on trust and reliance in clinical decision support systems. In 2015 International Conference on Healthcare Informatics, pp. 160-169, Oct 2015. doi: 10.1109/9C..2215...6.
Chacha Chen, Shi Feng, Amit Sharma, and Chenhao Tan. Machine explanations and human understanding (2022). URL: http://arxxv. org/abs/2202.044 042222222 2022.
Yiling Chen and Bo Waggoner. Informational substitutes. 2016 IEEE 57th Annual Symposium on Foundations of Computer Science (FOCS), pp. 239-247. IEEE, 2016.
Chun- Chun-Wei Chiang and Ming Yin. You'd better stop! understanding human reliance on machine learning models under covariate shift. In Proceedings of the 13th ACM Web Science Conference 2021, pp. 120- 129, 2021.
Nina Corvelo Benz and Manuel Rodriguez. Hlmann-lignned calibration for ai-assisted decision making. Advances in Neural Information Processing Systems, 36: 16644946-4463,, 2023.
Dean De Cock. Ames, iowa: Alternative to the boston housing data as an end of semester regression project. Journal of Statistics Education, 19(3), 2011.
Giovanni De Toni, Nastaran Okati, Suhas Thejaswi, Eleni Straitouri, and Manuel Rodriguez. Towards human-ai complementarity with prediction sets. Advances in Neural Information Processing Systems, 37:31380-31400 2024.
Brian Dolhansky, Joanna Bitton, Ben Pflaum, Jikuo Lu, Russ Howes, Menglin Wang, and Cristian Can- ton Ferrer. The deepfake detection challenge (dfdc) dataset. arXiv preprint arXiv:2006. 07397, 2020.
Alexey Dosovitskiy. An image is worth 16x 16 words: Transformers for image recognition at scale. arXiv preprint arXiv: 2010. 1929, 2020.
Drew Fudenberg, Jon Kleinberg, Annie I Liang, and Sendhil Mullainathan. Measuring the completeness of economic models. Journal of Political Economy, 130(4):956-990, 2022.
Ary L. Goldberger, Luis AN Amaral, Leon Glass, Jeffrey Hausdorff, Plamen Ivanov, Roger Mark, Joseph E Mietus, George B Moody, Chung-Kan Peng, and H Eugene Stanley. Physiobank, physiotoolkit, and physionet: components new research resource for complex physiologic signals. circulation, 101(23):e215-c22) 2000.
Ben Green and Yiling Chen. The principles and limits of algorithm-in-the-le decision making. Proceedings of the ACM on Human-Compptee Interaction, 3(CSCW):1-24, 2019.
Matthew Groh, Ziv Epstein, Chaz Firestone, and Rosalind Picard. Deepfake detection by human crowds, machines, and machine-informe crowds. Proceedings of the National Academy of Sciences, 119(1)=22 10013119, 2022.
12

---

Published as a conference paper at ICLR 2026
William M Grove, David H Zald, Boyd S Lebow, Beth E Snitz, and Chad Nelson. Clinical versus mechanical prediction: A veta--nnalysi.. Psychological Assessment, 12(1):19, 2000.
Ziyang Guo, Yifan Wu, Jason D Hartline, and Jessica Hullman. A decision theoretic framework for measuring reliance. In The 2024 ACM Conference on Fairness, Accountability, and Transparency, pp. 211-222-226 2024.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Paul A Heidenreich, Biykem Bozkurt, David Aguilar, Larry A Allen, Joni Byun, Monica M Colvin, Anita Deswal, Mark H Drazner, Shannon M Dunlay, Linda R Evers, et al. 2022 aha/acc/hfssa guide- line for the management of heart failure: a report of the american college of cardiology/amerrcan heart association joint committee clinical practice guidelines. Journal of the American College 2022.
of Cardiology, 79(17):2233-e)
Patrick Hemmer, Max Schemmer, Niklas Kühl, Michael Vössing, and Gerhard Satzger. On the effect of information asymmetry human-ai teams. arXiv preprint arXiv:2205.0 04467, 2022.
Kenneth Holstein, Maria Dr-Arteaga, Lakshmi Tumati, and Yanghuidi Cheng. Toward supporting perceptual complementarity in human-ai collaboration via reflection on unobservables. Proceedings of the ACM on Human-Compute Interaction, 7CCCWW 1::1200-22 2023.
Lunjia Hu and Yifan Wu. Predict to minimize swap regret for all payoff-bounded tasks. arXiv preprint arXiv:2404. 13503, 2024.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7700-4708, 2017.
Xuanxiang Huang and Joao Karques-Silva. On the failings of shapley values for explainability. International Journal of Approximate Reasoning, 171:109112. 2024.
Jessica Hullman and Andrew Gelman. Designing for interactive exploratory data analysis requires theories of graphical inference. Harvard Data Science Review, 333)10--162, 2021.
Jessica Hullman, Ziyang Guo, and Berk Ustun, Explanations are means to an end. arXiv preprint arXiv:2506.22744 2025.
Jeremy Irvin, Pranav Rajpurkar, Michael Ko, Yifan Yu, Silviana Ciurea- Chris Chute, Henrik Marklund, Behzad Haghgoo, Robyn Ball, Katie Shpanskaya, et al. Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison. Proceedings of the AAAI conference on artificial intelligence, volume 33, pp. 590-597, 2019.
Maia Jacobs, Melanie F Pradier. Thomas McCoy Jr, Roy H Perlis, Finale Doshi- Velez, and Krzysztof Z. Gajos. How machine-learning recommendations influence clinician treatment selec- tions: the example of antidepressant selection. Translational psychiatry, 11(1):108, 2021.
Alistair EW Johnson, Tom Pollard, Seth Berkowitz, Nathaniel R Greenbaum, Matthew P Lungren, Chih-ying Deng, Roger G Mark, and Steven Horng. Mimic-cxr, de-identified publicly available database of chest radiographs with free-text reports. Scientific data, 6(1):317,2) 2019.
Alistair EW Johnson, Lucas Bulgarelli, Lu Shen, Alvin Gayles, Ayad Shammout, Steven Horng, Tom J Pollard. Sicheng Hao, Benjamin Moody, Brian Gow, et al. Mimic-iv, a freely accessible electronic health record dataset. Scientific data, 10(1):1, 2023.
Vijay Keswani. Matthew Lease, and Krishnaram Kenthapadi. Towards unbiased and accurate deferral to multiple experts. In Proceedings of the 2021 AAAI/ACM Conference on Al, Ethics, and Society, pp. 154-165, 2021.
Vijay Keswani, Matthew Lease, and Krishnaram Kenthapadi. Designing closed human- in the-loop deferral pipelines. arXiv preprint arXiv:2202.047/38 2022.
13

---

Published as a conference paper at ICLR 2026
Bobby Kleinberg, Renato Paes Leme, Jon Schneider, and Yifeng Teng. U-calibration: Forecasting for an unknown agent. In The Thirty Sixth Annual Conference on Learning Theory, pp. 5143-51444 PMLR. 2023.
Jon Kleinberg, Jens Ludwig, Sendhil Mullainathan, and Ziad Obermeyer. Prediction policy problems. American Economic Review, 105(5):911-495, 2015..
Jon Kleinberg, Himabindu Lakkaraju, Jure Leskovec, Jens Ludwig, and Sendhil Mullainathan. Human decisions and machine predictions. The quarterly journal of economics, 133(1):237-293, 2018.
Igor Kononenko. Machine learning for medical diagnosis: history, state the art and perspective. Artificial Intelligence in medicine, 23(1):89-109 2001.
Vivian Lai and Chenhao Tan. On human predictions with explanations and predictions of machine learning models: A case study on deception detection. In Proceedings of the conference on fairness, accountability, and transparency. pp. 29-38, 2019.
Himabindu Lakkaraju, Jon Kleinberg, Jure Leskovec, Jens Ludwig, and Sendhil Mullainathan. The selective labels problem: Evaluating algorithmic predictions in the presence of unobservables. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 275-284, 2017.
Yingkai Li, Jason D Hartline, Liren Shan, and Yifan Wu. Optimization of scoring rules, In Proceedings of the 23rd ACM Conference on Economics and Computation, pp. 988-989, 2022.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows, In Proceedings of the IEEE/CVF international conference on computer vision, pp. 10012- 2021.
Scott M Lundberg and Su-In Lee, A unified approach to interpreting model predictions. In Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Gamett (eds.), Advances in Neural Information Processing Systems 30, pp. 4765-4774. Curran Associates, Inc., 2017.
Scott M Lundberg, Gabriel G Erion, and Su-In Lee. Consistent individualized feature attribution for tree ensembles. arXiv preprint arXiv: 100.xv8002.0:00 888.00888.. 2018.
David Madras, Toni Pitassi, and Richard Zemel. Predict responsibly: improving faimess and accuracy by learning defer. Advances in neural information processing systems, 31, 2018.
Bryce Melaughlin and Jann Spiess. Algorithmic assistance with recommendaiion-deeendenn prefer- ences. In Proceedings of the 24th ACM Conference Economics and Computatioon EC '23, pp. 991, New York, NY, USA, 2023. Association for Computing Machinery. ISBN 9798400701077, doi: 10. 4455/558050735597773
BRRlatts:://dee org 11.00...44451 35805377 11111/1.111113350551110 3597775
Paul E Meehl. Clinical Versus Statistical Prediction: Theoretical Analysis and a Review of the Evidence. University of Minnesota Press, 1954.
Hussein Mozannar, Gagan Bansal, Adam Fourney, and Eric Horvitz. When to show suggestion? integrating human feedback in ai-assisted programming. In Proceedings of the AAAI Conference Artificial Intelligence, volume 38, pp. 10137-10144, 2024a.
Hussein Mozannar, Jimin Lee, Dennis Wei, Prasanna Sattigeri, Subhro Das, and David Sontag. Effective human-ai teams via learned natural language rules and onboarding. Advances in Neural Information Processing Systems, 36, 2024b.
Christian Mueller, Kenneth McDonald, Rudolf A de Boer, Alan Maisel, John GF Cleland, Nikola Kozhuharov, Andrew JS Coats, Marco Metra, Alexandre Mebazaa, Frank Ruschitzka, et Heart failure association of the european society of cardiology practical guidance on the use of natriuretic peptide concentrations. Eurppean journal of heart failure, (1(6):715-731, 2019.
Sendhil Mullainathan and Ziad Obermeyer. Diagnosing physician error: A machine learning approach to law-value health care. The Quarterly Journal of Economics, 137(2):679-727, 2022.
14

---

Published as a conference paper at ICLR 2026
Nastaran Okati, Abir De, and Manuel Rodriguez. Differentiable learning under triage. Advances in Neural Information Processing Systems, 34:9140-9133 11141, 2021.
Samir Passi and Mihaela Vorvoreanu. Overreliance ai literature review. Microsoft Research, 339: 340, 2022.
Forough Boursabzi-Sangdeee Daniel G Goldstein, Jake M Hofman, Jennifer Wortman Vaughan, and Hanna Wallach. Manipulating and measuring model interpretability. In Proceedings of the 2021 CHI conference on human factors in computing systems, pp. 1-52, 2021.
Maithra Raghu, Katy Blumer, Greg Corrado, Jon Kleinberg, Ziad Obermeyer, and Sendhil Mul- lainathan. The algorithmic automation problem: Prediction, triage, and human effort. arXiv preprint arXiv:1903.1 122200, 2019.
Pranav Rajpurkar, Jeremy Irvin, Robyn L Ball, Kaylie Zhu, Brandon Yang. Hershel Mehta, Tony Duan, Daisy Ding, Aarti Bagul, Curtis P Langlotz, et al. Deep learning for chest radiograph diagnosis: A retrospective comparison of the chexnext algorithm to practicing radiologists. PLOS medicine, 15(1 (5(11):e102268, 2018.
Ashesh Rambachan. Identifying prediction mistakes in observational data. The Quarterly Journal of Economics, pp. qjae013, 2024.
Charvi Rastogi, Liu Leqi, Kenneth Holstein, and Hoda Heidari. A taxonomy of human and ml strengths in decision-maker to investigate human-ml complementarity. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, volume 11, pp. 127-139, 2023.
Aaron Roth and Mirah Shi. Forecasting for swap regret for all downstream agents. In Proceedings of the 25th ACM Conference on Economics and Computation, pp. 466-488, 2024.
Max Schemmer, Patrick Hemmer, Maximilian Nitsche, Niklas Kühl, and Michael Vössing. A meta-analysis of the utility explainable artificial intelligence in human-ai decision-making In Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 617-626, 2022.
Lloyd S Shapley. A value for n-person games. Contributions to the Theory of Games, 2, 1953.
Advik Shreekumar. X-raying experts: Decomposing predictable mistakes in radiology. 2025.
Eleni Straitouri, Lequn Wang, Nastaran Okati, and Manuel Gomez Rodriguez. Improving expert predictions with conformal prediction. In International Conference on Machine Learning, pp. 32633-32653, PMLR, 2023.
Erik Strumbelj and Igor Kononenko. An efficient explanation of individual classifications using game theory. The Journal of Machine Learning Research, 11:1-18, 2010.
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2818 1-88222262111 2016.
Yu-Xing Tang, You-Bao Tang, Yifan Peng, Ke Yan, Mohammadhadi Bagheri, Bernadette A Redd, Catherine Brandon, Zhiyong Lu, Mei Han, Jing Xiao, et Automated abnormality classification of chest radiographs using deep convolutional neural networks. NPJ digital medicine, 3(1):70, 2020.
Michelle Vaccaro and Jim Waldo. The effects of mixing machine learning and human judgment. Communications of the ACM, 62(11):104-110, 2019.
Michelle Vaccaro, Abdullah Almaatouq, and Thomas Malone. When combinations of humans and ai are useful: A systematic review and meta-analysis. Nature Human Behaviour, pp. 1-11, 2024.
Kailas Vodrahalli, Roxana Daneshjou. Tobias Gerstenberg, and James Zou. Do humans trust advice more comes from ai? an analysis of human-ai interactions. In Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 763-777, 2022.
15

---

Published as a conference paper at ICLR 2026
Bryan Wilder, Eric Horvitz, and Ece Kamar. Learning to complement humans. In Proceedings of the Twenty-Ninth International Conference on International Joint Conferences on Artificial Intelligence, pp- 1526-1533, 2021.
Yifan Wu, Ziyang Guo, Michalis Mamakos, Jason Hartline, and Jessica Hullman. The rational agent benchmark for data visualization. IEEE transactions on visualization and computer graphics, 2023.
Kaipeng Zhang, Zhanpeng Zhang, Zhifeng Li, and Yu Qiao. Joint face detection and alignment using multitask cascaded convolutional networks. IEEE signal processing letters, 23(10): 1499-1503, 2016.
16

---

Published as a conference paper at ICLR 2026
A THE COMBINATORIAL NATURE OF THE VALUE OF SIGNALS
When decision-maken are provided with multiple signals, the signals have the combinatorial property by nature. Acknowledged by recent works in decision theory and game theory (Chen & Waggoner 2016), one signal may have no information value by itself, but it can be complementary to other signals to provide information value. For example, two signals Σn and Σg are uniformly random bits and the state = Σn ① Σz, the XOR of Σ1 and Σg. In this case, neither of the signals offers information value on its own, but knowing both leads the maximum payoff. Though we did not explicitly observe the complementation between signals in our survey of human-Al decision-making tasks (see the results of deepfake detection in supplementary materials), we want note that our framework can be extended to consider this complementation between signals. We use the Shapley value (Shapley 1953) to interpret the contribution to the ACIV of each basic signal. & is the average of the marginal contribution of signal in every combination with other signals.
E ("I') !)
(n
o(V) = n E.JV
(ACIVT.S (V'UV;:") -VCIVT (9)
The following algorithm provides polynomial-time approximation of the Shapley value of ACIV. Under the assumption of submodularity, it orders the signals the same as the Shapley value.
Algorithm 2 Greedy algorithm for marginal gain of ACTV
1: V = =DD*
2: 4* = [}
3: for 1 n do
4: Φ ACIV(E,; V*) for each j
5: j arg max, Æ 0/
6: Dj. = max, s.l. vv 0f
7:
add E,. to V
8: add 6y- to $*
9: end for
10: output 05-
B ROBUST ANALYSIS OF INFORMATION ORDER
Our approach assumes a decision problem input and evaluates agents" decisions and use of information on this problem. However, evaluators may face ambiguity around the appropriate decision problem specification, and in particular, the appropriate scoring rule. In particular, ambiguity can arise in payoff functions; doctors, for example, penalize false negative results differently when diagnosing younger versus older patients (Mclaughlin & Spiess 2023). Blackwell's comparison of signals (Blackwell et al.) 1951) is an ideal tool for addressing ambiguity about the payoff function, as it defines signal V1 more informative than V22 if V1 has higher information value all possible decision problems. We identify this partial order by decomposing the space of decision problems via basis of proper scoring rules (Li et al. 2022) Kleinberg al., 2023).
Definition B.1 (Blackwell Order of Information). A signal Vi is Blackwell more informative than V/2 11 achieves a higher best-attainable payoff on any decision problems:
R (Vi) 2 R=.S(V)) (V2),
where (V) denotes the expected performance of the rational DM payoff function S and information structure æ when observing V.
The Blackwell order is evaluated over all possible decision problems, which cannot be tested directly. Fortunately, we only need to test over all proper scoring rules since any decision problem can be represented by equivalent proper scoring rule, and the space of proper scoring rules be characterized by a set of V-shaped scoring rules. A V-shaped scoring rule is parameterized by the kink of the piecewise-lined utility function.
17

---

Published as a conference paper at ICLR 2026
Definition B.2. (V-shaped scoring rule) A V-shaped scoring rule with kink μ (0, 3 is defined as
w {
2/
i
else,
When yí' E (2: 1), the V-shaped scoring rule can be symmetrically defined by Sμ = 11- y, w).
Intuitively, the kink j represents the threshold belief where the decision-makes switches between two actions. Larger y means that the decision-maken will prefer more. The closer 1a 0.5, the more indifferent the decision-maker is to d = 0 or d 1.
Proposition B.3 shows that V1 achieves higher information value on the basis of V-shaped proper scoring rules than 22, then Vi is Blackwell more informative than V2. Proposition B.3 follows from fact that any best-responding payoff can be linearly decomposed into the payoff on V-shaped scoring rules.
Proposition B.3 (Hu Wu 2024). If (0,1)
R**S.(() 2 R (*2)
then V1 is Blackwell more informative than V/-
Extending this to ACIV, V1 offers higher complementary value than 1/2 under the Blackwell order ACIV=.5M (V1; 2 ACIVT.5, (V/2; D*), Vp (0,1)
This definition allows us to rank signals (or sets of signals) without needing to commit to a specific payoff function. We present case in Appendix G
C ALGORITHM TO ESTIMATE ILIV
Algorithm 3 A method to calculate the ILIV of v' on the instances of
1: Input: Observed realizations (vx, dy wi M test counterfactual signal realization v', test signal realization predictive algorithm A. payoff function S, decision space D, and state space f.
2: Output: ILIV" (v'; D*)
3: à + A((u; di w 1
4: a + A({d wi }=
5: (i1, .a....,Jkk]
the indices of instances where Uy for any E [k]
6: for = to k do
7: P: â(v', d)))
8:
d arg maX Eu (Sd,
9:
P a² (dd)))
10: deh t arg max E Bunpy(((,ww)
11: 8; - S(d wi) - S(() wi)
12: end for
13: return Σl=
D OBSERVATIONAL STUDY ON THE EFFECT OF ACIV ON HUMAN- AI TEAM PERFORMANCE
In this section, we present the results of an observational study on the effect of ACIV on human-AI team performance using the dataset from Vodrahalli et al. (2022).
Data description. We experiment with the publicly available Human-Al Interactions dataset (Vodrahalli et al. 2022). The dataset comprises 34,783 unique predictions from 1,088 different human participants on four different binary prediction tasks ("Art", "Sarcasm". "Cities" and "Census") In each of the tasks, human participants provide confidence values about their predictions before (d" and after + receiving Al advice from a classifier in the form of the classifier's confidence values (d.l). dz AAI AS and d'l are in the range of [0, 1).
18

---

Published as a conference paper at ICLR 2026
Results. We present the ACIV of the Al confidence values (dA1) over human decisions (d") measured by the payoff function as the mean squared error (MSE) in Figure[4 and show the reduction of the mean squared error (MSE) of the human-Al team over the human-alowe baseline in Table We find that hetween these four tasks, the ACIV of the Al over human decisions predicts the improvement of the human-Al team performance the human-alone baseline.
Humman-complementtrryyyyyy information of different
0.175
T
0.150
0.125
0.100
0.050
0.025
0.000
cires
sarcasm
cansus
Figure 4: The ACIV of the Al confidence values human decisions for different tasks. The bars are 95% confidence intervals.
Table 2:
The ACTV of the Al confidence values over human decisions and the reduction of mean squared error (MSE) of the human-Al Al team over the human-alone baseline for different tasks.
Task
Human-complemenntary Info
Human MSE
Human+Al MSE
Reduction in MSE
Art Cities
0.1772 [0. 1747, 0.1798)
0.243
0.1847
0.0583
0.0919 [0.0860, 0.0978]
0.1955
0.1615
0.034
Sarcasm
0.1625 [0. 0.1662]
0.2137
0.1767
0.037
Census
0.1147 [0.1077, 0.1217]
0.1997
0.1861
0.0136
E STATISTICAL TESTS ON EXPERIMENT RESULTS
We calculate the p-values of the significance tests with Welch's 's t-test, with a = 0.05. The degrees of freedom (v) of Welch's t-test calculated with the following formula:
N2
+
N₂
(10)
x7 NF
2
(N
where 81 and &2 are the standard deviations of the two samples, and N₁ and N2 are the sizes of the two samples (see sample sizes and standard deviations in T5. We used Bonferroni correction as the multiple comparisons correction to control the overall type error rate. The results are shown in Table 4
19

---

Published as a conference paper at ICLR 2026
Table 3: 95% confidence intervals for the reduction of APE with different AI models and explanations in Figure
No explanation
SHAP
ILIV-SHAP + SHAP
5.96%
5.88% 5.47%, 6.28%]
6.94%
All
5.50%, 6. 6.42%]
66.5%% 7.38%]
6.33%
6.22%
5.31% |4.80% 5.83%]
A12
5.90% 6.76%]
5. 5.71%, 6.72%]
Table 4: P-values for significance tests reduction in APE between experimental conditions (after Bonferroni correction).
Null Hypothesis (Ha)
P-value
AII + ILIV-SHAP + SHAP AI1 + SHAP
0.002
All + ILIV-SHAP + SHAP < All + No explanation
0.010
All < A12
0.555
All + ILIV-SHAP + SHAP < A12 + IIIV-SHAP SHAP
<0.001
COGNITIVE LOADS AND ANCHORING EFFECTS IN EXPERIMENT
We also examine the spent time on task and the degree of anchoring on the human versus Al decisions by looking at the distance between human-Al team decisions and Al or human-aloo decisions.
Time spent. Table 6 shows that there is no significant increase in the duration of the experiment for ILIV-SHAP (23.3 [13.1, 32.1] minutes for All with ILIV-SHAP and 22.2 [11.9, 19, 28.0] minutes for A12 with ILIV-SHAP, where the square brackets show the 25% and 75% quantiles).
Anchoring effects. Table alaso shows that ILIV-SHAP does not increase anchoring on the Al or human alone decisions, while the presence of SHAP alone tends to make the participants anchor more on the Al model. With no explanation, the participants anchor more on their own decisions in the first round.
G DEMONSTRATION I: MODEL COMPARISON ON CHEST RADIOGRAPH DIAGNOSIS
We apply our framework to a well-known cardiac dysfunction diagnosis task (Rajpurkar et al. 2018 Tang et 2020 Shreekumar 2025). We demonstrate how our framework can be used in model evaluation for analyzing how much complementary information value set of possible Al models offers to the radiology reports written by experts.
G.1 DATA AND MODEL
We data from the MIMIC dataset (Goldberger et al. 2000). which contains anonymized electronic health records from Beth Israel Deaconess Medical Center (BIDMC), a large teaching hospital in Boston, Massachusetts, affiliated with Harvard Medical School. Specifically, we utilize chest x-ray images and radiology reports from the MIMIC-CXR database (Johnson et al. 2019) merged with patient and visit information from the broader MIMIC-IV database (Johnson al. 2023). The payoff- related state, cardiac dysfunction E {0, 1}. is coded based two common tests, the NT-proBNP and the troponin, using the age-specific cutoff' from Mueller et al. (2019) and Heidenreich et al. (2022). We label the radiology reports by rule-based tool (Irvin et al. 2019 and use the labels the human decisions (without Al assistance) in the diagnosis task to solve the problem of computational feasibility with high-dimensional textual reports. The labels are represented by the symptoms as positive, negative, or uncertain, i.c., d E {+. ?, -]. We fine-tuned five deep-learning models on the cardiac dysfunction diagnosis task, Tssioorransormmr (Dosovitskiy 2020). Tranforonfrrmer (Liu
20

---

Published as a conference paper at ICLR 2026
Table 5: Sample sizes and standard deviations of the reduction of APE for each experimental condition
Model
Explanation
Participants
Observations (N)
Standard Deviation
ILIV-SHAP + SHAP
70
840
0.0653
AII
SHAP
79
948
0.0642
No explanation
69
828
0.0668
Total (all explanations)
218
2,616
0.0656
ILIV-SHAP + SHAP
67
804
0.0746
A12
SHAP
62
744
0.0679
No explanation
74
888
0.0673
Total (all explanations)
203
2,436
0.0700
Total
421
5,052
0.0678
Table 6: The number of participants (after filtering based the criteria in the pre-registration), the mean duration of the experiment, and the anchoring on AI and human for each condition. Anchoring on AI is calculated as the mean of the absolute difference between the human-Al team's 's prediction and the Al's prediction normalized by the actual sale price, |dHumm--Al / and anchoring on
human is calculated as the mean of the absolute difference between the buman-AI team's 's prediction and the human's 's prediction normalized by the actual sale price, Human-AA H //w. The square brackets indicate the 25th and 75th percentiles.
Condition
participants
(minutes)
Anchoring (J)
Anchoring Human (U)
All ILIN-SHAP+SHAP
70
23.3 [13.1, 32.1]
0.303 00.0277. 0.253]
0.404 [0.0746, 0.491]
All SHAP
79
21.3 [12.0. 27.9)
0.211 (0.0201. 0.222)
0.427 (0.0721. 0.533]
No Explanation
[14.5, 28.4)
0.226 (0.0276, 0.258)
0.367 (0.0740, 0.483)
A12 ILIY-SHAP+SHAP
67
22.2 [11.9, 28.0]
0.206 [0 (0.0273, 0.260)
0.458 [0.08 (0.0840, 0.613]
A12 SHAP
62
23.1 [13.2, 27.0]
0.203 (0.0215. 0.224)
0.456 (0.0777. 0.602]
No Explanation
22.5 (1.57.3055
0.259 (0.0265. 0.260)
0.434 (0.0611. 0.501]
ct al. 2021), ResNet (He et al., 2016), Inception-v3 (Szegedy et al. 2016), and DenseNet (Huang al. 2017). Our training set contains 12,228 images, and the validation set contains 6,115 images. On a hold-out test set with 12,229 images, the AUC achieved by the five models is: DenseNet with 0.77, Inception v3 with 0.76, ResNet with 0.77, SwinTrannffrmee with 0.78. and VisionTransformer
with 0.80.
We consider Brier score, a.k.a., quadratic score, as the payoff function: S(w, d) 1 (w d)2 The scale of the quadratic score is [0, 1] and a random guess (d ~ Bernoulli(0.)) achieves 0.75 payofT. We use the quadratic score instead of the mean absolute error that is usually used in cardiac dysfunction diagnosis task because the quadratic score is proper scoring rule where truthfully reporting the belief maximizes the payoff We also conduct robust analysis considering various V-shaped payoff functions with different kinks on discretized grid of [0, 1] with step of 0.01. We use the hold-out test set to estimate the data-generee getrtreennnnnnngggg process, which defines the joint distribution of state, human decisions, and Al models' predictions.
We construct the scale of performances by no-information bound and a full- aall-information bound. The no- in-fformtaiion bound is R. (.)) the baseline as we define the information value. The full- information bound defined as the expected payoff of a rational DM who has access to signals, human label from radiology report and predictions from five Al models.
G.2 RESULTS
Can the AI models complement human judgment? We first analyze the agent-complemenntary information values Figure 5) using the Brier score as the payoff function. We find that all AI models provide complementary information value the aforementioned human judgment. As shown
We prefer proper scoring rule that the rational decision-maker's strategy is to reveal their true belief, ensuring that the signal's information value accurately reflects role in forming beliefs.
21

---

Published as a conference paper at ICLR 2026
Payoff
R=*(D"))
RT )mm)
D*-0
No Information
Full Information
D*
D*eReeNee50
D*
Expected payoff
Figure 5: Information value of all fieep-lcarinn models calculated under framework. The first row represents the human-alone decisions (without considering any Al predictions as additional signals). The other rows the combinations of the human-alone decisions and the Al predictions from different pre-trained models. We list the AI predictions alone to show the Al-complementary information value offered by human decisions.
in Figure (comparison between RT. (D)tummn u DAI) and RT. ( all Al models capture at least 20% of the total available information value (across all Al model and human decisions) that is not exploited by human decisions. This motivates deploying an AI to assist humans in this scenario.
In the other direction. the human decisions also provide complementary information to all Al models, comparing R*.5( Humman) with R (T)L) in Figure 5 This observation might inspire, for example, further investigation of the information the humans can access to that not represented Al training data.
Which AI model offers the most decision-relevane information over human judgments? Fig- ure 5 shows that Tranforrre contains slightly higher information value than the other models, and Inception v3 contains slightly lower information value than the other models. We assess the stability of VisionsTransfommer's superiority over the other Al models test if there is Blackwell ordering of models. By Proposition B.3 we test the payoff of models on all V-shaped scoring rules, shown in Figure 6 Across all the V-shaped payoff functions, find that Vision Transformer is Blackwell more informative and Inception v3 is Blackwell less informative than all other models. The VisionTransformer achieves higher information value all V-shaped scoring rules, implying higher information value on all decision problems.
many possible losses
H DEMONSTRATION II: BEHAVIORAL ANALYSIS ON DEEPFAKE DETECTION
We apply our framework analyze deepfake video detection task (Dolhansky al. 2020), where participants are asked to judge whether a video was created by generative AI, including with the assistance an Al model.
H.1 DATA AND MODEI.
We define the information model on the experiment data of Groh et al. 2022). Non-expert participants (n=5,524) were recruited through Prolific and asked to examine the videos. They reported their decisions in two rounds. They first reviewed the video and reported an initial decision Hummn without access the AI model. Then, in second round, they were provided with the recommendation (DAI) of a multitask cascaded convolutional neural network (Zhang et al. 2016), with estimated 65% accuracy on holdout data, and chose whether change their initial decision. This produced final decision ( Both decisions were elicited as a percentage indicating how confident the participant was that the video deepfake, measured in 1% increments: (0%, 1%.... 100%).
22

---

Published as a conference paper at ICLR 2026
V
DenseNer121
ResNet50
Swintians/ormer
Inception
0.02
pon rranstormm
-0.02
(ooooddd on geaelhymmee colermn
0.02
UersoNer121
-0.11
-0.02
0.02
-0.c1
-c02
0.01
-0.01
0.02
0.01
udlplocui
0
-0.01
-0.02
0.25
6.75
0.25 0.50
0.75
0.25 0:50
0.75
0.05
0.50
0.75
Figure 6: Robust analysis for demonstration I on all V-shaped payoff functions. The kink is shown on the x-axis. Each subplot displays the difference between the ACIV on the row model over human decisions and the ACIV on the column model over human decisions. A positive value (colored in blue) at y indicates the model on the row contains more informative than the model on the column under the evaluation of V-shaped scoring rule with kink μ. The subplots are symmetric along the diagonal, e.g.. (1, 2) subplot and (2, 1) subplot display same distribution with opposite signs.
Payoff
RAD U1
RD
Only
D U two, people
No Information
Full
D blurry
Du darkkkin.indiiiidee
Du flik
dark
boatiingdistraction
D u greiny
Expected payoff
Figure 7: Information value calculated under framework in the information model defined by the experiment Groh et al. (2022). Basic signals include the seven video level features and three types of agent decisions. The baseline the left represents the expected payoff given information, ie.. RR))(0), and the benchmark on the right represents the expected payoff given all available information, i.c., RT.S ((V1, ([E...., En, Iulmmn Uulmmmi DAA)). All the payoffs are caleulated by RT.S(.) (.), where is the signal on the y-axis.
We round the predictions from the Al model to the same 100-scale probability scale available to study participants.
23

---

Published as a conference paper at ICLR 2026
We use the Brier score the payoff function, with the binary payoff-related state: {0. 1) {genuine, fake). This choice differs from the mean absolute error used by /roh et al. (2022), but again the quadratic score because it is proper scoring rule where truthfully reporting the belief maximizes the score.
We identify set of features that were implicitly available to all three agents (human, AI, and human- AI). Because the video signal is high dimensional, we make use of seven video-level features using manually coded labels by Groh et al. (2022): graininess, blurriness, darkness, presence of a flickering face, presence of two people, presence of a floating distraction, and the presence of an individual with dark skin, all of which are labeled as binary indictors. These are the basic signals in our framework. We estimate the gettartnnrgting process π using the realizations of signals, state, first-round human decisions, Al predictions, and second-round human-Al decisions. The no-information bound the same as Appendix while the full-information bound is defined as the expected payoff of a rational DM who has access to all video-level features and three agents' decisions.
H.2 RESULTS
How much decision-relevant information do agents' decisions offer? We first compare the information value of the AI predictions to that of the human decisions in the first round (without AI assistance). Figure a) shows that AT predictions provide about 65% of the total possible information value over the no-ofoomation baseline, while human decisions only provide about 15%. Because the no information baseline, 0.75, is equivalent to random guess drawn from Bernoulli human decisions are only weakly informative for the problem.
We next consider the human-AT decisions. Given that the Al predictions contain a significant portion of the total possible information value, we might hope that when participants have access to the AI predictions, their performance will be close to the full information baseline. However, the information value of the human-Al decisions only achieves small proportion of the total possible information value (30%). This is consistent with the findings of Guo et al. (2024) that humans are bad at distinguishing when AI predictions are correct.
How much additional decision-relevant information do the available features offer over agents' decisions? To understand what information might improve human decisions, we assess the ACTVs of different signals over different agents. This describes the additional information value in the signal after conditioning on the existing information in the agents' decisions. As shown on the fifth row in Figure 7 the presence of flickering face offers larger ACIV over human decisions than over AI predictions, meaning that human decisions could improve by a greater amount if they were to incorporate this information. Meanwhile, as shown an individual with dark skin offers larger ACIV over AI predictions than over human decisions, suggesting that humans make greater use of this information. This suggests that the AT and human rely on differing information to make their initial predictions, where the Al relies more on information associated with the presence of a flickering face while human participants rely more on information associated with the presence of an individual with dark skin.
the fourth row in Figure 7 the presence of
By comparing the ACIV' of different signals over human decisions and human-Al decisions, we also find that simply displaying AI predictions to humans did not lead to the Al-assisted humans exploiting the observed signals in their decisions. As shown in Figure with the assistance of AI, the ACIVs of all signals over the human-Al teams' decisions do not change much compared to the ACTVs over human decisions, with the exception of a slight improvement in the presence of a flickering face. This finding further confirms the hypothesis that humans are simply relying on AI predictions without processing the information contained in them.
1 SENSITIVITY ANALYSIS OF THE RATIONAL BELIEF ESTIMATOR
In this section, we first present a theoretical analysis on the quality of the ACIV estimator by connecting the calibration error of the given A. Then, present empirical analysis the chest X-ray diagnosis demonstration with different algorithms (linear regression (LR), gradient boosting methods (GBM), and neural network (NN) as A.
24

---

Published as a conference paper at ICLR 2026
1.1 THEORETICAL ANALYSIS
In this section, we theoretically show how the quality of the ACIV estimator (Algorithm D is related to the calibration error of the rational belief estimator a = A( (v, dt. w; wx){=1) We use regret of the decision maker to represent the quality of the estimation on the rational decision rules, i.e., how much payoff can be improved if we correct the decisions in hindsigh 12 Because the ACIV is defined as the improvement of the best-attainable performance with the signal and the agent decision over the one with the agent decision alone, the decisions d' and d'h in Algorithm | should have small regret as possible. We first derive our definition of the regret of Algorithm | Jusing the swap regret by Roth & Shi 2024
Definition 1.1 (Swap Regret (Roth & Shi 2024 ). Given a set of observations {(r,, d, wy w))
decision rule d(-), and decision task with payoff function S, the swap regret of the DM is:
max SwAPs(d, [(,,, w;)) D:D-D
(( S(d(o, d(). wi)l
(11)
The swap function σ is permutation of the action space D that maps the action of d" (1; d to another action. We define the regret of the ACIV estimator as the difference between the ACIV under the best-responding decision rules-d"* and dnd the ACIV under the estimated decision rules-d" and drb.
Definition I.2 (Regret of the ACIV estimator). Given empirical distribution as the data-generating process, π Uniform (v,, db wi w=))))) the estimated decision rules d' V D D and dro
D + D from Algorithm ! and decision task with payoff function S, the regret of the ACIV estimator is:
REGACIV (d VV'VDD)) ACIVESS "a (V: ACI at (V;D")
(12)
where A d denotes the ACIV estimated under the rational decision rules d' and dob d and db are the optimal decision rules that get from d" and dh
d (r)),)) o² (dt (d())) where of arg
slold (v, dz)), wi))
o:D-D
(r)())) o (ar(())) where o.b arg max
((a (d))) (d))))
o:DD.D
Lemma L.3. The regret of the ACIV estimator is upper bounded by the swap regret of d::
REGACIV s, Vv, < SwAPs(d", U1u di wi (1)
(13)
Proof.
REGACIVs,, s,,"'' (d', "T,db,V,D. E w 1S (v,d")), (o (v, (v,d)) dd), wi)l
E (d)) wi]] E w) (S (d)) will
Ei w/ (v. wi)l E (v (S (dd (v, d'), will
(vi, S(o(d) (vi, ()), S(d S(d d d())
max oDDD =WWAPs(d" (10,, di wi)
(14)
We use the notion of regret to quantify the quality of our ACTV estimator because the perfect decisions are usually unidentifiable with finite signals.
25

---

Published as a conference paper at ICLR 2026
Claim 1.4 (Kleinberg et al. (2023), Theorem 12). Given decision task with payoff function
S: DxΩ + [0. 11, if the DM responds by taking d(u, d²) ) = d" (a(v, d")), where a is an estimator of the probability of the payoff state given the signal and action and d' (p) arg max ( w)] the best response to the probability p. the swap regret of the DM is bounded by the expected calibration error (ECE) of the estimator
SWAPs(d Alus, (i) S 2ECE(a, [(e,, & wi))
(15)
Theorem 1.5. Given a decision task with bounded payaff S : D S + [Mi, M2), the regret the ACIV estimator is bounded by the expected calibration error of the estimator = A(tr,, , di )=))
REGACIV (dd,, d DV S 2(M2 - MI)EC {(e;, di. wi)
(16)
The proof normalizes the payoff in Lemma I.3 into [0, 1] and then applies Claim I.4
Theorem L.5 shows that, in Algorithm when we choose a predictive algorithm A that yields low ECE, the ACIV estimated by Algorithm is close to the ACIV under the optimal decision rules.
1.2 EMPTRICAL ANALYSIS
In this section, we take the chest X-ray diagnosis task in Appendix G as an example to empirically an- alyze the estimation of the full information value by Algorithm| [ with different modeling approaches (linear regression (LR), gradient boosting methods (GBM), and neural network (NN)).
Task and data. We use the chest X-ray diagnosis task in Appendix as the example. The task is to predict the probability of the presence of a disease given a chest X-ray image. The decision space D A 1] and the payoff space is Ω {0, 1}. The payoff function is S(d, w) = (d w) The signal space is high-dimensional feature vector of the chest X-ray image. We split the dataset that we used Appendix G into 70/30 train/test split.
Predictive Models.
We use the following models:
Linear Regression (LR) from scikit -latn package.
Extreme Gradient Boosting (GBM) from xgboost package with a_estimators 100, max_depth 6, and learning_rate 0.3.
MLP classifier (NN) from scikit learn package with 2 hidden layers of 256 and 64 neurons each.
Evaluation Metrics.
We report the following evaluation metrics for each estimator:
Brier Score:
1 n Σ (p: wx)²
Accuracy: E (1 2 0.5) = w()
Expected Calibration Error (ECE): 1 E p. -wwppll
Table 7: Performance of different algorithms for ACIV estimation in the chest X-ray diagnosis task.
Metrics for the estimators include Brier Score, Accuracy F1-Score, and Expected Calibration Error (ECE). We report the estimated ACIV (V:V():) 0) for each estimator with Algorithm
Model
Brier Score
Accuracy
ECE +
ACIV (V: 0) 1
Linear Regression (LR)
0.834
0.775
0.039
0.086
Gradient Boosting (GBM)
0.841
0.771
0.026
0.108
Neural Network (NN)
0.853
0.789
0.036
0.0757
The results are shown in Table 7 We can see that the GBM estimator achieves the best performance in terms of ECE, while the NN estimator achieves the best performance in terms of Brier Score and Accuracy. This validates our theoretical result: ECE is good proxy for the regret of the estimation of the ACIV by Algorithm Secrrihmmmoomppeee compared the Brier Score and Accuracy.
Results.
26

---

Published as a conference paper at ICLR 2026
J SCREENSHOTS OF THE EXPERIMENT
Trial number: 11/12, Round: 1/2
Year Built
2004
# of Fireplaces
0
Car Capacity in Garage
2.0
Above Grade Living Area
2084 sq ft
Feature
420
Feature
140
What is your guess for the sale price of this house? (Please use the following slider to choose the answer and then click on the Next button to continue.)
B7
236
310
458
582
6B1
755
Price ($K)
Figure 8: Screenshot of the Luman-alone trials in the first round of the experiment.
27

---

Published as a conference paper at ICLR 2026
Trial number: 1/12, Round: 2/2
Year Built
1920
of Fireplaces
0
Car Capacity in Garage
2.0
Above Grade Living Area
1196 sq ft
Feature
500
Feature
120
Model prediction: $144.1K
Your previous guess: $160K
What is your guess for the sale price of this house? (Please use the following slider to choose the answer and then click on the Next button to continue.)
13
87
181
238
310
384
458
532
607
681
765
Price ($k)
Figure 9: Screenshot of the no explanation condition in the second round of the experiment.
28

---

Published as a conference paper at ICLR 2026
Trial number: 1/12, Round: 2/2
Feature name
Value
Impact Al prediction
Year Built
2007
+S7.7K
of Fireplaces
$$7.2K
Car Capacity in Garage
3.0
4$17.6K
Above Grade Living
1542 sq ft
$$0.1K
Area
Feature X
360
+$7K
Feature Y
160
+$31.7K
Model prediction: $286.1K
Your previous guess: $160K
What is your guess for the sale price of this house? (Please use the following slider to choose the answer and then click on the Next button to continue.)
13
87
161
236
310
384
458
532
607
681
755
Price ($k)
Figure 10: Screenshot of the SHAP condition of All in the second round of the experiment.
29

---

Published as a conference paper at ICLR 2026
Trial number: 1/12, Round: 2/2
Feature name
Value
Impact on Al prediction
Year Built
1940
$$1.55
of Fireplaces
+$10K
Car Capacity in Garage
2.0
-S0.9K
Above Grade Living
2480 sq ft
+$35.2K
Area
Feature X
1500
$0K
Feature Y
140
$0K
Model prediction: $238.7K
Your previous guess: $160K
What is your guess for the sale price of this house? (Please use the following slider to choose the answer and then click on the Next button to continue.)
13
87
NE1
236
310
384
458
532
607
881
755
Price (sk)
Figure 11: Screenshot of the SHAP condition of AI2 in the second round of the experiment.
30

---

Published as a conference paper at ICLR 2026
Trial number: 2/12, Round: 2/2
Feature name
Value
Impact on Al prediction
Feature Y
100
$$6.2K
# of Fireplaces
-$13.3K
Year Built
1900
$$7.5K
Car Capacity Garage
2.0
$$1.1K
Feature X
380
+$4.8K
Above Grade Living
1627sq
+$1K
Area
Model prediction: $134.9K
Your previous guess: $160K
Highlighted features are where the model gets significant amount of humann-compereentary information.
What is your guess for the sale price of this house? (Please use the following slider to choose the answer and then click on the Next button to continue.)
161
236
310
384
755
Price (sk)
Figure 12: Screenshot of the ILIV-SHAP + SHAP condition of All in the second round of the experiment.
31

---

Published as a conference paper at ICLR 2026
Trial number: 1/12, Round: 2/2
Feature name
Value
Impact on Al prediction
Year Built
1941
-$1..KK
Above Grade Living
1376sq
$$2.1K
Area
Car Capacity in Garage
1.0
$$3.7K
# of Fireplaces
$3.1K
Feature
1500
$0K
Feature
100
$0K
Model prediction: $135.3K
Your previous guess: $160K
Highlighted features are where the model gets significant
amount of human-cooplemeentrry information.
What is your guess for the sale price of this house? (Please use the following slider to choose the answer and then click on the Next button to continue.)
13
236
3E4
458
532
607
B8'
755
Price (sk)
Figure 13: Screenshot of the ILIV-SHAP + SHAP condition of AI2 in the second round of the experiment.
32


Reviews:
[{"review_id": "2s5qLAyaLZ", "reviewer": "Reviewer_3aDh", "summary": "The paper considers scenarios where multiple agents make a decision together, as in human-AI decision-making. The authors develop new metrics, ACIV and ILIV, which measure how much informational value the decision of an agent adds to the decision of another. Based on ILIV, they develop a new interpretability method called ILIV-SHAP, which highlights how much each feature contributes to the added informational value.", "questions": "ACIV measures how much one decision-maker (D^1) adds information to the decisions of another (D^2) by looking at how the optimal decisions made observing only D^2 differ from the optimal decisions made observing only D^1 and D^2. This seems like a sound measure, especially when the goal is to perform optimally, however isn’t it indirect in the sense that it involves a third decision-maker? What are the challenges associated with trying to quantify the impact of D^1 on D^2 directly?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The goal of human-AI systems is to achieve complementary performance and the paper makes an important contribution to this effort, quantifying the complementary value of decisions made by different agents.\n\nThe proposed metrics are defined rigorously and presented very clearly. In particular, starting with the conceptually simple ACIV, and then extending it to the instance-wise case, ILIV, and finally incorporating it to SHAP is a good way of organizing the technical parts of the paper (with increasing complexity).\n\nPapers contributing metrics often struggle to provide concrete use cases. However, this paper uses ILIV with SHAP to create a new interpretability method, which is shown to improve human-AI accuracy with human-subject experiments.\n\nThe setup for the human-subject experiments is well thought out. The authors have created two versions of the AI agent with the same accuracy but varying ACIV and then observed ILIV-SHAP leads to greater reduction in errors when used with the higher ACIV model. This observation nicely ties all the ideas in the paper together.", "weaknesses": "Motivation of the demonstrations in Section 6 is not clear. For instance, experiments for chest X-ray diagnosis and deepfake detection both vary different dimensions (AI models for chest X-rays and features for deepfakes) and report different metrics (ACIVs of combined decisions for chest X-rays and ACIVs of decisions combined with different features for deepfakes). It is not clear why the two settings are set up differently and what hypothesis drove the design of these setups.\n\nWhat I particularly found hard to understand with these setups: ACIV relative to an empty set is just an expected reward value that doesn’t consider complementarity. Although Figure 2a just reports ACIV relative to empty sets, I can look at the difference between green and orange or green and blue to reason about the complementary information each agent provides to the other. But I was not able to figure out what differences were relevant in Figure 2b.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Qs5l64lmZq", "reviewer": "Reviewer_C2Li", "summary": "The paper attempts to answer the question of whether complementing information improves human-AI collaboration. A preregistered between-subjects experiment with six conditions is performed. A novel instance-level explanation technique is proposed. Together with AI predictions, the new explanation technique appears to help reduce human-AI decision error.", "questions": "1. Two fixed features, namely \"year remodeled\" and \"material and finish\", are selected as complementing information hidden from the human participants. To make the paper stronger, a varying subset of features of different sizes should be tested.   \n\n2. Noise is introduced to the selected complementing attributes to bring down the accuracy of AI1. The authors claim that  \"model accuracy differences would not confound our results\". Is this true?  Even if both models have similar accuracy, one model may consistently make higher predictions while the other produces random error. Humans may catch the consistency in AI1 and take advantage of it. To clarify, experiments should be set up so that AI1 is trained without noise, with a small amount of noise, and with a large amount of noise. This also helps to understand and quantify the impact of noise on the value of information. \n\n3. Were the participants aware that the highlighted features based on the ILIV-SHAP value are linked to human-complementary information? If so, this may incentivize the participants to always follow the AI as the AI knows more and consequently helps generate higher pay-off. \n\n4. For AI2, instead of using 0k as the SHAP value for features X and Y, a random value should be used. Otherwise, the participants would question the quality of AI and become more skeptical about AI2's predictions.  \n\n\nMinor comments:\n\nOn page 3, is there any difference between Pr[w] and pi(w)? Pi is referred to as a probability distribution (on page 3) and then an information model (on page 4). Are these two the same concepts?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The subject of the study is timely in the quick expansion of AI applications. \nHypotheses are tested with a wide choice of experimental conditions.", "weaknesses": "The setup of the experiment needs significant improvement. \nThe claims presented in the paper are inconclusive.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QIfhMtYSM1", "reviewer": "Reviewer_r9GS", "summary": "This paper offers a decision-theoretic approach to analyze (and improve) human-AI decision-making by quantify the \"value of information\" employed by each (AI or human) agent. It introduces a Bayesian decision-theoretic framework for evaluating the performance of agents against a \"rational\" benchmark that represents the best-attainable payoff given a set of signals. It also proposes new metrics based on this framework: Agent-Complementary Information Value (ACIV), to quantify the global value an (AI) agent adds to (human's) existing information, and Instance-Level ACIV (ILIV), to quantify this value for a specific decision. ACIV is formalized as an upper bound using a learned surrogate for the unknown data-generating process, while ILIV is operationalized by a counterfactual payoff definition and an estimator. The authors further propose ILIV-SHAP, a variant of popular SHAP explanation method to highlight features that contribute most to the AI's \"complementary\" information value, rather than just its prediction. The authors validate this framework through a crowdsourced experiment on house price prediction, claiming that ACIV can identify the better AI teammate and that ILIV-SHAP improves team performance more than standard SHAP. Demonstrations on chest X-ray and deepfake detection are also provided.", "questions": "- **Theoretical Inaccuracy in Appendix:** The appendix introduces a greedy algorithm (Algorithm 3) and claims that \"it orders the signals the same as the Shapley value\". This appears incorrect. A greedy algorithm finds a single ordering that approximates the maximum set value, whereas the Shapley value is a unique value attribution derived from *averaging* marginal contributions over *all possible* orderings. The claim indicates a conceptual misunderstanding and should be removed/corrected.\n\n- **Causal overreach in demonstrations.** In deepfake detection, the paper infers *why* humans underuse certain information (e.g., \"simply relying on AI predictions\"), but ACIV is *observational* and cannot identify causal cognitive mechanisms. Conclusions should be tempered.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- **[S1] Novel and Important Problem:** The paper tackles a central problem in human-AI interaction: the failure to achieve complementarity.. The motivation to move beyond simple accuracy metrics and develop a formal theory of \"information value\" is both strong and timely.\n- **[S2] Crisp formalism:** The core idea of defining a Bayesian rational benchmark, $R(V)$, as the best-attainable performance is theoretically sound and provides an elegant upper bound to quantify agent sub-optimality, as noted in prior work (Guo et al., 2024). Algorithmic recipes make the framework implementable in principle, and the proposal of ILIV-SHAP that mirrors SHAP while targeting complementarity is conceptually interesting (offering a nice shift from \"why did the AI make this prediction?\" to \"why should the human listen to this prediction?\").\n- **[S3] Broader demonstrations.** The chest-X-ray and deepfake case studies help situate the framework beyond the main user study.", "weaknesses": "-  **[W1] Weak and Confounded Experimental Validation of ILIV-SHAP:** The primary experiment (Section 5) designed to validate ILIV-SHAP (RQ2) suffers from multiple issues, making it impossible to attribute the observed performance lift to the proposed method.\n    * **Problematic Baseline:** The results for the main AI model (AI1) show that \"No explanation\" (6.50% error reduction) significantly *outperformed* \"SHAP\" alone (5.20%). This critical finding, which suggests standard SHAP was actively *harmful* to user performance, is reported in the plot but never discussed. The paper's headline claim that ILIV-SHAP + SHAP (7.11%) is superior to SHAP (5.20%) is misleading, as it uses a broken baseline. The more meaningful comparison to \"No Explanation\" (7.11% vs. 6.50%) shows a much smaller improvement, which is neither statistically nor practically significant.\n    * The experiment does not compare \"SHAP\" vs. \"ILIV-SHAP,\" and appears to run into **additive information confound** instead. It compares \"SHAP\" (control) vs. \"ILIV-SHAP *and* SHAP\" (treatment). The treatment group received strictly more information (two explanations) than the control (one). The reported improvement could simply be an effect of information dosage, not the specific content of ILIV-SHAP. The two conditions are not presented identically, leading to possible **presentational confound**. The SHAP condition is a standard table (Fig. 9). The ILIV-SHAP + SHAP condition includes *visual highlighting* of specific rows and a *direct textual hint* (Fig. 11). This simple, salient cue is a powerful intervention on its own and is not part of the core ILIV-SHAP contribution. The performance gain is likely attributable to this simple UI intervention, not a deep understanding of information value.\n\n- **[W2] The Practical Utility of ACIV is Overstated:** The data shows that while ACIV may predict the direction of improvement, it is a poor predictor of the *magnitude*. In the main experiment (Fig 1, left), AI1 has more than double the ACIV of AI2. This massive theoretical gap in complementarity translated to a *trivial difference* in actual team performance (6.27% vs 5.65% error reduction, collapsing across explanations). The observational study in Appendix (Table 1) shows the same weak link. The Art task has the highest ACIV (0.1772) and highest MSE reduction (0.0583). However, \"Sarcasm\" has a similarly high ACIV (0.1625) but its MSE reduction (0.037) is much smaller, and nearly identical to that of \"Cities\" (0.034), which had a far lower ACIV (0.0919). This weak correlation between the *metric's magnitude* and the *outcome's magnitude* severely limits its practical value for model selection.\n\n- **[W3] Weak \"Human-AI\" Framing and Reliance on an Idealistic Human Model:** The paper's \"human-AI\" framing is a significant weakness, as the core framework is general to any two agents (e.g., AI-AI) and fails to capture the unique and well-documented imperfections of human cognition.\n    * The framework's reliance on a \"rational Bayesian DM\" as the benchmark for optimal performance is an idealistic setup that is known to be a poor model for real human decision-makers, who are subject to cognitive biases and do not perform perfect Bayesian updates.\n    * Because the \"human\" component is modeled so abstractly, the paper fails to clearly position its contribution within specific, established human-AI collaboration paradigms, such as AI-assisted decision-making (where the human makes the final choice) versus learning-to-defer (where the system decides between agents) vs other \"human-in-the-loop\" options. Clear positioning would dictate which of the human-AI decision-making methods and baselines are relevant, and could require deeper engagement with additional literature (Noti et al., Zhang et al., Mahmood et al. etc.).\n    * A more rigorous evaluation in a controlled AI-AI setting might have been more effective at validating the core information-theoretic framework, as the current human experiment introduces significant, unmodeled \"human\" noise that obscures the results.\n\n- **[W4] Serious Practical and Computational Hurdles are Ignored:** The entire framework relies on a regression algorithm $\\mathcal{A}$ to proxy the \"rational Bayesian DM\" and estimate the posterior $Pr[\\omega | V]$.\n    * **Estimator Sensitivity:** The accuracy of ACIV and ILIV is critically dependent on the choice and quality of this surrogate model $\\mathcal{A}$. The paper provides no sensitivity analysis on this choice. If $\\mathcal{A}$ is a poor estimator, the entire framework produces meaningless numbers.\n    * **Intractability of ILIV-SHAP:** The ILIV-SHAP calculation (Definition 4.1) appears computationally intractable. It requires a combinatorial number of calls to the ILIV function. The ILIV function itself (Algorithm 2) requires conditioning on an exact signal match $v_j = v$. In any high-dimensional or continuous-feature domain, the probability of an exact match is zero. This means the set of matching instances $k$ will be 1 or 0, making the estimate undefined or infinitely high-variance. This is not a minor detail, and suggests the proposed method is unusable for most non-trivial problems.\n\n\n[Noti et al.] \"Learning When to Advise Human Decision Makers.\" In Proceedings of the 2023 International Joint Conference on Artificial Intelligence.\n\n[Zhang et al.]. \"Learning to Complement with Multiple Humans.\" arXiv preprint arXiv:2311.13172 (2023).\n\n[Mahmood et al.] \"Designing behavior-aware AI to improve the human-AI team performance in AI-assisted decision making.\" In Proceedings of the 2024 International Joint Conference on Artificial Intelligence.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "q9JACDwPSx", "reviewer": "Reviewer_A4pM", "summary": "The paper proposes a decision-theoretic framework to quantify the *value of information* in human-AI decision workflows. It formalizes (i) *Agent-Complementary Information Value (ACIV)* (the marginal payoff gain from adding a signal (V) beyond what is already encoded in an agent's decisions ($D_b$)) and (ii) an *instance-level* analogue (*ILIV*) to analyze specific realizations. Building on ILIV, the authors introduce *ILIV-SHAP*, an explanation method that attributes feature importance to *complementary* information rather than to predictive contribution alone. A preregistered Prolific study on *house-price prediction* ($n=421$) crosses two models with different ex-ante ACIV (AI1 high vs. AI2 low) and three explanation conditions (ILIV-SHAP+SHAP / SHAP / None). Results indicate that (a) the higher-ACIV model yields larger error reductions for human-AI teams, and (b) *ILIV-SHAP+SHAP* outperforms *SHAP* alone in reducing absolute percentage error (APE). The framework is further demonstrated on *chest X-ray diagnosis* and *deepfake detection* to show how ACIV/ILIV expose complementarity patterns between humans and models.", "questions": "1.  How sensitive are ACIV/ILIV estimates to the choice of model class and calibration? Could you report a *model-sweep* (e.g., LR vs. GBM vs. NN) and reliability diagrams for posterior quality used in Eq. (1)/(2)? \n2.  What exact *thresholding* produced the highlighted features, and how were ties/instability handled across permutations in the SHAP-style computation? Please include a *UI figure* and an ablation where only ILIV-driven highlights (without SHAP bars) are shown. \n3. Did you measure *time-on-task*, confidence, or reliance behaviors to ensure gains weren't due to simple anchoring on model outputs? Any heterogeneity across participants (e.g., numeracy)? \n4. Can you replicate the user-study result on a setting where human-visible proxies to X/Y exist (weaker complementarity), and test whether ILIV-SHAP still outperforms SHAP?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "* *Clear formalization of 'information value' in teaming.* The ACIV/ILIV definitions make the complementarity question precise, grounding it in Bayesian decision theory and proper scoring rules (incl. a robustness view). This provides a principled upper-bound benchmark and a way to diagnose under-use of available signals. \n* *Practical algorithms and instance-level lens.* Algorithms 1-2 give workable estimators for ACIV/ILIV, enabling empirical analysis without requiring full generative models of all signals.  \n* *Explanation innovation (ILIV-SHAP).* Reframing attribution around *decision-relevant payoff gains* (rather than prediction deltas) is original and well-motivated; the method preserves SHAP axioms while targeting complementarity. \n* *Empirical evidence for design claims.* In the crowdsourced study, the high-ACIV model (AI1) shows bigger APE reductions than the low-ACIV model (AI2), and ILIV-SHAP+SHAP yields larger gains than SHAP alone for AI1 (e.g., 7.11% vs. 5.20% for SHAP; CI reported). The analysis includes hierarchical modeling and multiple-comparison controls.  \n* *External demonstrations.* The chest-X-ray and deepfake case studies illustrate how the framework can compare agents and reveal which features/signals are under- or over-utilized by humans vs. models.", "weaknesses": "1. *Human-AI complementarity definition*. In the related work section, the *human-AI complementarity* term is, in my opinion, not properly defined and there is also lack of related and important related work missing [1-5]. In this area, complementarity refers to team decisions that, in expectation, outperform both the human and the AI alone [Bansal et al 2021a(already cited),1,2], rather than just analyzing the *information value and use*, as author's state in the related work. However, imo, authors' use a better definition in the first paragraph of the introduction. \n2. *Complementarity by design related work*. While I understand the authors' analysis of the value of additional information from one of the agents, I believe they also make a claim in the paper that existing complementarity approaches \"do not account for the best achievable performance given the information available at the time of the decision\" (l. 41). I find this to be an overstatement and a shortcoming in the related work section. There are work in the literature proposing existing complementarity approaches **by design**. These methods exploit information asymmetry (where humans possess additional contextual knowledge, which the authors identify as a central aspect of human-AI complementarity) and offer principled methods with **provable guarantees** in decision-making about the better perfomance of the teaming. Specific applications of these methods include classification [3,4] or matching problems [5]. These methods effectively identify parameters that model collaboration, achieving what the authors define as the \"best achievable performance given the information available at the time of the decision\". I recommend revising the Related Work section to accurately position your contribution in relation to this body of research, highlighting the differences in goals, assumptions, and guarantees. Anticipating author's comments, I want to emphasize that I do not see these approaches as substitutes for the authors' contribution. Rather, *my concern is with how the introduction and related work is positioned*: some of the current statements underrepresent what prior work already achieves in this space. In fact, the authors' proposed framework (which focuses on quantifying and analyzing the value of additional information) could even complement and strengthen these existing methods. I appreciate that the authors are not proposing a new method per se but a general framework, and I believe clarifying this positioning would make the paper's contribution more precise and impactful.\n3. *Estimator dependence and potential leakage.* ACIV/ILIV are estimated via learned regressors ($\\hat a, \\hat a_b$). The main text does not detail safeguards (strict train/validation/test splits aligned with the team-vs-human comparisons, calibration of posteriors, or sensitivity to the choice of (A)). Since the conclusions rely on these regressors approximating posteriors used in the rational benchmark, a *sensitivity analysis* (alternate model classes, calibration checks, and ablations on (A)) would strengthen validity.\n4. *Flow of theory section.* The notation and explanation is dense and introduces granularly: mixing state, signals, decisions, and payoffs  are presented and used before an anchoring example. The exposition would benefit from a *running toy example* (e.g., the umbrella example) threaded through Defs. 3.1-3.3 and Algorithms 1-2, plus a *table of symbols* and an 'intuitions first' paragraph for ACIV vs. ILIV. \n5. *SHAP computation*. There is no optimization on the SHAP computation: shap is O(2^n) in the number of features, so for high-dimensional data this can be very slow. Some approximation methods exist (e.g., sampling-based), but they are not discussed. In addition, the paper does not address the potential for bias in the SHAP values (e.g. not stable to noise) [6], which could impact the fairness of the model.\n6. *Intervention realism in the main user study.* The two 'uninterpretable' features (Feature X/Y) are relabeled to be intentionally opaque, which creates complementarity by construction. This is a clean testbed but may *overstate* ILIV-SHAP benefits relative to realistic domains where human observables partially correlate with those features. I'd appreciate if authors discuss this and add a variant where 'semi-interpretable' proxies are available to humans.\n7. *Cognitive load and UI details for ILIV-SHAP.* The paper states that ILIV-SHAP 'ranked and highlighted' features exceeding a threshold, but there's limited information on *interface design*, thresholds, and how users processed both SHAP and ILIV-SHAP simultaneously. Add representative screenshots of the UI and an *attention/time* analysis to verify that added information did not induce confusion or anchoring.\n8. *Ethical nuance in DeepFake study.* In the deepfake study, the discussion of the 'dark skin' feature as informative should more explicitly acknowledge fairness and representation concerns and potential spurious correlations when translating these insights into deployed systems. Even if out of scope, a short cautionary note would be responsible. Authors could add a brief fairness statement in the deepfake section.\n\n*Minor*:\n\n* Some key p-values and sample sizes appear only in Appendix D/Table 2-3. Bring the primary contrasts (AI1 vs. AI2; ILIV-SHAP+SHAP vs. SHAP) with effect sizes and corrected p-values into the main text to improve readability and soundness of the results.\n* Fig 1 left and Fig 2 are a bit hard to read due to small fonts.\n* Bring key *CIs/p-values* and *N per condition* from Appendix D into the main text/figure caption.\n* ACIV abbreviation is mentioned in the abstract before being defined.\n* Missing space in line 135 \"...available to a DM, including...\"\n\nIn sum, the paper tackles an important and timely problem (measuring complementarity in human-AI teams) with a principled framework (ACIV/ILIV) and a neat explanatory tool (ILIV-SHAP). The conceptual contribution is solid and potentially influential. The empirical study is well-designed and aligns with the theory, and the demonstrations are informative. I'm slightly below the bar primarily due to (i) limited sensitivity analyses for the learned estimators behind ACIV/ILIV, (ii) the constructed complementarity in the user study, (iii) presentation density in the theory sections, and (iv) concerns in the related work and introduction. Addressing the listed weaknesses would likely move this into acceptance.\n\n> [1] Bayesian modeling of human-AI complementarity. PNAS 2022.\n\n> [2] Complementarity in human-AI collaboration: Concept, sources, and evidence. EJIS 2025.\n\n> [3] Improving Expert Predictions with Conformal Prediction. ICML 2023.\n\n> [4] Towards human-AI complementarity with prediction sets. NeurIPS 2024.\n\n> [5] Towards Human-AI Complementarity in Matching Tasks. HLDM @ ECML-PKDD 2025.\n\n> [6] On the failings of Shapley values for explainability. IJAR 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)