When Does Uncertainty Matter?: Understanding the Impact of Predictive Uncertainty in ML Assisted Decision Making
Authors: Sean McGrath
Parth Mehta
Alexandra Zytek
Isaac Lage
Himabindu Lakkaraju
Venue: Accepted by TMLR ()

Abstract: As machine learning (ML) models are increasingly being employed to assist human decision makers, it becomes critical to provide these decision makers with relevant inputs which can help them decide if and how to incorporate model predictions into their decision making. For instance, communicating the uncertainty associated with model predictions could potentially be helpful in this regard. In this work, we carry out user studies (1,330 responses from 190 participants) to systematically assess how people with differing levels of expertise respond to different types of predictive uncertainty (i.e., posterior predictive distributions with different shapes and variances) in the context of ML assisted decision making for predicting apartment rental prices. We found that showing posterior predictive distributions led to smaller disagreements with the ML model's predictions, regardless of the shapes and variances of the posterior predictive distributions we considered, and that these effects may be sensitive to expertise in both ML and the domain. This suggests that posterior predictive distributions can potentially serve as useful decision aids which should be used with caution and take into account the type of distribution and the expertise of the human.

Published in Transactions on Machine Learning Research (06/2023)
When Does Uncertainty Matter?: Understanding the Im- pact of Predictive Uncertainty in ML Assisted Decision Mak- ing
Sean McGrath Harvard University
seen www__mmgruhh@gg..vrrrr.... edu
Parth Mchta Harvard University
mehla.parthi 170llaa com
Alexandra Zytek
zytcko@miit. edu
Massachusetts Institute of Technology
Isaac Lage Harvard University
Sstackage@@g.arrvadd.oo
Himabindu Lakkaraju
hlakkareju@hba edu
Harvard Universily
Reviewed on OpenReview: https: /ttp://ppppe openreview. net/ forum? id=pbs22hJmB0
Abstract
As machine learning (ML) models are increasingly being employed to assist human decision makers, it becomes critical provide these decision makers with relevant inputs which can help them decide if and how to incorporate model predictions into their decision making. For instance, communicating the uncertainty associated with model predictions could po- tentially be helpful in this regard. In this work, we carry out user studies (1,330 responses from 190 participants) to systematically assess how people with differing levels expertise respond to different types of predictive uncertainty (i.e., posterior predictive distributions with different shapes and variances) the context of ML. assisted decision making for pre- dicting apartment rental prices. We found that showing posterior predictive distributions led to smaller disagreements with the ML model's predictions, regardless of the shapes and variances of the posterior predictive distributions we considered, and that these effects may be scnsitive to expertise in both ML and the domain. This suggests that posterior predictive distributions can potentially serve as useful decision aids which should be used with caution and take into account the type of distribution and the expertise of the human.
1 Introduction
As machine learning (ML) models are increasingly being deployed in critical domains such healthcare and criminal justice, there has been growing emphasis the need for human interpretable models and predictions which enable decision makers to decide if and how much to rely these predictions. In order to interpret model predictions, the following two complementary approaches have been proposed in litcrature. The first paradigm involves building inherently simpler models such as decision trees poass/lssss/eeee (Letham el al. 2015 Rudin 2019 Lakkaraju et al. 2016), point systems that can be memorized (Ustun Rudin 2016)) or gencralized additive models in which the impact of each feature on the model's prediction explicitly given (Caruana al. 2015). However, complex models such as deep neural networks and random forests seem achieve higher accuracy compared to these inherently simpler models in several real world settings (Ribeiro)

---

Published in Transactions on Machine Learning Research (06/2023)
et al. 2016 Therefore, alternate approach of constructing post hoc explanations to interpret these complex models has been proposed in literature (Ribeiro et al., 2016; Lundberg & Lee 2017). Both simple models and post hoc explanations are attempts at creating more interpretable decisions aids.
In addition to the aforementioned simple models and explanations, other auxiliary information such as uncertainty associated with predictions could also potentially serve as useful decision aids. While there are reasons to believe that conveying information pertaining to predictive uncertainty might help decision makers in figuring out how to incorporate model predictions into their decision making, there is little empirical work that systematically explores the validity of this hypothesis (Bhatt et al. 2021). While some prior research has examined how predictive uncertainty in various contexts ranging from weather forecasting to public transit scheduling can be communicated to the public (Greis et al. 2016; Roulston et al.) 2006 Fernandes et al. 2018], these works largely focus on the cases where uncertainty can be expressed using a low variance normal distribution. However, reality, predictive uncertainties could be much more complex to reason about (e.g.. posterior prodictive distributions which are bimodal or normal with high variance). In such cases, interpreting predictive uncertainly and incorporating it in decision making is non-trivial, and is heavily driven by the end user's domain expertise and familiarity with ML.
To illustrate, let us consider the following example: if an ML model predicts that Alice should sell her car for $20, 000, should she place an advertisement. for that amount or rely on her background knowledge to price the car herself? If she knew the model was highly certain, the rational choice may be to defer to its prediction, but if is highly uncertain, she may instead prefer her own. However this picture is complicated by both Alice's own level of expertise, and the form of the uncertainty associated with the prediction by the model. For example, what if Alice is an expert used car appraiser? Or what if the model is highly certain that the car will sell for either $17, 000 or $23, 000? A rational choice for Alice might look different between these cases, and it is unclear whether the human decision maker will even follow rational strategy. Therefore, it is important to systematically study how different types of predictive uncertainty (e.g., posterior prodictive distributions with different shapes and variancos) and diffcrent contexts (namely decision makers with varying degrees of domain expertise and familiarity with ML) impact decision making.
In this paper, we explore how decision making is impacted when decision makers (end users) are shown estimates of predictive uncortainty. More spocifically, we consider posterior prodictive distributions our measure of uncertainty, and study how posterior predictive distributions with various shapes and variances impact decision making in cases where decision makers have varying degrees of expertise in ML and the problem domain. We consider the setting where decision makers are only given limited information for the task at hand, and are shown the output of a ML model (which may consider features not available to the decision makers). While the goal is to understand how uncertainty affects the decision making process as a whole, in this work, we focus on systematically measuring how closely participants agree with the predictions of the model based on the factors described above. This allows us to separate the impact of uncertainty from the quality of the ML model. To the best of our knowledge, this work makes one of the first attempts at systematically exploring the aforementioned rescarch questions.
To find answers to the above questions, we conduct a user study with 1,330 responses from 190 participants where each participant is asked to predict monthly rental prices of apartments in Cambridge, Massachusetts (MA). Participants were recruited from the platform Prolific as well as from rescarch groups in various fields across multiple universities. We found that: 1) Showing posterior predictive distributions (regardless of their shapes and variances) led to smaller disagreements with the ML model's predictions compared to showing uncertainty estimates at all. 2) Showing model predictions (whether or not posterior predictive distributions were also shown) helped equalized differences between people with and without domain expertise. 3) People with ML. expertise agreed with the model most when provided with posterior predictive distributions that were normally distributed and had low variance. This suggests that posterior predictive distributions can potentially serve as useful decision aids which should be used with caution and take into account the type of distribution and the expertise of the human.
2

---

Published in Transactions on Machine Learning Research (06/2023)
2
Related Work
ML Assisted Decision Making: Machine learning is increasingly being employed to assist human decision makers. For example, Kleinberg et al. (0117) examine ML assisted decisions in the context of the the criminal justice system, and Giannini et al. (2019) studies how ML can assist in clinical care context. Consequently, there has been a lot of work in how decision makers make use of the output of ML models (Lai et al, 2021]. In some cases, users may trust models even when the model's predictions are inaccurate (Poursabzi- Sangdeh et al., 2021). In other cases, users will ignore a model's prediction even when is expected to perform better than humans (Yin et al.] 2019). More generally, Skktka et al. (1999) found that people made more errors when aided by highly accurate automation, suggesting the existence of an automation bias that affects people's decision making with automated aids. We explore how showing uncertainty information for specific prediction alongside an MI. prediction can alfect the downstream decisions made by human decision maker.
Model Interpretability: Prior research has suggested that model interpretability can be extremely helpful in MT. assisted decision making (Doshi- Velez & Kim 2017). Different. classes of interpretable models have
been proposed (Wang & Rudin 2015
Zeng et al.] 2017: Letham et al.] 2015 Bien & Tibshirani] 2009;
Lakkaraju a 2016 Lou et al.) 2012 Caruana et al.; 2015) and studied in the context of human decision
bzirraazz--aandeeh et al. 2021]. While these simpler interpretable models are often casier for users to interact with, complex models such as deep neural networks and random forests are
making (Lage et al. 2019:
often shown to achieve higher accuracy (Ribeiro et al.) 2016). This motivated the development of post hoc explanation methods. including perturbation-base
local explanation methods (Ribeiro et. al.] 2016 2018 Lundberg & Lee 2017: Slack et al., 2021), gradient-based based local explanation methods (Selvaraju et al. 2017
Simonyan ct al. 2014; Smilkov et al.] 2017 Sundararajan et al.] 2017). global explanation mothods (Bastani
et al. 2017 7 Lakkaraju et al. 2019), and other methods (Koh & Liingg 2017
However, these post hoc techniques have been shown to have flaws Rudin 2019) argued that post hoc explanations are not reliable, as these explanations are not necessarily faithful
to the underlying models and present correlations rather than information about the original computation.
There has also been recent work
on exploring vulnerabilities of black box explanations (Adebayo et al. 2018 Slack et al. 2020
Lakkaraju &
Bastani 2020: Rudin 2019
Dombrowski et al. 2019). Moreover, there is a growing literature on evaluating
the effectiveness of post hoc explanations for ML assisted decision making Doshi-Velez & Kim 2017 Kaur
et al. [020: Bhatt et al. 2020: Hong al., 2020 Lakkaraju & Bastani 2020 Pouusabzii-Sandeeh et al. 2021;
Buçinna ct al., 2020 Krishna et al. 2022) which has highlighted number of challenges and subtletics. For
instance, while prior studies have shown that decision makers can perform better when provided with ML predictions and corresponding explanations in various tasks such as medical diagnosis (Cai et al.) 2019
Lundborg et al. 2018), data annotation (Schmidt & Bicssmann] 2019). and decoption detection (Lai & Tan 2019), Bansal et al. (2021) argued that such improvements may be due to decision makers simply following the recommendations of highly accurate ML, models rather than placing so-called appropriate reliance on the models (Lee & Seel 2004).. Indeed, [ansal al. (2021) found that decision makers were more likely to follow the model's prediction when provided with explanations, regardless of the model's correctness.
We focus on studying the impact of predictive uncertainty on user behavior since it is often more easily attainable than a faithful and interpretable explanation. Based on insights from the explainable Al literature [Buçinca et al. 2020), we focus on the impact of predictive uncertainty on user behavior in an actual decision- making lask rather on than proxy tasks or subjective measures such as user trust or preference.
Predictive Uncertainty as Auxiliary Input: Bhatt et al. (2021) outline key considerations for conveying uncertainty for ML assisted decision making. Many works have empirically studied how to best communi- cate uncertainty to decision makers in tasks ranging from reacting to weather forecasts to catching public transportation (Correll& & Gleicher [014 Kay et al.] 2016 Greis et al. 2016 Roulston et al. 2006 Fernandes 2018 Kirschenbaum el al. 2014] LefTrang& & Müller 2021 Koval & Jansen 2022). For example, Greis et al. on weather forecasts. They also found that the performance was bost when showing a simpler uncertainty visualization than the full distribution. Roulston et al. (2006) finds that showing participants error bars in addition to point estimate can help them make a larger profit in a game where they must decide when to
et al.
2016 finds that people earned more when shown uncertainty estimates in a farming game that depends
3

---

Published in Transactions on Machine Learning Research (06/2023)
salt roads based on snow forecast. They also find, however, that showing explicit probabilities does not lead to further improvements. Fernandes et al. (0018) explores the question of which types of uncertainty vi- sualizations help users make the 'best' decisions according to a specified payoff function when catching a bus. They explore different uncertainty visualizations and find that all of them allow people to improve over time, and their preferred method quantile dotplots resulted in the most consistent performance. [Krschenbauum et al. (2014) studied the impact of spatial vs. tabular uncertainty representation on submarine detection and found that spatial uncertainty raised the performance of non-experts almost to that of experts. Motivated in part by a preprint of our work, Leffrang& & Müller (2021) studied the impact of showing 95% confidence interval plots and ensemble displays in time series forecasts of the number of hospital beds occupied by COVID-19 patients. They found that users were less willing to follow the model's prediction when shown more salient visualizations of uncertainly, and called for further studies evaluating the impact of conveying uncertainty visualizations. Our work differs from previous research in that we systematically study how decision makers are impacted when shown posterior prodictive distributions under several scttings. We also consider expert decision makers unlike most of these works who may be able to work with more complex information about the probability distribution (Spiegelhalter et al.. 2011
Finally, some work has been conducted on how people make decisions with MI. models that present uncer- tainty information Zhou et al. 2017 Arshad et al. 2015). For instance, Zhou et al. (017)) conducted user study to evaluate how uncertainty intervals around a point estimate affects users' predictive decision making. They found that showing users such uncertainty intervals increased trust, but only under conditions with low cognitive load. We study the impact of showing uncertainty information on downstream decisions, rather than relying trust a proxy for team performance. We also explore different features of the uncertainty distribution that may affect human decisions.
3
Goals and Study Design
3.1
Goals
In this project, we seek to understand how people make decisions when provided with the uncertainty of an ML model as an input to their decision process. The uncertainty of an ML model is auxiliary information provided by many ML models that can be used as an input to human decision making in addition to the model's prediction. Specifically, we explore how different factors related to the model's uncertainty distribution impact ML assisted decision making. We measure this through user agreement with the model's prediction before and after being shown the uncertainty information, which allows us to isolate the important factor of agreement with the machine prediction from whether the prediction is correct. We also study how ML assisted decision making differs depending on user expertise in the domain or in ML. We describe each of these factors in more detail below.
We operationalize the notion of predictive uncertainty in a Bayesian setting where predictions may be given as point estimates or as entire posterior predictive distributions. We show uncertainty information as a visualization of the model's full posterior predictive distribution, in addition to its prediction marked at the mean of the posterior predictive.
We study the effect of 2 factors related to the uncertainty distribution: the shape of the uncertainty dis- tribution, and its variance. Different shapes of distributions have different properties that may both affect how much people are willing to take them into account when making decisions, and what the optimal way to take them into account may be. We study 3 shapes of posterior predictives: a normal distribution, a skewed distribution, and a bimodal distribution. A normal distribution is commonly studied in past work (e.g.. |ernaandes et al. (2018)), and has the property that its mean is equal to its mode. This is not true however for the bimodal distribution, which has 2 modes and very little probability near its mcan, or the skewed distribution which has a mode to one side of the mean and long tail of reasonably high probabil- ity outcomes. Whether people will still update their estimmaess to reflect the ML model's prediction when presented with these distributions with more complicated properties is an open question.
We also explore the effect of variance on people's behavior. We wish to answer the question of how much uncertainty affects people's decisions when the variance of the posterior distribution is larger vs. when it is
4

---

Published in Transactions on Machine Learning Research (06/2023)
smaller. Are people more likely to agree with an ML prediction when the uncertainty around the prediction is low? That is, do they correctly interpret and respond to uncertainty? And how does this compare to when only shown a point estimate? We study this in the context of the normal distribution.
Moreover, we explore how decision differs between experts (in either the domain or in ML) and non-experts. Will people with differing levels of MT. expertise feel more or less compelled to follow the machine's 's prediction regardless of the provided uncertainty? Will they respond more or less strongly to the presented uncertainty estimates? Orthogonally, will people who know more about the domain study feel more confident in their prediction and therefore place comparatively less weight the machine's prediction? We study the relationship between these notions of expertise and decision making with various forms of uncertainty.
Finally, we study the correlation between subjective measures of trust and perceived usefulness of the ML model and participant agreement with the model. In summary, our main research questions are:
Does showing predietive uncertainty affect how closely people follow model predictions?
Does the effect of showing predictive uncertainty depend on the type of uncertainty either the shape or the variance of the distribution?
Do participant with more expertise, either in the domain in MI, more closely follow model predictions?
3.2
Study Design
In this section, we describe the details of the user study that we carried out to answer the research questions outlined in the previous section.
3.2.1 Domain and Model
Our user study was based on predicting the monthly rental prices of apartments in Cambridge, MA. Par- ticipants were provided with the number of square feet, number of bedrooms, and the relative size of each apartment (i.c., whether the apartment size is smaller than average, average, larger than average for its number of bedrooms). We provided only these features to the participants to align with typical real world scenarios where decision makers only have partial information available.
To set realistic apartment prices, we collected details such as the monthly rental listing price, square footage, and number of bedrooms of 75 apartments in Cambridge, MA from Zillow.com in November 2019. A linear regression model was fit this data to estimate the monthly rental listing price using the square footage and number of bedrooms as features. This model was then used to set the point estimates for each of the apartments in the study.
The posteriors were not generated from the model, but were instead hand generated to embody the properties we aim to study. These are described in more detail below.
All participants in the study were shown the same 10 hypothetical apartments. Specifically, the number of bedrooms and square foet for the 10 hypothetical apartments were respectively (1, 500), (1, 800), (2, 500). (2, 800), (2, 1100), (3, 800), (3, 1100), (3, 1400), (4, 1400), and (4, 1700). Three of these (namely, (2, 500), (3, 1100), (4, 1700)) were used practice examples for participants to familiarize themselves with the study (see Section 3.2.3 for more details).
3.2.2 Conditions
We consider 5 conditions in our study each of which corresponds to different properties of the posterior predictive distribution shown to the participant:
No Uncertainty: Participants were only given access the model's point estimate of the predicted rental price for each apartment.
5

---

Published in Transactions on Machine Learning Research (06/2023)
In adoition to its estimate of the price, the machine learning model may also tell how uncertain is about its estimate. If prevides information, you prices on the x axis, and the probablity the machine leaming model places on the price the y-axis.
graph with opartment
The higher blue line for a certain price, more confident the machine learning model is that that's what the apartmnnt costs. The dotteo black Ihee represents the machine learning models average estimote the coartment price.
Uncerrointy can look differrnt in different circumstances. The exomples below correspond to qualitatively different types of uncertainty that the machine learning model may have.
High prisailty
Mos 1000
Law prbabiitt - Te:p mro-saa Aall
Meas- 1000
Lave pobasiity
350
<To
1500
350
Prodiced Rost
Prefkred Rene
High Soskiii
Now MAMAAA 3 FEEA Enneeoon
Mous 1000
Lowe pestaliiny
1500 Prodiced Romt
750
a6e
Prediced Remt
Figure 1; The uncertainty tutorial page given to participants before completing the task. The tutorial shows examples of uncertainty distributions that correspond to the types of uncertainty shown different conditions. This same page is shown regardless of the condition the partieipant is randomized to. The text tells participants how to interpret the and y-axis. The dotted black line marks the mean of each distribution.
Normal, Low Variance: Participants were shown normally distributed posterior with a standard deviation (SD) of 80 for each apartment.
Normal, High Variance: Participants were shown normally distributed posterior with SD of 250 for each apartment.
Skew:
Participants were shown a right-skewed posterior generated from shifted and scaled beta distribution with a SD of 250 for each apartment.
Bimodal: Participants were shown bimodal posterior generated from a mixture two normal distributions with equal weights with SD of 250 for each apartment.
Note that each participant was shown the same type of (or absence of a) the posterior distribution throughout the study, as showing a participant different types of posterior distributions may affect the participant's 's behavior in subsequent. apartments pricing questions in the study.

---

Published in Transactions on Machine Learning Research (06/2023)
Participants were randomly assigned with equal probability to one of the 5 conditions. The same set of apartments was shown in each condition, although the order of the apartments was randomized. The mean of the distribution for each apartment was held fixed across conditions, and the distribution that the participant had been randomly assigned to was scaled to have that mean. The mean was marked on the visualization as the model's prediction, and it corresponds to the point prediction given in the no uncertainly case.
3.2.3 Procedure
At the beginning of the study, all participants were trained on how to interpret posterior distributions with a tutorial page (See Figure D. This shows examples of different distributions of uncertainty (corresponding to the 4 conditions where participants are shown uncertainty in our experiment), and describes how to read the figures showing the uncertainly distribution. Participants are shown the same tutorial regardless of which condition they are randomized to.
To allow participants to calibrate their estimates of apartment rental prices and become familiar with the performance of the model, all participants were then laken through a practice run. Each participant was shown 3 example apartments and asked to predict their prices. Participants were told that the model slightly overestimated the true rental price in one case (prediction = $2, 644, truth = $2,560), slightly underestimated the true rental price in one case (prediction $3,561, truth- $3,601), and moderately overestimated the true rental price in one case (prediction = $4,800, truth 84,290) the that (i) participants are not incentivized to blindly agree with the ML model, and (ii) participants could not easily decide that the model always over or underestimated the true value.
After finishing this practice run, participants then completed the following task for new apartments. For each apartment, the participant was asked for two estimates of rental price. Specifically, the participant was first presented with the square footage, number of rooms, and relative size of each apartment, and then asked to make a prediction about the monthly rental cost of the apartment. Then, the participant was presented with the prediction of the ML model, and the corresponding postcrior predictive distribution wherever applicable. They were then asked to update their original prediction based on this information. They were reminded of their original prediction when completing this part. This design allowed us to quantify the impact of showing participants the model output by the difference between the participant's 's first and second estimates; This was suggested as good practice by prior work (Poursabzi-Sangdee et al. 2021). . Participants were not given leedback on whether or not they were correct after any of these trials. Also, participants were not told what features the model used to predict the price of the apartments.
At the end of the survey, participants were asked to rate their trust in and the usefulness of the model using 5-point Likert scale [Likert] 1932). They were also asked to rade their familiarity with MI. by choosing one of the following options- "No understanding or little understanding", "Worked with it a few times", "Worked with it many times or on a larger project". Participants were also asked they ever lived in off- campus housing in Cambridge, MA or surrounding areas. The response to this question conveys participants' domain expertise in Cambridge rental listings, since people who have lived off campus in the Cambridge area are more likely to be familiar with apartment rental prices in the area than those who have not.
Participants were paid $3 for completing the study and could earn up to additional $30 based their performance in the study. Participants were told that their performance is measured based on the average distance between each apartment's 's true price and their first and second estimates. For the purposes of distributing bonus payments, we set the true price of the apartments to the model predictions, although recall that users were not told the true price of the 7 apartments at any point in the study. In each of the two phases of our study (see Section 3.2.5), the three participants with the lowest average distance received the $30 bonus. The median time to complete the study was approximately 10 minutes.
3.2.4 Interface
The interface (sce Figure 2 for an example depicting the Normal, High Variance condition) contains three important elements to aid participants with the study. First, it contains the model's estimate of the apartment price in bold (A). Second, if a participant was randomly assigned to an uncertainty group, the
7

---

Published in Transactions on Machine Learning Research (06/2023)
Question out
This apartment has bedrooms and 800 square feet. This apartment is smaller than average for bedrooms.
The model predicts that this apartment should cost $3258 month.
The uncertainty clistribution of the modelfs prediction looks like:
300
4000
Prodkced Remt
B
Based on the information above. how would you price this apartment month (in US Dollars)?
Your previous answer was 2900
C
Figure 2: Example of the interface for participant in the Normal, High Variance group after providing an initial estimate of the monthly rent of the apartment. shows the model's prediction, shows the visualization of the model's prediction uncertainty, C shows the participant's first response so they can reference it, and D shows the apartment's 's features.
interface shows the corresponding uncertainty distribution in color (B). Third, participants are reminded of their previous estimate in bold so they may choose condition on it (C). It also contains the apartment's features (D).
3.2.5 Participants
In order to largel participants with different degrees of ML and domain expertise for our study, recruited participants in two w First, we recruited 95 participants which were mainly students and researchers spanning various fields (machine learning, biology, physical sciences, history, etc.) across multiple univer- sities. We recruited these participants by advertising in multiple mailing lists and reaching out to various faculty across multiple universities. Additionally, we recruited an additional 95 participants on the plat- form Prolific. We required Prolific participants to have a minimal approval rate of 98%, have completed minimum of 250 previous submissions the Prolific platform, and be located in the United States.
This study was approved by our institution's IRB.
8

---

Published in Transactions on Machine Learning Research (06/2023)
3.3 Analysis
We look at 3 metrics to understand the extent to which seeing predictive uncertainty affects people's decisions: the distance (in the L1 norm) between the first estimate (before seeing any model outputs) and the second estimate (after seeing the prediction and uncertainty wherever applicable) we call this the "update"; the distance between the second estimate (after seeing the model's prediction) and the model's prediction we call this the "final disagreement"; and the distance between the first estimate (before seeing the model's prodiction) and the model's prodiction we call this the "initial disagrecment". The update allows us to determine the extent to which participants updated their estimate based on the model's prediction and uncertainty information. The final disagreement measures the extent to which people agree with what the model predicted. The initial disagrocment allows us to understand any bascline differences between participants of varying levels of expertise before they were shown the model's prediction and uncertainty information.
Our primary analyses use mixed-effects regression models for these three metrics. Unlike regression models with only fixed effects, mixed effects models allow one to properly account for correlation between repeated measurements from participants in the study (e-g., some participants may consistently change their estimates large amount while others may consistently change their estimates by small amount) (Bates et al. 2015 A random intercept was included for the participant ID in all models. No other random elfects were included in the models, All the relevant factors including the type of predictive uncertainty, background in ML, experience living in Cambridge, self-reported usefulncss and self-reported trust in the model were included as fixed effects in their respective analyses. All p-values reported are derived from a one-way ANOVA based on these models. We report p-values that are less than 0.05 as significant and p-values in the interval [0.05, 0.10] as marginal. The bar plots illustrate the sample mean of the outcome in the relevant group of participants +1 standard error based on the fitted model.
4 Results
In this section, we describe in detail the results we observed. Specifically, we analyze the data along 2 dimensions: (1) how decision making differed between experts (either in the domain or MI.) and non-experts, and (2) how decision making differed when shown different types of model uncertainty. We describe each of these in turn below, then run an analysis of the effect of model uncertainty on decision making stratified by participant expertise. In Appendix A analyze the association between subjective metries confidence and trust - and how users updated their estimates to agrec with the machine prediction.
4.1 Comparing Decision Making Between Experts and Non-Experts
Our first set of research questions are about how people's willingness to update their estimates differed based on their familiarity with either the domain (apartment rentals in Cambridge, MA) or ML models. Table [ captures the number of responses (i.e., the total mumber of apartment pricing examples completed) and participants from each category of ML expertise and domain expertise. Figure illustrates the magnitude of the update (left), final disagreement (middle), and initial disagreement (right) among participants in each of the two categories of expertise in Cambridge apartment prices (top) and two categories of ML expertise (bottom). Throughout, we combined the ML expertise categorics of "No understanding or little understanding" and "Worked with it a few times" into a single category which we refer to as "No MT. background / Some ML background", and we relabelled the category "Worked with it many times or on a larger project" to "Strong ML background" for clarily.
We found that participants with domain expertise in Cambridge apartment prices had lower initial disagree- ment and smaller updates compared to those without such expertise. However, the final disagreements of both groups were similar. Participants who have previously lived in or around Cambridge made initial esti- mates that were closer to the model's prediction (smaller initial disagreement) even before having seen the prodiction compared to those who never lived in Cambridge (significant: F81, = 14.4761, p = 0.0002). This likely reflects their increased familiarity of rental prices in Cambridge. However, we also observed that domain experts had smaller updates (significant: F11, 23.0224, p < 0.0001), and there was
9

---

Published in Transactions on Machine Learning Research (06/2023)
Table 1: Number of responses and participants with each level of expertise for the domain and ML. We have reasonable number of responses and participants with each level of expertise.
Responses
Participants
%
(N = 1, 330)
(N = 190)
Experience living in Cambridge Yes
483
69
36.32%
No Experience with ML
847
121
63.68%
Strong background
364
52
27.37%
Some no background
966
138
72.63%
Domain Expertise
Never in Cambridge. MA
Lived in Cambrndge, MA
200
150
Magnitude of update
Magnitude disagreement
Magnitude initial disagroement
ML Expertise
No buckgrouud/ Some MI. background
Strong ML bockground
150
250
350
50
150
100
200
300
400
Magnitude update
Magnitude of final disagreement
Magnitude initial disagreement
Figure 3: Magnitude of update (left), final disagreement (center) and initial disagreement (right) stratified by expertise in domain (top) and ML (bottom). Bar width corresponds to the mean, and standard errors are shown in black. Those without domain expertise had higher initial disagreement, and updated their predictions more, but they had similar final disagreement to those with domain expertise. Similar trends held for the analysis comparing ML exports and non-experts.
significant. difference between domain experts and non-experts in their final disagreement with the model (F(1,, = 0.8631, 0.3541). This suggests that the ML model equalized differences in the initial disagreement between experts and non-experts by bringing participants' final disagreement with the model into the same range, regardless of expertise.
We found that participants with more ML expertise updated their estimates less after seeing the model's prediction (signéficant: F(1,188) 5.1232, p 0.0248). Similar to the analysis comparing domain experts in Cambridge apartment prices to non-experts, (i) the final disagreement did not significantly differ between the ML experts and non-experts, and (ii) the ML non-experts had smaller initial disagreement (marginally significant: F(1, 188) = 3. 3.1833, p 0.0760). These trends may be altributed to the ML expert group having a greater portion of participants with domain expertise in Cambridge compared to the ML non- expert group. When accounting for domain expertise (i.c., by including a fixed effect term for domain expertise in our mixed-effect regression models), there were no significant differences between ML experts and non-experts in their initial disagreement, magnitude of update, final disagreement.
10

---

Published in Transactions on Machine Learning Research (06/2023)
4.2 Impact of Type of Uncertainty Decision Making
Our second set of research questions are about the extent to which showing uncertainly influences the decisions people make, and how the type of uncertainty modulates that effect. To this end, we plot the magnitude of the update (left), final disagreement (middle), and initial disagreement (right) for each of the five conditions corresponding to different kind of uncertainty or no uncertainty (Figure
[Low Variance)
Normel
(ligh Variance)
Skow
Bimodal
Uncertainty
100
200
300
400
100
200
100
300
400
Magnitude of update
Magnitude disagreement
Magnitude initial disagroement
Figure 4: Magnitude of update (left), final disagreement (center) and initial disagreement (right) stratified by uncertainty condition. Bar width corresponds to the mean, and standard errors are shown in black. The final disagreement differs between conditions, with highest agreement for normal-low variance, and lowest agreement for no uncertainly.
We found that the type of uncertainty affected participants final disagreement with the model. Furthermore, showing uncerlainly resulted in final estimates that were farthest from model predictions. The type of uncertainty had an effect on how closely participants' second estimate agreed with the machine's 's prediction (size the final disagreement) (marginally significant: F(4, 185) = 2.2149, 9, 0.0690). It can be seen from Figure f that the condition where participants were shown a normal distribution with low variance moved people closest to the model's prediction, however even cases like the normal distribution with high variance where the model is clearly uncertain, or the bimodal distribution where there is very little probability mass the model's prediction, people were still moving their second estimates closer to the model's prediction compared lo when they were shown no uncertainly. More specifically, participants shown uncertainty had an average final disagreement $49 smaller than those not shown uncertainty (significant: F(1, 188) = 8.3989, 0.0042).
We do not see scallstically significant differences in the magnitude of the update (F(4, 185) 1.5239, p= 0.1969). This could be tied high variance in the original estimate before being shown the prediction, although these are also not significantly different (F(4, 185) 1.2734, p 0.2820), suggesting that our randomization worked.
4.3 Impact of Type of Uncertainty on Decision Making Stratified by Expertise
Finally, we present an analysis of the impact of uncertainty type stratified by participant expertise. The impact of the type of uncertainty on agreement with the prediction may be modulated by expertiso. Table 2 shows the number of responses and participants in each uncertainty group, stratified by different levels of domain and ML expertise. While the sample sizes are small to draw strong conclusions, this analysis helps gain further context on how expertise and uncertainty type interact, and suggests areas for future rescarch. Figure 5 illustrates the magnitude of the update (left), final disagreement (middle), and initial disagreement (right) for each uncertainty type/no uncertainty among participants in each of the two categories of domain expertise. Figure shows analogous results with respect to ML expertise.
Uncertainty type by domain expertise: While we found that the magnitude of the update and initial dis- agreement much higher for people without domain expertise, the relatine trends of the impact of the
11

---

Published in Transactions on Machine Learning Research (06/2023)
Table 2: Number of responses and participants (within brackets) in each uncertainty group, stratified by
expertise for the domain and for ML. Most groups have more than 70 responses and 10 participants.
Normal
Normal
Expertise
No Uncertainty
Skew
Bimodal
(Low Variance)
(High Variance)
Domain
Yes
112 (16)
105 (15)
98 (14)
77 (11)
91 (13)
No
168 (24)
154 (22)
154 (22)
189 (27)
182 (26)
ML
Strong
105 (15)
63 (9)
63 (9)
77 (11)
56 (8)
Some or no
175 (25)
196 (28)
189 (27)
189 (27)
217 (31)
Domain: Expert
Normal
(Low Variance)
Normal
(High Variance)
Skew
Rimodill
Uncertainty
100
200
300
400
50
100
150
200
250
100
300
500
Magnitude of update
Magnitude final disugreement
Magnitude initiial disagreement
Domain: Non-Expert
(Law Variance)
Normal (ligh Variance)
Skow
Dimodia
Uncertuinty
100
400
250
Magnitude of update
Magnituds final disagrcement
Magnitude initial disagreement
Figure 5: Magnitude of update (left), final disagreement (center) and initial disagreement (right) stratified by uncertainty condition and by domain experts (top) and non-experts (bottom). Bar width corresponds to the mean, and standard errors are shown black. The trends in the magnitude of update across uncertainty types were similar for domain experts and non-experts, however non-experts had larger updates across conditions. Non-experts and experts had similar final disagreements.
type of uncertainty similar for domain experts and non-ezperts. The normal low-variance and skew conditions had amongst the largest updates for both domain experts and non-experts, and the normal high variance condition had the smallest updates for both groups. For both domain experts and non-experts, the initial disagreement. was similar between the different uncertainty conditions, as one would expect due to the randomization.
Uncertainty type by domain ezpertise: We found that participants in the no uncertainty condition had the largest final disngreement for both domain expertise groups. The final difference between the participants'
12

---

Published in Transactions on Machine Learning Research (06/2023)
ML: Strong Background
Normal (Low Variance)
Normal (High Variance)
Skew
Binodial
No Uncertainty
100
200
300
400
0
50
100
150
200
250
0
100
300
500
Magnitude of update
Magnitude of final disagreement
Magnitude of initial disagreement
ML: Some or No Background
Normal (Low Variance)
Normal (ligh Variance)
Skow
Bimodal
Uncertainty
100
200
300
400
50
100
150
200
250
100
300
500
Magnitude of update
Magnitude of final disagreement
Magnitude of initial disagreement
Figure 6: Magnitude of update (left), final disagreement (center) and initial disagreement (right) stratified by uncertainty condition and by background in ML. Bar width corresponds to the mean, and standard errors are shown in black. In the ML expert group, the type of uncertainty affected the magnitude of the update (significant: F(4,47)
3.0754, p 0.0249) and the final disagreement (significant: F(4, 47) 3.6643, p 0.0112). ML experts updated their estimates the most when shown the normal low-variance uncertainty, while all other types of uncertainty had a somewhat similar effect to not showing any at all. There are no comparable marked differences for the non-experts.
estimates and the model's predictions were relatively consistent between the different types of uncertainty for both groups. For domain non-experts, the final disagreement was largest in the no uncertainty condition. While this was also true for domain experts, it was not as marked.
Uncertainly type by ML. expertise: We found that ML experts updated their original estimates more with the normal low voriance condition than for any other, while there were no clear trends for ML non-experts. From the left column of Figure G the normal low variance condition had much larger updates for experts (top) than those with only some ML experience or no ML experience (bottom). For the experts, the updates were much larger for this condition than for any of the others, while the updates were somewhat similar for all of the other uncertainty conditions and the no uncertainty condition. Perhaps ML experts are used to working with normal distributions and trust the prediction when the ML model is relatively certain, but do not agree to the same extent ML models that do not present their uncertainty, or present uncertainty estimates that have high variance and possibly hard-to-interpret distributions. For the non-cxperts, the trends were not as clear, and the updates appear relatively similar for the different uncertainty conditions. Perhaps participants with less experience with ML do not consider the variance of the distribution as much and have less of prior on which uncertainty displays from the model are reliable. Further exploring this hypothesis is interesting future work, as suggests that ML non-experts are perhaps less likely respond appropriately to uncertainty.
13

---

Published in Transactions on Machine Learning Research (06/2023)
5 Discussion and Conclusions
We studied how conveying predictive uncertainty to end users impacts decisions in the context of ML assisted decision making. In particular, we explored the extent to which different properties of the posterior predictive distribution affected participant agreement with the ML model's prediction in an apartment pricing task. We also explored how participant agreement with the ML model's prediction differed between experts and non-experts. Better understanding when humans agree with machine predictions, and how human expertise and machine uncertainty influence this agreement, lays the foundation for successful human-AT teamwork.
We presented new insights about how uncertainty can affect ML assisted decisions. Most interestingly, show- ing posterior predictive distributions (regardless of the shape and variance of the distribution) increased par- ticipant agreement with model's prodictions compared to showing no uncertainty estimates at all. However, we also found that people with ML expertise agreed with the model most when provided with posterior predictive distributions that were normally distributed and had low variance. This suggests that the level of human expertise may modulate the effect of showing posterior predictive distributions, and therefore should be taken into account.
5.1 Limitations and Future Directions
While our study makes one of the first attempls at investigating the impact of different types of posterior predictive distributions in ML assisted decision making, it has limitations which may affect the generaliz- ability of our findings. First, while the domain experts in our study had significantly more knowledge of apartment prices in Cambridge compared to the non-experts as evidenced by having significantly smaller initial disagreement with the model, it would be a good idea to consider users with a higher degree of domain expertise such as local real estate agents. In this case, we would expect greater differences in decision making between domain experts and non-experts. Second, as our study focuses entirely on the task of estimating apartment prices, results may differ for other decision making lasks and in other contexts. For example, human behavior studies have found that participant behavior can vary considerably between different tasks or in different contexts due to factors such as risk sensitivity (Lakkaraju & Bastani 2020 Poursabzi-Sangee et al. 2021 Attali et al. 2011). Third, the two-stage nature of our study design (i.e., participants providing price estimates prior to and after seeing the model output) may be susceptible to self-priming effects (i.e., participants' price estimates after sceing the model output may be affected by having to provide initial price estimates without seeing the model oulput). To help miligate potential self-prining effects, we offered $30 incentive for users to make the most accurate apartment price predictions. Furthermore, this study design has been suggested as good practice by prior work (Poursabzi-Sangdch ot al. 2021
Besides the lype of posterior predictive distribution and expertise of the decision makers, there may be other important factors that affect the impact of predictive uncertainty in ML assisted decision making. One such factor may be model performance. For example, if decision makers are aware that the model has excellent performance, they may heavily rely on the model regardless of their expertise or the shape or variance of the posterior predictive distribution. Exploring the impact of model performance in this context would be an interesting avenue for further work.
The setting we considered is ML assisted decision making under limited information. In this context, one can consider scenarios where users have less than, more than, or the same features as the ML. model. Each of these three scenarios has been studied by seminal works in the ML assisted decision making literature. For example, Poursabzi-Sangdeh et al. (2021) studied scenarios where users have the same features as the ML model and scenarios where users have more features than the ML model. As one of the first studies investigating how different types of posterior predictive distributions affect ML assisted decision making (and how such effects vary between domain/ MLL experts and non-experts), we did not want to introduce ionalional complications arising in the scenario where users have fewer features than the ML model. For instance, if the ML model inchudes some additional features (not available to the user) that are highly influential for a few apartments, model predictions may diverge heavily from experts' understanding and create distrust in the model. Exploring these additional scenarios would be interesting future directions.
14

---

Published in Transactions on Machine Learning Research (06/2023)
Finally, note that there are several sources of predictive uncertainty, which have been categorized into data uncertainty (also referred to as aleatoric uncertainty), model uncertainty (also referred to epistemic uncertainty), and distributional uncertainty (Malinin & Gales) 2018). Prior works have highlighted the importance of these different sources of uncertainty (c.g., Dusenberry et al. (2020) on model uncertainty). While our work focused on posterior predictive distributions which captures data and model uncertainly (Dusenberry al.) 2020) it would be interesting to investigate the impact of presenting all these three of these types of uncertainty to decision makers.
Acknowledgements
We would like to thank the reviewers and action editor for their comments which helped us improve the paper. SM and IL were supported by the National Science Foundation Graduate Research Fellowship Program under Grant No. DGE1745303. SM was also supported by the National Library Of Medicine of the National Institutes of Health under Award Number T32I.M12241 and Fonds de recherche du Québec - Nature et technologies BIX research scholarship. HL is supported in part by NSF awards # 2008461 and # #11S- 2040989, and rescarch awards from Google, JP Morgan, Amazon, Bayer, Harvard Data Science Initiative, and D Institule at Harvard. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the funding agencies.
References
Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, and Been Kim. Sanity checks for saliency maps. In Advances in Neural Information Processing Systems, pp. 9505-5505555 9515, 2018.
Syed Z Arshad, Jianlong Zhou, Constant Bridon, Fang Chen, and Yang Wang. Investigating user confidence for uncertainty presentation in predictive decision making. In Proceedings of the Annual Meeting of the Australion Special Interest Group for Computer Human Interaction, pp. 352-3600 2015.
Yigal Attali, Zvika Neeman, and Analia Schlosser. Rise to the challenge or not give a damn: differential performance in high vs. low stakes tests. Discussion Paper 5693, Institute for the Study of Labor (IZA), 2011.
Gagan Bansal, Tongshuang Wu, Joyce Zhou, Raymond Fok, Besmira Nushi, Ece Kamar, Marco Tulio Ribeiro, and Daniel Weld. Does the whole exceed its parts? The effect of Al explanations complementary team performance. In Proceedings of the 2021 CHI Conference on Human Factors in Compuling Systems, pp.
1-16, 2021.
Osbert Bastani, Carolyn Kim, and Hamsa Bastani. Interpretability via model extraction. ww.Xi:1706.0999 09777. 2017.
Douglas Bates, Martin Mächler, Ben Bolker, and Steve Walker. Fitting lincar mixed-effects models using 1me4. Journal of Slatistical Soflware, Articles, 67(1):1 48, 2015. ISSN 1548-7660. doi: 1...8667/jss.v06. 101. URL https: wttp::/wwwtttooottt/// wwwpeefttttrrrgeeeeee wwwp:o//////000//
Umang Bhatt, Alice Xiang, Shubham Sharma, Adrian Weller, Ankur Taly, Yunhan Jia, Joydeep Ghosh, Ruchir Puri, José MF Moura, and Peter Eckersley. Explainable machine learning in deployment. In Proceedings of the 2020 Conference Fairness, Accountability, and Transparency, pp. 648-651, 2020.
Umang Bhatt, Javier Antorán, Yunfong Zhang, Q Vera Liao, Prasanna Sattigeri, Riccardo Fogliato, Gabrielle Melançon, Ranganath Krishnan, Jason Stanley, Omesh Tickoo, et al. Uncertainty as form of trans- parency: Measuring, communicating, and using uncertainty. Tn Proceedings of the 2021 AAAT/ACM Conference A1, Ethics, and Society, pp. 401 413, 2021.
Jacob Bien and Robert Tibshirani. arXiv:0908.2200 2284, 2009.
Classification by cover: The prototype vector machine.
15

---

Published in Transactions on Machine Learning Research (06/2023)
Zana Buginca, Phoebe Lin, Krzysztof Z Gajos, and Elena L Glassman. Proxy tasks and subjective measures can be misleading in evaluating explainable AT systems. In Proceedings of the 25th International Conference on Intelligent User Interfaces, pp. 454 464, 2020.
Carric Cai, Samantha Winter, David Steiner, Lauren Wilcox, and Michael Terry. "Hello AT:: uncovering the ouboarding needs of medical practitioners for human-Al collaborative decision-making, Proceedings of the ACM on Human- Computerr Interaction, 3(CSCW):1 2019.
Rich Caruana, Yin Lou, Johannes Gehrke, Paul Koch, Mare Sturm, and Noemie Elhadad. Intelligible models for healtheare: Predicting pncumonia risk and hospital 30-day readmission. In Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Dala Mining, pp. 1721 1730. ACM, 2015.
Michael Correll and Michael Gleicher. Error bars considered harmful: Exploring alternate encodings for mcan and crror. IEEE Transactions on Visualization and Computer Graphics, 20(12):2142-2133 2014.
Aun-Kathrin Dombrowski, Maximilian Alber, Christopher Anders, Marcel Ackermann, Robert Müller, and Pan Kessel, Explanations can be manipulated and geometry is to blame. WyXiv:196.07998. 2019.
Finale Doshi-Velez and Been Kim. Towards a rigorous science of interpretable machine learning.
arXiv: 1rXXiv1102.000 08608, 2017.
Michael W Dusenberry, Dustin Tran, Edward Choi, Jonas Kemp, Jeremy Nixon, Ghassen Jerfel, Katherine Heller, and Andrew M Dai. Analyzing the role of model uncertainty for electronic health records. In Proceedings of the ACM Conference on Health, Inference, and Learning, pp. 204-213, 2020.
Michael Fernandes, Logan Walls, Sean Munson, Jessica Hullman, and Matthew Kay. Uncertainty displays using quantile dotplots or edfs improve transit docision-making, In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, New York, NY, USA, 2018. Association for Computing Machin- ery. ISBN 9781450356206. doi: 10.1145/3173374..177777 URL http::///ddl..rrr10111515553
3173718
Heather M. Giannini, Jennifer C. Ginestra, Corey Chivers, Michael Draugelis, Asaf Hanish, William D. Schweickert, Barry D. Fuchs, Laurie Meadows, Michael Lynch, Patrick J. Donnelly, Kimberly Pavan, Neil O. Fishman, C. William III Hanson, and Craig A. Umscheid. machine learning algorithm to predict severe sepsis and septic shock: Development, implemennaation, and impact on clinical practice*. Critical Care Medicine, 47(11), 2019. URL latzps://journals 1ww 11000/A Machine Leennaaeaaaeaaaannggg...nn_ -earningg.Agoritttimn. to Predict_ Serrdiict_Seeee...t 22 aspx
Miriam Greis, Passant El. Agroudy, Hendrik Schuff, Tonja Machulla, and Albrecht Schmidt. Decision-makiin under uncertainty: How the amount of presented uncertainty influences user behavior. In Proceedings of the 9th Nordic Conference on ComputCr Interaction, New York, NY, USA, 2016. Association for Computing Machinery. ISBN 97814503446631 doi: 211144/977148529999 ezp-prod1. hul. harvard. 1t.
URL datps://doi-or
2971535
Sungsoo Ray Hong, Jessica Hullman, and Enrico Bertini. Human factors in model interprotability: Industry practices, challenges, and needs. Proceedings of the ACM on Himan-Compiun Interaction, 4(CSCW1): 1-26, 2020.
Harmanpreet
Kaur, Harsha Nori, Samuel Jenkins, Rich Caruana, Hanna Wallach, and Jennifer Wort- man Vaughan. Interpreting interpretability: Understanding data scientists' use of interpretability tools for machine learning. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, pp. 1-14, 2020.
Matthew Kay, Tara Kola, Jessica R Hullman, and Scan A Munson. When (ish) my bus? uscr-centered visualizations of uncertainty in everyday, mobile predictive systems. In Proceedings of the 2016 CHI Conference Human Factors in Compuling Systems, pp. 5092 5103, 2016.
16

---

Published in Transactions on Machine Learning Research (06/2023)
Susan S. Kirschenbaum, J. Gregory Trafton, Christian D. Schumn, and Susan B. Trickett. Visualizing uncer- tainty: The impact on performance. Human Factors, 56(3):509-52) 520, 2014. doi: 101117//00152201134400000. PMID: 24930172.
URL https://doi.org/// http:///...111700000228111443300
Jon Kleinberg, Himabindu Lakkaraju, Jure Leskovec, Jens Ludwig, and Sendhil Mullainathan. Human
decisions and machine predictions. The Quarterly Journal of Economics, 133(1):237 293, 08 2017. ISSN 0033-5533. doi: 10.1093/qii /01/0/////////ia gj//qjx032. URL https://doi...... http///o..oog/////// org/10. (098///j/qjxo0//
Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In Proceedings of the 34th International Conference Machine Learning, pp. 1885-1894, 2017.
Morgane Koval and Yvonne Jansen. Do you see what you mean? Using predictive visualizations to re- duce optimism in duration estimates. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems, pp. 1-19, 2022.
Satyapriya Krishna, Tessa Han, Alex Gu, Javin Pombra, Shahin Jabbari, Steven Wu, and Himabindu Lakkaraju. The disagrocment problem in explainable machine lcarning: practitioner's perspective. arXiv preprint ar Xrv::22202.011 01602, 2022.
Isaac Lage, Emily Chen, Jeffrey Hc, Menaka Narayanan, Been Kim, Sam Gershman, and Finale Doshi- Velez. Human evaluation of models built for interpretability. In Proceedings of the Seventh AAAI Conference on Human Computation and Crowdsourcing, HCOMP '19, New York, NY, USA, 2019. AAAT Press.
Vivian Lai and Chenhao Tan. On human predictions with explanations and predictions of machine learning models: A case study on deception detection. In Proceedings of the Conference on Foirness, Accountability, and Transporency, pp. 29 38, 2019.
Vivian Lai, Chacha Chen, Q Vera Liao, Alison Smith-Renner, and Chenhao Tan. Towards a science of hunan-AI decision making; A survey of empirical studies. arxi preprint urXiv:2112.1144 2021.
Himabindu Lakkaraju and Osbert Bastani. "How do I fool you?": Manipulating user trust via misleading black box explanations. In Proceedings of the 3rd A AAAT/ACM Conference AI, Ethics, and Sociely, pp. 79-85, 2020.
Himabindu Lakkaraju, Stephen Bach, and Jure Leskovec. Interpretable decision sets: A joint framework for description and prediction. In Proceedings of the 22nd ACM SIGKDD International Conference Knowledge Discovery and Data Mining, pp. 1675-1684, 2016.
Himabindu Lakkaraju, Ece Kamar, Rich Caruana, and Jure Leskovee. Faithful and customizable explanations of black box models. In Proceedings of the 2nd AAAI/ACM Conference on Artificial Intelligence, Ethics, and Sociely, pp. 131-138, 2019.
John D Lee and Katrina A See. Trust in automation: Designing for appropriate reliance. Human Factors, 46(1):50- 80, 2004.
Dirk Leffrang and Oliver Müller. Should follow this model? The effect of uncertainty visualization on the acceptance of time series forecasts. In 2021 IEEE Workshop on TRust and EXpertise in Visual Analytics (TREX), pp. 20 26. IEEE, 2021.
Benjamin Letham, Cynthia Rudin, Tyler H. McCormick, and David Madigan. Interpretable classifiers using rules and bayesian analysis: Building a better stroke prediction model. Arnals of Applied Statistics, 2015.
Rensis Likert. A technique for the measurement of attitudes. Archives of Psychology, 1932.
Yin Lou, Rich Caruana, and Johannes Gehrke. Intelligible models for classification and regression. In Pro- ceedings of the 18th ACM SIGKDD International Conference Knowledge Discovery and Data Mining, pp. 150-155 158, 2012.
Scott Lundberg and Su-In Lee. unified approach to interpreting model predictions. In Advances Neural Information Processing Systems, pp. 4765 -7744 2017.
17

---

Published in Transactions on Machine Learning Research (06/2023)
Scott M Lundberg, Bala Nair, Monica S Vavilala, Mayumi Horibe, Michael J Eisses, Trevor Adams, David E Liston, Daniel King-Waa Wai Low, Shu-Fang Newman, Jerry Kim, et al. Explainable machine-learning pre- dictions for the prevention of hypoxaemia during surgery. Nature Biomedical Engineering, 2(10):749 760, 2018.
Andrey Malinin and Mark Gales. Predictive uncertainty estimation via prior networks. Advances in Neural Information Processing Systems, 31, 2018.
Forough Poursabzi-Sangleh Daniel G Goldstein, Jake M Hofman, Jemnifer Wortman Wortman Vaughan, and Hanna Wallach. Manipulating and measuring model interpretability. In Proceedings of the 2021 CHI conference on human factors in computing systems, pp. 1-52, 2021.
Marco Ribeiro, Sameer Singh, and Carlos Guestrin. Anchors: In Proceedings of the 32nd AAAT Conference Artificial Intelligence, pp. 1527-1535, 2018.
agnestc explanations.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. "Why should trust you?": Explaining the prodictions of any classifier. In Proceedings of the 22nd ACM SIGKDD, pp. 1135-1144, 2016.
Mark S. Roulston, Gary E. Bolton, Andrew N. Kleit, and Addison L. Collins. A laboratory study of the benefits of including uncortainty information in weather forecasts. Weather and Forecasting, 21(1): 116 122, 2006. doi: 10.1175/ 10.1175/WAPSST 1..8..11 URL https://aoi. 11tp//00.115///A888..11
Cynthia Rudin. Stop explaining black box machine learning models for high stakes decisions and use inter- pretable models instead. Nature Machine Intelligence, (5):2006 2019.
Philipp Schmidt and Felix Biessmann. Quantifying interpretability and trust in machine learning systems. arXiv preprint ar Xi:: 11011999115... 2019.
Ramprasaath Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE International Conference on Computer Vision, pp. 618 626, 2017.
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. In Proceedings of the 2nd International Conference on Learning Representations, 2014.
Linda J. Skitka, Kathleen L. Mosier, and Mark Burdick. Does automation bias decision-making?
In-
ternotional Journal of Human Coman-C Studies, 51(5):991 1006, 1999. ISSN 1071-5819. doi: https://doi.org/ wtt.://.o///1.1/////111 (000//jccc.999.0222. URL http::/www..iiieneeeeee http::/wwwaaaaeee.......oeeeaeea $10715819999902555
Dylan Slack, Sophie Hilgard, Emily Jia, Sameer Singh, and Himabindu Lakkaraju. Fooling LIME and SHAP: adversarial attacks on post hoc explanation methods. In Proceeding of the 3rd AAAI/ACM Conference on AI, Ethics, and Sociely,, pp. 180-186, 2020.
Dylan Slack, Anna Hilgard, Sameer Singh, and Himabindu Lakkaraju. Reliable post hoc explanations: Modeling uncertainty in explainability. Advances Neural Information Processing Systems, 34:9391- 9404, 2021.
Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda B. Viégas, and Martin Wattenberg. SmoothGrad: removing noise by adding noise. In ICML Workshop on Visualization for Deep Learning, 2017.
David Spiegelhalter, Mike Pearson, and Ian Short. Visualizing uncertainty about the future. Science, :33(6043):1393 1400, 2011. ISSN 0036-8075. doi: /00/3/33/448/33333
URL https://science.
sciencemag
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In Proceedings of the 34th International Conference on Machine Learning, pp. 3319-3328, 2017.
Berk Ustun and Cynthia Rudin. Learning optimized risk scores. aixiv preprint arx 1610.0016. 2016.
18

---

Published in Transactions on Machine Learning Research (06/2023)
Fulton Wang and Cyuthia Rudin. Falling rule lists. In Proceedings of the 18th International Conference Arlificial Intellignnce and Statisties, pp. 1013-1022) 2015.
Ming Yin, Jennifer Wortman Vaughan, and Hanna Wallach. Understanding the effect of accuracy on in machine learning models. In Proceedings of the 2019 CHI Conference Human Factors in Computing
Systems, CHT '19, pp. 279:1 279:12, New York, NY, USA, 2019. ACM. ISBN 978-1-4503-5970-22 doi: 10.1145/33290005333000 33000000000. URL http:://di...eeeem wwmoi.acm..o http:://doo..a..org////0. 00g/10. 1145/ 30.55//2906055.0 /155///0..115/2290005333330 3300509
Jiaming Zeng, Berk Ustun, and Cynthia Rudin. Interpretable classification models for recidivism prediction. Journal of the Royal Statistical Society: Series A (Statistics in Society), 180(3):689-77) 2017.
Jianlong Zhou, Syed Z Arshad, Simon Luo, and Fang Chen. Effects of uncertainty and cognitive load on user trust in predictive decision making. In TFIP Conference on Human- Computer Interaction, pp. 23-39. Springer, 2017.
A Correlations Between Self-Reported Measures and Decision Making
The final question we explored to what extent self-reported measures of trust and usefulness correlated with people's actual decisions. Table 3 captures details of participant ratings indicating how much they trust the model and how useful they find it. We combined the usefulness categories of "Very useless" and "Somewhat useless" into a single category called "Useless", and we combined the categories of "Somewhat useful" and "Very usefull" into a single category called "Usefull" Similarly, we combined trust categories of "None at all" and "A little" into a category called "Little/Nond" and we combined the categorics "A lot" and "A great deal" into category called "Lot". In the top panel of Figure 7 we give the average magnitude of update (left panel), final disagreement (middle panel), and initial disagreement (right. panel) among participants in each category of self-reported trust. The bottom panel of Figure 7 gives the analogous results for self-reported usefulness.
We found that self-reported trust correlated with the magnitude of the final disagreement (significant: 187, = 4.5907, p 0.0113), but the trend was less clear for usefubness. The magnitude of the final disagreement largely decreased with trust, as one may expect. While the magnitude of the update increased with trust, the trend was not significant (F(2, 187) 1.8490, p 0.1603). We did not observe any statisti- cally significant differences in any of the three considered metrics between the groups defined by self-reported usefulness of the model.
Table 3: Number of responses and participants with each level of self-reported trust, and perceived usefulness of the model. Most participants trusted the model moderately or lot, and most participants found the model useful.
Responses
Participants
Self-reported measure
%
(N = 1,330)
(N = 190)
Trust in the model
Little/Non
168
24
12.63%
Moderate amount
616
88
46.32%
Lot
546
78
41.05%
Usefulness of the model
Useless
70
10
5.26%
Neither useful nor useless
112
16
8.42%
Useful
1148
164
86.32%
19

---

Published in Transactions on Machine Learning Research (06/2023)
Trust
Moderate aucoun
Lot
50
150
350
50
100
150
200
100
200
300
400
Magnitude of update
Magnitude final disagreement
Magnitude initial disagroement
Usefulness
Islless
Neither useful
Useful
100
200
300
400
0
50
100
150
200
0
100
200
300
400
500
Magnitude of update
Magnitude of final disagreement
Magnitude of iniiiiil disagreement
Figure 7: Magnitude of update (left), final disagreement (center) and initial disagreement (right) stratified by self-reported trust (top) and self-reported usefulness (bottom). Bar width corresponds to the mean, and standard errors shown in black. The magnitude of the final disagreement was significantly different between the groups defined by self-reported trust. There were no significant differences between the groups defined by self-reported usefulness.
B Additional Study Details
B.1
Additional Participant Details
Recall that we recruited two groups of participants in our study. The first group of participants (i.c., the students and researchers) were recruited by advertising in mailing lists and contacting faculty across multiple universities during July Setteebbee 2020. There were a total of 95 participants this group.
The second group of participants were recruited from the platform Prolific during April 2023. A total of 95 individuals from this group completed the study. There were four participants who did not complete the study. None of their responses were included in analyses. The average age of the study participants from Prolific was 37.14 years (range: 19 years to 66 years). A total of 41 (43.16%) of these participants were male and 54 (56.84% were female.
B.2 Additional Data Analysis Details
There were three cases where participant made a clear typo in their response. In each of these cases, the participant appeared to have accidentally included an extra 0 or omilted a in their apartment rental price estimate, These three cases are detailed below.
One participant gave a second estimate of $200, which we took be $2,000 in our analysos. This corresponded the apartment with bedroom and 500 square feet. This participant's first estimate was $1,500 and the model prediction was $2,105.
20

---

Published in Transactions on Machine Learning Research (06/2023)
One participant gave second estimate of $44,000, which we took to be $4,400 in our analyses. This corresponded to the apartment with 4 bedrooms and 1,400 square feet. This participant's 's first estimate was $4,300 and the model prediction was $4,498.
One participant gave first estimate of $200, which we took to be $2,000 in analyses. This corresponded to the apartment with bedroom and 800 square feet. This participant's second estimate was 82,400 and the model prediction was $2,405.
21


Reviews:
[{"review_id": "MULELaqt7J", "reviewer": "Reviewer_m3a9", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- According to the paper, this is the first attempt to address the question \"How does various factors (e.g. distribution type, variance, human expertise) affect how people agree with model prediction?\". The paper uses a well thought-out procedure to eliminate the factor of model prediction performance, and only focuses on the question. While this seems like a strength on the one hand (i.e. simplifies the question and the complexity of the required analysis), it also sets a hard limitation to the depth of the research, which I'll elaborate in the Weaknesses.\n- The analysis of the results is overall quite reasonable, and the paper does a great job of summarizing the entire experiment in a few simple statements.\n\nWeaknesses:\n- As noted above, this paper simplifies the question \"how does uncertainty affect human decision?\" by eliminating the factor of model performance from the entire process. In other words, this paper only focuses on how distribution types and the magnitude of variance affects human decision, coupled with human expertise. To guarantee this, during the practice run, the participants were shown three model predictions where the model under-estimates and over-estimates at least one time respectively, so that people would not blindly believe the model predictions during the actual run. Although a clever approach, this seems to over-simplify how people actually engage with ML models in the real world. I believe model performance would play a crucial role in how people react to model uncertainty. For example, with a prediction model that people know to possess poor accuracy, I assume human expertise would play a much bigger role than distribution types or variance magnitude. With a very accurate model (better than most domain experts), maybe none of the factors (human expertise, distribution types, variance magnitude) would play a significant role. \n- Another aspect this paper do not address in terms of model uncertainty is different types of uncertainty: epistemic uncertainty (which is also called model uncertainty), aleatoric uncertainty (a.k.a. data uncertainty). Following this nomenclature, this paper is in fact, studying data uncertainty, not model uncertainty. Some works [1] even go as far as counting out-of-distribution (distributional uncertainty) as a third type of uncertainty. If people are given all two (or three) types of uncertainty information, they would behave very differently compared to when given only aleatoric uncertainty (as this paper). For example, in safety critical tasks such as medical prediction, epistemic uncertainty could play as big a role as aleatoric uncertainty [2].\n\nReferences:\n[1] Malinin, Andrey, and Mark Gales. \"Predictive uncertainty estimation via prior networks.\" Advances in neural information processing systems 31 (2018).\n[2] Dusenberry, M.W., Tran, D., Choi, E., Kemp, J., Nixon, J., Jerfel, G., Heller, K. and Dai, A.M., 2020, April. Analyzing the role of model uncertainty for electronic health records. In Proceedings of the ACM Conference on Health, Inference, and Learning (pp. 204-213).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work conducts a set of experiments to answer the question \"How do people respond to model uncertainty?\" in multiple dimensions such as the effect of expertise on the task/ML, the effect of distribution type and variance. The paper uses apartment rental price regression as the main task, and recruits 95 people to make an initial prediction, see the model output (along with its uncertainty), and update their prediction. The paper collects the survey results and conducts multiple analyses.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- If the paper could include a limitation section to address the above concerns, that would make this paper stronger.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "dtdMZGBn4n", "reviewer": "Reviewer_s9ZU", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n * Timely and important problem: how should ML model outputs be presented to users, and what impact do different ways of communicating predictions to users have?\n * Well written paper\n * A number of clearly stated hypotheses that are tested\n\n**Weaknesses:**\n * Number of participants and/or apartments rated too low. Though aggregate results are technically speaking stat. significant, the stratified results in Fig. 5 (where experts and non-experts are shown independently) suggest experts and non-experts differ in their adjustments. Unfortunately the stratified cohorts in Fig. 5 are much too small (e.g., 6 or even just 3 participants in the Non-Expert category for some distributions). See Requested Changes for a more detailed discussion, but the effects and discrepancies in Fig. 5 cannot be ignored solely with the argument that aggregate statistics in Fig. 3 yield a good enough p-value. Since recording additional participants in the study is very quick and cheap (3$ per participant for rating 7 apartments) I strongly recommend collecting at least two, to three times more data.\n * Generalizability of findings is very limited. The paper is written as a general investigation into how different types of uncertainty of ML predictors affect human predictions that get to work with the model outputs. Main claims in the paper are stated very generally. This is in stark contrast to the experimental results, which investigate a single, very simple task, and 5 kinds of ML model uncertainty types (4 hand-crafted distributions). This needs to be addressed by toning down the claims and adding a well written limitations section or, even better, adding more different tasks to the study.\n * No true expert group. The expert group in the study is only marginally more experienced in the task than the non-experts. The study would greatly benefit from including another group that is recruited from local property agents (specialising in rental properties). I would not be surprised if these actual experts, who have high confidence in their predictions, would adjust a lot less based on seeing the model’s predictions - producing a much more pronounced difference between experts and non-experts and maybe being less sensitive to the type distribution of the model prediction.\n\n**Overall verdict:** The problem that the paper contributes to answering is timely and important, and the main idea behind the study design in the paper is good. I personally would interpret the current results as a promising proof-of-concept (where the results for the main hypotheses look promising, but some effects in the stratified data can currently not be explained and are likely due to low sample size, but this needs to be investigated since these effects bleed into the aggregate data). The next step after a proof-of-concept is to conduct a larger scale study, and I think the work is ready to do this (and compared to many other behavioral studies, collecting more data is relatively quick and cheap in this case). The paper would greatly benefit from this; the findings presented would be more reliable and convincing. To me personally, the work currently does not pass the threshold for publication - though the paper is well written and the main approach sound, there is more needed than “just getting the p-values for the aggregate results right”. Some of the claims and findings in the paper are not reliable enough, and the unexplained effects (which the paper itself attributes most likely to having too small a sample size) need to be addressed. Luckily this can be done very easily and straightforwardly by collecting more data. I think there is good potential for conducting a well-rounded, exemplary empirical study, but the current state of the empirical work falls short of the high standards needed for good behavioral studies with humans. I want to strongly encourage the authors to spend the extra work to make this a strong behavioral study that stands the test of time.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper investigates to which degree humans, when predicting rental prices of apartments, adjust their predictions when being presented with the prediction of a ML model. Importantly, the paper studies the effect of showing the ML model prediction with different degrees of certainty, represented by different distributions around the mean prediction. The main finding is that showing the model’s uncertainty estimates, instead of just the mean, has an effect on the degree to which humans adjust their predictions. The study also finds that the adjustment of predictions is modulated by the shape of the distribution (e.g., low-variance Normal leads to lager adjustment compared to high-variance Normal). The results are based on a study with 95 participants, that each predicted the rental price of 7 apartments (knowing only the size and number of bedrooms). Additionally, the paper investigates whether having more or less domain knowledge has an influence on the prediction adjustments; and similarly for having little, moderate, and significant experience with ML.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "**Improvements / Major comments:**\n\n1. Instead of reporting all measurements (initial and final disagreement and magnitude of update) in absolute values (dollars), report results in terms of quantities normalized by the true prize of the apartment. E.g., if the initial prediction is 1200USD for an apartment that costs 1000USD, report the initial disagreement as 1.2. This ensures that predictions and adjustments for apartments of very different sizes (and thus absolute rent) are comparable. A side-effect might be that some of the stat. tests are no longer stat. significant - which I would take as a strong hint that the effect is too weak or the sample size is too low.\n\n2. Record more participants. Since recording for this study is neither time-consuming nor expensive (compared to typical behavioral studies), I suggest to double or triple the number of participants. Without additional data, the paper is left with the results in Fig. 3 (which look valid, but do not show the paper’s main point of investigation: different predictor distributions), and the results in Fig. 4, but since Fig. 3 and Fig. 5 suggest that experts and non-experts update differently, the stratified results are very important.\n\n3. I think Fig. 5 is the main figure of the current study (because Fig. 4 aggregates over potentially different update mechanisms between experts and non-experts). Unfortunately it has some shortcomings: (A) within each group there is no reason to expect a difference in initial agreement (right column), but there is clearly one for the non-experts (blue bar). This is a very strong hint that the sample size for each stratified cohort is too low. (B) The largest adjustment is seen for the non-experts in the grey condition (Normal, low variance) - but this is based on 3 participants only according to Table 2. That number is concerningly low. Both issues A and B can be easily fixed by increasing the number of participants.\n\n4. No actual expert group. The current study would be much stronger with an actual expert group (ideally recruited from local property agents). While I do not consider this a hard requirement for publication, I think it would make the work much stronger. Alternatively the task could also be switched to something where it is easier to recruit experts and non-experts from a student/faculty population.\n\n5. Generality of findings needs to be toned down. The current work presents limited results on a single task, but is written as if findings were to hold in general. This is unjustified. Particularly with human prediction and decision-making, there is a long history of behavioral studies that show that behavior of the same participant can change drastically across tasks or even framing the same task differently (e.g. due to priming effects, risk-sensitity, etc.). Ideally, the work would present results that hold across a range of tasks (the same participants could easily be recorded to answer more questions from different tasks). I would be willing to accept a paper with a single task only, if executed very well and if these limitations are clearly discussed.\n\n6. Add a limitations section. Provide a critical summary of the claims that can be made given the study. Clearly state and discuss current limitations and unexplained results (or results hypothesised to result from too small sample size). Discuss how this affects the generality of findings.\n\n7. Ablations: show that there is no difference in final disagreement due to the two-stage process (i.e. no self-priming by having to give an answer before seeing the prediction). This can easily be done by having one group which sees the model prediction right away before having to make a prediction. There should be no stat. significant difference in final disagreement between this control group and the standard group; otherwise having to make a prediction first becomes essential/detrimental to the process and would need to be factored in. I find this point important and would certainly add it if additional data recording is done, but since it is somewhat tangential to the question of different uncertainty representations, I would give this point lowest priority\n\n**Minor comments:**\n\nThe following sentence is a bit ambiguous: “Specifically, for each participant, we computed the average distance between each apartment’s true price (i.e. the model’s prediction) and the participant’s first and second estimates.”. I guess participants were only instructed to match the “true price” and not the model’s prediction (which they saw in the trials and could have trivially matched) - the authors used the model’s prediction rather than the actual true price (a month later) to compute the bonus payouts only. Is this correct?\n\nMinor point for discussion: It is unclear that, when asking humans for a number, humans report their mean estimate (they might have a skewed distribution and report the median) - similarly, after seeing the model’s prediction humans might put more emphasis on the median and not the mean. It is thus unclear whether humans should always be expected to adjust towards the mean (e.g. in the bimodal case).\n\nIdeally, provide some measure of quality/accuracy of the ML predictions (if the model performs badly, it would be unsurprising that experts do not update their predictions much compared to non-experts; the maximal version here would be to conduct an additional ablation study with a good model and a bad model to show that experts can spot the bad model and adjust less and non-experts cannot).\n", "broader_impact_concerns": "No concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Lpoi713ssG", "reviewer": "Reviewer_pcWi", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n1. Systematically studies the effects of different predictive uncertainty distributions on human-ML agreement, across experts and non-experts. This distinguishes the submission from prior work.\n1. The paper is very clearly written and easy to read\n1. Appropriate statistical methods are used and the authors also refrain from using them and making strong claims in the \"qualitative analysis\" of Section 4.3 where the sample sizes are too small\n\n## Weaknesses\n1. The prediction task is based on only two input features, number of bedrooms and square footage. To me, this is too simple to represent meaningful applications of ML-assisted decision-making, where the data is high-dimensional and complex enough to motivate the use of ML models (beyond linear regression) for assistance. In short, I think two features is too few to draw generalizable insights. It also seems too simple for the specific application of rent prediction, where even unassisted humans consider many more features than number of bedrooms and square footage. \n1. The experimental design somewhat confounds the effects of showing predictions and uncertainty because the two are always presented together, except for the no-uncertainty experimental condition. I thought from paragraph 1 in Section 3.1 that \"agreement with the model's prediction [is measured] before and after being shown the uncertainty information\" alone, which would indeed isolate the effect of uncertainty. \n1. The number of participants is too small to reliably measure effects across all the dimensions considered, specifically the interaction of predictive uncertainty distribution with human expertise in Section 4.3. This is especially true of the group with no/weak ML background. It is presumably easier to recruit more of such non-ML experts.\n    1. On a related note, what about exposing each participant to more than one experimental condition, and increasing the number of apartments shown accordingly? 10 apartments did strike me as low. More importantly, this could increase the overall number of observations and it might also allow for greater statistical power in comparing the conditions. The reason for the latter is that for observations from the same participant but under different conditions, the between-participant variation is removed.\n1. If I understand correctly, the additional $30 payment incentivizes agreement with the ML model since the \"true price\" is taken to be the ML prediction. I am a bit concerned that this incentive biases all the measurements of agreement.\n\n## Minor Comments and Questions\n- Introduction, paragraph 1: \"complimentary\" --> \"complementary\" \n- Introduction, car selling example: This example may be complicated by the game theory aspect, i.e., the best asking price may not equal the predicted selling price depending on how prospective buyers may respond.\n- Section 3.1, paragraph 4: The wording \"when that uncertainty is larger and encompasses more plausible predictions vs. when it is smaller and more targeted\" suggests to me that there is also a choice of range over which to show uncertainty (e.g. choosing lower and upper quantiles that bound a range, or the width of a prediction interval), in addition to the intrinsic variance of the distribution. But the former choice is not considered in this paper so I would suggest rephrasing.\n- For the bimodal distribution, how far apart are the two means and what are the two individual standard deviations? Giving the overall SD does not fully specify it.\n- How long did the participants take on average to complete the study?\n- Figure 2: The sentence \"this apartment is smaller than average for 3 bedrooms\" is technically additional information beyond the two feature values (specifically information about the conditional distribution of square footage).\n- Section 3.3: Is the random intercept for participant ID the only random effect in the mixed-effects model?\n- Figure 3 caption: \"without domain expertise have lower initial disagreement\" --> \"higher\"", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This submission studies the effects of communicating predictive uncertainty in ML-assisted human decision-making. User studies are conducted to determine the effect of different distributions of predictive uncertainty on human agreement with the ML prediction. The effects of human expertise in both ML and the application domain (apartment rent prediction) are also studied. The main findings are that showing any distribution of predictive uncertainty increases agreement with the ML prediction, with the closest agreement under low-variance normal distributions. This trend is especially noticeable for people who are not domain experts.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Weaknesses 1 (too few input features) and 3 (too few observations, which could be remedied in more than one way) are critical for me.\n- Addressing weakness 2 would obviously sharpen the focus on uncertainty, but I could live without it if the authors have a good reason, given the presence of the no-uncertainty condition (especially if each participant is exposed to multiple conditions). \n- Weakness 4 is concerning but not a deal-breaker.\n- Addressing the minor comments and questions would strengthen the work.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The initial reviews were in broad agreement about the strengths and weaknesses of this paper. The reviewers found that while the proposed methodology was sound and the initial manuscript might make a good contribution to TMLR, it also had a few significant weaknesses. 

The authors turned around and effectively provided a major revision within the discussion period. In a short amount of time they were able to address most of the weaknesses identified by reviewers. Notably, they were able to double the sample size of their study which allowed for stronger conclusions. They updated their analysis with this new data and updated the manuscript taking other reviewer comments into consideration. They also addressed other limitations of their work by adapting some of their claims (e.g., with respect to generality of the findings) and also adding a one-page discussion of limitations (Section 5.1).

The number of available features to the ML model (two) remains an important perceived limitation of this paper. This is clear from reviewers' comments and from our private discussion. I agree that this setting may not be that realistic (in this setting other features are likely available and so one would likely model them). I also do not know of the possible effects of increasing the number of features on the paper's findings apart from making the model more accurate. Having said that, I believe this point is less about correctness and more about significance of the paper and its results.

Given TMLR's evaluation criteria I find the paper should be accepted with minor revisions.


Minor revisions. I would like the authors to consider the following two changes:  

- I found that some of the discussion you had with the reviewers regarding the above limitation (number of features provided to the model) was insightful and would be useful to add to the paper. In particular:
   - *Positioniong of your work with respect to the existing literature.* In your response you wrote 
      > "The setting we consider is ML assisted decision making under limited information. In this context, one can consider scenarios where users have less than, more than, or the same features as the ML model. Each of these three scenarios has been studied by seminal works in the ML assisted decision making literature. For example, Poursabzi-Sangdeh et al. (2021) studied scenarios where users have the same features as the ML model and scenarios where users have more features than the ML model (in which case their ML model is also based on just two features). Kleinberg et al. (2017) consider scenarios where users have more features than the ML model." 

       I did not find that this view was clearly exposed in the paper. Would it be possible to add it? If it is already there and I have missed it please kindly let me know. 

    - *Discussing possible effects of using different features for the model and for humans.* In your reply to reviewers you also write
      > "As one of the first studies investigating how different types of posterior predictive distributions affect ML assisted decision making (and how such effects vary between domain/ML experts and non-experts), we did not want to introduce additional complications arising in the scenario where users have fewer features than the ML model. For instance, if the ML model includes some additional features (not available to the user) that are highly influential for a few apartments, model predictions may diverge heavily from experts’ understanding and create distrust in the model. We agree with the reviewer that exploring these additional scenarios would be interesting future directions. We will add a paragraph in the Discussion describing these additional scenarios one may consider."

        This seems to relate to some of the discussion in the second paragraph of Section 5.1. I would suggest adding some of the above to it (or somewhere else in the same section). 

- Fixing the typos in the new text (in blue).
Decision: Accept (poster)