Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper shows that calibrated models are not always the best for human-AI systems. Optimizing the ML model alone with human *uncalibrate* the model, but resulting in better performance. The paper provides rigorous experiments and related details. 

strengths_and_weaknesses: **Strength**
1. The paper provides extensive and detailed experiment results that challenge the common belief of "better-calibrated models lead to better human-AI systems."


**Weakness**
1. The experiments are a little bit toy, with only two actions.
questions: 1. The paper shows that it's important to optimize the ML model with humans or potentially a model for human behaviors. It reminds me of the model-based RL problem. How does the reliability of the model for human behaviors affect the final human-AI performance?


**Suggestions**
1. Since the paper is related to human perceptions of the model's confidence, it will be great to have some human studies or interviews
limitations: The paper is easy to follow and the idea is well justified. As described in the last section, the experiment settings are limited and do not capture many real-world cases.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper focuses on optimizing AI advice when AI is used as second opinion for a human decision maker. They propose a model for human interaction with AI that can be learned from data into two parts: an activation model and an integration model. They then provide a strategy to modify AI confidence given this human model to maximize task performance. They validate their results on four experimental tasks where they show that their approach of optimizing AI confidence increases performance compared to showing humans calibrated probabilities. 
strengths_and_weaknesses: Originality: The learning of the activation and integration models from human data are novel, even if the formulation is not as novel. However, the optimization procedure for AI advice is novel. Moreover, this is one of the first papers to do human-AI optimization directly. 

Quality: The AI advice optimization procedure is technically sound. The experimental tasks are not very ecologically valid as they are not representative of tasks people will use AI for. 

Clarity: The paper is well written. I think the human-AI optimization procedure can be better illustrated with a figure to show how $\alpha,\beta$ feed into both the activation and integration models. I would also include Figure 6 from the appendix in the main body. 

Significance: To the best of my knowledge, this paper is one of the first to do AI optimization for AI assisted decision making and produce a positive result. I think this will inspire future work in this area, as I can see many extensions to this work. However, for practical significance, the authors need to strongly argue for why misleading users is okay, I strongly encourage them to reward the language of the intro and abstract as to have this paper be an academic exploration rather than a method AI designers should develop to improve human-AI teams in practice

questions: “One potential question about our approach is: does modifying the AI’s confidence constitute misleading the user? We do not view this as misleading since a human’s assessment of confidence may differ from the statistical notion of calibrated probability.” – This is the most major point about this work. Even though I am positive about this paper, it is absolutely misleading the human and such approaches cannot be ethically implemented in practice as they are intentionally misleading. I believe even lay-people understand the notion 

Why the two-stage interaction process instead of providing the AI advice from the beginning to mimic more real-world deployments?

What is baseline human accuracy and AI accuracy on all four tasks? Is there a baseline human-AI team, for example a learning to defer baseline, and results for that? It would be quite significant to show that humans deciding when to use AI advice is better than AI deciding when to use human advice. 

Why weren’t alpha and beta optimized given the user demographics? Was there insufficient data? Furthermore, since alpha and beta weren’t personalized, then why does the activate/integrate functions need to be personalized?

The fact that “asks with higher advice accuracy (Art and Cities) exhibit larger increases in accuracy and confidence” is well documented in the literature since absolute AI advice accuracy is higher. Is there evidence of a relative increase in human accuracy (rather than absolute)?

Rather than misleading the users, what would you expect the effect of an intervention where users are taught to adapt their activation/integration of AI to revise their own internal model, could such an intervention mitigate the need for AI optimization and uncalibrated model?

What would you think the effect of telling users that one model is calibrated, and one is uncalibrated? For example, if you randomly split the user base that received the optimized advice, and told one part that the advice is calibrated (lie) and the other that it is uncalibrated (truth), what differences would you see? 

limitations: I think the authors should not encourage practice to adopt uncalibrated models, and should highlight that a limitation of their work, but still highlight that their work is an interesting exploration.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper proposes a method for adjusting model confidence such that the output model decision/recommendation it is best leveraged by a human. The method is based on the observation that humans may not be able to interpret confidence in the form of probabilistic estimate accurately, and may therefore prefer less calibrated notions of confidence. The authors show this via a set of user studies in 4 different domains. They use the data from these user studies to learn a model that re calibrates confidence depending human behavior, and they show that this benefits final team performance. 
strengths_and_weaknesses: Strengths:

S1- The paper studies the problem of adjusting AI behavior specifically to improve human ai collaboration. This is a problem that deserves more attention in the community, especially for high-stake domains where the model is intended to augment a human and not operate autonomously.

S2- It verifies the observation on "uncalibration helping team performance" on several datasets and user studies. 

S3- The method is practical to use and can even be used on an already trained model, assuming that that model is somewhat calibrated.

Weaknesses:

W1- The paper is missing comparisons with some important or obvious baselines. For example, comparing with a simple and pure step function (figure 3a) where the model only reveals its answer, this being an extreme case of uncalibration. Most importantly, there could have been a comparison with related work [1] mentioned in the literature. Technically, this is the analogous version of this paper which attempts to achieve similar results but during model optimization and not post hoc. That said, the relationship of the paper with [1] needs to be made more clear as the goals are really similar. The authors may also want to check the following paper [*] from the same authors, which provides evidence that humans prefer clean, non-stochastic error boundaries. Uncalibration as proposed by this work falls within the same framework, if the error boundaries are defined by using confidence.

[*] Beyond accuracy: The role of mental models in human-AI team performance

W2- Except [1], authors could also discuss post-hoc calibration in related work given that the proposed method is indeed a post-hoc calibration method for HAI. Note that the paper often claims for this to be an optimization approach but since it operates only on model confidences it is not such. This is not necessarily negative (in fact sometimes may be advantageous), but the discussion and framing of the approach needs to be made more precise.

W3- Depending on the utility function, uncalibration can be dangerous, if for example mistakes are associated with a much higher cost than the reward for correct predictions. This limitation needs to be discussed accordingly.

W4- The proposed approach may not work as well when the original confidence scores from the model are not calibrated. In this experiment they are reasonably calibrated since the model output is the majority vote of crowd answers. However, if this is not the case, the post-hoc calibration may not change much the situation or it may make it worse.
questions: My main questions are related to W1 (comparison to baselines) and W4 (what happens when the initial model is not calibrated). 

Morevoer, there is an overall question on how these results would change when the model accuracy varies. One can assume that if the model has low accuracy, it will be less often useful to the user and gains in team performance will be lower. There could be a discussion around these in the limitations section.
limitations: The most important item to address here is the fact that depending on the cost\reward function of accurate and non-accurate decisions, uncalibration may lead to over reliance, which in the real world may have implications on safety.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: In the context of AI-assisted decision-making, the authors propose a method to adjust the confidence of the AI advice, that a human user observes on a given predictive task, with the goal to improve the user's accuracy on that task. For this purpose, they assume a two-stage cognitive mechanism for the user's processing of the reported AI confidence.  In the first stage, the users decide whether to use the advice or not, and if they do so, in the second stage, they decide to what extent they will use it. The authors fit a function for each stage using empirical data that they gathered on four binary classification settings. Then, they leverage these fitted functions to find which AI-confidence modification function--among a parametric family of functions-- minimizes the user's binary cross entropy loss on a specific training data set. Finally, the authors demonstrate  with simulations as well as experiments with real human users, that adjusting the AI confidence with such an optimized modification function, appears to improve not only the user's accuracy but also their confidence on their predictions. 
strengths_and_weaknesses:  The authors appear to be the first to make an attempt to manipulate the reported confidence of a decision-aid system, with the goal to maximize the user's accuracy when advised by such a system. Even though the idea that the reported system's confidence does affect the user's trust on the system's advice is not new and has been studied in prior works [1], [2], [3], the paper seems to be the first to formalize the problem of adjusting the reported confidence to increase the user's accuracy, opening an interesting new perspective for future works.

Overall the paper seems mostly well-structured and clearly written. There is a meticulous description on the experimental setting to gather user's data used to fit the activation and integration models as well as a detailed analysis of the models themselves. The formalization of the setting is also detailed, however the optimization objective and procedure could be described more clearly with more details, since for example conditioning on x on Eq.2, but not in equations between lines 249-250 confuses the reader. The authors provide theoretical examples, simulations and experimental results, with adequate analysis, based on which they evaluate their proposed method.  

The idea of adjusting the reported confidence of an AI decision-aid system, with the goal to benefit the user's accuracy, as well as the experimental results do seem promising and opens different perspectives for future work. Overall, one might talk about an interesting and important first attempt to manipulate a system's reported confidence, to benefit the users.


[1] Sunita Sah, Don A Moore, and Robert J MacCoun. Cheap talk and credibility: The consequences of confidence and accuracy on advisor credibility and persuasiveness. Organizational Behavior and Human Decision Processes, 121(2):246–255, 2013.

[2] Zhang, Yunfeng, Q. Vera Liao, and Rachel KE Bellamy. "Effect of confidence and explanation on accuracy and trust calibration in AI-assisted decision making." In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency, pp. 295-305., 2020.

[3]Rechkemmer, Amy, and Ming Yin. "When Confidence Meets Accuracy: Exploring the Effects of Multiple Performance Indicators on Trust in Machine Learning Models." In CHI Conference on Human Factors in Computing Systems, pp. 1-14. 2022.
questions: 1. In the study the authors used prior aggregated human predictions as a proxy for the AI model, since the four datasets used in the experimental setting are too limited for training actual AI models. However, the sarcasm dataset as well as the census dataset are subsets of much larger datasets, namely the Reddit sarcasm dataset [1] and the Census dataset [2], that are suitable for actual model training.  Definitely, training (and calibrating)  models from scratch for the purposes of the study are beyond the scope of the paper, but given that the existing  state-of-the-art model [3] on the Reddit sarcasm dataset, reports F1 score > 70%, why such an off-the-shelf model, or similar ones were not used  as the AI model giving advice for the correspondent dataset? 

2. In figure 2(b)  the initial response and advice are in $[-1,1]$ whereas in figures 3(b), and  9(b) they are in $[0,1]$. The authors do mention in line 196 that $r_1$ and $r_2$ are probabilities normalized in $[-1,1]$ and it would be helpful if the $r_1$ range was consistent across plots, or clearly specified otherwise.

3. The objective introduced in Eq. 2 is conditioned on the vector $\boldsymbol x$ of initial response $r_1$, advice A and user's features $u$. Also in Appendix C.4 it is mentioned that each advice-initial response pair was optimized separately. The objective between lines 249-250 refers to the expected value of the binary cross entropy loss on a given dataset, not conditioning on x or the advice-initial response pair. Is there an optimal $\alpha^*$, $\beta^*$ for each advice-initial response pair? Is there one for each $\boldsymbol x$ or there is one optimal $\alpha^*$, $\beta^*$ for a given dataset? From line 276, it seems that there is one $\alpha^*$, $\beta^*$ given a training dataset, though conditioning on x in Eq. (2) and stating in Appendix that optimization happens for each $(A,r_1)$ pair are rather confusing.

4. Minor typos: 
       In line 533 "to modification" --> "no modification" ?
       In line 540 "for modify advice" --> "for modified advice"?



[1]  Mikhail Khodak, Nikunj Saunshi, and Kiran Vodrahalli. 2017. A large self-annotated corpus for sarcasm. arXiv preprint arXiv:1704.05579 (2017).

[2] Alex West and Anusha Praturu. 2019. Enhancing the Census Income Prediction Dataset. https://people.ischool.berkeley.edu/~alexwest/w210_census_income_html/.

[3] Taha Shangipour ataei, Soroush Javdan, and Behrouz Minaei-Bidgoli. 2020. Applying Transformers and Aspect-based Sentiment Analysis approaches on Sarcasm Detection. In Proceedings of the Second Workshop on Figurative Language Processing, pages 67–71, Online. Association for Computational Linguistics.
limitations: One key limitation to the proposed method is the need of accurately modeling human behavior, based on which the authors optimize the Ai's reported confidence with respect to the user's accuracy. The authors do acknowledge this limitation and clarify that computing such a human model  is not within the focus of their work. They also consider as future work a robust human model with which they could account for the model's error. Another limitation, that the authors consider as future work, is the fact that they used only an AI model with an accuracy higher than each user's accuracy, while not considering cases in which users' accuracy would exceed the model's accuracy. Furthermore, the proposed optimization is based on the choice of the training dataset D, which could be addressed as a limitation; it would be helpful if the authors could add some discussion or example of how a distribution shift on the test data could affect the user's performance or clarify if their method should only be applied in absence of such a shift.
review_title: null

Meta-review: metareview: The paper studies the problem of optimizing an AI agent's reported confidence to improve the overall performance of a human user in a collaborative human-AI decision system. The main idea is to train a human behavior model based on experimental data that can be used to simulate the joint human-AI system and then optimize the AI agent's reported confidence based on this simulated system. The paper provides rigorous experimental evaluation involving human participants, and the results demonstrate the importance of jointly optimizing the human-AI system. The reviewers acknowledged that the paper considers an important problem and provides new insights on how to calibrate AI advice for human use. However, the reviewers also raised several concerns and questions in their initial reviews. We want to thank the authors for their detailed responses and for actively engaging with the reviewers during the discussion phase. The reviewers appreciated the responses, which helped in answering their key questions. The reviewers have an overall positive assessment of the paper, and there is a consensus for acceptance. The reviewers have provided detailed feedback in their reviews, and we strongly encourage the authors to incorporate this feedback when preparing the final version of the paper.
confidence: Certain
Decision: Accept