Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: An interesting paper which focuses on formalising the sequential problem of online decision mediation (ODM)  - and proposes a specialised solution (uncertainty-modulated policy for intervention and requisition - UMPIRE) to tackling pitfalls which traditional bandit algorithms face in ODM. The paper is relevant and very promising for describing potential solutions for real-world and real-time decision making, especially in safety-critical applications (e.g. clinical diagnosis), wherein, a human-in-the-loop approach can benefit from providing an effective interface between  human mistakes and expert feedback - as the paper also mentions.
strengths_and_weaknesses: A very well written and comprehensive paper, well-supported by experiments and comparison with other baseline models/algorithms, particularly on real-world datasets/benchmarks which makes it very interesting to read.

The main weakness of the paper is that it is very verbose and text-heavy at some places e.g. in Section 2.2. Can these e.g. "learning with rejection" be better encapsulated (or relevant papers from past literature cited instead of a complete re-definition from scratch?). This would greatly help improve readability of the paper.

questions: Where could this proposed solution be used outside of clinical decision support? Can the authors provide few more examples in the Conclusions section e.g. could it be used in industry for real-time decision support of engineering systems (e.g. during operations & maintenance) etc.?
 
limitations: What are the main drawbacks/limitations of UMPIRE? It is not clearly mentioned in the paper's Section 5 - it would have been useful to have a separate "sub-heading" for Limitations e.g. in a Discussion section as it is inadvertent that the proposed algorithm may suffer from pitfalls which need to be clearly outlined with simplistic language (and comprehensively). 

Update: Thanks for addressing this reviewer's comments - the paper is now substantially clearer and comprehensive. I have changed my score from 6 to 7.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The work proposes a solution to the following problem setting: Given an input instance, an imperfect human provides a provisional decision/prediction. A mediator model decides whether to accept, to intervene by showing its own decision (at a cost), or whether to defer (at a cost) to another human expert. The instances come in a streaming fashion. The proposed approach (UMPIRE) trades off system risk (final decision) for model risk (ability of the mediator to make predictions and learn about the deferring strategy). The algorithm is compared with several related works, mainly bandit algorithms, and shown to mostly outperform. A detailed sensitivity analysis is also presented.
strengths_and_weaknesses: Strengths:

S1- Optimizing for overall system accuracy instead of autonomous model predictions is an important albeit overlooked direction. 

S2- The paper is very well written, and results are clearly presented.

S3- The paper also does a very good job at contrasting and comparing with related work.

Weaknesses:
W1- While optimizing for overall system accuracy is important, the choice of keeping an imperfect human decision maker in the loop is not very well motivated from both an ethical and mathematical standpoint. If I am not mistaken, the imperfect human decision maker is simulated with a probability of 0.5 (line 304). Thus, it is questionable why keep this actor in the loop on the first place and why not change the whole interaction altogether to have the model interact only with the expert. This is in fact the setting that other works in human-ai decision making also consider (e.g. [24] non-streaming or [*] streaming). Otherwise, if there are any other reasons to why this workflow may still make sense, at least the experiments need to be adjusted to a probability of error > 0.5 for the human. 

[24] Consistent estimators for learning to defer to an expert
[*] Pre-emptive learning-to-defer for sequential medical decision-making under uncertainty

questions: My questions are mostly related to W1 above. The human appears to be somewhat redundant in the workflow, also because they seem to have no say if their decision is overridden by the model's decision. What is the motivation for choosing \alpha = 0.5?

In addition, it would be useful to see the progress of how z is distributed across timesteps. Does the model decide to request for the expert's prediction mostly at the beginning? When does it start to request less and in which cases in the timeline it decides to just accept the human decision?
limitations: -Literature in this field is somewhat divided between work that assumes that a model does the routing (deferring) and work that assumes that the human is in control for when to defer to an AI or when to ask for help or when to decide to integrate AI recommendations in their decision (see examples below).  There probably needs to be a discussion here or in the introduction that describes when\why it makes sense to sometimes have a model that controls the routing. 

Teaching Humans When To Defer to a Classifier via Exemplars

Is the most accurate ai the best teammate? optimizing ai for teamwork

Updates in human-ai teams: Understanding and addressing the performance/compatibility tradeoff

- There may also need to be a more extended discussion on why pessimistic bayesian sampling becomes a competitive option when the expert noise increases. This detail also needs to be mentioned in the introduction or abstract as an important insight for real-world applications, where indeed most often than not expert decisions are also not perfect.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper studied a new problem: online decision mediation, in which the system (mediator policy) decides whether to follow human decision, or take an intervention, or request help from the expert based on human behaviors at every step. To learn a good mediator policy, the proposed solution UMPIRE balances the immediate loss terms against future improvement. They also validate the advantage of UMPIRE through empirical studies. 
strengths_and_weaknesses: Strength:
The big idea of online decision mediation framework is novel and interesting. Requesting advice from experts costs high in practice, while human behaviors are not always optimal and needs to be corrected if necessary. Thus, it’s very worthwhile to think about mediator policies. I think this paper proposed nice formulation and methodologies for this practical scenario.

Weakness:
1. Many parts are not clear.
    - in equation (3), what is Y?
    - How did you derive equation (5)?
2. The main theoretical result (Theorem 1) is hard to interpret. From the right side of equation (6), I cannot tell how much the improvement is. It will be helpful if authors can give some specific examples to show the scaling. 
3. This paper is not easy to read and follow. Existing works and new techniques are often mixed in sections. 

Overall though I think this paper is not that theoretically strong, but ODM and their methodologies are good contributions to the community.

questions: Please refer to the weakness section. Moreover, Figures are too small to read. 


limitations: Yes.
review_title: null

Meta-review: metareview: Thank you for submitting your paper to NeurIPS! This paper studies sequential decision-making for mediation -- given actions chosen by an imperfect human, it decides whether to accept the decision, intervene with an alternative, or request a costly expert opinion. The authors identify an exploration-exploitation tradeoff for this problem (costly to obtain labels, but improves future accuracy) and build on the bandit literature to propose the new UMPIRE policy. The model and algorithmic approach seem sound for optimizing overall system accuracy, and the reviewers especially appreciated the extensive experiments on real datasets. I am pleased to recommend acceptance. However, please be sure to add more discussion of the limitations into the main paper (as promised in the response); even if the entire discussion doesn't fit in the main text, please add pointers in the main paper to Appendix E.
 
There are also some technical choices that were brought up by the senior program committee that would be useful to discuss since they may have usability/impact implications. First, what are implications of the choice of a scalarized objective that trades off the error metric with the expert cost (Eq. 3), say compared with something that's based on constraints? Does the scalarized objective do a good job approximating how we expect decisions to be triaged in practice? In what applications? Second, the regime k_req =m/m-1 - gamma is mentioned as interesting (in Remark 1) when combined with a 0/1 loss. Could you please expand on applications where the 0/1 loss is appropriate and the interpretation of the “interesting regime” in that case?
confidence: Certain
Decision: Accept