Online Decision Mediation
Authors: Daniel Jarrett
Alihan Hüyük
Mihaela van der Schaar
Venue: NeurIPS 2022 Accept (2022)
Keywords: Decision System
Decision Mediation
Decision Support

Abstract: Consider learning a decision support assistant to serve as an intermediary between (oracle) expert behavior and (imperfect) human behavior: At each time, the algorithm observes an action chosen by a fallible agent, and decides whether to *accept* that agent's decision, *intervene* with an alternative, or *request* the expert's opinion. For instance, in clinical diagnosis, fully-autonomous machine behavior is often beyond ethical affordances, thus real-world decision support is often limited to monitoring and forecasting. Instead, such an intermediary would strike a prudent balance between the former (purely prescriptive) and latter (purely descriptive) approaches, while providing an efficient interface between human mistakes and expert feedback. In this work, we first formalize the sequential problem of *online decision mediation*---that is, of simultaneously learning and evaluating mediator policies from scratch with *abstentive feedback*: In each round, deferring to the oracle obviates the risk of error, but incurs an upfront penalty, and reveals the otherwise hidden expert action as a new training data point. Second, we motivate and propose a solution that seeks to trade off (immediate) loss terms against (future) improvements in generalization error; in doing so, we identify why conventional bandit algorithms may fail. Finally, through experiments and sensitivities on a variety of datasets, we illustrate consistent gains over applicable benchmarks on performance measures with respect to the mediator policy, the learned model, and the decision-making system as a whole.

Online Decision Mediation
Daniel Jarrett'
Alihan Hüyük'
Mihaela van der Schaar 1,2,3
Department of Applied Mathematics and Theoretical Physics 'University of Cambridge, UCLA, 3 AAlan Turing Institute
[dkj25, ah2075, mv472] @cam. ac.uk
Abstract
Consider learning a decision support assistant to serve as an intermediary between (oracle) expert behavior and (imperfect) human behavior: At each time, the algo- rithm observes an action chosen by fallible agent, and decides whether to accept that agent's 's decision, intervene with alternative, or request the expert's ''s opinion. For instance, in clinical diagnosis, fully-autonone autonomuus machine behavior is often bey- ond ethical affordances, thus real-world decision support is often limited to monitor- ing and forecasting. Instead, such an intermediary would strike a prudent balance between the former (purely prescriptive) and latter (purely descriptive) approaches, while providing an efficient interface between human mistakes and expert feedback. In this work, first formalize the sequential problem of online decision mediation that is, of simultaneously learning and evaluating mediator policies from scratch with abstentive feedback: In each round, deferring the oracle obviates the risk of error, but incurs an upfront penalty, and reveals the otherwise hidden expert action as a new training data point. Second, we motivate and propose a solution that seeks to trade off (immediate) loss terms against (future) improvements in generalization error; in doing so, we identify why conventional bandit algorithms may fail. Finally, through experiments and sensitivities on variety of datasets, we illustrate consis- tent gains over applicable benchmarks on performance measures with respect the mediator policy, the learned model, and the decision-maker system as whole.
Introduction
Research in data-driven decision support has burgeoned in recent years, with proposed applications in wide variety domains such as finance [1. psychology (2), and medicine (3) Most work on machine learning for decision support falls into two categories: On one hand, descriptive approaches deal with monitoring, forecasting, and learning interpretable parameterizations of observed human behavior (+8) While such tools can help audit and debug decision-making making, they play a limited role in directly guiding human behavior. On the other hand, prescriptive approaches deal with systems that behave autonomously, optimally, and with minimal manual control [912]. While such tools can reduce the need for expert input, they are often at odds with ethical minsiddrattons-especillll in high- stakes settings such as healthcare [3 I8). Instead, we argue for third: We believe machine learning decision support has a viable role as an intermediary between (oracle) expert behavior and (imperfect) human behavior- Mhavior-tthaiie is, as "assistant", which would strike a prudent balance between the above two approaches, while providing an efficient interface between human mistakes and expert feedback.
Online Decision Mediation In this paper, we consider learning and evaluating mediator policies for online decision support from scratch: At each time step, upon observing the context vector of an incoming instance, the mediator policy decides whether accept the human's 's action, intervene with its own output, or request the expert's opinion-and this determines which action is ultimately taken. Deferring to the oracle obviates the risk of error, but incurs an upfront penalty, and reveals the other- wise hidden expert action as a new training data point. As our running example, consider the task of
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

early diagnosis in Alzheimer's disease, where the action is to diagnose each incoming patient as cog- nitively normal, mildly impaired, or at risk of dementia [9 20). Our problem setting is distinguished primarily by three key characteristics: (1) Instances i.e. patients-arrive in a streaming process, and actions must be taken immediately and sequentially. (2) Feedhack-i.e. true diagnoses- only available in an abstentive manner, meaning ground truths are only revealed if the oracle is deferred to. cumulative regret-is computed in an online fashion, without convenient separa- tion into "training" versus "testing" phases. This setting is challenging but general, and applicable wherever domain experts are resource-constrained e.g. the costs of definitive examinations are high.
(3) Evaluation
Contributions In the sequel, we first formalize this sequential problem of online decision mediation ("ODM"), and establish its unique challenges versus more conventional problem settings (Section 2 Second, we identify why conventional bandit algorithms may fail, and describe our proposed solution, uncertainty-modde
policy for intervention and requisition ("UMPIRE"), which seeks to trade off (immediate) loss terms against expected (future) improvements in generalization error (Section [ Finally, through experiments and sensitivities on variety of real-world datasets, we illustrate consis- tent gains over applicable benchmarks to the mediator policy, the learned model, and the entire decision- makinn system as a unit (Section +.
comprehensive set of performance measures with respect
Implications Humans are heterogeneous, and mistakes require timely correction. Machines are also fallible, and models require timely learning. The implications are clear: Rather than pitting computers against climicians, an efficient mediator should augment clinician capabilities by leveraging costly but informative expert resources. By focusing on the (human- cuuman-expper--aeee expeeeeppeeeeeeeeeeeeee meddttto))) decision-makin system as a whole, we take a first step towards more methodical integration of machines "into the loop". Moreover, the technical problem itself combines diverse challenges from sequential decision-making. learning with rejection, and active learning, thus opening the door to multiple avenues of further work.
2
Online Decision Mediation
2.1 Problem Formulation
We use uppercase for random variables, and lowercase for specific values. Let X denote the input variable, taking on values æ E X. and let Y denote the target variable, taking on values y E y. In line with related fields, X may synonymously be referred to as "contexts", "features", and "states" depend- ing on the underlying task, and Y may likewise be referred to as "actions", "decisions", and "labels". Per motivating example, we shall adopt the "context-action terminology for consistency, although our framework applies to any decision task that requires mapping inputs to specific outputs. Following most related settings, we focus on discrete action spaces, and leave continuous actions for future work.
Human, Expert, and Mediator Let p(X) denote an exogenous distribution from which streaming sequence of contexts (X,) e is drawn and indexed by time step. We consider three decision-makers: human, an expert, and a mediator. In each round, a human action is drawn as Y, ~ (-)x,) from an (unknown and possibly stochastic) human policy स E I1. For instance, this is the noisy diagnosis is- sued by an apprentice clinician. If prompted, an expert action may likewise be drawn as Y2 ~ т, (- (-1X,) from an (unknown and possibly stochastic) expert policy π. II. For instance, this is the final diagno- sis issued by a senior doctor- that is, should they indeed be appointed to conduct a full examination of the patient. Finally, a "mediator" is identified by the tuple (#t o), consisting of a (leareed model policy t E IT- from which model actions Y, ~ #--| (x) may be drawn-as as well as a mediator policy & E &:
Definition 1 (Decision System) Let S= म. n.. () denote the decision system as whole. Given an incoming (X, Y), the mediator policy defines a distribution ((||X, Y) over the space of mediator :0, (0, 1, 2), consisting of options accept (2 = 0), intervene (z= 1), and request (2 =2). Let actions Z :==
5(Y- -y) be the Dirac delta centered at y; drawing Z ~ 6(-1X, Y) induces the overall system policy:
Ts( [x Ý) 12 V) 1[2=1jo(Y max, , ( + 1[2=2] (-Ix)
Intervening incurs some cost kim (e.g. inconveniencing the apprentice clinician to reconsider/alter their decision). Requesting incurs some cost kire (e.g. appointing the senior doctor to provide their opinion), but also reveals the otherwise hidden ground-truth action: Let D. = ((X,, Y=): Z, = 2) denote the cumulative dataset of requested points, taking on values d, D. U, x y)S, and constitutes the training set with which the model policy स is defined. Thus feedback (for learning) is "abstentive" in that it is only observable when the system "abstains" in favor of deferring the decision to the expert.
2

---

Risk and Evaluation Among other aspects, our objective of interest differs from supervised learning in two important ways: First, we are chiefly interested in the performance of the decision system S. instead of the model & per se. Second, learning and evaluation are both conducted online. By way of contrast, consider first the familiar supervised learning objective, which is simply concerned with mini- mizing the generalization error of the model over the underlying data distribution, or the "model risk":
R(#) E
X~p
&(Y, ((|xx)
(2)
where 1/ A()) is some choice of loss function. In decision problems, this most commonly takes the form of the zero-one loss E(Y, ((-[z)) fo1 (Y, arg max; i (y][z)). or in some cases-if a
surrogate loss is required-thr the cross-entropy L # [[[[=)))==== log 7 [1)); we shall use the former to be consistent with comparable literature. Now, our main focus is not the model risk, but the "system risk":
Definition 2 (System Risk) Let :: (X V ft, Ta, P. Rin)) denote the mediation setting. Given mediator (ft, o), the system risk in each round t is the expected error of the induced system policy (i.e. having selected from human, model, and expert actions), plus the upfront cost of mediator actions:
Ø) =E
[6(2,= 1.), 0(Y - Y)))+
Xiro (
VENTT (-px)
(Z,=1|X,, (eY T( X,)) + kim) + - Y.)
Then the online decision mediation ("ODM") problem is to select mediator (f, Φ) to minimize (cumu- lative) regret over a
horizon. Importantly, note that this is more challenging ob- jective than simply minimizing the generalization error of the model, system, or some asymptotic com- plexity thereof: Here we have no separation between "training" versus "testing", since losses begin accumulating from the very first step of the sequential process. The regret at any round n is given by:
Regret [)) Ei (Rr Q1) - (T(,,
where we assume realizability such that the best-in-class mediator is defined as the tuple consisting of the expert policy т, and the greedy mediator policy Φ. (i.e. always choosing Z to minimize each round's immediate system risk). Note the above notation makes it explicit that both the model policy and mediator policy evolve as sequences ;= ] and x= [ that-in general-deport on Dş).
Remark 1 (Assumptions) For ease of exposition, we assume all mistakes are equally important, that kint Kieg are constants, and expert action classes are more or less balanced over the input distribution; allowing relaxations is straightforward and left for future work. To eliminate the more trivial cases, we assume < kim <Rirqg and operate the common rejection regime where kirg = m-1 - for some small 7 0, with m being the number of actions in )): this induces the most interesting tradeoff setting where abstention is neither excessive nor immediately ruled out by the greedy policy. (In our experi- ments, shall empirically consider range of sensitivities). We assume nothing about म,, for instance it is even stationary. Lastly, we assume Do is randomly seeded with one example per action class.
2.2 Related Work
The ODM problem lies the confluence of three classes of learning problems while being distinct from all: (i) learning with rejection, (ii) stream-based active learning, and (iii) stochastic contextual bandits. As before, we employ generic notation and make note of synonymous terminology as appropriate.
Learning with Rejection Compared to standard supervised learning, learning with rejection is a problem setting that endows the algorithm-dduring during test time-with the option to "reject" their own prediction in favor of expert advice [21)26] 26). This is variously referred to as the option to "abstain" from decision, or to "defer" to an oracle. Exereising incurs an abstention cost kabs- but enables avoiding misclassification when the model is uncertain. Typically, a solution consists of model policy स and rejection policy y defining a distribution (U'X) over the space of actions t (1, consisting of the options not abstain (u 1) and abstain (u=2) While the rejection option is similar to that in ODM, learning proceeds from static dataset, evaluation focuses on model risk, and like supervised learning labelss in the batch are always available. An online variant of this setting shares more similarity with the ODM problem by focusing on minimizing the cumulative loss over the course of learning, instead of simply minimizing the held-out performance of the algorithm (27 30) However, a key distinction from us that feedback is not an active choice, and expert labels are always streamed (or when the model does not defer to the expert, which is exactly the opposite of our setting).
3

---

Table I: Online Decision Mediation vs. Related Work. The ODM problem is distinguished by three key factors:
(1) Learning is brsedd (so exploratory considerations cannot benefit from any pool-based comparison).
E {0. 1} 3-yield no learning signal all). tradeoff is explicitly measured by the cumulative loss). Subscripts are omitted from policy terms. Shaded terms denote those evaluated by the risk function, "a.s.c." de- notes asymptotic sample complexity, "feedback condition" indicates when ground-truths are revealed for learning, and "multi-class" indicates whether each setting is not restricted (theoretically empirically) to binary decisions.
(2) Feedback is ahstentive (so some exploitative cettions-precisee
(3) Evaluation is online (so the
Problem
Adtiv
Reques
Components
Rick Flanction of Interest
Minimization Interest
Fredhark
Qaline Esaal
Muti class
Setting
Evalusted)
Condition
Supervised
Learning
R X-y (0)4)))) (OLLXYD (()A)) ME2IORRN
R(
(nls)
Leaming with
Rejection
(p/a)
Online Leaming
R: (ILX)) ((())) (CUA)AAN (R, 00) RR(s.....). (always)
Rojection
(Steamn-on)
Learing
Active
x (0)))))))
R(() a.c.
2-2
Active
RIF s36
Z=2
Abstemton
Deal Purpase
Learning
6.6-6
R(F,0)aae.
Z=2
(Stochastic Contes
tuel)) Bandits
48
of
(alunge)
V
Learning
Active
Kina 1012 X.). o().
() R.(..))
Z-2
V
with Contest 53
Apple Tasting
X
R. Ex (6)) x(X-)]
E,(R(() (R(6669)--
Z-2 X
Reinfonced Active
R. Xime 6X
(always)
/
Leurning 56 09
R. [o(O),, 60 9:9+
Online Decision
Mediation
1
/ R..t, 8.600
2-2
6OIX, SO)0OS. ((X.)) + Ra) -.2 KiMkn
Stream-based Active Learning In contrast with standard incremental learning, ground truths in stream-based active learning are unobserved unless actively "acquired" by the algorithm during training (31-36). This is variously referred to as the option to "request" or "query" the oracle for its decision. Like supervised learning, the goal is to minimize test-time model risk, but with emphasis on reducing labeled and/or unlabeled asymptotic sample complexity. Typically, solution consists of a model मे and an acquisition policy & defining a distribution oZ| over the space of actions 2 (1, consisting of the options not request (2 1) and request (2= While the active request aspect is similar to that in ODM, evaluation focuses on model risk, so ø is not evaluated in the objective itself; moreover, the model has ability to abstain from a prediction. One variant includes abstention to enable the algorithm-during test time-to reject their own prediction (37-40). However, the objective remains to minimize test-time loss and its asymptotic complexity, which contrasts with our focus on evaluating cumulative losses over the entire process. A second variant called dual purpose learning is perhaps more similar in that the model abstains from a decision just when the expert is queried for its decision [41] (i.c. the acquisition policy o coincides with the rejection policy vi, and Z Z=),. But the risk function of interest is still like the usual test-time model risk, but now conditioned on 1=1, 1, so & only enters the objective as a conditioning term to omit points where the model abstains.
Stochastic Contextual Bandits Lastly, ODM bears resemblance to stochastic contextual bandits, class of sequential decision problems with the goal of minimizing (cumulative) regret defined in terms of an arbitrary "reward" function X x y R (42-48) A main difference from ODM is that bandit rewards and are always observed as feedback for learning after each round, but no "expert" actions (viz. best-in-class policy) are available to be queried. One variant incorporates elements of active learning by stipulating that feedback must be requested at some cost kipoa
(49552): ; in a special case dubbed apple tasting. model decisions are tied to acquisition decisions (i.e. the acquisition policy ó coincides with the model policy म. and = (53455) But unlike ODM, the model has no option to abstain, and there is no tradcoff' between requesting information and making predictions. second variant turns around and treats active learning itself as bandit problem [56 ; in the streaming, contextual case (56). the model policy itself actually not evaluated at all: Instead, the acquisition policy is rewarded positively just when querying the expert turns out to be "useful" (i.e. Y Y). and punished just when querying the expert turns out to be "redundant" (i.e. Y= YY)); moreover, feedback (for øi in this setting) is always observed. While these works are similar to ODM in focusing on the online eval- uation objective of cumulative regret, the essential element of abstentive feedback- is central to our motivation for solution to ODM to mediate efficiently using expert resources -ss missing.

---

Table contexttuliss ODM versus related work: Our setting combines the challenges from each, and uniquely characterized by the three key aspects alluded in Section [ streaming instances, absten- tive feedback, and online evaluation. In Section 3 we argue that a good algorithm must appropriately account for these challenges simultaneously. In Section E we verify empirically that neglecting any of them results in poor performance. (Additionally, since we are motivated from the perspective of decision support as an intermediary between humans, models, and experts, note that-as another practical component-ODX. also extends Z with the option to accept human decisions Y ~ =(-lX))
3 Mediator Policies
In light of the preceding discussion, is clear a good mediator should satisfy the following criteria.
The first deals with immediate loss, the second with future loss, and the third with trading off the two:
It should accept or intervene only when errors are thereby unlikely (cf. learning with rejection).
It should acquire ground truths when uncertainty may thereby be reduced (ef. active learning).
It should balance such exploration and exploitation adaptively over time (cf. contextual bandits).
Greedy Mediator It is instructive to first examine the greedy policy. Given any model policy Å,, the greedy mediator policy o, simply chooses Z to minimize the immediate (i.e. one-step) system risk, which balances immediate probabilities of error with immediate costs of intervention/requiitioon
Φ (Z|x. = 6(Z - argmin, (1 (1 - (\(1x)
(5)
+ 1== +((1x) + km) + 12==2)
where 6(Z 2)) denotes the Dirac delta centered at 2, and Y == arg max, At (ylx). It is clear that such mediator policy is optimal in terms of regret if the model policy were already perfect (i.e. if ft = n.), or if the model policy were otherwise fixed (e.g. if 22 no longer provided any feedback for learning).
Passive Exploration But what if the model is not perfect, and must incorporate new data points for learning? Actually, the greedy policy already "inadvertently" performs sort of passive exploration: Whenever the target probabilities ( concentrated, o. would request from the expert, which may reduce uncertainty for points similar to X. However, &h may learn too slowly: If-at any point- the model is even slightly erroneously confident (e.g. (3 [X) 1 +E for any y' * y, for any E 0), then it would simply not query the expert, and may commit similar mistakes again later. This is because it fails to distinguish between aleatoric and epistemic uncertainty: Not only do we wish to defer (viz. abstain) when the former is high at X, but we also wish to defer (viz. learn) when the latter may be reduced by knowing the ground truth at X. So the question is: Can we explore in manner that better balances these (present vs. future) demands?
--yymm/ [X]] for a context X are not sufficiently
3.1 Bandit Mediator Policies
Prima facie. this resembles bandit tradcoff, the immediate question becomes: Can we simply formulate this as a specific instance of contextual bandits? Consider an ODM problem with m actions in V: This gives m 2 "arms" in total (i.c. accept, request, and intervene arm for each of the m underlying actions). However, precisely due to the nature of abstentive feedback, the answer is no:
Loss vs. Feedback In conventional bandit problems, there is no distinction between the notions "loss" and "feedback"" In each round, when an arm ye is pulled in response to a context zt, the negative reward- -v( W1) serves dual purposes: It is always incurred into the performance measure (viz. re- gret), and it always observed as new data point for learning (viz. reinforcement). In ODM, however, "loss" and "feedback" are distinct quantities: In each round, when a mediator action 2, is chosen in response to a context-action pair (Z,, (t). the resulting loss is always incurred (viz. Definition 2 but no information whatsoever (i.c. not even the loss) is observed as feedback for learning unless 2, = 2.
Exploring without Learning should now be apparent why bandit algorithms may suffer in ODM. The crux of the issue here is that only one arm can actually provide any new information (i.e. the request arm). So the role of "exploration" here is very different: Bandit strategies would "explore" different arms pointlessly with no learning occurring most of the time. It is easy to see that this applies to all manner of algorithms such as e-greedy policies, posterior sampling, and strategies that rely on optimism in the face of uncertainty (the latter actually leading to strictly fewer request arms being pulled!). In Appendix C we formally show that ODM is actually a distinct and concretely "harder" problem than contextual bandits (Definition B
precisely due to the nature of abstentive feedback.
5

---

3.2 UMPIRE Mediator Policy
Given the previous discussion, simple "e-request" mediator policy may seem promising: It executes the greedy policy o, but with probability e opts to request from the expert. Learning thus occurs more frequently, and there are no exploratory actions that yield no learning. But important problem is that not all exploratory actions are equally useful: The value of any requested information surely depends on the context Tú; by randomizing "indiscriminatel""
e-request strategy does not account for this.
We propose more principled basis for interpolating between exploration and exploitation that we term cerrrtiittt-moddddcee policy for intervention and requisition ("UMPIRE"). The main idea is that we might be willing to pay more to request an oracle action now, if means we can more confidently rely on model predictions in the future (i.e. by accepting or intervening autonomously). So our motivation is to explicitly trade off (immediate) system risk with expected improvements in the (future) model risk.
To do so, we need a method that allows us to estimate the latter. Operating in the probabilistic setting, let W denote the parameter variable, taking on values in w E W, such that we can write at (Y(X)= := p( w), and can also speak of the marginal p D, X) = = |p)) ( [X, W). Now, denote the expected model risk with R(D)= == Ew-p(.1D ((tw)) note that this itself a random variable due to its dependence on D. Consider the t-th round of play: If the mediator chooses = E (0. 1 }. then But in deciding whether to choose 2 =2,, 2, we wish to capture a measure of how much this risk might possibly improve- that is, if we were to reveal (and learn from) the ground-truth label Y, So we are interested in how far R R(D,) De) can end up relative to R(d,-1). 1). where Y, p(-pppp(1,111))
d so we have [
The following result gives such an upper bound expected improvement:
Theorem (Expected Improvement) Let R be bounded as [-b, b) for instance, by centering 6or- Let I[W; Y xze] denote the mutual information between W and Y, conditioned on d;- and st. and let Wo denote the principal branch of the product logarithm function. Then (proof in Appendix C
R. R(de-1) - Ey.-p(1dd-1..x,) (Rt__
Z. 2] S 2b
1) (6)
This motivates straightforward technique: Define g g 2b (ol 1))-1 1), and let kreg: and
denote some tradeoff coefficient. Then we can designate Rireg == (1 rg (I[W;
simply use kieeq in place of kreq wherever it appears- but keeping the greedy mediator otherwise intact. UMPIRE thus has one hyperparameter ,; in our experiments we simply set its value as the normalizing constant Ki = (2b(e:W m- 1)/e))++1-11---- 1eee--1)11)) 1))-1 which has the effect of keeping all costs non-negative.
Interpretation Since g is monotonically increasing. Theorem D jis naturally interpreted as translating an -nformation-theorrtiic criterion (i.e. the mutual information) into decision- -theorett criterion (i.c. the expected improvement posterior risk) which is what we require. particular, the argument to g expands as I[W Yild-1. 2;} = H[W interpretation of how much the (epistemic) uncertainty in the model policy expected to decrease <,>pp((d,,-1- is revealed; this view is reminiscent of entropy- based approaches to active learning (59 60). Observe that when deployed, G, = (I[W;;; Yz Dr-1, X.)) is large in the beginning, so kirq small and UMPIRE behaves like standard incremental learning. In the limit of a perfect model, G. goes to zero, so kiny = kioq and UMPIRE behaves the same as the (optimal) greedy mediator (T., (b.,)...
(d,-1..,) [ Te, Yi). which has the
3.3 Practical Implementation
Some practical remarks deserve mention. First, UMPIRE is compatible with any choice of probabilistic modeling technique, such as Gaussian processes, Bayesian neural networks, and tropouut-assed ap- proximations. Second, since integration over parameter posteriors is generally intractable, we use stan- dard Monte-Carlo sampling to compute expectations: Let denote the number of samples taken from the posterior, and let ( (= indicate the set of samples drawn from (W).). In computing the value of 9t, observe that the inner expression Ey:-<p(-dde_ Hi[W dd- It, Yr] requires retraining the model policy on every possible value of Y,. Instead, we can rely on the symmetry of mutual infor- mation and expand 1[Y;; W W|d 2z] H[Y, dd- xx] E -p-..
[I,, WJ. so we can write:
ilW; Yelde-1, = 11 Ei= P(Y, xe, wi wr, 1) - Σ H (Y, (r,, wx)-1)]
where we define H H)x(Y))=- Eyey p(y) p(y), giving us a more efficient way to compute the ex- pression without such retraining (see Appendix C for detail). Lastly, to guarantee consistency we can easily still request from the expert with some small probability (i.e. in the same way the "e-request" policy above does so over the greedy policy). Algorithm [ summarizes UMPIRE applied to ODM.

---

Algorithm UMPIRE Mediator
b for Online Decision Mediation
1: Hyperparameters: tradeoff coefficient R, Monte-Carlo samples
2: Input: initial dataset do, cost of intervention kim. cost of requisition kneq
3: for each round 1 = I,... do
4: Te X1
Ve - Yc TT(122)) = p(Y (21, wal-1)
D human action
6:
#[Y[2;))
7:
V/ + urgmax, ((yxx)(()))))
D. model action
8:
[[: Yelr,,1/, i Σ (VV Wa,--]]] - (Y, wis--)]
(Z[ (t) 8(Z arg (1 [()])=+)) [1 #(0 mx) Rima
10:
21 t
2(1xx1.0))
+1|/=22| (1 kg([[W;; Y Yi|di- 3x Keag)
11:
if = 2 then
12:
t Yı ~ -T (-(22)
D expert action
13:
de s de-1 ( 2t)}
14:
else de t de-1
15:
Output: Vi f of 11===100 1=/=/////
b (final) system action
Empirical Results
Three aspects of UMPIRE deserve investigation: (a) Performance: Does it work? Section 4.1 compares it to existing methods. validating its role in decision support by most consistently improving decisions. (b) Source of Gain: Why does it work? Section 4.2 deconstructs the key characteristics of UMPIRE, verifying the importance of each. (c) Sensitivity Analysis: Finally, Section 4.3 assesses the sensitivity of UMPIRE and benchmarks to the expert's ' stochasticity, costs of request, and number of samples used.
Datasets We experiment with six environments. In GaussSine, synthetic points generated in three categories by rounding a sinusoidal latent function on 2D Gaussian input [6T]. In HighEnergy. the task is to identify signals in high energy particles registered in Cherenkov gamma telescope (62]. In Mot MotionCapture, the task is to recognize hand postures from data recorded by glove markers on users 633] In LunarLander, the task with the expert defined as a PPO2 agent 665 66) trained on the true reward. In Alzheimers, the task is to perform early diagnosis of patients in the Alzheimer's s Disease Neuroimaging Initiative study (67 as cognitively normal, mildly impaired, or at risk of dementia [0] Lastly, in SysticFing Fiirossis.
perform actions in the OpenAl gym 664 Atari environment,
the task is to perform diagnosis of patients enrolled in the UK Cystic Fibrosis registry (68] as their
GOLD grading in chronic obstructive pulmonary disease 69 See Appendix
B for additional detail.
minimal baseline, Human always accepts : = 0, thus constituting the starting point for performance comparison. Random draws: z random. Supervised picks = {0, 1} based solely
Benchmarks We consider adaptations of algorithms from related work. First
#'s output, and = w.p. e, and thus resembles supervised learning. Cost-Sensitive is the greedy (ft, Φ.) from Section & which additionally accounts for costs Kint, Kioq. Thompson S Sampling ing [46) draws from the posterior p W (d) and selects greedily using (T,,ó.) the Full version also uses that sampled model when predicting. Epsilon-Greedy [47 is greedy but draws z random w.p. E; the Request version is the smarter "c- request" from Section 3.2 Pessimistis Bayesian Sampling adapts OBS to ODM by revers- ing the direction of optimism such that the tendency to request actually increases. Bayesian Active Request adapts Bayesian active learning [60] to ODM by requesting w.p. ~ expected reduction in en- tropy. To further highlight the advantage of our proposed criterion, Matched Decaying Request is an artificially boosted benchmark that is similar to e-request-but where is a decay function that has the benefit of matching the effective request rate of UMPIRE: This done post-hoc by searching for a polynomial function that best models UMPIRE's request pattern. See Appendix B for additional detail.
Experiment Setup Each experiment run consists of n 20000 rounds of interactions (except for the synthetic GaussSine, for which n == 500), and this is repeated for a total of 10 runs with random seeds. For all algorithms, the underlying model policy is implemented identically using birichlet-based Gaussian process classifiers [6 (0- We simulate fallible human decisions as random perturba- tions of the ground truth with some probability a. As mentioned in Section 2.T] we let kneg = m-1 -? for some small 7 > 0: To do so, we simply set kreg to m-1 rounded down to the nearest decimal point. However, we shall perform additional sensitivities on this below. all experiments, we set kim 0. 0. a= 2. 10% where applicable, and == Kgg as noted in Section 3.2 Regret is defined with respect to the oracle mediator (m., dh),, for which n, approximated by training on the full dataset in advance. Performance metrics for each benchmark are reported means and standard deviations across all runs.
7

---

(a) GaussSine
(b) Highnneryy
(c) MottonCapture
(d) LunarLander
(e) Alzheinars
(1) Cysticc ibrosis
Figure 1: Performance (System): Regrets. Numbers are plotted as cumulative sums of system loss less oracle loss.
(a) GaussSine
(b) HighEnergy
(c) MotionCapture
(d) LunarLander
(e) Alzhemers
(1) cysticrbbrosis
Figure 2: Performance (Model): Heldout Mistakes. Models are evaluated on heldout data once every n/ 10 rounds.
CiseeBive
Heghturrgy Alc. 5V
Lwweeaatiionnn Dc
Landee
Atahermers
CyskiePibreett
Bexhrak
Ear
Ric
Er
Dac AXC
Alc
Irc Acc. Fa.
Acpervlaed Lasd
25.0
19:08
35:013
1::01
11
211
176
45%
206-111
156
162-1
2012.0
533
200
26611
Full
Dhappacs sumpling
Compline
01-3
106.1 11
0-6-1
33:.100 10
385
31614
110
T38
13
-17
335-34
MG-441
565-1
311.31
605-
132.
T14 096113
Teal Bovheeeeeee
14:11
7164
2745
766-
107-73
764-117
138.114
-47
194-.31
704--47
235
425.233
Moseend and Respend
Actine
15:.6
2544
366-
15:111
12.
13
11
3164
188-33
05.17
132-13
446135
541 6.55
12
115
00-11
164.33 1954
UMPERE
423
9.3
614
T12215
113:15
1545
071.33
02:17
(M112
341116 312
135217
10213
122.12 TI9AES
Table 2: Performance (Mediator): Erroncous acceptance, excessive intervention, and abstention shortfall at f=n..
4.1 Performance
Recall from Section T we motivated ODM from the view of decision support: To best improve Decision-making by the (human-expert
system as whole. Primarily, then, we evaluate the performance of the decision system-ther that is, in terms of system regret (Equation Figure I shows results for all algorithms, with Huran (i.c. without support) also shown for comparison: UMPIRE con- sistently accumulates lower regret. This is our objective function, and this is the main takeaway. Sec- ondarily, we can also assess the (heldout) performance of the model policy- thict is, if it were tasked with acting autonomously at any point. Figure _ shows the rate of heldout mistakes: UMPIRE appears to induce more efficient learning. As another auxiliary, behaves-ther is, it accepts erroneously 0 but x y), intervenes excessively (z (==1 1 but oracle 2, E {0, 2)). or abstains not conservatively enough (2 E {0, 1} while ÿ 4 y and i o y). Table 2 shows the results, likewise with UMPIRE as best. (Note that this not simply due to more frequent requests, since Mat ched Decaying Request selects z just often UMPIRE using dynamic é, tuned post-hoc). For more comprehensive measures of the system (loss, regret, mistakes) and model (heldout cross entropy, mistakes, AUROC, and AUPRC), see Figures X g
can also consider how the mediator policy
and in Appendix
4.2 Source of Gain
Recall from Section 2 that ODM combines the challenges from all three related settings, and from Section 3 that UMPIRE is designed with three corresponding desiderata in mind. We now examine each aspect's 's contribution to final performance. TableeBenmerrates some settings isolate the following: (i) Does it act exploit á in cost- sensitive manner ("CS"), like in learning with rejection? (ii) Does it attempt explore deliberately in addition ("DE"), like in bandit algorithms? (iii) Does it leverage uncertainty awareness in decision decision-making ("UA"), similar to active learning? (Abbreviations: Synamic-Request w. w.p. KG requests w.p. kG,, and
Scttiss
CS
DE
LA
She Zequuas
Dasaiow Dequaett
Spsiion Zuquoett wich
synamic-eqques wip.
Table 3: Source-of-Gain Legend.
with ME requests with matched E,, i.c same as Mat ched Decaying Request from above). Figure shows results for the decision system in terms of system regret, and secondarily Figure f Ishows results for the learned model in terms of heldout mistakes: It is apparent that all three aspects are crucial for performance, and cannot be neglected. For more comprehensive source-of-gan -aiin evaluation metrics for the system, model, and mediator, see Figures 15 16
9 20 and 21
and Table 5able5Sin Appendix
4.3 Sensitivity Analysis
I Lastly, we examine UMPIRE's sensitivity to several factors. First, the expert might be more or less stochastic depending on the environment. We simulate this by progressively injecting additional noise to the latent function in GaussSine. Figure 5 shows the results: The advantage of UMPIRE is most
8

---

(a) GaussSine
(b) HighEnergy
(c) MotionCapture
(d) Lunartander
(e) Alzheimers
(1) CysticPibrosis
Figure Source of Gain (System): Regrets, Numbers plotted cumulative sums system loss less oracle loss.
(a) GaussSine
(b) Nighhneerg
(c) MotionCapture
(d) Lonarlander
(c) Alzheiees
(f) Cystic/ abrosis
Figure Source of Gain (Model); Heldout Mistakes. Models evaluated on heldout data once every n. 10 rounds.
(x) Noise
(b)s Noise
+20% Noise
Noise
(4)+ Noise
+50% Noise
Figure Expert Stochasticity vs. Performance: Regret at various noise levels added ground truths (GaussSine)
GaussSine
(b) HighEnergy
MotionCapture
LunarLander
(e) Alsbenness
(i) CysticFibrosie
Figure 6: Cost of Request vs. Performance: Episodic cumulative regret at round (=n at various costs of request.
(b) Nighhnergy MotinCCppuur
CysticFäbrosie
Figure 7: Number of Samples vs. Performance: Regret using various numbers of posterior samples s in UMPIRE.
pronounced at lower noise levels, and decreases as noise levels rise; this makes sense, as our estimates of uncertainty become entangled with noise. For more comprehensive results, Figures 22
24 and 25 in Appendix A Second, we examine the consequence of different costs of request. Recall
that have been operating the regime where kreg = m 7 for small 7 >>0 22 25
m/
We now allow the cost of request to vary from = 0 to m as sensitivities. (For completeness, we
actually go slightly beyond the standard threshold and go as high as - 0.005 < 0). o the results: The advantage of UMPIRE appears most pronounced at higher costs, and decreases in the opposite direction; this makes sense, as cheaper costs mean abstention becomes more trivial decision. For more comprehensive results, see Figures 26
27 28 and 20 in Appendix A Finally, recall that Equation f requires estimating the mutual information by Monte-Carle samples. Figure shows the sensitivity of performance to the practical choice of sample size s: Observe that performance appears to stabilize with reasonable choices of 64 and above. See Appendix Afoor additional detail.
Additional Sensitivities the above experiments, the imperfect human decisions are simulated with a= However, it should be clear that the specific pattern frequency of mistakes does not affect the basic structure of the problem, as long as the imperfect decisions stochastically deviate from the ex- pert's with some probability between zero and one. Appendices E.4 perform complete re-run of all experiments under the same conditions as before-but now setting a To
g Similarly, we can verify
that our intuitions still hold when the cost of intervention Am 0.O. which corresponds to the special case where the machine behaves "autonomously" for all intents and purposes, except where it requests input from the expert. Appendices E.5 E.6 perform experiments for the cartesian product of settings E {0.5, 0.7, 0.9, 1.0) and kim E (0.0, 0.1I, using the GaussSine environment. Across all sensitivi- ties, is easy to see that UMPIRE still consistently accumulates lower regret (our primary metric of interest), well as outperforming comparators with respect to to the rest of the performance measures.

---

5 Discussion
Research in machine learning for decision decision-making is proliferating. such as in data-driven clinical de- cision support- but much focus is exclusively placed on comparing computers versus clinicians [73]: Less explored is how machines can serve adjuncts to make decision systems more efficient as unit. In this work, we take a first step by formalizing ODM as a sequential problem, proposing UMPIRE as a potential solution, and demonstrating the importance of considering all aspects of this unique setting. While this perspective is general, the ethical responsibility for a decision e.g. signing off on a diagno- dis-often must ultimately fall on humans: To remain vigilant of potential bias in societal impact, it is thus crucial to examine the complementary problem of "closing the loop" by considering how humans themselves may in turn interpret feedback to modify their own behavior [4]. Moreover, future work may explore generalizing ODM and its solutions such as to settings with differential costs of mistake or class imbalances, or to consider aspects of interpretability in model policies for human feedback.
Applications In addition to clinical decision support, the ODM problem setting is applicable to any scenario where "imperfect" decision- makers are the front-line decision-makers, and "oracle" decision- makers are available as expert supervision- supervision-albett with limited availability, and where learning feedback is abstentive. This situation arises in many settings where the responsibility for a decision must ultimately fall on a person (i.e. the imperfect human or the expert), but a machine is available for learning and issuing recommendations. The following are some potential examples of such:
Suppose junior employee signs off on the quality of a product batch. The mediator can decide to (1) accept the sign-off as is, or (2) recommend a re-inspection for the batch, due to a disagreeing autonomous prediction as the product quality, or (3) recommend that more senior employee take over and issue their more qualified assessment.
Product Inspection:
Content Moderation: Suppose users in a social network can report suspected content viola- tions in real time. The mediator can decide (1) accept and act on user's report as is, or (2) recommend that the user re-classify the content due to a disagreeing assessment as to its appro- priateness, or (3) recommend that an internal moderator take over and issue their judgement.
option in spoken- dialog system. The mediator can decide to (1) accept and act on the user's ' option as is, or (2) recommend that the user re-select option from the same menu, or (3) re-route the customer to a phone conversation with an actual (human) customer representative to continue the work.
Spoken-Dialog System: Suppose customer selects possibly-nonsenss
Finally, note that ODM is also applicable in settings where there is no imperfect human involved, so the machine simply makes decisions autonomously and learns from selective expert feedback; this is sim- ply the setting where kimt 00. Importantly, however, this does not alter the basic structure of the ODM problem, whose hardness distinguished by the fact that expert feedback is costly and abstentive.
There are two main limitations of our analysis: First, ODM is an online learning frame- work. In general, it is known that online learning may not perform well during early time steps when the learner's 's decisions are largely exploratory, especially if learning proceeds "from scratch" " which is the setting we operate in. In this sense, UMPIRE as solution is also not immune this challenge. Therefore, future work would benefit from examining the potential to nol learn from scratch-e.g. to "warm-start" learner using existing data, which extensive literature on imitation learning. Second, we must recognize that there are two sides to human- machine interactions: While ODM focuses on how machines should best propose recommendations to humans, there is also the complementary aspect of how/whether humans actually incorporate such rec- ommendations into their behavior. Ignoring this second aspect may lead to models that are accurate but not necessarily best at proposing recommendations that are most likely to be complied with -hhihhhhchhhhh would severely undermine the practical utility of such a model. Therefore, future work would also benefit from jointly studying how humans and machines should behave in a "mutually-awarc fashion.
Limitations
which
be done using variety of methods from the
Acknowledgments
We would like to thank the reviewers for their comments, suggestions, and generous feedback. This work was supported by Alzheimer's Research UK, The Alan Turing Institute under EPSRC grant number 1P/N510129/1, the United States Office of Naval Research (ONR), as well as the National Science Foundation (NSF) under grant numbers 1407712, 1462245, 1524417, 1533983, and 1722516.
10

---

References
[]] Gernmanno Teles, Joel JPC Rodrigues, Kashif Saleem, Sergei Kozlov, and Ricardo AL Rabêlo. Machine learning and decision support system on credit scoring. Neural Computing and Applications, 22(14):809-99826 2020.
[2] Bernhard Kratzwald, Suzana Ilié, Mathias Kraus, Stefan Feuerriegel, and Helmut Prendinger.
Deep learning for affective computing: Text-based emotion recognition in decision support. Decision Support Systems, 115:24-33 35, 2018.
[3] Alistair EW Johnson, Mohammad M Ghassemi, Shamim Nemati, Katherine E Nichaus, David A Clifton, and Gari D Clifford. Machine learning and decision support in critical care. Proceedings of the IEEE, 104(2)4444400) 2016.
[4] Daniel Jarrett, Alihan Hüyük, and Mihaela Van Der Schaar. Inverse decision modeling: Learning interpretable representations of behavior. International Conference on Machine Learning (ICML), 2021.
[5] Alihan Hüyük, Daniel Jarrett, Cem Tekin, and Mihaela van der Schaar. Explaining by imitating: Understanding decisions by interpretable policy learning. International Conference on Learning Representations (ICLR), 2021.
[6] Constantin A Rothkopf and Christos Dimitrakakis. Preference elicitation and inverse rein- forcement learning. Joint European conference on machine learning and knowledge discovery in databases (ECML), 2011.
[7] Iván Sánchez Fernánde7, Arnold J Sansevere, Marina Gaínza-Lein, Kush Kapur, and Tobias Loddenkemper. Machine learning for outcome prediction in clectroencephaalgraap (ceg)- monitored children in the intensive care unit. Journal of child neurology, 33(8):446-533, 2018.
[8] Bryan Lim and Mihaela van der Schaar. Disease-atlas: Navigating disease trajectories using deep learning. In Machine Learning for Healthcare Conference, pages 137-160. PMLR, 2018.
[9] Ahmad EL Sallab, Mohammed Abdou, Etienne Perot, and Senthil Yogamani. Deep rein- forcement learning framework for autonomous driving. Electronic Imaging. 2017(19):70-76. 2017.
[10] Tim Lustberg, Johan Soest, Mark Gooding, Devis Peressutti, Paul Aljabar, Judith der Stoep. Wouter van Elmpt, and Andre Dekker. Clinical evaluation of atlas and deep learning based automatic contouring for lung cancer. Radiotherapy and Oncology, 126(2)311-317 2018.
[11] Katerina Lepenioti, Minas Pertselakis, Alexandros Bousdekis, Andreas Louca, Fenareti Lam- pathaki, Dimitris Apostolou, Gregoris Mentzas, and Stathis Anastasiou. Machine learning for predictive and prescriptive analytics of operational data in smart manufacturing. In Inter- national Conference on Advanced Information Systems Engineering, pages 5-16. Springer, 2020.
[12] Eyke Hüllerreeirr Prescriptive machine learning for automated decision making: Challenges and opportunities. arXiv preprint 1rXiv:2122.0826: 2021.
[13] Samuele Lo Piano. Ethical principles in machine learning and artificial intelligence: cases from the field and possible ways forward. Nature, 7(1):1-7, 2020.
[14] Nina Schwalbe and Brian Wahl. Artificial intelligence and the future of global health. The Lancet, 395 (95(10236))1579-1
2020.
[15] Emanuele Neri, Francesca Coppola, Vittorio Miele, Corrado Bibbolino, and Roberto Grassi. Artificial intelligence: Who is responsible for the diagnosis?, 2020.
[16] Abdullah Awaysheh, Jeffrey Wilcke, François Elvinger, Loren Rees, Weiguo Fan, and Kurt L Zimmerman. Review of medical decision support and machine-learning methods. Veterinary pathology, 56(4):512-525 2019.
[17] Daniel Jarrett, Fleanor Stride, Katherine Vallis, and Mark J Gooding. Applications and limitations of machine learning in radiation oncology. The Brätish journal of radiology, 92(1 2000:0019001,, 2019.
[18] Saif Khairat, David Marc, William Crosby, Ali Al Sanousi, et al. Reasons for physicians not adopting clinical decision support systems: critical analysis. JMIR medical informatics, 6(2):e8912, 2018
11

---

[19] Bennett P Leifer. Early diagnosis of alzheimer's disease: clinical and economic benefits. Journal of the American Geriatrics Sociery, 51(552):8281-S255 2003.
[20] Susanne G Mueller, Michael W Weiner, Leon Thal, Ronald C Petersen, Clifford R Jack, William Jagust, John Q Trojanowski, Arthur W Toga, and Laurel Beckett. Ways toward an early diagnosis in alzheimer's ' disease: the alzheimer's ' disease neuroimaging initiative (adni). Alzheimer's & Dementia, 1(1):55 66, 2005.
[21] Corinna Cortes, Giulia DeSalvo, and Mehryar Mohri. Learning with rejection. In International Conference on Algorithmic Learning Theory, pages 67-82. Springer, 2016.
[22] Harish G Ramaswany, Ambuj Tewari, and Shivani Agarwal. Consistent algorithms for multiclass classification with an abstain option. Electronic Journal of Statistics, 12(1):530-
554, 2018.
[23] Chenri Ni, Nontawat Charoenphakdee, Junya Honda, and Masashi Sugiyama. On the calibra- tion of multiclass classification with rejection. Advances in Neural Information Processing Systems, 32, 2019.
[24] Hussein Mozannar and David Sontag. Consistent estimators for learning to defer to an expert. In International Conference Machine Learning, pages 7076-7087. PMLR, 2020.
[25] Nontawat Charoenphakdee, Zhenghang Cui, Yivan Zhang, and Masashi Sugiyama. Classifi- cation with rejection based on cost-sensitive classification. In International Conference on Machine Learning. pages 1507-1517. PMLR, 2021.
[26] Kilian Hendrickx, Lorenzo Perini, Dries Van der Plas, Wannes Meert, and Jesse Davis, Machine learning with a reject option: A survey. arXiv preprint 0rXiv:2107.11277: 2021.
[27] Gergely Neu and Nikita Zhivotovskiy. Fast rates for online prediction with abstention. In Conference on Learning Theory, pages 3030-3048. PMLR, 2020.
[28] Amin Sayedi, Morteza Zadimoghaddam, and Avrim Blum. Trading off mistakes and don't- know predictions. Advances in Neural Information Processing Systems, 23, 2010.
[29] Corinna Cortes, Giulia DeSalvo, Claudio Gentile, Mehryar Mohri, and Scott Yang. Online learning with abstention. In international conference on machine learning, pages 1059-1067. PMLR, 2018.
[30] Chicheng Zhang and Kamalika Chaudhuri. The extended littlestone's dimension for learning with mistakes and abstentions. In Conference on Learning Theory, pages 1584-1616. PMLR, 2016.
[31] Les Atlas, David Cohn, and Richard Ladner. Training connectionist networks with queries and selective sampling. Advances in neural information processing systems, 1989.
[32] David Cohn, Les Atlas, and Richard Ladner. Improving generalization with active learning. Machine learning, 15(2):201-22)
1994.
[33] Sanjoy Dasgupta, Daniel J Hsu, and Claire Monteleoni. A general agnostic active learning algorithm. Advances in neural information processing systems, 20, 2007.
[34] Alina Beygelzimer, Sanjoy Dasgupta, and John Langford. Importance weighted active learning. In Proceedings of the 26th annual international conference on machine learning, pages 49-56,
2009.
[35] Burr Settles. Active learning, 2012.
[36] Giulia DeSalvo, Claudio Gentile, and Tobias Sommer Thune. Online active learning with surrogate loss functions. Advances in Neural Information Processing Systems, 34, 2021.
[37] Shubhanshu Shekhar, Mohammad Ghavamzadeh, and Tara Javidi. Active learning for binary classification with abstention. arXiv preprint arXiv:1906 00303, 2019.
[38] Nikita Puchkin and Nikita Zhivotovskiy. Exponential savings in agnostic active learning through abstention. In Conference on Learning Theory, pages 3806-3832. PMLR, 2021.
[39] Shubhanshu Shekhar, Mohammad Ghavamzadeh, and Tara Javidi. Active learning for classifi- cation with abstention. IEEE Journal on Selected Areas in Information Theory, 229) 2021.
[40] Yinglun Zhu and Robert Nowak. Efficient active learning with abstention. arXiv preprint 2xXv:22044000443, 2022.
[41] Kareem Amin, Giulia DeSalvo, and Afshin Rostamizadeh. Learning with labeling induced abstentions. Advances in Neural Information Processing Systems, 34, 2021.
12

---

[42] Wei Chu, Lihong Li, Lev Reyzin, and Robert Schapire. Contextual bandits with linear payoff functions. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, pages 204- JMLR Workshop and Conference Proceedings, 2011.
[43] Emilie Kaufmann, Olivier Cappé, and Aurélien Garivier. On bayesian upper confidence bounds for bandit problems. In Artificial intelligence and statistics, pages 592-600. PMLR, 2012.
[44] Benedict C May, Nathan Korda, Anthony Lee, and David S Leslie. Optimistic bayesian sampling in contextual-bande bandii problems. Journal of Machine Learning Research, 13:2069- 2106. 2012.
[45] Shipra Agrawal and Navin Goyal. Thompson sampling for contextual bandits with linear payoffs. In International conference machine learning, pages 127-135. PMLR, 2013.
[46] Daniel Russo, Benjamin Van Roy, Abbas Kazerouni, Ian Osband, Zheng Wen, et al. A tutorial on thompson sampling. Foundations and Trends@ in Machine Learning, 11(1):1-96, 2018.
[47] Tor Lattimore and Csaba Szepesvári. Bandit algorithms. Cambridge University Press, 2020.
[48] Alihan Hüyük, Daniel Jarrett, and Mihaela van der Schaar. Inverse contextual bandits: Learning how behavior evolves over time. arXiv preprint arXiv:2107.06317, 2021.
[49] Linqi Song and Jie Xu. A contextual bandit approach for stream-based active learning. arXiv preprint arXiv: 170.::671..755. 2017.
[50] András Antos, Varun Grover, and Csaba Szepesvári. Active learning in multi-armed bandits.
In International conference algorithmic learning theory. Springer, 2009.
[51] Alexandra Carpentier, Alessandro Lazaric, Mohammad Ghavamzadeh, Rémi Munos, and Peter Auer. Upper--ooffidence-bbude algorithms for active learning in multi-armed bandits. In International Conference on Algorithmic Learning Theory. Springer, 2015.
[52] Linqi Song, Jie Xu, and Congduan Li. Active learning for streaming data in a contextual bandit framework. In Proceedings of the 20/9 5th International Conference on Computing and Data Engineering, pages 29-35, 2019.
[53] David P Helmbold, Nick Littlestone, and Philip M Long. Apple tasting and nearly one-sided learning. In Proceedings., 33rd Annual Symposium on Foundations of Computer Science, pages 493-502. IEEE Computer Society, 1992.
[54] David P Helmbold. Nicholas Littlestone, and Philip M Long. Apple tasting. Information and Computation, 161(2):85-139, 2000.
[55] James A Grant and David S Leslie. Apple tasting revisited: Bayesian approaches to partially monitored online binary classification. arXiv preprint Stti:2109.144122 2021.
[56) Sarah Wassermann, Thibaut Cuvelier, and Pedro Casas. Ral-improving stream-based active learning by reinforcement learning. In European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML-PKDD) Workshop on Interactive Adaptive Learning (IAL), 2019.
[57] Ravi Ganti and Alexander G Gray. Building bridges: Viewing active learning from the multi-armed bandit lens. arXiv preprint arXiv: 11.300666300 2013.
[58] Djallel Bouneffouf, Romain Laroche, Tanguy Urvoy, Raphael Féraud, and Robin Allesiardo. Contextual bandit for active learning: Active thompson sampling. In International Conference on Neural Information Processing, pages 405-412. Springer, 2014.
[59] David JC MacKay, Information- based objective functions for active data selection, Neural computation, (4):599-604 1992.
[60] Neil Houlsby, Ferene Huszár, Zoubin Ghahramani, and Máté Lengyel. Bayesian active learning for classification and preference learning. arXiv preprint arXiv:1112.5745 2011.
[61] Geoff Pleiss, Jacob R. Gardner, Kilian Q. Weinberger, Andrew Gordon Wilson, and Max Balandat. Exact gp regression on classification labels. GPyTorch Examples, 2020.
[62] RK Bock, A Chilingarian. M Gaug, F Hakl, Th Hengstebeck, M Jifina, J Klaschka, E Kotrě, P Savický, Towers, et al. Methods for multidimensional event classification: case study using images from a cherenkov gamma-ray telescope. Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment, (16(2-3)551--525 2004.
13

---

[63] Andrew Gardner, Jinko Kanno, Christian A Duncan, and Rastko Selmic. Measuring distance between unordered sets of different sizes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 137-143, 2014.
[64]
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang. and Wojciech Zaremba. Openai gym. OpenAl, 2016.
[65]
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. International Conference on Machine Learning (ICML), 2015.
[66]
Antonin Raffin. RI baselines zoo: A collection of pre-trained reinforcement learning agents. 200| 2018.
https: /tapa:///iihuune com/ www.aaeerrraaaeeeeeaaaeaaaaeeeee
[67] Laurel A Beckett, Michael C Donohue, Cathy Wang, et al. The alzheimer's disease neu- roimaging initiative phase 2: Increasing the length, breadth, and depth our understanding. Alzheimer's & Dementia, 11(7):823-831, 2015.
[68] David Maylor-Robinson Olia Archangelidi, Siobhán Carr, Rebecca Cosgriff, Elaine Gunn, Ruth H Keogh, Amy MacDougall, Simon Newsome, Daniela K Schlüter, Sanja Stanojevic, et al. Data resource profile: the uk cystic fibrosis registry, International journal of epidemiology. 47(1):9-10e, 2018.
[69] Federico P Gómez and Roberto Rodriguez-Roisin Global initiative for chronic obstructive lung disease (gold) guidelines for chronic obstructive pulmonary disease. Current opinion in pulmonary medicine, 8(2):11-86, 2002.
[70] Carl Edward Rasmussen and Christopher KI Williams. Gaussian processes for machine learning. 2006. Cited on, page 95, 2014.
[71] Jacob R Gardner, Geoff Pleiss, David Bindel, Kilian Q Weinberger, and Andrew Gordon Wilson. Gpytorch: Blackbox matrix- matrixx gaussian process inference with gpu acceleration. In Advances in Neural Information Processing Systems, 2018.
[72] Dimitrios Milios, Raffaello Camoriano, Pietro Michiardi, Lorenzo Rosasco, and Maurizio Fil- ippone. Dirichlet-based gaussian processes for large-scale calibrated classification. Advances in Neural Information Processing Systems, 31, 2018.
[73] Baptiste Vasey, Stephan Ursprung. Benjamin Beddoe, Elliott Taylor, Neale Marlow, Nicole Bilbro, Peter Watkinson, and Peter McCulloch. Association of clinician diagnostic performance with machine learning-basse decision support systems: a systematic review. JAMA network open, 1etpee222266222127707 2021.
[74] Yuchao Qin, Fergus Imrie, Alihan Hüyük, Daniel Jarrett, Mihaela van der Schaar, et al. Closing the loop in medical decision support by understanding clinical decision makkin:: A case study on organ transplantation. Advances in Neural Information Processing Systems, 34, 2021.
[75] Yarin Gal, Riashat Islam, and Zoubin Ghahramani. Deep bayesian active learning with image data. In International Conference on Machine Learning, pages 1 1183-1 192. PMLR, 2017.
[76] Andreas Kirsch, Joost Van Amersfoort, and Yarin Gal. Batchbald: Efficient and diverse batch acquisition for deep bayesian active learning. Advances in neural information processing systems, 32, 2019.
[77] Andrew Jesson, Panagiotis Tigas, Joost van Amersfoort, Andreas Kirsch, Uri Shalit, and Yarin Gal. Causal-bald: Deep bayesian active learning of outcomes infer treatment-effects from observational data. Advances in Neural Information Processing Systems, 34, 2021.
[78] Hideaki Ishibashi and Hideitsu Hino. Stopping criterion for active learning based on deter- ministic generalization bounds. In International Conference on Artificial Intelligence and Statistics, pages 386-397. PMIR, 2020.
[79] Gábor Bartók and Csaba Szepesvári. Partial monitoring with side information. In International Conference on Algorithmic Learning Theory, pages 305-319. Springer, 2012.
[80] Johannes Kirschner, Tor Lattimore, and Andreas Krause. Information directed sampling for linear partial monitoring. In Conference on Learning Theory, pages 2328 2269..9.... PMLR, 2020.
[81] Hongju Park and Mohamad Kazem Shirani Faradonbeh. Analysis of thompson sampling for partially observable contextual multi-armed bandits. IEEE Control Systems Letters, 6:2150- 2155, 2021.
[82] Guy Tennenholtz, Uri Shalit, Shie Mannor, and Yonathan Efroni. Bandits with partially observable confounded data. In Uncertainty Artificial Intelligence, pages 430-439 PMLR, 2021.
14

---

[83]
Rohin Shah, Pedro Freire, Neel Alex, Rachel Freedman, Dmitrii Krasheninnikov. Lawrence Chan, Michael D Dennis, Pieter Abbeel, Anca Dragan, and Stuart Russell. Benefits of assistance over reward learning. 2020.
[84] Allan Dafoe, Edward Hughes, Yoram Bachrach, Tantum Collins, Kevin R McKee, Joel Z. Leibo, Kate Larson, and Thore Graepel. Open problems in cooperative ai. arXiv preprint arXiv:2012.08633 2020.
[85] Dorsa Sadigh, Anca D Dragan, Shankar Sastry, and Sanjit A Seshia. Active preference-based learning of reward functions. functiios 2017.
[86] Sören Mindermann, Rohin Shah, Adam Gleave, and Dylan Hadfield-Menell Active inverse reward design. arXiv preprint arXiv: 1809.030066 2018.
[87] Erdem BIyIk, Malayandi Palan, Nicholas C Landolfi, Dylan P Losey, and Dorsa Sadigh. Asking easy questions; A user-friendly approach to active reward learning, arXiv preprint 2019.
[88] Nils Wilde, Dana Kulić, and Stephen L Smith. Active preference learning using maximum regret. In 2020 IEEE/RSI International Conference on Intelligent Robots and Systems (IROS), pages 10952-10959. IEEE, 2020.
[89] Dylan Hadfield Menell,, Stuart Russell, Pieter Abbeel, and Anca Dragan. Cooperative inverse reinforcement learning. Advances in neural information processing systems, 29, 2016.
[90] Dhruv Malik, Malayandi Palaniappan, Jaime Fisac, Dylan Hadfield-Menell, Stuart Russell, and Anca Dragan. An efficient, generalized bellman update for cooperative inverse reinforcement learning. In International Conference on Machine Learning, pages 3394-3402. PMLR, 2018. [91] Mark Woodward, Chelsea Finn, and Karol Hausman. Learning to interactively learn and assist In Proceedings of the AAAI conference artificial intelligence, volume 34, pages 2020.
[92]
Dylan Hladfield-Menele The principal-agent alignment problem in artificial intelligence. 2021. [93] Umaa Rebbapragada, Carla E Brodley, Damien Sulla-Menashe, and Mark A Friedl. Active label correction. In 2012 IEEE 12th International Conference on Data Mining, pages 1080- 1085, IEEE, 2012.
[94] Ruth Urner, Shai Ben David, and Ohad Shamir. Learning from weak teachers. In Artificial intelligence and statistics, pages 1252-1260. PMLR, 2012.
[95] Jan Kremer, Fei Sha, and Christian lgel. Robust active label correction. In International conference on artificial intelligence and statistics, pages 308-316. PMLR, 2018.
[96] Mattia Zeni Wanyi Zhang, Enrico Bignotti, Andrea Passerini. and Fausto Giunchiglia. Fixing mislabeling by human annotators leveraging conflict resolution and prior knowledge. Proceed- ings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 3(1):1-23, 2019.
[97] Andrea Bontempelli, Stefano Teso, Fausto Giunchiglia, and Andrea Passerini. Learning in the wild with incremental skeptical gaussian processes. arXiv preprint arXiv:201 1.00022, 2020.
[98] Stefano Teso, Andrea Bontempelli, Fausto Giunchiglia, and Andrea Passerini. Interactive label cleaning with example-based explanations. Advances Neural Information Processing Systems, 34, 2021.
[99] Chicheng Zhang and Kamalika Chaudhuri. Active learning from weak and strong labelers. Advances in Neural Information Processing Systems, 28, 2015.
[100] Songbai Yan, Kamalika Chaudhuri, and Tara Javidi. Active learning from imperfect labelers. Advances in Neural Information Processing Systems, 29, 2016.
[101] Shalmali Joshi, Sonali Parbhoo, and Finale Velei- Pre-emptive learning-to-defer for sequential medical decision-making under uncertainty. arXiv preprint mrXiv:2109.063
2021.
[102] Hussein Mozannar, Arvind Satyanarayan, and David Sontag. Teaching humans when to defer to a classifier via exemplars. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 5323-5331, 2022.
[103] Gagan Bansal, Besmira Nushi, Ece Kamar, Eric Horvitz, and Daniel S Weld. Is the most accurate ai the best teammate? optimizing ai for teamwork. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 1 1400-11444, 2021.
[104] Gagan Bansal, Besmira Nushi, Ece Kamar, Daniel S Weld, Walter S Lasecki, and Eric Horvitz. Updates in human-ai teams: Understanding and addressing the perfomaanc/compattilltyy tradeoff. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 2019.
15

---

Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(h) Did you describe the limitations of your work? [Yes] See Section S
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See Section S
(d) Have you read the ethics review guidelines and ensured that your paper conforms them?
[Yes]
you are including theoretical results...
(a) Did you state the full of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)? [No] The code will be made available upon acceptance.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Section I and Appendix [
(c) Did you report eтror bars (e.g.. with respect to the random seed after running experiments multiple times)? [Yes] Every result includes error bars across 10 runs.
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, cloud provider)? [N/A] This work is not computation-heavy the details are not pertinent. (All work was done on MacBook Pro, 13-inch, 2017).
4. you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you 're using/curating? [N/A]
Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applica-
ble? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board
(IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]


Reviews:
[{"review_id": "qSfZlpamKZ", "reviewer": "Reviewer_X8r8", "summary": "This paper studied a new problem: online decision mediation, in which the system (mediator policy) decides whether to follow human decision, or take an intervention, or request help from the expert based on human behaviors at every step. To learn a good mediator policy, the proposed solution UMPIRE balances the immediate loss terms against future improvement. They also validate the advantage of UMPIRE through empirical studies. ", "questions": "Please refer to the weakness section. Moreover, Figures are too small to read. \n\n", "limitations": "Yes.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\nThe big idea of online decision mediation framework is novel and interesting. Requesting advice from experts costs high in practice, while human behaviors are not always optimal and needs to be corrected if necessary. Thus, it’s very worthwhile to think about mediator policies. I think this paper proposed nice formulation and methodologies for this practical scenario.\n\nWeakness:\n1. Many parts are not clear.\n    - in equation (3), what is Y?\n    - How did you derive equation (5)?\n2. The main theoretical result (Theorem 1) is hard to interpret. From the right side of equation (6), I cannot tell how much the improvement is. It will be helpful if authors can give some specific examples to show the scaling. \n3. This paper is not easy to read and follow. Existing works and new techniques are often mixed in sections. \n\nOverall though I think this paper is not that theoretically strong, but ODM and their methodologies are good contributions to the community.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "kOs7dYMAvuw", "reviewer": "Reviewer_EaPK", "summary": "An interesting paper which focuses on formalising the sequential problem of online decision mediation (ODM)  - and proposes a specialised solution (uncertainty-modulated policy for intervention and requisition - UMPIRE) to tackling pitfalls which traditional bandit algorithms face in ODM. The paper is relevant and very promising for describing potential solutions for real-world and real-time decision making, especially in safety-critical applications (e.g. clinical diagnosis), wherein, a human-in-the-loop approach can benefit from providing an effective interface between  human mistakes and expert feedback - as the paper also mentions.", "questions": "Where could this proposed solution be used outside of clinical decision support? Can the authors provide few more examples in the Conclusions section e.g. could it be used in industry for real-time decision support of engineering systems (e.g. during operations & maintenance) etc.?\n ", "limitations": "What are the main drawbacks/limitations of UMPIRE? It is not clearly mentioned in the paper's Section 5 - it would have been useful to have a separate \"sub-heading\" for Limitations e.g. in a Discussion section as it is inadvertent that the proposed algorithm may suffer from pitfalls which need to be clearly outlined with simplistic language (and comprehensively). \n\nUpdate: Thanks for addressing this reviewer's comments - the paper is now substantially clearer and comprehensive. I have changed my score from 6 to 7.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "A very well written and comprehensive paper, well-supported by experiments and comparison with other baseline models/algorithms, particularly on real-world datasets/benchmarks which makes it very interesting to read.\n\nThe main weakness of the paper is that it is very verbose and text-heavy at some places e.g. in Section 2.2. Can these e.g. \"learning with rejection\" be better encapsulated (or relevant papers from past literature cited instead of a complete re-definition from scratch?). This would greatly help improve readability of the paper.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "iMC7U44FwM7", "reviewer": "Reviewer_Vsuz", "summary": "The work proposes a solution to the following problem setting: Given an input instance, an imperfect human provides a provisional decision/prediction. A mediator model decides whether to accept, to intervene by showing its own decision (at a cost), or whether to defer (at a cost) to another human expert. The instances come in a streaming fashion. The proposed approach (UMPIRE) trades off system risk (final decision) for model risk (ability of the mediator to make predictions and learn about the deferring strategy). The algorithm is compared with several related works, mainly bandit algorithms, and shown to mostly outperform. A detailed sensitivity analysis is also presented.", "questions": "My questions are mostly related to W1 above. The human appears to be somewhat redundant in the workflow, also because they seem to have no say if their decision is overridden by the model's decision. What is the motivation for choosing \\alpha = 0.5?\n\nIn addition, it would be useful to see the progress of how z is distributed across timesteps. Does the model decide to request for the expert's prediction mostly at the beginning? When does it start to request less and in which cases in the timeline it decides to just accept the human decision?", "limitations": "-Literature in this field is somewhat divided between work that assumes that a model does the routing (deferring) and work that assumes that the human is in control for when to defer to an AI or when to ask for help or when to decide to integrate AI recommendations in their decision (see examples below).  There probably needs to be a discussion here or in the introduction that describes when\\why it makes sense to sometimes have a model that controls the routing. \n\nTeaching Humans When To Defer to a Classifier via Exemplars\n\nIs the most accurate ai the best teammate? optimizing ai for teamwork\n\nUpdates in human-ai teams: Understanding and addressing the performance/compatibility tradeoff\n\n- There may also need to be a more extended discussion on why pessimistic bayesian sampling becomes a competitive option when the expert noise increases. This detail also needs to be mentioned in the introduction or abstract as an important insight for real-world applications, where indeed most often than not expert decisions are also not perfect.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nS1- Optimizing for overall system accuracy instead of autonomous model predictions is an important albeit overlooked direction. \n\nS2- The paper is very well written, and results are clearly presented.\n\nS3- The paper also does a very good job at contrasting and comparing with related work.\n\nWeaknesses:\nW1- While optimizing for overall system accuracy is important, the choice of keeping an imperfect human decision maker in the loop is not very well motivated from both an ethical and mathematical standpoint. If I am not mistaken, the imperfect human decision maker is simulated with a probability of 0.5 (line 304). Thus, it is questionable why keep this actor in the loop on the first place and why not change the whole interaction altogether to have the model interact only with the expert. This is in fact the setting that other works in human-ai decision making also consider (e.g. [24] non-streaming or [*] streaming). Otherwise, if there are any other reasons to why this workflow may still make sense, at least the experiments need to be adjusted to a probability of error > 0.5 for the human. \n\n[24] Consistent estimators for learning to defer to an expert\n[*] Pre-emptive learning-to-defer for sequential medical decision-making under uncertainty\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept