Reward Design for Justifiable Sequential Decision-Making
Authors: Aleksa Sukovic
Goran Radanovic
Venue: ICLR 2024 poster (2024)
Keywords: reinforcement learning
reward design
alignment
preference-based learning
Primary area: reinforcement learning
TL;DR: This paper proposes the use of a reward model defined as the outcome of a two-player zero-sum debate game, where agents compete to justify made decisions with supporting evidence according to human preferences.

Abstract: Equipping agents with the capacity to justify made decisions using supporting evidence represents a cornerstone of accountable decision-making. Furthermore, ensuring that justifications are in line with human expectations and societal norms is vital, especially in high-stakes situations such as healthcare. In this work, we propose the use of a debate-based reward model for reinforcement learning agents, where the outcome of a zero-sum debate game quantifies the justifiability of a decision in a particular state. This reward model is then used to train a justifiable policy, whose decisions can be more easily corroborated with supporting evidence. In the debate game, two argumentative agents take turns providing supporting evidence for two competing decisions. Given the proposed evidence, a proxy of a human judge evaluates which decision is better justified. We demonstrate the potential of our approach in learning policies for prescribing and justifying treatment decisions of septic patients. We show that augmenting the reward with the feedback signal generated by the debate-based reward model yields policies highly favored by the judge when compared to the policy obtained solely from the environment rewards, while hardly sacrificing any performance. Moreover, in terms of the overall performance and justifiability of trained policies, the debate-based feedback is comparable to the feedback obtained from an ideal judge proxy that evaluates decisions using the full information encoded in the state. This suggests that the debate game outputs key information contained in states that is most relevant for evaluating decisions, which in turn substantiates the practicality of combining our approach with human-in-the-loop evaluations. Lastly, we showcase that agents trained via multi-agent debate learn to propose evidence that is resilient to refutations and closely aligns with human preferences.

Published as a conference paper at ICLR 2024
REWARD DESIGN FOR JUSTIFIABLE SEQUENTIAL DECISION-MAKING
Aleksa Sukovie 1,2 Goran Radanovie
Max Planck Institute for Software Systems' 1 Saarland University" 2 lasukovic, gradanovic @mpi sws. org
ABSTRACT
Equipping agents with the capacity to justify made decisions using supporting evidence represents a cornerstone of accountable decision-making. Furthermore, ensuring that justifications are in line with human expectations and societal norms is vital, especially in high-stakes situations such as healthcare. In this work, we propose the use of debate-based reward model for reinforcement learning agents, where the outcome of zero-sum debate game quantifies the justifiability of a de- cision in particular state. This reward model is then used to train a justifiable pol- icy, whose decisions can be more easily corroborated with supporting evidence. In the debate game, two argumentative agents take turns providing supporting ev- idence for two competing decisions. Given the proposed evidence, proxy of a human judge evaluates which decision is better justified. We demonstrate the po- tential of our approach in learning policies for prescribing and justifying treatment decisions of septic patients. We show that augmenting the reward with the feed- back signal generated by the debate-based reward model yields policies highly favored by the judge when compared to the policy obtained solely from the envi- ronment rewards, while hardly sacrificing any performance. Moreover, in terms of the overall performance and justifiability of trained policies, the debate- feedback is comparable to the feedback obtained from ideal judge proxy that evaluates decisions using the full information encoded in the state. This sug- gests that the debate game outputs key information contained in states that is most relevant for evaluating decisions, which in turn substantiates the practicality of combining our approach with human-in connann----thh-loe tn-the-loop evaluations. Lastly, we show- case that agents trained via multi-agent debate learn to propose evidence that is resilient refutations and closely aligns with human preferences.
INTRODUCTION
Reinforcement learning (RL) has been achieving impressive successes in wide range of complex domains. However, specifying a reward function which incentivizes RL agents to exhibit desired behavior remains difficult (Leike et al. 2018). Prior work proposes several approaches that address these difficulties (Kwon et al.) [023] Bahdanau et al., 2018; Jothimurugan et al., 2019), including those based on learning from pairwise preferences (Christiano et al. (011]). However, such reward models are not informative enough for agents to learn how to substantiate their decisions with sup- porting evidence a key property needed for accountable Docision-making (Bovens) 2007). Hence, we ask the following question: How can we design rewards that incentivize agents to carry out given task, but through decisions that can be justified?
To answer this questions, we consider setting in which an RL agent acts as principal, influencing the state of the world, while human agent acts as a verifier responsible for validating the justifia- bility of the decisions taken by the RL agent, based on the evidence provided. This scenario mirrors common real-world situations where accountability critical, including healthcare scenarios where doctors are tasked with scrutinizing the validity of automated decisions. We recognize three impor- tant properties that the provided evidence should satisfy. First, it should reflect the underlying state of the world, i.e., include the key information based on which action was taken. A naive solution is to provide the full state as evidence. However, this discards the fact that the human, as a suboptimal- decision maker, may not easily reason about the taken decision because the state might be too large

---

Published as a conference paper at ICLR 2024
Tt
A1
π
J
n B
A2
To obtain a debate reward r in the state St, two argumentative agents A₁ and A2 take turns proposing supporting evidence (depicted as triangles) for two decisions, up to a predefined limit (here, 3 evidence each). Then, positive debate reward is issued whenever proxy of judge I considers action a,, taken by the justifiable policy N', better justified than action a. taken by the baseline policy BB This reward is then mixed with the environment reward rf via debate coefficient X, yielding the final reward r, used to train the justifiable agent.
Figure I:
or otherwise incomprehensible. This brings us to the second property: the provided evidence should also be concise and only reflect the most relevant information. The third property builds on the second one: given that the provided evidence contains only partial information about the state, this information should not be easy to refute. More specifically, additional information about the un- derlying state, e-g., additional evidence, should not change the human's judgment. Therefore, the overall challenge is to design a framework which can enable such justifications through a reward model that incentivizes both performance and justifiability.
To tackle this challenge, we consider a framework which modifics the environment rewards by mix- ing them with rewards coming from debate-based reward model (see Figure[p. Dp. Each debate-based reward is defined through the outcome of two-player zero-sum debate game. More specifically, we let two argumentative agents debate by taking turns providing supporting evidence contingent the current state, each corroborating a decision made by one of two competing policies. Based on the proposed set of evidence, the human then states their preference over these decisions, and this preference defines the debate reward. In this setup, one decision comes from a baseline policy, while the other comes from a justifiable policy that we optimize. Our approach builds upon the work of Irving et al. (2018). but we consider sequential decision-making problems and utilize debate to quantify a decision's justifiability. To this end, we recognize two main technical challenges. First, learning proxy of a human judge that evaluates decisions solely based on the proposed evidence, with comparable performance to methods requiring full state exposure. Second, learning a represen- tation of argumentative strategies that are solutions to different instances of the debate game. These two components are needed to enable efficient learning of the justifiable policy.
Contributions. Our contributions are follows. (i) We formalize the problem of justifiable makin,,, modeling debate as an extensive-form game (Sec. 33. (ii) We provide a method for learning proxy of human judge that evaluates a decision's justifiability using proposed evidence (Sec. (iii) We propose an approach to learning contextualized argumentative strategies that constitute approximate solutions of the debate games (Sec. 4.3). (iv) We conduct extensive empir- ical evaluation of our approach on real-world problem of treating sepsis, testing the performance and justifiability of policies trained through our framework (Sec. 5.2). well the effectiveness
and robustness of argumentative agents (Sec. 5.3 Sec. 5.4 and Sec. 5.5)
2 RELATED WORK
Debate. Irving et al. (2018) first outlined theoretical and practical implications of debate as a method for training aligned agents. Debate has also been used to improve factuality of large language models (Du et al. 2023), reason in knowledge graphs (Hildebrandt et al. (2020). and aid in Q&A tasks (Perez) al. a 2019). We build on this line of work by bringing the utility of debate in sequential decision- making problems, where it is used as an approach of quantifying justifiability of made decisions. In addition, by leveraging learning from pairwise preferences as Christiano et al. (011)) our
OOr code is publicly available at wut.hhhw..oooeesssoooooeoeeeeeee

---

Published as a conference paper at ICLR 2024
framework enables encoding of human judgments in form of the preferred evidence that renders decision justified.
Reward Design. Previous work proposes several approaches that address difficulties of reward design, based on natural language (Kwon et al. 2023] Bahdanau et al., 2018), mile-based methods
(Jothimunugan et al. (2199) and preferences (Biyik & Sadigh 2018 Christiano et al. 2017). Further- more, there is a line of work considering interpretable reward design, including reward sparsification (Devidze al., 202T) and reward decomposition (Jozaapaitis et al.) 2019 [ica et al. 2020). Dif- ferently, we define a reward model as an outcome of a zero-sum debate game, which in itself is interpretable, and learn contextualized policies that solve it. Most similar to our methodology is learning from pairwise preferences as in [Cristiano et al. (017), only we assume comparisons are made over justifying evidence, without exposure to the underlying state or trajectory.
Explainable AI and Other Related Work. Our approach is most similar to the attribution based techniques for explaining the inner workings of models. In such approaches, contributions of input features are quantified numerically (Lundberg & Lee, 2017 Ribeiro et al. (0066) visually repre- sented as heatmaps (Selvaraju et al. 2017 Mundhenk al. 2019). This line research has received significant attention, also in the context of explaining decisions of RL agents (Ragodos et al. 2022] Bastani et al. 2018). However, one limitation of these explanations is the inability to further align approach them with human preferences (Hadfield 2021 Brundage et al. 2020). In contrast, additionally enables human-specifiie justifications, facilitating the incorporation of preferences into their generation. There is also line of work that examines approaches for adaptation of agent's recommendations (actions) to a baseline human policy in an expert-in-loop setup & Pauphilet 2022 Faros al. 2023). Differently, we consider problem of learning to take actions that can be justified, where the agent itself acts as a primary decision-maker in the environment.
3 FORMAL SETUP
We consider sequential decision- making problem, where an agent interacts with an environment over a series of time-steps, modeled by a discrete-time Markov Decision Processes (MDP). The episode begins by sampling starting state from the initial state distribution p. At each time-step an agent observes the current state E S, takes action a A and receives an environment reward (s;, a.). The environment then transitions to a successor state 8141 with a probability specified by the transition function T( (5,411$2,,;))
3.1 AGENTS
We consider two kinds of agents that operate in the defined MDP: baseline and justifiable agent.
Baseline Agent. The baseline agent aims to maximize the expected discounted value of the envi- ronment's return given R Ei-1 7'r (st,ar), where ? E [0, 1] is a user-specified discount factor. Its optimal aatien-value function, defined as Q" a) = max, E [R|s; = = a. T). is the maximum expected discounted return obtained after taking action a state s. We will refer to deterministic policy that maximizes the expected value of R as the baseline policy x which satisfies n B (s) E argmax, (*(s,
Justifiable Agent. While the baseline agent learns to solve the environment, its decisions may not be easy to justify when evaluated by a human. To design an agent that accounts for the justifiability of its decisions, we consider a reward defined as weighted combination of the environment reward rf and the debate reward rd, which encapsulates human judgment of justifiability and is specified in the next subsection. The expected return is then defined as:
- R, R, -'= T-1 7a ((1-)) X) (s,, az) + X r (st, ar a
where [0, 1] is debate coefficient, balancing between environment and debate rewards, and at is the action of the baseline agent in state The value of X = 0.0 corresponds to the return of the baseline policy, whereas for the value > = 1.0 we obtain a setup similar Christiano et al. (017)), where the agent relies only on the debate reward model. We refer to an agent that maxi- mizes the expected value of R, as the justifiable agent, denote its optimal action-value function as
In practice, we require the haseline policy to be well-perf.rming. but not necessarily optimal.
3

---

Published as a conference paper at ICLR 2024
Qj(s,a) a) max, E [R)|s: 8, 8, a, π| and its deterministic policy x (s) Q3 as the justifiable policy.
3.2 REWARD MODELING VIA DEBATE
Our objective is to learn reward model, denoted as (s,, at, af),, which quantifies the justifiability of a decision. To this end, we introduce a reward model based on debate game. this model, (st,, (z. o.)) represents the value of a debate game induced by tuple (st, dt, af"). In more details, for a given tuple (St, at, az)) the induced debate game is formulated as a two-player zero-sum extensive-form game (Shoham & Leyton-Browe the decision ay in state St, while the second player argues for taking the baseline decision of in st.
2008), in which the first player argues for taking
Debate Game. With N denote set of nodes in finite, rooted game tree. The action space is represented by a finite set of evidence E, and each node n E N consists of evidence proposed thus far in the game
{e) S E. Additionally, the edges successors of each node define actions (evidence) {e : e E E ) available to the acting player, where we disallow evidence repetition. The debate game is evidence proposed up until the current point and have a complete knowledge about the state of the game. The game proceeds as players take turns: in turn l, player i = mod 2+11 proposes evidence The total number of turns L is assumed to be even and significantly smaller than the evidence
inffectiifformationn game: all times, the players have ambiguity about the
f F7
set, i.e., L < [E) After the last turn, a terminal node nn (e), ei,ee,... ei...., e/2,/2 e2,/2) fen.. } is evaluated. The players' utilities are un (RL) (2 = U(ac, of {Enz }), with U defined as:
+1,
Ja, {et) [aB
B (e))
Uia a =
0,
Ja, (e))))) ( J (a {e})
1,
otherwise
Here, J is a model of human judge that, for a given decision and evidence (e). outputs numerical value [a, {e}) E R quantifying how justifiable a is under evidence {e}.
Strategies and Solution Concept. player's strategy o² . E outputs available evidence ('(n) E \ n in a given node n and E¹ is the set of all strategies of the player i. Based on the utility function, we additionally define G( o0) }²), St, at, at) as the payoff (utility) of the first player, conditioned on both players following the strategy profile for ,o²), Then, set of the best responses of the first (resp. second) player to its opponent strategy o² (resp. a²) is defined as b¹ (0(2)
arg max (( 0²), 8,, 0z,, az (resp. b² (0 = arg ming²ega G _ _0a) az, af)) A G((a))
strategy profile ō = { o²} is a pure-strategy Nash equilibrium if o² (o Because the debate game is perfect-information extensive-form game, pure-strategy Nash equilibrium exists (Shoham & Leyton-Bre Brown| 2008), and due to its zero-sum structure, can be obtained by solving the following max-min optimization problem: max, min,2 G((o', ,0²) St. ae, a.)). We refer to G((ō', _02] St, ae, a. as the value of the game and define yd (st, a,, a.") ) i.e., rd (St, de; at = a G((a") _, 81, ae; a²)). where 0 is a scaling coefficient.
to be equal to it,
)2),,
4 LEARNING FRAMEWORK
To effectively debate game outcome during training of justifiable policies, it is necessary to devise a model a human judge J (Sec. (.2) that encapsulates justifiability judgment and addi- tionally learn argumentative policies that approximate Nash equilibrium and are able to generalize across different instances of the debate game. With ("((,,, at, a we denote an approximation of (St, ae; af)). obtained by running the argumentative policies from Sec. 4.3 in the debate game induced by (St, ae, aa)) In all our experiments, we set a 5.
In our implementation of the dehate game, we randomly chose which player has the first tum,, i.e., +++ r) mod + 1, where 7 M({0, 1}). This only affects the order of the evidence in zy. For two-player zero-sum game, the value of the game (or payoff) is unique (von Neumann & Morgen- stem 1947
4

---

Published as a conference paper at ICLR 2024
4.1 PREFERENCE DATASET
We assume the human judgments are collected in a preference dataset D of tuples (st, ao, d1 ,)), where 8, is state, do f di are two decisions and p E [0, indicates which of the decision is more justified in a particular state. The value of p 0 (resp. = 1) indicates that og (resp. 01) is more preferred.
4.2 JUDGE MODEL
Because asking for human feedback is expensive, we aim to learn a judge model from the dataset 'D of preferences that can used to evaluate the outcome of debate games. The judge, parametrized with learnable parameter 0 Rd is defined how justifiable a decision a is, given evidence {e}. Similar Christiano et al. (2017), we addition- ally assume that justifiability preference for decision dg over decision d1 follows the Bradley- model (Bradley Terry 1952):
scalar reward function Ja(a. {e]] IR quantifying
exp Ja(uo, {et)
P(ao an, {et)= exp Jo(aa {e]) + exp Jo(a1,
Here, we require the judge to quantify the level of justifiability given all evidence at once. Note the lack of dependency on the state sr: the judge evaluates only proposed evidence, whereas the argumentative agents are in charge of providing those evidence, contingent on the state. We optimize the parameters by minimizing the cross-entropy loss between preference-predittie and labels from the dataset. See App. C.I for more details.
4.3 ARGUMENTATIVE AGENT
Our overarching goal is to learn a generalizable argumentative policy that is able to solve any de- bate game, conditioned on its defining tuple (st, at, a()). This is difficult feat, the evidence set available to the agent is contingent on the state st. To this end, we can treat the debate game an instance of a contextualized extensive form game (Sessa al. 2020), where the tuple (&,, at, aa)) as a context = E 2. In a general case, the justifiable agent n' observes the state 8; and takes action ae which, paired with an action a. the baseline agent B would have taken, sets the debate game context = {St, az, a. In the specific case of offline reinforcement learning we consider here, the contexts are sampled i.i.d. from the static dataset throughout train- ing of both, argumentative and justifiable agents. Therefore, given a sample from the preference dataset (st, ap, a1,p) ~ D, we set the context to 2 (st, (s, 01-p). Player i's contextual strategy, parametrized with learnable parameter Φ; IRd, defined 0z Z + E', mapping a con- text to the strategy of the player for the induced debate game 5 To learn parameters Φi, we solve: max ERd, min Eg~D[G((o (2),0 0 (=)},
consider
4.4 METHIOD
Judge. The judge To is parametrized by weights 0 E Rd of neural network with two fully- connected layers of size 256, using parametric relu (He al. (015)) activation and batch normal- ization (Ioffe& Szegedy 2015). The network receives a vector x in (l where only the values of evidence {e} are shown, while the rest are set to zero. In addition. a binary mask of the same dimen- sion, wherein all elements corresponding to the evidence {e} are assigned value of one, while the remaining elements are set to zero, as well as one-hot encoded action are passed. The learning is done for a total of 100 epochs using batches of 64 comparisons sampled from the preference dataset D, Adam optimizer and learning rate of 5e-4. See App. C.I for more details.
A Argumentative Agent. We represent parameters E Rdd of the argumentative agent (-12), as weights of a neural network composed of 2 fully-connected layers of size 512 with leaky-relu activation function (Maas et al.) 2013) with slope 1c-2. The network takes as input a 44-dim state, one-hot encoded decision for which the agent argues, well as binary mask of currently proposed evidence. The evidence (action) space of the policy is discrete and has 44 choices, each corresponding to exactly state feature. To train the agent, use PPO (Schulman et al. 2017)
"Fqiivleetlly will also denote the contextual strategy of player of (22(-12) :N + for 2 Z.
5

---

Published as a conference paper at ICLR 2024
0-0.0
0-0.25
A-0.50
0-0.75
2-1.0
clinician
1.00
Preference Over Baseeia
0.50
0.00
0.40
Perfomarc
0.70
0.80
Mortality
0.30
Mortality
6,30
0.60
0.20
0.20
0.20
0.40
100
230 lterations
100
450
505
k-0.55
X-0.5 X-8.55 1-10
0,0.5.00
-0.11
0.20
0.01
0.00
-0.50
-0.75
0.00
0.5
A50
Policy Veriant
Excess
Excess
(a) WIS evaluation
(b) Preferred baseline
(c) IV dose excess
(d) VC dose excess
Figure 2: Evaluation of justifiable policies. For (b)-(d), the confidence intervals represent #2 stan- dard errors the mean over random seeds. (a) Policy performance as measured by WIS evaluation on held-out test set with 1 terminal rewards for every patient discharge or death. The mean and standard deviation are reported over random seeds. (b) Percent of times judge preferred decisions of justifiable policies (i.e.. À > 0.0) compared to those of the baseline policy (i.e., À 0.0). (c) (d) Observed patient mortality (y-axis) against variations in IV/VC treatment doses prescribed by clinicians compared the recommendations of learned policies (x-axis).
and examine two optimization strategies, namely self-play and maxmin. We train the self-play debate by letting the agent argue with a copy of itsell]] updating that copy every 100k steps, and repeating the procedure for 500 generations. For maxmin debate, we use 2 different set of weights, é1 and 62 to represent the agents' policies. The first agent i.e., Φ1 trained for 4k steps, followed by training of the second agent i.e., @2 for 100k steps, repeating this procedure for 500 generations. This approach of overfitting the second agent the current version of the first ensured learning of defense strategies against very strong adversary. See App.
C.2 for more details.
To learn a justifiable policy x (s) E arg max, (', a), we consider model-free reinforcement learning methods based on Q-learning. Our approach builds on top of Raghu et al.) (017)) and based on a variant Deep-Q networks (Mnih et al. 2015), specifically double-deep Q-network with the dueling architecture (Wang et al. 2016 Van Hasselt et al.) 2016). The final network consists of 2 fully-connected layers of size 128 using leaky-relu activation function with slope 1c-2. The learning is done in batches of 256 (s, a,, s') tuples sampled from a Prioritized Experience Replay buffer (Schaul et al., 2013) using a learning rate 1e-4, for a total of 25k iterations, evaluating the policy every 50 iterations from the debate, i.e., pd, we augment the reward from the replay buffer as described in Sec. 3.1 Modifying just the observed reward, without changing the state dynamics, has been shown to be sufficient to induce learning of an alternative policy (Ma et al., (019).. See App. details, including additional loss terms and a list of all hyperparameters.
Justifiable Agent.
held-out test set. When incorporating signal
C.3 for more
EXPERIMENTS
In our experiments, aim empirically evaluate the properties of debate method for reward specification. To this end, we perform quantitative and qualitative evaluation of justifiable policies (Sec. (5.2), analyze the effect of pairwise comparison only over proposed evidence on performance and justifiability (Sec. (.3)). examine the necessity of multi-agent debate in learning to provide evidence that resilient refutations (Sec. [5.4), and compare efficiency in providing supporting evidence for a decision of argumentative policies and SHAP
method (Sec. 5.5) 5.5).
5.1 ENVIRONMENTAL SETUP
Sepsis. Data for our cohort were obtained following steps outlined in Komorowski et al. (018)) utilizing MIMIC-III v1.4 database (Johnson al. 2016). We focus analysis patients that fulfill Sepsi-3 criteria (Singer et al., 2016). 18,585 in total. The patient vector consists of 40 contin- uous and 4 discrete features, and the action space consists of 5 x 5 discrete choices of intravenous
'In terms of learaalee parameters, this implies Φ2 Φ1-

---

Published as a conference paper at ICLR 2024
(IV) fluids and vasopressors (VC). As in Raghu al. (2017). the environment rewards are clinically guided. We set the environment reward r" to 15 for terminal states resulting in patient survival or death, respectively, and shape the intermediate rewards based on the SOFA score (measure of organ failure). See App. for more details.
Preference Dataset. To make comprehensive evaluation possible, we define a synthetic ground- truth preference by making an assumption that a human judge always prefers a treatment prescribed by the clinician. Therefore, clinician's actions are the justified actions in our experiments. More formally, we bootstrap the dataset of preferences 'D by matching every pair (st. 4x) from the cohort with an alternative action a, =((A), a, # at sampled uniform-randon random from the action space, initializing the preference variable p to point to the true action at as taken by the climician (see App. D.5 for alternative dataset bootstrapping methods). The number of evidence fixed to 6 (~ 13.6% of the full state) (see App. D.I for results with 7. = 4 evidence). The dataset is split into chunks of 70%, 15%, 15% used for training, validation, and testing respectively. We report all our results on held-out test set, not seen by any of the agents nor the judge. When training judge model, we additionally augment a tuple (st, ag; P) with an evidence set {e}. To generate it, we sample predetermined number of state features uniform-rande is inspired by Irving et al. (2018) and ensures evidence is contingent on the state (Sec. (32) On the test set, the judge is able correctly recover the underlying preference from the dataset with relatively low accuracy of 65%.
i.e., {e} ((s)), from the state, which
Baselines. When comparing effectiveness of treatment policies, we consider two baselines. First, we consider the observed reward of the clinician from the dataset (depicted as gray horizontal line in plots). Second, the baseline policy (Sec. 3.I) serves as an indicator of the optimal treatment policy. To demonstrate the robustness of multi-agent debate, we introduce an isolated argumentative agent. This agent aims to find an evidence set {e} that maximizes Happ {e}). ). for a given ag. To achieve this, we solve a search problem akin to the debate game by applying reinforcement learning (see App. for more details). Lastly, we use SHAP when comparing effectiveness of debate to a feature-importance approach in providing supporting evidence for a decision.
5.2 EXPERIMENT 1: EFFECTIVENESS OF TASKS POLICIES
To examine the potential of specifying the reward as an adversarial game and its effect on the quality of learned behaviors, we train several policies by varying the debate coefficient X.
Quantitative Evaluation. In Plot 2a we evaluate the performance of different justifiable policies on held-out test set during the course of training using weighted importance sampling (WIS) (Sutton) & Barto 2018) Likewise, in Plot 2bb we show the judge's preference over decisions made by justifialle policies (i.e., À > 0.0), compared to the baseline policy (i.e., = 0.0), when the two were different The observed inherent trade-off between performance and justifiability suggests that tuning the debate coefficient important in practice, and we further elaborate on this in See.
6
Qualitative Evaluation. addition to quantitative evaluation, we perform qualitative analysis showcase
similar to Raghu et al. (2017). Plots sh correlation between observed mortality and difference between the optimal doses suggested by policies, and the actual doses prescribed by the clinicians. For all trained policies, the lowest mortality is observed when the difference is zero, thus further showcasing their potential validity. It is also encouraging to see that the policy trained solely with debate rewards (i.e., À = 1.0) remains quantitatively and qualitatively competitive in addition to being highly favored by the judge, even though relies only on debate rewards.
5.3 EXPERIMENT 2: DEBATE-BASED FEEDBACK VS. STATE- BAATED FEEDBACK
Because the judge is evaluating justifiability using only proposed evidence, a natural question to ask is how much does this affect the performance and alignment of trained policies. To provide an answer, we train a new judge that evaluates decisions based on the full state, namely J'(a, st). We then use this judge instead of pd to train new justifiable policy using state-based feedback, one
We discuss different approaches for defining of evidence in Sec.
*The behavior policy used in WIS was derived via behavior cloning, further described in App. See Plot App. D for detailed breakdown of actions proposed by justifiable policies.
C33
7

---

Published as a conference paper at ICLR 2024
1.00
1.00
1.00
1.00
Perommarso
0.80
0.80
Derfomeat
0.800
0.00
0.20
0.70
0.70
0.73
0.60
0.60
0.60
0.50
0.50
0.50
0.50
feodback
0.30
0.30
0.30
0.39
Iterations
lterations
lterations
lerrtions
(a) A 0.25
(b) À = 0.50
(c) A 0.75
(d) A = 1.0
Figure 3: Performance of policies trained with state-based feedback compared to debate- bastte feed- back, as measured by the weighted importance sampling evaluation on held-out test set with #1 terminal rewards for every patient discharge or death. The mean and standard deviation are reported over 5 random seeds.
for each A. Given a sample (st, 00,00, a1, D, we consider a policy л aligned to the ground-truth preference if Q7 (8, ap) 2 Q (1p0)p)).
The results shown in Plot for various policies trained with state- and debate-based feedback. We see that, even though the debate-based approach uses only ~ 13.6% of the full state, the achieved level of alignment remains similar. Likewise. in Plots Ba 3d we observe matching trend when comes to performance of the policies, although here we additionally note that the difference seems to increase in favor of policies trained with the state-based feedback as we increase the debate coefficient X. These results seem promising, as they indicate one can expect to obtain a competitive level of alignment and performance, while requiring the judge to elicit preference over relatively small number of evidence.
5.4 EXPERIMENT 3: EFFECTIVENESS OF ARGUMENTATIVE POLICIES
Preference Recovery Rate. We recall the judge's 's accuracy in correctly predicting the preferred ac- tion dp from the preference dataset was 65%. To evaluate the effectiveness of argumentative policies, for each sample (st, ag, d1.p) ~ D we measure the judge's accuracy in predicting the more justified action ap. when evidence {e} is provided by one of the argumentative policies. In Plot 4b (green) we show the judge's s accuracy when different argumentative agents propose = 6 required evidence for action ap, averaged across 1000 different debate games. The judge's s accuracy is boosted from 65% to near 90%, demonstrating that agents can significantly amplify the capabilities of an otherwise limited judge.
Robust Argumentation. A good supporting evidence both convincing and not easily refutable by counterarguments. To test the robustness of the proposed evidence, we train 3 adversarial confuser agents [0] each targeting one of the three (frozen) argumentative agents. The goal of the confuser convince the judge the opposing action 01 01-p- For L to obtain evidence (e}, the agent (maxmin or self-play) and its corresponding confuser take turns and propose 3 evidence each. The isolated agent trained to first propose remainder (see App. C.2. for more details and App. [.3] for additional results). Plot 4b (red) shows judge's accuracy this setting for 1000 different debate games. We observe that the isolated argumentative agent (Sec. 5.1. is not resilient to refutations, enabling the confuser to bring the judge's 's accuracy down to 38% effectively convincing of the opposite of its preference. Differently, both maxmin and self-play agents managed to keep the judge's accuracy to about 85%.
evidence, followed by the confuser proposing the
5.5 EXPERIMENT 4: COMPARISON TO SHAP-BASED EXPLANATIONS
widely used approach to analyze black-bor machine learning models is through feature-attribution techniques. We aim to demonstrate that these explanations may not necessarily be effective when used as supporting evidence. We focus on the SHAP framework (Lundberg & Lee (017)), specifi-
cally in providing justifications for decisions of various justifiable policies (Sec. 5.2) To justify
"Confuser agents the same architecture argumentative agents, further described in App-
8

---

Published as a conference paper at ICLR 2024
Frac Of Allgned Decisions
1.00
0.80
Judge Accurecy
0.50
0.05
0.00
0.60
0.60
0.85
0.40
5.40
0.00
0.20
0.20
0.20
A-0.25
A-0.30
AH0.77
0.00
manmin
sulfplay
0.65
(a) State/debate feedback
(b) Preference recovery
(c) Comparison to SHAP
Figure 4: (a) Fraction of aligned decisions of policies trained with debate- and state-based feedback. Confidence intervals (CI) represent #2 standard eπrors of the mean over 5 random seeds. (b) Ac- curacy of the judge in predicting the preferred action, with and without the confuser agent, with CI representing +2 standard errors of the mean estimate. (c) Effectiveness of SHAP- explana- tions when used to justify a decision, as measured by the judge's 's accuracy, with CI representing +2 standard errors of the mean estimate.
decision using SHAP, we select the top 6 state features, as ranked by their Shapely values. For ar- gumentative models, we either run the debate between ag and aj (for maxmin and self-play agents) or propose 6 evidence in isolation (for isolated agent). In Plot we show the judge's accuracy against different approaches of proposing the evidence {e), across 1000 comparisons (&t: Go, 41.P) sampled uniform-randon from the test set. The SHAP-bases evidence do improve the accuracy to about 70%, but nevertheless fall short compared to the argumentative agents.
6 DISCUSSION
In this work, we proposed use of a debate-based reward model as a general method of specifying desired behavior and necessary evidence to justify it. In this section, we take a step back and touch upon couple of aspects that are relevant to future research based on this work.
Eliciting Preferences. The success of debate depends on the human's 's capability of judging its out- come, a process which may be affected by one's 's belief's and biases. Extra care must be taken when collecting large datasets of preferences, as belief bias |T| is known to alter the results of judgment in human evaluations (Anderson & Hartzler| (014) and is amplified in time-limited situations (Evans) & Curtis-Holmes, (205)). which human annotators frequently encounter. Furthermore, in our exper- iments, we assumed existence of single preference (clinician's 's decision). However, preferences collected from multiple raters will undoubtedly yield a higher variability this respect. Motivated by positive results seen in this work, future research could undertake further studies that examine the effectiveness of eliciting preferences over partial state visibility through use of the debate as an amplification method.
Defining Arguments is Difficult. We have focused on debate assuming a well-defined argument space. While state features one possible and easy choice, finding clear and expressive arguments presents a challenge, impacting the applicability of debate. In the context of RL, one potential alternative is considering previous trajectories in support of the current decision. In domains involy- ing text generation, evidence could be defined as sentences, paragraphs, or references supporting a claim. A potentially interesting research direction is examine the utility of a debate in a domain of human-ai collaboration, specifically in sequential decision maainng tasks.
Practical Considerations. We would like to emphasize that when deploying our framework in practice, it is important to account for the context in which the system is being employed. The performance -osomannc-jusiffabbiittyy trade-off from our experiments suggests that a special care ought to be given to selecting hyperparameters, in particular, those that weight the importance of the environ- ment rewards and debate-based rewards. In practice, this means that reward designer has to assess the potentially differing objectives encoded in these values as well as the accuracy of the proxy judge model, prior to the training process.
A belief bias represents tendency to judge the strength of arguments based on the plausibility of the conclusion, instead based on how strongly they support the conclusion.
9

---

Published as a conference paper at ICLR 2024
ETHICS STATEMENT
We acknowledge potential misuse of the debate framework for malevolent purposes, such as decep- tion. In the advent of AI systems that surpass human performance in many tasks, novel approaches must be developed which enable defense against malicious agents. While recent results indicate that debate is useful in thwarting malicious use of Al systems, further research paramount in ensuring one can detect and defend against nefarious purposes. Moreover, the reliance on human judgments introduces the possibility of capturing their biases. Subsequently, designed reward function can in- centivize argumentative agents to amplify those biases or learn to leverage them to achieve their objective. This particularly important for practical considerations (Sec. (6). where reward de- signer is tasked with assessing and handling the caaprmaannnc-sttiaabbllle trade-off. It is important to note that our results do not hint at solutions for dealing with these challenges; they only demon- strate that the trade-off exists. It is therefore of great importance that future research investigates novel algorithmic approaches and methods to tackle such challenges.
ACKNOWLEDGMENTS
This research was, part, funded by the Deutsche Forschungsgemminschattt (DFG, German Re- search Foundation) project number 467367360. We thank the anonymous reviewers for their valuable comments and suggestions.
REFERENCES
Richard B Anderson and Beth M Hartzler. Belief bias in the perception of sample size adequacy. Thinking & Reasoning, 20(3):297-314, 2014.
Dzmitry Bahdanau, Felix Hill, Jan Leike, Edward Hughes, Arian Hosseini, Pushmeet Kohli, and Edward Grefenstette. Learning to understand goal specifications by modelling reward. arXiv preprint 10X6v:1806.01944 2018
Osbert Bastani, Yewen Pu, and Armando Solar-Lezama. Verifiable reinforcement learning via policy extraction. Advances in Neural Information Processing Systems, 31, 2018.
Ioana Bica, Daniel Jarrett, Alihan Hüyük, and Mihaela van der Schaar. Learning" what-if" explana- tions for sequential decision- decision-making. arXiv preprint vrXiv:2007.1353: 2020.
Erdem Biyik and Dorsa Sadigh. Batch active preference-based learning of reward functions. In Conference robot learning, pp- 519-528. PMLR, 2018.
Mark Bovens. Analysing and assessing accountability: A conceptual framework 1. European law journal, 13(4):447-465 2007.
Ralph Allan Bradley and Milton Terry. Rank analysis of incomplete block designs: I. The method of paired comparisons. Biometrika, 39(3/4):324 345, 1952.
Miles Brundage, Shahar Avin, Jasmine Wang, Haydn Belfield, Gretchen Krueger, Gillian Hadfield, Heidy Khlaaf, Jingying Yang. Helen Toner, Ruth Fong, et al. Toward trustworthy ai development: mechanisms for supporting verifiable claims. arXiv preprint vrXiv:2004.07213. 2020.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in Neural Information Processing Systems, 30, 2017.
Jonathan Cohen, Jean-Louis Vincent, Neill Adhikari, Flavia Machado, Derek C Angus, Thierry Calandra, Katia Jaton, Stefano Giulieri, Julie Delaloye, Steven Opal, Kevin Tracey, Tom van der Poll, and Eric Pelfrenne Sepsis: a roadmap for future research. Lancet Infect Dis, 15(5):
581-614, April 2015.
R Phillip Dellinger, Jean M Carlet, Henry Masur, Herwig Gerlach, Thierry Calandra, Jonathan Cohen, Juan Gea- Bana-aaaccocce Didier Keh, John Marshall, Margaret M Parker, Graham Ramsay, Janice I. Zimmerman, Jean-Louis Vincent, Mitchell M Levy, and Surviving Sepsis Campaign
10

---

Published as a conference paper at ICLR 2024
Management Guidelines Committee. Surviving sepsis campaign guidelines for management of severe sepsis and septic shock. Crit Care Med, 32(3):858 883, March 2004.
Rati Devidze, Goran Radanovic, Parameswaran Kamalaruban, and Adish Singla. Explicable reward design for reinforcement learning agents. Advances in Neural Information Processing Systems, 34:20118 18-20131, 2021.
Yilun Du, Shuang Li, Antonio Torralba, Joshua Tenenbaum, and Igor Mordatch. Improv- ing factuality and reasoning in language models through multiagent debate. arXiv preprint arXiv:2305. 14325, 2023.
Jonathan St BT Evans and Jodie Curtis-Holmes. Rapid responding increases belief bias: Evidence for the dual-process theory of reasoning. Thinking Reasoning, 11(4):382-89, 2005.
Joannis Faros, Aditya Dave, and Andreas A Malikopoulos. q-learning approach for adherence- aware recommendations. arXiv preprint arXiv:2309.06519 2023.
Julien Grand-Clément and Jean Pauphilet. The best decisions are not the best advice: Making adherence-awee aware recommendations. arXiv preprint arXiv:2209.01874 2022.
Gillian K Hadfield. Explanation and justification: Ai Deecision-making law, and the rights of eitizens. University of Toronto/Schwartz Reisman Institute for Technology and Society, 18, 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification, In Proceedings of the IEEE international conference on computer vision, pp. 1026-1034, 2015.
Marcel Hildebrandt, Jorge Andres Quintero Serna, Yunpu Ma, Martin Ringsquandl, Mitchell Joblin, and Volker Tresp. Reasoning on knowledge graphs with debate dynamics. In Proceedings of the AAAA Conference on Artificial Intelligence, volume 34, pp. 4123 4131,, 2020.
Yong Huang, Rui Cao, and Amir Rahmani. Reinforcement learning for sepsis treatment: A continu- ous action space solution. In Machine Learning for Healthcare Conference, pp. 631-647. PMLR, 2022.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning, pp. 4p..48--0 448-456. pmlr, 2015.
Geoffrey Irving, Paul Christiano, and Dario Amodei. Ai safety via debate. arXiv preprint arXiv:1805.00899, 2018.
Alistair EW Johnson, Tom J Pollard, Lu Shen, Li-wei Lehman, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark. Mimic-iii, freely accessible critical care database. Scientific data, 3(1):1-9, 2016.
Kishor Jothimurugan, Rajeev Alur, and Osbert Bastani. A composable specification language for reinforcement learning tasks. Advances in Neural Information Processing Systems, 32, 2019.
Juozapaitis, A Koul, A Fern, Erwig, and Doshi-Velez. Explainable reinforcement learning via reward decomposition. ijcai. In Proceedings the International Joint Conference Artificial Intelligence. A Workshop on Explainable Artificial Intelligence. 2019.
Matthieu Komorowski, Leo A Celi, Omar Badawi, Anthony C Gordon, and A Aldo Faisal. The arti- ficial intelligence clinician learns optimal treatment strategies for sepsis in intensive care. Nature medicine, 24(11):
2018.
Minae Kwon, Sang Michael Xie, Kalesha Bullard, and Dorsa Sadigh. Reward design with language models. arXiv preprint arXiv:2303.00000 2023.
Jan Leike, David Krueger, Tom Everitt. Miljan Martic, Vishal Maini, and Shane Legg. Scal- able agent alignment via reward modeling: A research direction. arxiv 2018. arXiv preprint arXiv: 181 1.0781,, 2018.
11

---

Published as a conference paper at ICLR 2024
Scott M Lundberg and Su-In Lee. A unified approach interpreting model predictions. Advances in Neural Information Processing Systems, 30, 2017.
Yuzhe Ma, Xuezhou Zhang, Wen Sun, and Jerry Zhu. Policy poisoning batch reinforcement learning and control. Advances in Neural Information Processing Systems, 32, 2019.
Andrew L Maas, Awni Y Hannun, Andrew Y Ng. et al. Rectifier nonlinearities improve neural network acoustic models. In International Conference on Machine Learning, volume 30, pp. 3. Atlanta, GA, 2013.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 5 518(7540):529555 2015.
Nathan Mundhenk, Barry Y Chen, and Gerald Friedland. Efficient saliency maps for explainable ai. arXiv preprint arXiv: 1911.1111...... 2019.
Ethan Perez, Siddharth Karamcheti, Rob Fergus, Jason Weston, Douwe Kiela, and Kyunghyun Cho. Finding generalizable evidence by learning to convince q&aa models. arXiv preprint arXiv:1909.05880 05863, 2019.
Aniruddh Raghu, Matthieu Komorowski, Imran Ahmed, Leo Celi, Peter Szolovits, and Marzyeh Ghassemi. Deep reinforcement learning for sepsis treatment. arXiv preprint arXiv:1711.0960.. 2017.
Ronilo Ragodos, Tong Wang, Qihang Lin, and Xun Zhou. Protox: Explaining a reinforcement learning agent via prototyping. Advances in Neural Information Processing Systems, 35:27239- 27252, 2022.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. "why should i trust you?" explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pp. 1135- 1144, 2016.
Tom Schaul, John Quan, Joannis Antonoglou, and David Silver. Prioritized experience replay. arXiv preprint arXiv:1511.052 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint 1rXiv:1707.06347, 2017.
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruy Batra. Grad-cam: Visual explanations from deep networks via gradient-based local- ization. In Proceedings of the IEEE international conference on computer vision, pp. 618-626, 2017.
Pier Giuseppe Sessa, Ilija Bogunovic, Andreas Krause, and Maryam Kamgarpour. Contextual games: Multi-agent learning with side information. Advances Neural Information Process- ing Systems, 33:21912-21922. 2020.
Yoav Shoham and Kevin Leyton-Brown Multiagent systems: Algorithmic, game-theoretic, and logical foundations. Cambridge University Press, 2008
Mervyn Singer, Clifforr S Deutschman, Christopher Warren Seymour, Manu Shankar-Hari, Djillali Annane, Michael Bauer, Rinaldo Bellomo, Gordon R Bernard, Jean- Maniel Chiche, Craig M Coopersmith, et al. The third international consensus definitions for sepsis and septic shock (sepsis-3). Jama, 315(8):801-88) 8800 2016.
Richard Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q- learning, In Proceedings of the AAAI conference artificial intelligence, volume 30, pp. 2094- -2100, 2016.
J. von Neumann and O. Morgenstemn Theory of Games and Economic Behavior. Princeton Univer- sity Press, 1947.
12

---

Published as a conference paper at ICLR 2024
Ziyu Wang, Tom Schaul, Matteo Hessel, Hado Hasselt, Mare Lanctot, and Nando Freitas. Dueling network architectures for deep reinforcement learning. In International Conference on Machine Learning. pp. 1995-2003, PMLR, 2016.
A LIST OF APPENDICES
In this section, we provide a brief description of the content provided in the appendices of the paper.
Appendix
B
provides details about the patient dataset and the defined environment:
0
B.
provides general background on sepsis;
0
B
B.2
provides details about the patient cohort;
B
provides details about the environment's action space;
provides details about the environment's reward structure.
Appendix provides details about the models:
provides details about the judge model;
provides details about the argumentative agents and their confusers;
provides details about justifiable and baseline agents.
Appendix
provides additional results:
D.I
provides results involving debates with L 4 evidence;
0
D.2
provides further analysis of justifiable agent's s actions;
0
D.3
provides additional results pertaining to the isolated agent;
0
D.4
provides results for agents using an alternative definition of the utility function;
D.5
provides analysis of alternative methods for the preference dataset definition.
B DATASET
B.I SEPSIS
Sepsis is a life-threatening condition, defined as severe infection leading to an acute organ dysfunc- tion, which in turn can cause cascade of changes that damage multiple organ systems (Singer
et al. (016)) It is also one of the leading causes of patient mortality (Cohen et al. 2015). Apart from administration of antibiotics and control of the infection source, critical challenge man- agement of sepsis lies in the administration of intravenous fluids (IV) and vasopressors (VC). While international efforts attempt provide general guidance (Dellinger et al.) (004), clinicians are
nevertheless tasked in devising individualized treatments based on specificities of patients.
first step towards automated management of septic patients was done seminal work of Ko- morowski et al.) (2018). The problem of treating sepsis was tackled by applying reinforcement learning to devise optimal treatment strategies for prescribing doses of intravenous fluids and vaso- pressors. The authors discretized the possible doses, which resulted in action-space consisting of 25 distinct choices. The patient data was obtained from the MIMIC-III dataset (Johnson et al. 2016)) and the authors extracted a subset of 48 patient features, discretized into 4h time windows. The pre- liminary results indicated that a learned policy was both quantitatively and qualitatively desirable. Since then, several other works extend and improve upon this line of research. For example, Raghu et al. (201)) proposed a continuous state-space approach based on deep reinforcement learn- ing, which we build in this work. Likewise, (uang al. (2022) proposed approach that is based on continuous action-space, thus enabling more granular control of prescribed doses.
B.2 PATIENT COHORT
To assemble our patient cohort, we utilize the MIMIC-II v1.4 database (Johnson et al., 2016), focusing our analysis on patients that fulfill the Sepsis-3 criteria (Singer et al., (016). Similar as in Komorowski et al. (2018). the sepsis is defined as a suspected existence of infection (indicated by the prescribed antibiotics) paired with mild evidence of organ dysfunction (indicated by the
13

---

Published as a conference paper at ICLR 2024
Table 1: List of features comprising the patient vector. We select 40 time-varying continuous fea- tures and 4 demographic discrete features.
Continuous features
SOFA
Calcium
Shock index
Urine output 4h
Urine output total
Cumulated balance
Glasgow coma scale
Heart rate
Systolic blood pressure
Mean blood pressure
Diastolic blood pressure
Total input fluids
Respiratory rate
Temperature
Fi02 fraction of inspired oxygen
Potassium
Sodium
Chloride
Glucose
Magnesium
SIRS
Hemoglobin
White blood cells count
Platelets count
Partial Thromboplastin Time
PH - Acidity
PaO2
PaCO2
Base excess
Bicarbonate
Lactate
PaO2/FiO2 Ratio
Oxygen saturation
BUN
Creatinine
SGOT
SGPT
Total bilirubin
International normalized ratio
Prothrombin time
Discrete features
Age
Gender
Weight
Mechanical ventilation
SOFA score 2 2). To extract, preprocess and impute the data we utilize the pipeline described in Komorowski et al. (2018), process which resulted in 18,585 different patients that comprised our cohort. The cohort is split into three chunks of sizes 70%, 15%, 15% representing the train, validation, and test splits. The observed mortality of the entire cohort was slightly above 6%, and the splits were selected to approximately preserve this ratio. The course of patient treatment represented as trajectory consisting of state-action pairs, terminating upon patient discharge or death. The average trajectory length was 13, with 2 being the smallest and 20 being the largest. The state is a 44 dimensional vector. comprised of 40 time- varying continuous and 4 demographic discrete features, shown in Table [ Patients' data were discretized using 4h time steps, where variables with multiple measurements within this window averaged (e.g., heart rate) summed (e.g., urine output) as appropriate.
B.3 ACTION SPACE
Argumentative Policies. The evidence (action) space of the argumentative policies is defined by the total number of state features, in our case, as listed in Table To prevent the agent from repeating already proposed arguments, we additionally employ action masking, setting the log probability of already presented arguments to negative infinity.
Baseline and Justifiable Policy. To devise an action space of policies treating sepsis, we follow previous work (Komorowski et al. 2018 Raghu et al. 2017) and focus on managing the total volume of intravenous fluids (IV) and maximum dose of vasopressors (VC) administered to the patient over a 4h discretization window. The dose of each treatment represented as one of four possible non- null choices derived from observed doses divided into four quartiles. We additionally define another choice, designated as an option 'no drug given'. The combination these produced 25 possible discrete actions, 5 per each treatment, comprising the action space of the policy.
14

---

Published as a conference paper at ICLR 2024
B.4 REWARDS
Sepsis. To define intermediate and terminal rewards for treating septic patients, following the work of Komorowski et al. (2018), defined the primary outcome of the treatment via 90-day patient mortality. Therefore, the agent's objective is to optimize for patient survival. To this end, we issue terminal enviromment rewards of +55 for every patient discharge and death, respectively. To stabi- lize the training of deep-rl policy, we also issue intermediate rewards that are clinically guided, as in Raghu et al. (2017). These rewards are comprised of fairly reliable indicators of the patient's overall health, namely the SOFA score (measure of organ failure) and patient's 's lactate levels (mea- sure of cell-hypoxia, which is usually higher septic patients). The rewards for intermediate time steps are then shaped as follows:
SOFA 0) C2(5tTT st. SOFA)) + Cytanh Lactate st sLactale) where Co. C1 and C2 are tunable parameters which we set to Cu 00.05,, C₁ -..2 and C2 = -2 following previous work of Raghu et al. (2017). Rewards defined in this way penalize both, the high SOFA scores and lactate values, as well as increases in these quantities.
r(se St+1) Cs st SOFA s
C MODELS
C.I JUDGE
We defined a judge as a scalar reward function Ta(a, {e]) E R that quantifies the level of support decision has by the set of evidences {e}. The judge parametrized by weights neural network with two hidden layers of size 256, using parametric relu (He et al. 2015) activation and batch normalization (Ioffe Szegedy, 2015). The network takes as input values of proposed evidence {e), binary mask wherein all elements corresponding to the evidence {e} are assigned value of one, while the remaining elements are set to zero, as well as a one-hot encoded action. The addition of a binary mask empirically led to a more stable learning. During training, we augment tuple (8t, do, d1, p) ~ with an evidence set of state features sampled uniforn-random from state St i.c., {e} U, anew for each training batch. To learn the parameter 0, we minimize the cross-entropy loss between preference preference-rrddiciooss and labels from the dataset:
Rdd of
min Es,, (0. [p- log P(a1 > ao. {e}) + (1 -1-pp DER
p) log P(ao > a1, {e})]
The learning done for a total of 100 epochs using a batch size of 64, Adam optimizer and learning rate of 5e-4.
C.2 ARGUMENTATION
All argumentative models utilize the same network architecture comprised of 2 hidden layers of size 512 with a leaky-relu activation function with slope of 1e-2. The network input consists of 44 dimensional patient state vector, the decision for which the agent arguing, as well as a binary mask of arguments proposed thus far. The action space of the agent is represented by all 44 state features (see App. B.3 for more details). For training, we use PPO (Schulman et al. 2017), running the procedure for 1M steps using the Adam optimizer with learning rate 5e-4 and a batch size of 128. The discount factor was empirically tuned and set to 0.9. The full list of hyperparameters and their considered tuning ranges is given in Table During training. the agent's policy stochastie: the evidence is sampled from categorical distribution defined by its logits. To obtain a deterministic policy used in evaluations, we perform an argmax operator over obtained logits in a particular state.
C.2.1 ISOLATED AGENT
When investigating robustness of the multi-agent debate, we examine two different setups involving an isolated agent baseline: precommit (reported the main paper, Pbot|4b) and adaptive (reported in the App. D33 Plot (b)) The former represents easier case for the agent, but lacks the debate structure. The latter uses a full debate setup as described in Sec. 3.2 but evaluates the isolated agent in setup slightly different from one it was trained in.
Precommit. In this case, an isolated agent is trained to propose evidence {e} of size L/2 that for a given ap maximizes I' (ap: (e)), where J' is new judge trained to evaluate T./2 evidence. When
15

---

Published as a conference paper at ICLR 2024
Table 2: Hyperparameters used for argumentative agents. utilize the same parameters.
Unless otherwise indicated, all agents
Common parameters
Parameter name
Parameter value
Tuning range
Hidden dim
512
[256, 512]
Learning rate
5e-4
Deguniform[leS:]]]
Entropy coefficient
le-2
Clip range
0.1
[0.1, 0.2, 0.3, 0.4]
Discount
0.9
[0.8, 0.9, 0.95, 0.99]
GAE lambda
0.7
[0.7, 0.8, 0.9, 0.92, 0.95, 0.98, 0.99, 1.0)
VF weight
0.5
[0.3, 0.5, 0.65, 0.75]
Max grad norm
0.1
[0.3, 0.5, 0.6, 0.7, 0.8, 0.9, 1, 2, 5]
Normalize rewards
[true, false] [true, false]
Ortho init
true
Confuser parameters
Hidden dim
256
[256, 512]
Entropy coefficient
3e-4
Oggmiform[0000000000:0000
Clip range
0.4
[0.1, 0.2, 0.3, 0.4]
VF weight
0.65
[0.3, 0.5, 0.65, 0.75]
Max grad norm
[0.3, 0.5, 0.6, 0.7, 0.8, 0.9, 1, 2,
evaluating robustness (Plot 4h), the agent first proposes 1./2 evidence, followed by a confuser agent which proposes the remainder.
Adaptive. In this case, an isolated agent is trained to propose all evidence {e} that maximize J Tap, {e}), for a given ap. When evaluating robustness (Plot (6lot)6b), the isolated agent and its associ- ated confuser take turns and propose a total L/2 evidence each.
C.2.2 DEBATE AGENTS
For multi-agent scemarios, both maxmin and self-play agents use the architecture described in the beginning of Sec. C.2 but each modify the underlying optimization pipeline. To train self-play debate agent, we let the agent argue with a (frozen) copy of itself, updating it every 100k steps. This procedure is then repeated for fixed number of 500 generations. To train the maxmin debate agent, we parametrize the agent's ' opponent with a different set of weights Φ2 E Rd2, The procedure starts by training the main agent for 4k steps, followed by training of its opponent for 100k steps. Like in the previous case, the procedure is repeated for 500 generations. This approach is based on the bi-level optimization and allows for overfitting the opponent to the current version of the main agent, which ensures learning of defense strategies against very strong adversary.
C.2.3 CONFUSER AGENTS
The evaluation of robustness presented in Section 5.4 required learning three separate adver- saries (one for each argumentative agent), explicitly tasked in providing counterarguments that will confuse the judge. The architecture of these confuser agents is mostly the same that of the argu- mentative agents, shown in Table For a sample (s,, an, a1 p) ~ D. the confuser agent is rewarded positively, whenever the judge is convinced of the alternative (non-preferred) action.
16

---

Published as a conference paper at ICLR 2024
Table 3: Hyperparameters used for baseline and justifiable policies. Unless otherwise indicated. all policies use the same parameters. If a parameter does not specify tuning range, its value has either been selected based on previous work (e.g., buffer a and (3) or tuning was not necessary (e.g.. debate coefficient X).
Baseline policy
Parameter name
Parameter value
Tuning Range
Hidden dim
128
[64, 128, 256]
Learning rate
le-4
@ggnnform[le-S:1e..
Batch size
256
[128, 256, 512, 1024)
Polyak update
1e-3
[lc-1, 1e-2, 1e-3, le-4]
ReLu slope
le-2
[le-1, le-2, le-3]
Discount
0.99
Num. estimation step
6
[1,3, 6, 10, 15]
Terminal reward
15
Debate scaling coefficient
=
Debate coefficient
0.0
Buffer a
0.6
Buffer /
0.9
À E [0.25, 0.5, 0.75]
Num. estimation step
1.0
Num. estimation step
C.3 BASELINE AND JUSTIFIABLE AGENTS
Our approach for learning treatment policies for septic patients is based on continuous space models and builds on Raghu et al. (2017). The policy is based on a variant of Deep-Q networks (Mnih al. 2013), specifically double-deep (Van Hasselt al. 2016) Q-network, also employing the dueling architecture (Wang et al., 2016), where the estimated action-value function for pair (s. a) is split into separate value and advantage streams. The final network consists of 2 fully- connected layers of size 128 using leaky-relu activation functions with slope 1e-2. The learning done batches of 256 (s, a, r, s') tuples sampled from Prioritized experience replay buffer (Schaul al. 2015) with learning rate of le-4. Instead of periodically updating the target net- work, we leverage Polyak averaging with an update coefficient 5 set to 1e-3. The full list of used hyperparameters is given in Table Similar Raghu al. (2017), we augment the standard Q- network loss. First, we added a regularization term that penalizes Q-values outside the allowed threshold Qthresh +00. In addition, clip the target network outputs +22, which empirically proved to stabilize the learning. The final loss function we used is given by:
( =E (Q (Qdouble-tagge ((s, a; 0)) + B max (IQ(s, a; θ) Uhreshh- 0)
Qdouble-tageet =r + 7 Q(s', arg max, Q al: d):
Where 3 is user-specified coefficient we set to B 5.0 in all our experiments following Raghu let al. (2017). The learning done for a total of 25k iterations, evaluating the policy every 50 iterations using weighted importance sampling (WIS) (Sutton & Barto 2018) held-out test set.
Behavioral Policy. The calculation of the importance sampling ratio requires access to a so-called behavioral policy that generated offline samples. To obtain it, we train a behavior-lloning (BC) clinician policy to take in a state from the patient cohort and predict the action aj taken by the human clinician, minimizing the cross-entropy loss over the training dataset. The network consists of two colly-connected layers of size 64. We run the training with the Adam optimizer, using a learning rate of 1e-3 and weight decay set to 1e-1 for total of 100 epochs with batch size of 64.
17

---

Published as a conference paper at ICLR 2024
D ADDITIONAL RESULTS
D.1 SHORTER DEBATES
In the main text, we have seen positive results involving debate using 6 arguments. which amounts to ~ 13% of the entire state. In this section, we want to further examine the effectiveness of debate when limiting the number of evidence L = 4, amounting to 9% of the entire state.
The first question that arises when further limiting the amount of state visibility during debate is the impact it has on the quality of learned task policies. In Plots Sa we evaluate the performance of different justifiable policies on a held-out test set during the course of training using weighted importance sampling, for the case of 4 and 6 evidence limit. While the reduced number of evidence exposes the judge to only 9% of the entire state, parable to the case where = 6. Furthermore, in Plot 5i we also confirm that trained policies are significantly more preferred the baseline policy, a trend equivalent to the one saw in the main text. Apart from these quantitative evaluations, in Plots 5g and 5h we show the qualitative analysis of patient mortality from Sec. [.22] We confirm that the lowest mortality is observed when clinician prescribed doses recommended by justifiable policies, thus further signaling potential validity of policies trained with rewards stemming from debates with L. 4 evidence.
see that the achieved performance is com-
To evaluate the effectiveness of argumentative agents, we repeat the evaluation from Sec. 5.4 When exposed to 4 randomly selected evidence, the judge trained via the procedure outlined in App. C.1 achieves accuracy of ~59%. Plo(Se]shows the judge's accuracy when different argumentative agents propose L = 4 evidence [2] averaged over 1000 samples (st, 01 02 p) ~ DD Without the confuser, all three agents achieve performance similar to the case of L = 6 evidence, boosting the judge's s accuracy to almost 90%. In the setup involving an adversary, agents propose total of evidence each before the judge evaluates the outcome. The observed trend is also similar to the one from the main text (Sec. 5.4
D.2 PREFERENCE BREAKDOWN
To gain better understanding about the behavior of learned justifiable policies, further examine actions they propose. In particular, policy was preferred to that of the baseline policy (JP), percent of times when the action of the baseline policy was preferred to the action of the justifiable policy (BP), and percent of time when the two were equally preferred (EP) We see that the number of actions proposed by the justifiable agent and the baseline agent that are equally preferred decreases as the parameter À is increased: for 0.25 two agents chose the equally justifiable action about 49% of the times, for = 0.50 this number drops to 40,, whereas for À = 0.75 and À = 1.0 this number is 36% and 35% respectively. Out the remaining actions, the ones from justifiable policies were increasingly more preferred, as the parameter À was increased.
shows the percent of times an action from the justifiable
D.3 ISOLATED AGENT SETUP
In App. C.2.1 we described two setups which we consider when evaluating robustness of the isolated agent, namely precommit and adaptive. In Plot [bb] we show the accuracy of the judge in predicting the preferred action when evidence was proposed by an isolated agent trained with one of these approaches for 1000 different debate games. Without the confuser, both approaches amplify the capabilities of the judge, performing roughly equivalent. When faced with confuser agent, the precommit approach performs better, since the adaptive case, the agent is evaluated in a debate-like setup, which was not accounted for during training. In both cases, however, we observe that the isolated agent not robust to an adversary.
D.4 ALTERNATIVE UTILITY FUNCTIONS
In Sec. 3.2 have defined the utility function U to output binary values (-1,0,+1) +1} based on the justiñibility rewards obtained from the judge. One might ask if there are alternative ways of
Like in the main text, we use the precommit setup for the isolated agent (see App. C.2.1 for more details).
In our experiments, the judge deemed two actions equally justifiable whenever they were the same.
18

---

Published as a conference paper at ICLR 2024
0.90
0.90
0.90
6.50
Potormance
beffrrmrcce
Performance
0.70
0.70
0.70
6.70
0.60
0.50
evicence
0.50
denco
0.50
cx dence evidence cirician
6.50
evidence exidence
0.40
exidence
0.40
chriccinn
dence
clmittn
303
250
120
0.30
200
402
550
150
200
320
400
520
100
300
450
lterations
lterations
Per eerations
lterations
(a) À 0.25
(b) À = 0.5
(c) 0.75
(d) À = 1.0
1.00
Preference Over Besellie
1.00
0.50
2-0.0
0.50
Ae0.0
2=0
Aw0
Judge Accuracy
0.00
0.90
2-0.50
0.40
Mortaliy
A-
A-100
Mortality
0.30
Amb
Aw1.0
confuser
9.00
0.00
mAATTI play
0.50
00300
A-10
0.00
0.00
-0.00
0.50
Argumentative Agent
Policy Variant
(e) Convincing success
(f) Preferred to baseline
(g) IV dose excess
(h) VC dose excess
Figure 5: Quantitative and qualitative evaluation of policies trained using debate-basee rewards limited to L evidence. For (f)- the confidence intervals represent +2 standard errors of the mean over random seeds. (a)-(d) Performance of policies trained with L and L = 6 evidence, a held-out test set with +1 terminal rewards for every patient
as measured by WIS evaluation discharge or death. The mean and standard deviation are reported over 5 random seeds. (e) Accuracy of the judge in predicting the preferred action using 4 proposed evidence, with and without the confuser agent. The CI represent +2 standard errors of the mean estimate. (f) Percent of times judge preferred decisions of justifiable policies (i.e., À > 0.0) compared to those of the baseline policy (i.e., A = 0.0). (g) (h) Observed patient mortality (y-axis) against variations in IV/VC treatment doses prescribed by clinicians compared to the recommendations learned policies (x-axis).
1.00
Total Actons
0.50
3.60
Judge Accuracy.
Jadge Accuracy.
0.90
0.20
Judge
0.40
0.20
MIDiCU ConRUSor
0.20
contuser
5.00
0.00
0.00
0.00
Preference
Type
Argumentative Agent
Dataset
Generation Method
(a) Preference breakdown
(b) Isolated agent setup
(c) Convincing success
(d) Preference datasets
Figure 6: (a) Percent times when actions the justifiable policy were more preferred to those of the baseline policy (JP), less preferred (BP) and equally preferred (EP). (b) Accuracy of the judge in predicting the preferred action, with and without the confuser agent when evidence was proposed by two isolated agents trained in precommit and adaptive setup. The CI represents #2 standard emros of the mean estimate. (c) Accuracy of the judge in predicting the preferred action, when evidence was proposed by maxmin (MM) and self-play (SP) agents trained using the utility function proposed in Sec. 3.2 compared to agents trained using the utility function defined as a difference in predicted rewards proposed App D.4 MM-diff and SP-diff respectively. The CI represents +2 standard errors of the mean estimate. (d) Accuracy of the judge in predicting the preferred action with evidence sampled uniform-randon trained on datasets generated with different methods.
19

---

Published as a conference paper at ICLR 2024
defining U that preserve the zero-sum structure and potentially positively influence the learning. In this section, we consider one particular alternative that defines the utility function based on the difference between predicted rewards, which might provide more informative learning signal. In particular, we set uy (nz) = (22(n)) = U(a,, af (en, }), with U defined as U(a, af {et) = [(a, {e]) J[a {e]). We rerun the experiments from Sec. [5.4 for maxmin and self-play agents and show preference recovery success and robustness results in Plot [6c] We can see that the two approaches perform similarly in situations which do not involve confuser agent. However, seems that definig the utility using the difference in judge's 's rewards leads to slightly lower scores when debate agents are faced with an adversary.
D.5 ALTERNATIVE PREFERENCE DATASET
In Sec. 5.1 construct preference dataset by matching every pair (st, 0z) from the cohort with an alternative action a, ~ (A), a, * a sampled uniforn-random from the action space. In this section, we examine additional strategies one could take when constructing such synthetic dataset. In particular, we consider the random strategy we just described, paired with two alternative ones, namely exhaustive and offset. The exhaustive strategy pairs a, with all possible alternative actions, 24 in total. The offset strategy pairs ar with an alternative action that is in its neighborhood. To define a neighborhood, we recall that there are total of 5 choices for both, vasopressors (VC) and intravenous fluids (VC). Therefore, we can write a = 5 * IV + VC, where IV, VC E {0, 1, 3, 4). To obtain an alternative action, we consider changing IV and VC by an offset sampled uniform- random from set 1, 0, 1). for both IV and VC. We then train new judge for each of the datasets and show its accuracy in Plot (dd] The exhaustive variant represents the most informative, but also unrealistically large, dataset. Thh random variant represents somewhat of "middle ground" in terms of dataset difficulty. Lastly, the offset variant represents the most difficult case, as differences between two actions are more nuanced. However, while the achieved accuracies reflect the difficulty of the corresponding dataset, the capabilities of a trained judge model are roughly the same.
20


Reviews:
[{"review_id": "eMbgbc8kHn", "reviewer": "Reviewer_WJVW", "summary": "This paper considers the problem of designing RL agents that can provide evidence supporting their chosen actions ('justification') to a human observer - an important and timely problem that has not received enough attention. The authors propose an approach where a zero-sum debate extensive form game is played out between two agents, each attempting to convince a human judge (or proxy model) that their chosen action is most justified, by selecting 'evidence' (a subset of the state-space) to support the chosen action, contingent on the current state. The result of the human judge proxy model is a reward term which is mixed with the environmental reward, leading to an RL policy that trades off environmental reward and justifiability, as encoded by a dataset of human pairwise rankings of evidence proposals.\n\nThe paper is well written, with some minor grammatical errors. The proposed method is interesting and novel to my knowledge, and seems like a promising approach toward explainable RL policies. The empirical experiments are deep, but consider only one dataset - a medical problem of sepsis treatment. I have left some extensive comments and questions - but this is not a reflection of a weak paper, instead, this is due to my interest in the proposed method. The authors appear to be breaking new theoretical and empirical ground, and to this end the paper seems like a solid first effort in a promising direction.\n\nI'm familiar with single-agent RL (specifically, my background is Inverse Reinforcement Learning), but might be unaware of some prior literature that would be relevant when performing my review. I read the main paper and appendices carefully. I have not carefully checked any theoretical derivations.", "questions": "# Comments and questions\n\n * The use of an evidence-proposal debate setup for explainability is interesting and seems really promising to me. It seems like some deeper connections could be drawn to prior literature on the philosophical merits of debate as a form of explainability though. For instance, I'm curious about the following questions;\n   * Why did you choose to have a multi-step debate framework, rather than a single step of evidence proposal? What is the interpretation or significance of the trajectory through the debate game tree (is there any?), or is the reached leaf node the only relevant outcome of the debate?\n   * Similarly, how is a human RL designer or domain expert to interpret the proposal of multiple pieces of 'evidence' at each step of the debate game. If I understand correctly, evidence data are individual entries from the state-space vector (I think?). Is the interpretation of multiple pieces of evidence supposed to be a Boolean AND combination of each entry? (e.g. state feature A is present, AND B is present, therefore my action is better than the baseline policy)? I think a rich area for future exploration would be more structured evidence formats - e.g. \"A is present, and C is present, but B is not present, therefore action X is chosen\". Qualitative evaluations with domain experts would be beneficial in exploring this too.\n * There seem to be similarities between the debate-judge model and actor-critic RL approaches. Do you see any connections here? Could the proposed method be connected theoretically with A-C RL methods?\n * Similarly, as you note, the debate framework seems to have connections with pairwise preference ranking (e.g. Christiano et al. 2017). Given the popularity of RLHF at the moment, it might be illustrative to dig into the connections between this method and RLHF some more.\n * Fig 1. It appears that the baseline policy $\\pi_B$ never acts in the environment - is this correct?\n * The paper refers to the debate-based reward approach as 'reward shaping' several times - please be careful with this term. Reward shaping refers technically to a specific subset of reward transformation that value functions are invariant to (see [A]). Although people often do abuse this term to refer to arbitrary designer interventions to alter environmental reward, it is best to be more careful with the terminology here to avoid confusion.\n * I note the judge reward $J \\in \\mathbb{R}$ is unbounded - this concerns me as a infinite range within which to rank items poses difficulties for a human judge (see psychology and human factors literature for example). Does your method still work with bounds on the judge reward? E.g. $J \\in [-1, 1]$?\n * Can $\\pi$ and $\\pi_B$ output the same action and/or evidence? What happens in this situation? Is the judge proxy model able to output a 'no preference for either' choice?\n * Sec 4.4 mentions self-play is used during training. However I thought the agents being trained are deterministic. How can self-play be used in this case?\n * A compelling goal for future work would be the design of sample-efficient methods for specific MDPs that can operate with actual human-in-the loop judgements, rather than a proxy model.\n * The Sepsis reward design (App. B.3 is important - please provide references for the choice of $C_0, C_1, C_2$.\n * Given the specificity of your empirical experimental case-study, it would be helpful to have some more elaboration of the nature of the problem of sepsis control in the Appendix. I also assume the authors have some prior work or expertise or domain knowledge in this area. It would be beneficial to state if this is the case in the appendix to demonstrate that the research is being undertaken by an appropriately credentialed and/or experienced team (i.e. not computer scientists pretending to be doctors).\n\n# Minor comments and grammatical points\n\n * Sec 3. - The MDP specification is missing a starting state distribution.\n * Sec 3.2 Why does $a_t$ act first in the debate game, not $a^B_t$. Did you test the reverse design?\n * You set the hyper-parameter $\\alpha = 5$ in your experiments - why this value? Were other values tried?\n * Fig 3. caption - please define WIS so the reader doesn't have to refer to the main text.\n * App. B.2 - I believe you mean to say \"setting the [log] probability of presented arguments to negative infinity\".\n\n# References\n\n * [A] Ng, Andrew Y., Daishi Harada, and Stuart Russell. \"Policy invariance under reward transformations: Theory and application to reward shaping.\" Icml. Vol. 99. 1999.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 4, "strengths": "* A very important problem\n * A novel approach (debate-based justifications), and a solution method that seems promising\n * Clearly written, and the paper seems to represent high quality work\n * A compelling real-world empirical experimental case study\n * The comparison to SHAP as an alternative is great to see - and the proposed method fares well", "weaknesses": "* The method appears to require explicit knowledge of the optimal deterministic policy (called the baseline policy $\\pi_B$ as part of the debate framework (unless I have misunderstood something). This seems like a big limitation in terms of practically applying this method to larger-scale problems.\n * Sec. 4.2 mentions that the judge proxy model follows a 'Bradley-Terry' model (and provides a citation). This seems like an important point in the design of your method, but I'm not familiar with this model or the background or history here. Can you elaborate on why this model is chosen?\n * The training of the justifiable agent (and the argumentative agent) appears to use offline i.i.d. sampling of (s, a, a) tuples (Sec 4.3). This seems like it could pose over/under-representation problems for some MDPs with deep trajectories that might be less or more common (i.e. problems where the i.i.d. assumption is more heavily violated by the dynamics). Can you comment about this limitation of the method?\n * Due to the additional structure of the debate game, the method introduces a number of additional hyper-parameters (e.g. evidence size $L$, mixing coefficient $\\lambda$, scaling coefficient $\\alpha$, as well as architecture of the various policy models, to say nothing of the learning algorithm hyper-parameters. Because of this, evaluation on one or more additional MDP problems would strengthen the evidence for the efficacy of this method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "n6tlubzICJ", "reviewer": "Reviewer_8w5y", "summary": "This work describes a framework for producing RL agents which take \"justifiable\" actions. This is done by defining a \"debate game\" for a state and a pair of actions, in which two adversarial debaters (each representing one of the actions) take turns picking \"evidence\" (aka features or factors) from the state, each trying to maximize the chance that a synthetic \"judge\" which takes in all the evidence as input will rate their action higher. A justifiable action is thus an action for which the outcome of the action's debate game does indeed result in the judge rating the action highly. \n\nThe paper describes experiments performed in the RL environment of providing interventions for patients with Sepsis. The work compares a baseline (a standard RL agent trained to maximize reward in the environment) with the novel approach (an RL agent trained to maximize a combination of reward from the environment and reward from the judge's judgment of the result of the debate game for the chosen action vs. the baseline action). The optimal policies for the debate game are trained using self-play RL (the debate game is perfect-information). There is a dataset containing real clinicians' interventions in the Sepsis setting. The judge is trained to predict, for each state in the dataset, the clinician's actual action based on evidence which is sampled uniformly random from the available evidence.\n\nThe work is highly related to explainability. Its methodology is similar to research in learning pairwise preferences.\n\nMany experiments are performed, and there is ample discussion about motivations and future research.\n\n---\n\nPost-rebuttal period update: I originally rated this paper a 6. The authors addressed some comments I made, and performed additional experiments. If I could, I would raise my rating to something between 6, and 8 (but unfortunately such an integer does not exist). I still don't like that the experimental design doesn't exactly match the motivation, but the updated version more explicitly acknowledge this, and the paper is still an interesting proof-of-concept. Finally, with the addition of the new experiments, the paper contains so many experiments to investigate questions that one could want to know the answer to, that its value passes the bar for publication. Therefore, I decided to break the tie between 6 and 8 by rating the paper an 8.", "questions": "- In the abstract: the reward from the debate-based reward model yields \"effective policies highly favored by the judge when compared to [the baseline]\" -- the obvious question in the reader's mind is whether the \"effective policies\" actually yield higher environment reward than the baseline or not, and it would be nice to not have that ambiguity. If I'm reading the results correctly, they yielded lower performance, so maybe something like \"while being nearly as effective as [the baseline]\" or \"while barely sacrificing any performance\" would be nice.\n- Shouldn't the description of the Baseline Agent (Section 3.1) indicate that the baseline agent is learned and therefore in practice the baseline agent is approximately optimal, but not necessarily exactly optimal?\n- In the definition of the Justifiable Agent (Section 3.1), I think it would be helpful to explain what $a^B_t$ is. Even though it's described in the next section, it would clarify the dependence of $r^d$ and thus $R_J$ on the baseline agent's policy right away.\n- Debate Game (Section 3.2): I'm just curious: Did you consider defining U to be the difference between the judge's judgments instead of the binary +1, -1? If you did, why did you choose not to use it?\n- Using uniform random evidence to train the judge is surprising at first. Do you think it works well? Are there any alternatives? (can you keep training the judge in a loop while the argumentative agents train, perhaps?) An ideal dataset would actually include pairs of action + evidence, and human rankings over those, right?\n- Creating the synthetic dataset by comparing a random action with the clinician's action seems like it would result in a lot of pairs in the dataset where one action is clearly superior to the other. Is this true? Does this cause any issues?\n- In Section 5.4: \"not only can agents significantly amplify the capabilities of a judge, but they also manage to recognize and correctly convince it of its true underlying preference\" -- what are those two clauses referring to? I can't see how they're different.\n- Section 5.4: Am I reading correctly that the isolated argumentative agent is trained with L = 6, yet in this setting it's only allowed to submit 3 evidences? How does that work (does the isolated one just pick 3 without regards to the choice of confuser, while the confuser picks 1 evidence after each isolated choice)? Wouldn't it be more correct to train it with L = 3?", "limitations": "", "rating": 8, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "This is my first time hearing about debate agents, so my to eyes the very idea of using debate to train aligned agents is novel. However, this may not have been true if I had previously read one of the Debate papers in the Related Work section. Still, the research direction seems exciting, and this paper is a solid proof-of-concept exploring the specific idea of using a judge's evaluation of the result of a debate game as RL reward.\n\nThere were many experiments performed, and they answer much of the questions one would like to see answered about this work. \n\nThe writing is good. I especially enjoyed reading the Introduction and Formal Setup.", "weaknesses": "- The most glaring flaw is that the paper gives its setup as being motivated by requiring agents' actions to be justifiable to humans, but the actual methodology is a few steps removed from humans: both in the training (the synthetic judge is *not* trained on a dataset of humans ranking actions by their justifiability, but instead on a dataset of ground-truth clinician actions) and in evaluation (the evaluations measure the synthetic judge's judgments, instead of human judgments). I feel that the paper would be stronger with at least human evaluations, if not human preference data for training.\n  - Also, the abstract says that \"we showcase that agents trained via multi-agent debate learn to propose evidence that is resilient to refutations and *closely aligns with human preferences*.\" Which human preferences is this referring to? Does it just mean that the actions taken are similar to the actual clinicians' actions? Does this presuppose that clinicians' actions == justifiable actions? If so, is this stated anywhere?\n\n\n\nMinor:\n\n- The definition of the Preference Dataset (Section 4.1) seems strange: the preference indicates which of the two decisions is more justified by a given set of evidence in a particular state. But what is that set of evidence? Shouldn't that evidence be included in the tuple? Indeed, in the experiments, the preference dataset used does not fulfill this definition, since $p$ simply indicates the actions that the real clinicians took, not the action that they thought was more justifiable given the (sampled at random) evidence.\n\n\nNitpicks:\n- Perhaps footnote 3 is important enough to not be a footnote. I was searching for a bit to figure out where $a^B_t$ came from in the experiments.\n\n---\n\nI think this paper isn't perfect but it's a good proof-of-concept of an interesting idea, so I recommend it for (borderline) acceptance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ULZixpayvR", "reviewer": "Reviewer_2TSd", "summary": "General Approach:\nThis paper introduces a method of learning policies for sequential decisions that are justifiable to a (human) judge. The general technique is to use reward shaping, where the reward of an RL agent is determined jointly by the environment and by the judge, and parameterized by a coefficient that determines the balance between performance (determined by environment) and justifiability (determined by the judge). The novel aspect of the approach is the construction of the reward from the judge using a debate game. The reasons given for using a zero-sum debate game in extensive form is to ensure that (1) the evidence presented by the agent is sufficient yet concise, and (2) the argument (collection of evidence) should not be easy to refute by providing additional evidence to the judge.\n\nDebate Game:\nThe debate game is a zero-sum, perfect-information game between a baseline agent and an agent trying to learn more justifiable decisions, where the strategy space is composed of the evidence that each agent can introduce at each stage in the game, and the utilities/payoffs are determined by the judgement of the judge. Importantly, the number of steps in the game, and hence the size of the ultimate evidence set, is much smaller than the set of possible elements that can be added as evidence. The judge provides real-valued judgements J(action, evidence set). Two two agents each suggest an action, and then take turns introducing elements to evidence to justify their action, so they end up each introducing half of the evidence in the evidence set. The judge then judges each action with respect to the evidence set, and the agent whose action is preferred by the judge based on the evidence gets a utility of +1 (and the other -1). The reward from the judge is then proportional to the value of the game (or, rather, an approximation of it).\n\nAn empirical study is done on data for patients with Sepsis, including comparison to trainign with only environment-based and only debate-based rewards and agents who learn without an adversary.", "questions": "Isn’t the value of the game always in {1, -1}?\n\nWhy does J need to be real-valued? Isn’t it enough just to be a weak order over action-evidence pairs? The actual value of J seems irrelevant to the model.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Overall, the paper is well-organized, the problem is well-motivated, and some of the empirical results are highly compelling (i.e. Figs 2c and 2d). This approach provides one way of capturing the tradeoff between learning an optimally performing policy and one that is justifiable to a human agent, creating the ability to interpolate in between. The use of a debate game is a novel contribution the should, in theory, help train the RL agent to produce decisions whose justifications achieve robustness in the face of new information.", "weaknesses": "There are some weaknesses in the methodology and presentation of the work, but these are relatively minor. My biggest concern are the ethical considerations that are not addressed. My review would be to accept if not for the severe ethical implications of the work that have gone unstated.\n\nMethodology and Presentation:\nThe model of the game seems more complex than necessary, and bakes in assumptions that don’t have clear motivation. Specifically, both agents must provide equal amounts of information, in alternating fashion. Why not simply have a game where the justifying agent provides some number of pieces of evidence L and the adversarial (baseline or confuser) agent provides any number of additional pieces of information? What is the benefit of creating a game with L stages where the agents alternate with best response? The paper motivates the use of Nash Equilibria to ensure robustness against additional information, but no motivation is given for this multi-stage debate game instead of letting each agent make 1 move each.\n\nA missing benchmark is how often the benchmark and justifying agents choose the same action. This would be highly informative.\n\n\nUnless I misunderstand the methodology, the benchmark with the isolated agent does not seem entirely appropriate. The isolated agent is trained to provide L pieces of evidence and then tested in a setting where it provides L/2 and a confuser provides L/2. It is not surprising that it is vulnerable to a confuser that provides the evidence least-supportive of its L/2 pieces of information. A better benchmark would be to train the isolated in a setting where it only provides L/2 pieces of information, and then see how robust it is to the setting where L/2 adversarially selected pieces of information are added.\n\nIn Fig 2a and 3a-d, the clinician’s performance is compared to the learned policies according to WIS. But why would WIS be a reasonable way to measure the performance of the real-world clinician based on clinical decisions?\n\nThe payoff function G is introduced, but it is never specified how it relates to the utility function U.\n\nThe use of the “Bradley-Terry model” (based on the Boltzmann distribution) is never given a justification.\n\nAlso, in the real world, not all pieces of evidence have equal complexity to evaluate, not all combinations of evidence have equal complexity to evaluate.\n\nMinor Errors:\n\nTypo: (Section 3.1) \\gamma^{i} instead of \\gamma^t in multiple places\n\nTypo (Section 3.2):  after the definition of U the text uses J instead of \\mathcal{J}\n\nThere doesn’t seem to be any need to introduce the notation \\mathcal{E}(n). Just use \\mathcal{E} \\backslash n.\n\nTypo (Section 5.3, p2): “…even tough” —> “…even though”\n\nTypo (Figure 4): The legend labels for “with confused” and “without confused” appear to be inconsistent with the plots.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ib3BaEjhLj", "reviewer": "Reviewer_pkzb", "summary": "The paper introduces a date framework for reward-shaping and quantifying the justifiability of a decision at a given state. An application is outlined for treating sepsis patients.", "questions": "1. Bottom of page 1, “the human as a suboptimal decision maker”: do you mean that the RL agent is optimal? But if optimality refers to the actions deployed in practice, the RL agent will not be optimal either, no?\n\n2. The authors in [A,B] also consider sequential decision-making (MDPs)  influenced by a baseline policy and an agent to model partial adherence to an algorithmic recommendation. Is the setup in this submission close to the setup in [A,B]?\n\n3. Definition of $R_{J}$ : what is the expectation over? The transition probabilities (since policies are deterministic)? Is the decision to be optimized $a_{t}$ or $a^{B}_{t}$ ?\n\n4. p4: the definition of $\\mathbb{U}$ introduces the notation $\\mathcal{J}$ but the next sentence uses $J$. Please clarify.\n\n5. Section 4.4: can you please justify the exact architecture and neural nets used in the paper (two-fully connected layer, size 256, choice of parametric relu, etc.)? This section appears very ad-hoc, which it may be, and that is fine, but it requires some discussion/empirical justification.\n\n[A] Grand-Clément, J., & Pauphilet, J. (2022). The best decisions are not the best advice: Making adherence-aware recommendations. arXiv preprint arXiv:2209.01874. \n\n[B] Faros, I., Dave, A., & Malikopoulos, A. A. (2023). A Q-learning Approach for Adherence-Aware Recommendations. arXiv preprint arXiv:2309.06519.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "* Paper is clear and straight to the point. \n\n* The authors provide adequate motivation for their model, which is quite natural.\n\n* I believe that the problem the authors attempt to address is important for any practical use of RL.", "weaknesses": "* No real weakness. Some slight inaccuracies in the terminology/in some definitions (see my questions below) but this should be easily fixable.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)