Counterfactual harm
Authors: Jonathan Richens
Rory Beard
Daniel H. Thompson
Venue: NeurIPS 2022 Accept (2022)
Keywords: causality
safety
ethics
counterfactuals
decision theory
expected utility theory

Abstract: To act safely and ethically in the real world, agents must be able to reason about harm and avoid harmful actions. However, to date there is no statistical method for measuring harm and factoring it into algorithmic decisions. In this paper we propose the first formal definition of harm and benefit using causal models. We show that any factual definition of harm is incapable of identifying harmful actions in certain scenarios, and show that standard machine learning algorithms that cannot perform counterfactual reasoning are guaranteed to pursue harmful policies following distributional shifts. We use our definition of harm to devise a framework for harm-averse decision making using counterfactual objective functions. We demonstrate this framework on the problem of identifying optimal drug doses using a dose-response model learned from randomised control trial data. We find that the standard method of selecting doses using treatment effects results in unnecessarily harmful doses, while our counterfactual approach identifies doses that are significantly less harmful without sacrificing efficacy.

Counterfactual harm
Jonathan G. Richens"
Rory Beard Comind
DeepMind
Daniel H. Thompson
Babylon
Abstract
To act safely and ethically in the real world, agents must be able to reason about harm and avoid harmful actions. However, date there no statistical method for measuring harm and factoring it into algorithmic decisions. In this paper we propose the first formal definition of harm and benefit using causal models. We show that any factual definition of harm is incapable of identifying harmful actions in certain scenarios, and show that standard machine learning algorithms that cannot perform counterfactual reasoning are guaranteed to pursue harmful policies following distributional shifts. We use our definition of harm to devise a framework for harm-averse decision making using counterfactual objective functions. We demonstrate this framework on the problem of identifying optimal drug doses using a dose-response model learned from randomized control trial data. We find that the standard method of selecting doses using treatment effects results unnecessarily harmful doses, while counterfactual approach identifies doses that are significantly less harmful without sacrificing efficacy.
Introduction
Machine learning algorithms are increasingly used for making highly consequential decisions in areas such medicine I 2 3) criminal justice 4 5 6.. finance 7 g g and autonomous vehicles
[0]. These algorithms optimize for outcomes such as survival time, recidivism rate, and financial cost. However, these are purely factual objectives, depending only on what outcome occurs. On the other hand, human decision making incorporates preferences that depend not only on the outcome, but why it occurred.
For example, consider two treatments for a disease which, when left untreated, has 50% mortality rate. Treatment 1 has 60% chance of curing patient, and 40% chance of having no effect, with the disease progressing as if untreated. Treatment has an 80% chance of curing a patient and a 20% chance of killing them. Treatments and 2 have identical recovery rates, hence any agent that selects actions based solely on the factual outcome statistics (c.g. the effect of treatment on the treated (II) will not be able to distinguish between these two treatments. However, doctors systematically favour treatment as achieves the same recovery rate but never harms the patient-there are no patients that would have survived had they not been treated. On the other hand, doctors who administer treatment risk harming their patients-ther there are patients who die following treatment who would have lived had they not been treated. While treatments I and 2 have the same mortality rates, the resulting deaths have different causes (doctor or disease) which have different ethical implications.
Determining what would have happened to patient if the doctor hadn't 't treated them is counter- factual inference, which cannot identified from outcome statistics alone but requires knowledge of the functional causal relation between actions and outcomes [2]] However, standard machine
jonrichens @ @deeminddcom
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

learning algorithms are limited to making factual inferences e.g. learning correlations between actions and outcomes. How then can we express these ethical preferences as objective functions and ultimately incorporate them into machine learning algorithms? This question has some urgency, given that machine learning algorithms currently under development or in deployment will be unable to reason about harm and factor it into their decisions.
The concept of harm is foundational to many ethical codes and principles: from the Hippocratic oath [33 to environmental policy 14, and from the foundations of classical liberalism [15 to Asimov's 's laws of robotics [6.. Despite this ubiquity, there is no formal statistical definition of harm. We address this by translating the predominant account of harm [7 [8] into statistical measure- counterfactual harm. Our definition of harm enables us for the first time precisely answer the following basic questions,
Question Did the actions an agent cause harm, and if so, how much?
Question How much harm can we expect an action to cause prior to taking it?
Question How can we identify actions that balance the expected harms and benefits?
In section 2 we introduce concepts from causality, ethics, and expected utility theory used to derive our results. In section g we present our definitions of harm and benefit, and in section 4 we describe how harm-aversion can be factored into algorithmic decisions. In section we prove that algorithms based on standard statistical learning theory are incapable of reasoning about harm and are guaranteed to pursue harmful policies following certain distributional shifts, while our approach using counterfactual objective functions overcomes these problems. Finally, in section 6 we apply our methods to an interpretable machine learning model for determining optimal drug doses, and compare this to standard algorithms that maximize treatment effects.
2
Prelimiaries
In this section we introduce some basic principles from causal modelling, expected utility theory and ethics that we use to derive our results.
2.1
Structural causal models
We provide brief overview of structural causal models (SCMs) (see Chapter 7 of 19 and 20 for reviews). SCMs represent observable variables X as vertices directed acyelic graph (DAG), with directed edges representing causal relations between X' and their direct causes (parents), which includes observable (endogenous) parents pa, and a single exogenous noise variable E The value of each variable is assigned by deterministic function f. of these parents.
Definition 1 (Structural Causal Model (SCM)). A structural causal model M = FG specifies:
1. set of observable (endogenous) random variables X (XX, ,..., NN,,
2. a set of mutually independent noise (exogenous) variables E = (E (E?,...,BB
EN
with joint
distribution P(E = e) = 11 P(E = (assuming causal sufficiency
199
3. a directed acyclic graph G. whose nodes are the variables X with directed edges from pa,, E' to X', where pa, S X denotes the endogenous parents of X
4. set functions (mechanisms) F == {ff (fi...... fN where the collection F forms mapping from E to X, r² = Se(pa,, e²), for i I,
By recursively applying condition 4 every observable variable can be expressed as a function of the noise variables alone, (e) æ² where E = is the joint state of the noise variables, and the distribution over unobserved noise variables induces distribution over the observable variables,
P(E =
e:X(e)=r
where X = x is the joint state over X.

---

The power of SCMs is that they specify not only the joint distribution P(X = x) but also the distribution of X under all interventions, including incompatible interventions (counterfactuals), Interventions describe an external agent changing the underlying causal mechanisms F. For example, do( X = x:) denotes intervening to force X x regardless of the state of its parents, replacing xr1 == fi (pa,,et) e²) + x² := The variables in this cost-intervention SCM are denoted X or X and their interventional distribution given by ① under the updated mechanisms.
These interventional distributions quantify the effects of actions and are used to inform decisions. For example, the conditional average treatment effect (CATE) (II].
CATE(x) B|Yr=1|=] - [|Yr=
(2)
measures the expected value of outcome Y for treatment T 1 compared to control T 0 conditional on X, and is used in decision making from precision medicine to economics 22
23
Counterfactual distributions determine the probability that other outcomes would have occurred had some precondition been different. For example, for X, Y E {T, F), the causal effect P(Yx-T = T) insufficient to determine if do( X = 7) was a necessary cause of = (2 Instead this is captured by the counterfactual probability of necessity 25
PN = P(Yx=P -IX
which is the probability that Y would be false if X had been false (the counterfactual proposition). given that X and Y were both true (the factual data). For example, if PN then Y = T' would have occured regardless of X T, whereas if PN = 1 then Y = could not have occurred without X T. Equation (3) involves the joint distribution over incompatible states of Y under different interventions, Y = T and Yx=r= = F, which cannot be jointly measured and hence ( cannot be identified from data alone. However, E can be calculated from the SCM using (I for the combined factual and counterfactual propositions.
=
P(E e)
(4)
(x=F()==F Y(e)=T,X(e)=7
This ability to ascribe causal explanations for data has made SCMs and counterfactual inference
key ingredient in statistical definitions of blame, intent and responsibility 226
27 28 and has seen
applications in Al research ranging from explainability 29 30 safety
31
32 and fairness 33 to
reinforcement learning 34 35
and medical diagnosis 36
Our definitions and theoretical results are derived within the SCM framework as is standard in
causality research [19]. However, there a growing body of work developing deep learning models
that are capable of supporting counterfactual reasoning in complex domains including medical
imaging (37], visual question answering [38]. and vision-and- -ad-aand--angagg navigation robotics 39
and text generation 40] We discuss these approaches in relation to our work in Appendix L.I
2.2 The counterfactual comparative account of harm
The concept of harm is deeply embedded in ethical principles, codes and law. One famous example is the bioethical principle "first, do no harm" 41 asserting that the moral responsibility for doctors to benefit patients is superseded by their responsibility not harm them 42). Another example is
John Stuart Mill's harm principle which forms the basis of classical liberalism 15) and inspired the "zeroth law" of Asimov's 's fictional robot governed society, which states that "A robot may not harm humanity or, by inaction, allow humanity to come to harm" [[6].
Despite these attempts to codify harm into rules for governing human and AI behaviour, it is not immediately clear what we mean when we talk about harm. This can lead to confusion- for example, does "allow humanity to come to harm" describe agent causing harm by inaction 44 or failing to benefit? The meaning and measure of harm also has important practical ramifications. For example, establishing negligence in tort law requires establishing that significant harm has occurred 44
This has motivated work philosophy, ethics and law to rigorously define harm, with the most
widely accepted definition being the counterfactual comparative account (CCA) 35 46
3

---

Definition 2 (CCA). The counterfactual comparative account of harm and benefit states,
"An event or action harms (benefits) person overall if and only if she would have been on balance better (worse) off if e had not occurred, or had not been performed."
The CCA quantifies harm by comparing how well off person is (their factual outcome) compared to how well off they would have been (their counterfactual outcome) had the agent not acted. Generating these counterfactuals necessitates causal model of the agents actions and their consequences, and we determine how "better (worse) off" the person is using an expected utility analysis 47). While the wording of the CCA suggests that always compare to the counterfactual world where the agent takes no action, a more natural and general approach to measure harm compared a specific default action or policy. For example, when measuring the harm caused by medicine we may intuitively want to compare to the outcomes that would have occurred with a placebo, and for the harm caused by negligent medical decisions the natural choice is to compare standard clinical decision policy. To this end, we measure harm compared to a default action (or policy), see Appendix D for further discussion.
While the CCA is the predominant definition of harm and forms the basis of our results, alternative definitions have been proposed
motivated by scenarios where the CCA appears to give Surprisingly, we find these problematic scenarios are identical to those raised in the study of actual causality
48
40
counter-intuitive results 43
[4 and can be resolved with formal causal analysis. To our knowledge, this connection between the harm literature and actual causality has not been noted before, and we discuss these alternative definitions in Appendix C and how the arguments surrounding the CCA can be resolved in Appendix in the hope this will stimulate discussion between these two fields.
2.3 Decision theoretic setup
In this section we present our setup for calculating the CCA (Definition 2 The CCA describes a person (referred to as the user) being made better or worse off due to the actions of agent. We focus on the simplest case involving single agents performing single actions, though ultimately our definitions can be extended to multi-agent or sequential decision problems (as discussed in Appendix
L.3p. For causal modelling in these scenarios we refer the reader to 32 50 51
A
Figure 1: SCM M depicting the agent's action A, context X, outcomes Y and utility function U.
We describe the environment with a structural causal model M which specifies the distribution over environment states W as endogenous variables, P(w; M). For single actions the environment variables can be divided into three disjoint sets W A U X U (Figure a)); the agent's action A, outcome variables Y that are descendants of A and so can be influenced by the agent's action, and context variables that are non-descendants of nor in A. For simplicity we assume no unobserved confounders (causal sufficiency (19) in the derivation of our theoretical results. In the following we assume knowledge of the SCM, and in Appendix
we discuss this assumption and describe how tight upper bounds on the harm can be determined when the SCM is unknown.
The user's preferences over environment states specified by their utility function U(w) U(a, x, y). and for a given action the user's expected utility is
B(U x = P(yla, ())(a,
(5)
In sections we focus on the case where the agent acts to maximize the user's expected utility, with optimal actions amax arg max, a EU æ). While this is similar to the standard setup in expected note that U describes the preferences of the user, and in general the agent
utility theory 47 52 S3

---

can pursue any objective and could be entirely unaware of the user's preferences. In section 5 we extend our analysis agents pursuing arbitrary objectives.
Example I: conditional average treatment effects. Returning to our example from section [ consider agent choosing between placebo A treatments and 2. Y= indicates recovery and Y O mortality. In the trial, harm is measured by comparison to 'no treatment' = 0.
X describes any variables that can potentially influence the agent's choice of intervention and/or Y, c.g. the patient's s medical history.
Consider patients (users) whose preferences The expected utility [[U|a, x = Σy P(yla a, (a, x, y) = Σ, PP (x) = E.Ya where in the last line we have use the fact that there are no confounders between A and Y. The agent's policy is therefore to maximize the treatment effect Omax arg max, [ [z]], which is equivalent to dmax standard objective for identifying optimal treatments [54] [55]. In Appendix G we derive SCM model for the treatments described in
fully determined by survival, e.g. Uaa
arg max, { [z] E Y ] i.c. maximizing the CATE 2
the introduction, and show that a[ = E| = 0.8, iee. the expected utility and CATE is the same for treatments and 2.
3 Counterfactual harm
We now present our definitions for harm and benefit based on Definition - For the examples described in this paper we focus on simple environments with single outcome variables. for which harm can be determined by comparing the factual utility received by the user to the utility they would have received had the agent taken some default action Appendix
(Definition B} In D we describe how this default action can be determined. Some more complicated situations call for a path-dependent measure of harm, which we provide in Appendix A along with examples in Appendices B D In the following, counterfactual states are identified with an e.g. Z
First we define the harm caused by an action given we observe the outcome (Question 1).
Definition 3 (Counterfactual harm & benefit). The harm caused by action A = a given context X and outcome Y compared to the default action is.
Ma, v. ys M) P(Ya ="aa, =0 (0
(6)
and the expected benefit is b(a, x. yi M) P(Ys ,",,,,,, x, M)max U(,
where M is the SCM describing the environment and U is the user's utility.
The counterfactual harm (benefit) is the expected increase (decrease) in utility had the agent taken the default action A c. given they took action A in context X = z resulting in outcome Y y. Taking the max in ( and ? ensures that we only include counterfactual outcomes where the utility is higher (lower) in the counterfactual world. Note that Xa X X is not a descendant of A, so the factual and counterfactual contexts are identical [56]. While we focus on the standard decision theoretic analysis of actions, it is trivial extend our analysis naturally occurring events that not determined by an agent (e.g. by identify A with an event rather than an action).
To determine the expected harm or benefit of an action prior to taking it (Question 2) we cal- culate the expected value of O6 and 7 over the outcome distribution, e.g. B[hla, æ; M] = Jy P(yla, x; M)h(a, M). Note these form counterfactual decomposition of the expected utility.
Theorem 1 (harm-benenf trade-off). The difference in expected utility for action A and the
defoult action given by,
EEU 2d]
EU a Elbla, 2; M] Elhla, M]
(8)
(Proof: see Appendix
5

---

Example 2: Returning to Example I. the expected counterfactual harm for default action ā = 0 is. El la; M] = Ey,,, P(Ya y", y/a; M) max {0. y² y} = P(Y. L. = M; where we have used U(a,y) y) = y and the fact that Ya - and there are no latent confounders to P(Y, = y¹, Ya = y). The harm is therefore the probability that a patient dies following treatment A = a and would have recovered had they received no treatment. In Appendix G we show that P(Y, = Y1 = 0) 0 (treatment I causes zero harm) and P(Y, = 1, 0) 0.1 (treatment causes non zero harm), reflecting our intuition that treatment 2 is more harmful than treatment 1, despite having the same causal effect / CATE. Hence, the counterfactual harm allows us to differentiate between these two treatments.
give P(Y, y² y/a)
4 Harm in decision making
Now we have expressions for the expected harm and benefit, how can we incorporate these into the agent's 's decisions? Consider two actions a, a' such that Ebblaa, z;; MM] = Ebbla, x; M) + K and E'/aaa,, I;; MM = [[[ha,, 2; M] + K where K E IR. By Theorem D they must have the same expected utility, as |U la x = Iiblaa, 2; M - E[hhaa, x; Mi + EU la æ| = Ebbla, x; M] + K
E[hla, x;; MM] K + E ā z= E(Ula, zz.. Therefore expected utility maximizers are indifferent to harm, and will are willing to increase harm for an equal increase in benefit.
We say that an agent is harm averse if they are willing to risk causing harm only if they expect a comparatively greater benefit. Theorem I suggests simple way to overcome harm indifference in the expected utility by assigning a higher weight to the harm component of the decomposition maximize an adjusted expected utility E, x;; MM] == E [c] + E,bla, 2; M (I + A) a. x; M] where A E R. then if B[ Ja, z; M) = E[V x; MM and EThla', Iz; M] = ll x; M] + K then Blbla' x; M] = Elbl a, x; M] + (1 + A)K, and the agent is willing to trade harm for benefit at 1 : (1 + A) ratio. Taking inspiration from risk aversion [57] we refer to X as the agent's ' harm aversion. To achieve the desired trade-off we replace the utility function with the harm- carm-penalized utility (HPU),
If instead
Definition 4 (Harm penalized utility (HPU)). For utility U and an environment described by SCM M, the harm-penalized utility (IPU)) is,
V(a, F. y; M) U(a,,, Xh(a, x. yi M)
where h(a, x. y; M) is the harm (Definition 3 and X is the harm aversion.
For À = 0 the agent is harm indisferent and optimizes the standard expected utility. For À >>> 0 the agent is harm averse, and willing to reduce the expected utility to achieve greater reduction the expected harm. For À < O the agent is harm seeking, and is willing to reduce the expected utility in order to increase the expected harm.
Maximizing the expected HPU allows agents to choose actions that balance the expected benefits and harms (Question 2 section MD. We refer the expected HPU ( as a counterfactual objective function as V(a, x, y; M) is a counterfactual expectation that cannot be evaluated without know knowledge of M. We refer to the expected utility as a factual objective function, as it can be evaluated on data alone without knowledge of M. As we show in the following example, harm-aversion can lead to very different behaviours compared other factual approaches to safety such as risk aversion [57]
Example 3: assistant paradox. Alice invests $80 in the stock market, and expects a normally distributed return Y ~ (H,a²) with μ investment, and after a lengthy analysis the Al identifies three possible actions;
$100. She asks her new Al assistant to manage her
1. Multiply the investment by S K < 20, resulting in a return Y - KY.
2. Perform clever trade that Alice was unaware of, increasing her return by +10 with certainty, Y + Y + 10,
3. Cancel Alice's 's investment, returning $80.
Consider agents,
1. Agent maximizes expected return, choosing arg max, E[Y|a]
2. Agent 2 is risk-averse, in the sense of [57]. choosing arg max, (E| \Var[Y|a]),

---

3. Agent 3 is harm-aveers averse, choosing a = arg max, [E[Y|a] AE[M|a;MM]]
In Appendix H we derive the optimal policies for Agents 1-3, and calculate the harm with respect to the default action where the agent leaves Alice's investment unchanged (i.e. action 1, K = 1). Agent 1 chooses action 1 and K = 20, maximising Alice's return but also maximizing the expected g.g.-e.g. if Alice would have lost $100 she will now lose $2000. The standard approach in portfolio theory [8 is to use agent with an appropriate risk aversion A. However, agent 2 never chooses action 2 for any A, despite action 2 always increasing Alice's 's return (causing zero harm and non-zero benefit). Agent 2 will reduce or even cancel Alice's 's investment (for À > 0.0032) rather than take action 2. This is because the risk-averse objective is factual, and cannot differentiate between losses caused by the agent's is actions from those that would have occurred regardless of the agent's actions (i.e. due to exogenous fluctuations in return). Consequently, agent 2 treats all outcomes as being caused by its actions, and is willing to harm Alice by reducing or even cancelling her investment in order to minimize her risk. On the other hand agent 3 never reduces Alice's expected return, choosing either action 1 or action 2 depending on the degree of harm aversion.
The surprising behaviour of agent 2 highlights that factual objectives (c.g. risk aversion) cannot capture preferences that depend not only on the outcome but on its cause. Alice may assign a different weight to losses that are caused by her assistant's 's actions rather than by exogenous market fluctuations or her own actions-ee e.g. if she places a bet that would have made a return but was overridden by her assistant. Consequently she may prefer action 2 over action I or action 3 with K < 1. While this can be achieved using a factual objective function that favours action a priori (assuming Alice is aware of this action), next we show this approach fails to generalize and actually leads to harmful policies.
5
Counterfactual reasoning is necessary for harm aversion
In the previous sections we saw examples where expected utility maximizers harm the user. However, these examples describe specific environments and utility functions, and assume the agent maximizes the expected utility instead of some other (potentially safer) objective. this section we consider users with arbitrary utility functions and agents with arbitrary objectives to answer, when is maximizing the expected utility harmful? And are there other factual objective functions that can overcome this?
Factual objective functions can be expressed as the expected value of a real-valued function J(,, x. y). Examples include the expected utility, cost functions (59] and the cumulative discounted reward (60] Once J is specified the optimal actions in environment M are given by,
/.
amax arg max E[J(a, x]] = arg max
2.
(10)
a
Note that (T0 can be evaluated from data alone (i.e. from samples of the joint distribution of A, X and Y) without knowledge of M. In the following we relax our requirement that agents have some fixed harm aversion X, requiring only that they don't take strictly harmful actions.
Definition 5 (Harmful actions & policies). An action A is strictly harmful in context X = and environment M if there is a' + a such that RBU Jal, x]] 2 E(U[a, a] and R[hla', , x; M] < Elhla, z; M). A policy is strictly harmful if it assigns non-zero probability action.
streitly harmful
Definition 6 (Harmful objectives). An objective E[Jla, xi) is strictly harmful in environment M if there is a policy that maximizes the expected value of J and is strictly harmful.
Note that strictly harmful actions violate any degree of harm aversion À >> 0, as the agent is willing to choose actions that are strictly more harmful for no additional benefit. In examples and 2, treatment 2 is strictly harmful action and the CATE is strictly harmful objective for this decision task. First, we show that the expected HPU is not a strictly harmful objective in any environment,
Theorem 2. For any utility functions U, environment M and default action A = ā the expected HPU is not a strictly harmful objective for any X > O. (Proof: see Appendix K
It is vital that agents continue to pursue safe objectives following changes to the environment (distributional shifts). At the very least, agents should be able to re-train in the shifted environment without needing to tweak their objective functions. In SCMs distributional shifts are represented as interventions that change the exogenous noise distribution and/or the underlying causal mechanisms We focus on distributional shifts that change the outcome distribution P(yla, M) alone.
61 5
7

---

Definition 7 (Outcome distributional shift). For an environment described by SCM M. MMMM- M' is a shift in the outcome distribution if the exogenous noise distribution for Y changes P(e + P(e M) and/or the causal mechanism changes By (a, x., xx,e) # Jy [v,,,
By Theorem 2 the HPU never strictly harmful following any distributional shift, as it is not strictly harmful in any M'. On the other hand, we find that maximizing almost any factual utility function will result in harmful policies under distributional shifts. In the following we assume some degree of outcome dependence in the user's utility function.
Definition 8 (Outcome dependence). U(a, v) is outcome dependent for a set of actions C' = {aI...., a 3 in context X if V a, az E C, i + j. max, ( x, y) > miny U Uaa,, N. y).
If there is no outcome dependence then the optimal action is independent of the outcome distribution P(yla, 2r), and no learning is required to determine the optimal policy. Typically we are interested in tasks that require some degree of learning and hence outcome dependence.
Theorem 3. If for any context X = x the user's utility function is outcome dependent for the default action in that context a(x) and another action ā(æ), then there is an outcome distributional shift such that U is strictly harmful in the shifted environment. (Proof: see Appendix K
To understand the implications of Theorem 3 we can return to examples & 2 and no longer assume a specific utility function or outcome statistics. Theorem 3 jippies that for an agent maximizing the user's utility, there is always a distributional shift such that the agent will cause unnecessary harm to patient in the shifted environment, compared to not treating them (ā = 0) or any other default action. Unlike in examples 1 & 2 we allow the patient's 's utility function to depend on the treatment choice, and for treatments to vary in effectiveness. The only assumption we make is that whatever the user's utility function is, it not true that U(A = a, J. Y = 0) > U(A = 0, æ. Y = 1) V a, x. If this were true then the patient's 's preferences are dominated by the agents action choice alone, i.e. the user would always prefer to die so long as they are treated, rather than not being treated and surviving.
While robust harm aversion is not possible for expected utility maximizers in general, it seems likely first that we can train robustly harm-averse agents using other learning objectives. One possibility would be to use human interactions and demonstrations (3 64 35) that implicitly encode harm aversion, with Ja, xr y) representing user feedback or some learned (factual) reward model. In examples & 2 users with utility U(a, x,,r) y) = y could be altered to assign a higher cost to deaths following treatment 2 compared to treatment 1, ([A 2, Y 0) < J(A 1, Y 0), with J implicitly encoding the harm caused by treatment 2. If there is some choice of J that implicitly encodes this harm-aversion in all shifted environments, this would allow robust harm-averse agents to be trained using standard statistical learning techniques without needing to learn M or perform counterfactual inference. However, we now show this is not possible in general.
Theorem 4. for any context X = x the user's utility function is outcome dependent for the default action in that context ā(x) and two other actions a1 02 # a(x), then for any factual objective function J there an outcome distributional shift such that J is strictly harmful in the shifted environment. (Proof: see Appendix K
Theorem 4 has profound consequences for the possibility of training safe agents with standard machine learning algorithms. It implies that agents optimizing factual objective functions may appear safe in their training environments but are guaranteed pursue harmful policies (with respect to Definition
2 following certain distributional shifts, even if they are allowed to re-train. This is true regardless the (factual) objective function we choose for the agent, and applies to any user whose utility function has some basic degree of outcome dependence. This is particularly concerning as all standard machine learning algorithms use factual objective functions. In section x we give a concrete example of reasonable factual objective function and a distributional shift that renders it harmful.
Example 4: An AI assistant manages Alice's health including medical treatments, clinical testing and lifestyle interventions. These actions have causal effect on Alice's 's health outcomes- e.g. disease progression, severity of symptoms, and medical costs, and Alice's preferences depend on these outcomes and not just on the action chosen by the agent (outcome dependence). The agent maximizes a reward function over these health states and actions (factual objective), including feedback from Alice and clinicians in-the-loop which penalize harmful actions (implicit harm aversion) by comparing to standardised treatment rules (default actions). The agent appears to be safe and is deployed at scale. However, by Theorem 4 there exists distributionally shifted environment such that the agent's
8

---

0.6
Epected
8.5
8.0
0.2
A-0
1-100
100
7.0
34
(mghlaa)
18
0.94
0.95
0.06
10M
0.97
0.98
0.99
1.00
(b)
Figure 2: a) Expected HPU (Del for the dose-response model (TI v.s. dosage of for À = 0, A= 10 and 100. Dotted lines show optimal doses. b) [[ / 0max] v.s. EU / [max] for S 0max- Dotted lines show doses reducing harm by 50% and 90% and utility by 0.2% and 2.0% respectively.
optimal policy is strictly harmful (Definition 5 in that it chooses needlessly harmful treatments for the patient, and this is true even if we train to convergence in the shifted environment.
Illustration: Dose-response models
How does harm aversion affect behaviour and performance in real-world tasks? And how can reasonable objective functions result in harmful policies following distributional shifts? In this section we apply the methods developed in section 4 to the task of determining optimal treatments. This problem has received much attention in recent years following advances in causal machine learning personalized
techniques (66] and growing commercial interests including recommendation
67
medicine 55 epidemiology and econometrics
68
to name a few.
While harm is important for decision making in many of these areas, existing approaches reduce the problem to identifying treatment effects-e.g, maximizing the CATE @ -wich is indifferent to harm shown in sections and 5 This could result in needlessly harmful decisions being made, perhaps overlooking equally effective decisions that are far less harmful.
To demonstrate this from a meta-analysis of randomised control trial data [69] This dose-response model predicts the reduction in symptom severity Y following treatment with dose A (mg/day) of Aripiprazole using generalized additive model (GAM) 770 an interpretable machine learning model commonly used
model for determining the effectiveneess of the drug Aripiprazole learned
for respones analysis M
= (a,aaaa,,,00 = 01a + (2f(a) +
(11)
where 0; ~ NQ;, Vi) are random effects parameters, ~ (0, Vz) is a noise parameter, and f(a) cubic spline function (for details and parameters Appendix D.. We assume ((a,8) y) =
and measure harm in comparison to the zero dose ā = 0. In Appendix J we derive a closed-form expression for the expected harm in GAMs.
calculate expected HPU for O, 10, 100 (Figure 2 a)). this context a relatively large A is appropriate to ensure high ratio of expected benefit to harm required for medical interventions [72]. We find that the optimal dose is highly sensitive X, for example reducing from 19.3 mg/day to 17.3 mg/day for À 100. On the other hand, these lower-harm doses achieve similar improvement in symptom severity compared to the optimal dose. Figure 2 b) shows the trade-off between expected utility and harm relative to the optimal dosage @CATE arg max, E[Ya]. For almost-optimal doses we observe a steep harm gradient, with small improvements in symptom severity requiring large increases in harm. For example, choosing lower dose than dcATE only reducing the treatment effect by 2%. This demonstrates that ignoring harm while maximizing expected utility will often result in extreme harm values (an example of Goodhart's law (73)
reduce harm by 90% while
Finally, we consider identifying safe doses using a reasonable factual objective function (risk- aversion) and describe how this can result in harm following distributional shifts. Consider the

---

risk-averse objective [[J|a] = E[Ya] /Var(Ya]. As Var[Y,] increases with dose. this objective
also selects lower harm doses our example. However, consider a distributional shift where (I) gains an additional noise term, y [(a, 01, 22 1)) = 01a 02)(a) + + 11" (10 0.5a) where n ~ OO, 1), e.g. describing a shift where untreated patients (A = 0) have a high variation in outcome Y which decreases as the dose increases. It is simple to check that, for a = arg max, [B[U1a] [a]
and a' = arg max, [[]][a], []U[a] > E|U|a')] and Elh]a] < E]hla'] V 3 > O in this shifted environment. Hence risk averse agents are strictly harmful in the shifted environment (Definition 6).
Related work
In appendix we discuss two related works, path-specific counterfactual fairness 3$ 74 and
path-specific objectives n5 which similarly use counterfactual contrasts with a baseline to impose ethical constraints on objective functions.
Following the publication of our results an alternative causal definition of harm has been proposed 76 777. In the simplest settings with single actions and outcomes, this work defines an action as harmful in deterministic environment if i) the utility is lower than default value and ii) any other action the agent could have taken would have resulted in a higher utility. In Appendix E we describe this definition and present examples where it gives counter-intuitive results including attributing harm to unharmful actions. Furthermore, as described in [76] this definition intractable except for simple causal models with low-dimensional variables, which limits its applicability to machine learning models (for example, this definition cannot be used for the dose-response model in section 6 which space).
has continuous
8 Conclusion
One of the core assumptions of statistical learning theory is that human preferences can be expressed solely in terms of states of the world, while the underlying dynamical (i.e. causal) relations between these states can be ignored. This assumption allows goals to be expressed as factual objective functions and optimized on data without needing to learn causal models or perform counterfactual inference. According to this view, all behaviours we care about can be achieved using the first two levels of Pearl's hierarchy 78) In this article we have argued against this view. We proposed the first statistical definition of harm and showed that agents optimizing factual objective functions are incapable of robustly avoiding harmful actions in general. We have demonstrated how our definition of harm can be used to meaningfully reduce the harms caused by machine learning algorithms, using the example of a simple model for determining optimal drug doses. The ability to avoid causing harm is vital for the deployment of safe and ethical Al [79]. and our results add weight to view that advanced Al systems will be severely limited if they are incapable of making causal and counterfactual inferences [80)
Acknowledgements
The authors would like to thank the NeurIPS reviewers for their details comments and to thank Tom Everitt, Ryan Carey, Albert Buchard, Sander Beckers, Hana Chockler and Joseph Halper for their helpful discussions and comments on the manuscript.
References
[1] Jeffrey De Fauw, Joseph R Ledsam, Bernardino Pareera- Stanislav Nikolov, Nenad Tomasev, Sam Blackwell, Harry Askham, Xavier Glorot, Brendan O' D'Donoghue, Daniel Visentin, et al. Clinically applicable deep learning for diagnosis and referral in retinal disease. Nature medicine, (119):1342-1333 2018.
[2] Eric J Topol. High-performance medicine: the convergence of human and artificial intelligence. Nature medicine, 25(1 (544566,, 2019.
[3] Ziad Obermeyer, Brian Powers, Christine Vogeli, and Sendhil Mullainathan. Dissecting racial bias in an algorithm used to manage the health of populations. Science, 366(6466)):44-44553, 2019.
10

---

[4] Aleš Završnik. Algorithmic justice: Algorithms and big data in criminal justice settings. European Journal of Criminology, 18(5):623-44 642, 2021.
[5] Julia Angwin, Jeff Larson, Surya Mattu, and Lauren Kirchner. Machine bias. ProPublica, May, 23(2016):139-55, 2016.
[6] Danielle Leah Kehl and Samuel Ari Kessler. Algorithms in the criminal justice system: Assessing the use of risk assessments in sentencing. Responsive Communities Initiative, Berkman Klein Center for Internet & Society, Harvard Law School, 2017.
[7] Kristin Johnson, Frank Pasquale, and Jennifer Chapman. Artificial intelligence, machine learning, and bias in finance: toward responsible innovation. Fordham L. Rev., 88:499, 2019.
[8] Daniel Belanche, Luis V Casaló, and Carlos Flavián. Artificial intelligence in fintech: under- standing robo-advisors adoption among customers. Industrial Management & Data Systems, 2019.
[9] Peter Martey Addo, Dominique Guegan, and Bertrand Hassani. Credit risk analysis using machine and deep learning models. Risks, 6(2):38, 2018.
[10] Wilko Schwarting, Javier Alonso-Mora, and Daniela Rus. Planning and decision for autonomous vehicles. Annual Review of Control, Robotics, and Autonomous Systems, 1: 187-210, 2018.
[1]] James Heekman. Randomization and social policy evaluation. Evaluating welfare and training programs, 1:201-30, 1992.
[12] Alexander Balke and Judea Pearl. Counterfactual probabilities: Computational methods, bounds and applications. Uncertainty Proceedings 1994, pages 46-54, Elsevier, 1994.
[13] Daniel K Sokol. "first do no harm" revisited. Bmj, 347, 2013.
[14] Albert Lin. The unifying role of harm in environmental law, 2006 wis. L. Rev, 897:904, 2006.
[15] John Stuart Mill. On liberty and the subjection of women. Penguin UK, 2006.
[16] Isaac Asimov. I, robot, volume 1. Spectra, 2004.
[17] Justin Klocksiem. A defense of the counterfactual comparative account of harm. American Philosophical Quarterly, (9)4))225-330 2012.
[18] Joel Feinberg. Harm to others, volume 1. Oxford University Press on Demand, 1987.
[19] Judea Pearl. Causality. Cambridge university press, 2009.
[20] Elias Bareinboim, Juan D Correa, Duligur lbeling, and Thomas Icard. On pearl's hierarchy and the foundations of causal inference. ACM Special Volume Honor of Judea Pearl (provisional title), 2:4, 2020.
[21] Ilya Shpitser and Judea Pearl. Identification of conditional interventional distributions. arXiv preprint /206. 6876, 2012.
[22] Jason Abrevaya, Yu-Chin Hsu, and Robert P Lieli. Estimating conditional average treatment effects. Journal of Business & Economic Statistics, 3354),485-505, 2015.
[23] Uri Shalit, Fredrik D Johansson, and David Sontag. Estimating individual treatment effect: generalization bounds and algorithms. In International Conference on Machine Learning, pages 3076-3085. PMLR, 2017.
[24] Joseph Y Halpern. Actual causality. MIT Press, 2016.
[25] Jin Tian and Judea Pearl. Probabilities of causation: Bounds and identification. Annals of Mathematics and Artificial Intelligence, 28(1):287-313, 2000.
[26] Joseph Halpern and Max Kleiman-Weiner. Towards formal definitions of blameworthiness, intention, and moral responsibility. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
[27] Max Kleiman-Weiner, Tobias Gerstenberg, Sydney Levine, and Joshua B Tenenbaum. Infer- ence of intention and permissibility in moral decision making. In CogSei, 2015.
[28] David Lagnado, Tobias Gerstenberg, and Ro'i Zultan. Causal responsibility and counterfac- tuals. Cognitive science, 3736) 2013.

---

[29] Sandra Wachter, Brent Mittelstadt, and Chris Russell. Counterfactual explanations without opening the black box: Automated decisions and the gdpr. Harv. JL Tech.. 31:841, 2017.
[30] Prashan Madumal, Tim Miller, Liz Sonenberg, and Frank Vetere. Explainable reinforcement learning through a causal lens. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 1493-2500, 2020.
[31] Tom Everitt, Marcus Hutter, Ramana Kumar, and Victoria Krakovna. Reward tampering problems and solutions in reinforcement learning: A causal influence diagram perspective.
Synthese, pages 1-33, 2021.
[32] Tom Everitt, Ramana Kumar, Victoria Krakovna, and Shane Legg. Modeling agi safety frameworks with causal influence diagrams. arXiv preprint 1906v:990.00800) 2019.
[33] Matt Kusner, Joshua Loftus, Chris Russell, and Ricardo Silva. Counterfactual fairness, In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
[34] Andrew Forney, Judea Pearl, and Elias Bareinboim. Counterfactual data-fusion for online reinforcement learners. In International Conference on Machine Learning, pages 1 1156-1164.
PMLR, 2017.
[35] Lars Buesing. Theophane Weber. Yori Zwols, Sebastien Racaniere, Arthur Guez, Jean- Saan-Baptiste Lespiau, and Nicolas Heess. Woulda, coulda, shoulda: Coumterfactuullly-uuinnn policy search. arXiv preprint arXiv:1811.06272. 2018.
[36] Jonathan G Richens, Ciarán M Lee, and Saurabh Johri. Improving the accuracy of medical diagnosis with causal machine learning. Nature communications, I1(1):1-9, 2020.
[37] Nick Pawlowski. Daniel Coelho de Castro, and Ben Glocker. Deep structural causal models for tractable counterfactual inference. Advances in Neural Information Processing Systems, 33:857-869, 2020.
[38] Yulei Niu, Kaihua Tang, Hanwang Zhang, Zhiwu Lu, Xian-Sheng Hua, and Ji-Rong Wen. Counterfactual vqa: A cause-effect look language bias. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12700-12710, 2021.
[39] Amin Parvanch, Ehsan Abbasnejad, Damien Teney, Javen Qinfeng Shi, and Anton van den Hengel. Counterfactual assion-and-language navigation: Unravelling the unseen. Advances in Neural Information Processing Systems, 33:5296 0006--3372200 2007,2020.
[40] Nishtha Madaan, Inkit Padhi, Naveen Panwar, and Diptikalyan Saha. Generate your counter- factuals: Towards controlled counterfactual generation for text. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 13524, 2021.
[41] Cedric M Smith. Origin and uses of primum non nocere-abov all, do no harm! The Journal of Clinical Pharmacology. 45(4):371-377 2005.
[42] W David Ross. Foundations of ethics. Read Books Ltd, 2011.
[43] Ben Bradley. Doing away with harm 1, Philosophy and Phenomenological Research, 85(2):
390-412, 2012.
[44] Richard W Wright. Causation in tort law. Calif. L. Rev., 73:1735, 1985.
[45] Joel Feinberg. Wrongful life and the counterfactual element in harming. Social Philosophy and Policy, 4(1):145-178, 1986.
[46] Matthew Hanser. The metaphysics of harm. Philosophy and Phenomenological Research, 77 (2):421-450, 2008
[47] Oskar Morgenstern and John Von Neumann. Theory of games and economic behavior. Princeton university press, 1953.
[48] Alastair Noreross. Harming in context. Philosophical Studies, 123(1-2):149-177 2005.
[49] Elizabeth Harman. Harming as causing harm. In Harming future persons, pages 137-154.
Springer, 2009.
[50] A Philip Dawid. Influence diagrams for causal modelling and inference. International Statistical Review, 1612)1661-189 2002.
12

---

[51] David Heckerman and Ross Shachter. A decision- view of causality. In Uncertainty Proceedings 1994, pages 302-310. Elsevier, 1994.
[52] Leonard J Savage. The foundations of statistics. Courier Corporation, 1972.
[53] Peter C Fishburn. The foundations of expected utility, volume 31. Springer Science & Business Media, 2013.
[54] Mattia Prosperi, Yi Guo, Matt Sperrin, James S Koopman, Jae Min, Xing He, Shannan Rich, Mo Wang, Iain E Buchan, and Jiang Bian. Causal inference and counterfactual prediction in machine learning for actionable healthcare. Nature Machine Intelligence, 2(7):369 33--75,,,,, 2020.
[55] Ioana Bica, Ahmed Alaa, Craig Lambert, and Mihaela Van Der Schaar. From real-world patient data individualized treatment effects using machine learning: Current and future methods to address underlying challenges. Clinical Pharmacology & Therapeutics, 109(1):
87-100, 2021.
[56] llya Shpitser and Judea Pearl. What counterfactuals can be tested. arXiv:1206.5223 2012.
arXiv preprint
[57] Ronald Howard and James E Matheson, Risk-sensitive markov decision processes. Man- agement science, 18(7):5563369 1972.
[58] Harry M Markowitz. Portfolio selection. Yale university press, 1968.
[59] James O Berger. Statistical decision theory and Bayesian analysis. Springer Science & Business Media, 2013.
[60] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction, MIT press, 2018.
[61] Jonas Peters, Dominik Janzing, and Bernhard Schölkopf. Elements of causal inference: foundations and learning algorithums. The MIT Press, 2017.
[62] Bernhard Schölkopf, Francesco Locatello, Stefan Bauer, Nan Rosemary Ke, Nal Kalchbrenner, Anirudh Goyal, and Yoshua Bengio. Toward causal representation learning. Proceedings of the IEEE, 109(5):612-6))) 2344, 2021.
[63] Paul Christiano, Jan Leike, Tom B Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. arXiv preprint
2017.
[64] Jan Leike, David Krueger, Tom Everitt, Miljan Martic, Vishal Maini, and Shane Legg. Scalable agent alignment via reward modeling: a research direction. arXiv preprint arXiv: 18110071, 2018.
[65] Ashvin Nair, Bob McGrew, Marcin Andrychowicz, Wojciech Zaremba, and Pieter Abbeel. Overcoming exploration in reinforcement learning with demonstrations. In 2018 IEEE Inter- national Conference on Robotics and Automation (ICRA), pages 6292-6299. IEEE, 2018
[66] Liuyi Yao, Sheng Li, Yaliang Li, Mengdi Huai, Jing Gao, and Aidong Zhang, Representa- tion learning for treatment effect estimation from observational data. Advances in Neural Information Processing Systems, 31, 2018.
[67] Dawen Liang. Laurent Charlin, and David M Blei. Causal inference for recommendation. In Causation: Foundation to Application, Workshop at UAL. AUAI, 2016.
[68] Guido Imbens and Donald B Rubin. Causal inference in statistics, social, and biomedical sciences. Cambridge University Press, 2015.
[69] Alessio Crippa and Nicola Orsini. Dose-response meta-analysis of differences in means. BMC medical research methodology, 16(1):1-10, 2016
[70] Trevor J Hastie and Robert J Tibshirani. Generalized additive models. Routledge, 2017.
[71] Loic Desquilbet and François Mariotti. Dose-response analyses using restricted cubic spline functions in public health research. Statistics in medicine, 2909):1037 2010.
[72] FDA. Benefit-risk assessment in drug regulatory decision becision-making. Draft PDUFA VI imple- mentation Plan (FY Preseriptionnrrugusserrr Useree UCM60, 2885, 2018.
https://www. fda. go//ooooolollo//oor Industry/ UserFees/
[73] Simon Zhuang and Dylan Hadfield-Menel Consequences of misaligned ai, arXiv preprint arXiv:2102.03880 1021.
13

---

[74] Silvia Chiappa. Path- spehh-pecific counterfactual faimess.. In Proceedings of the AAAI Conference on Artificial Intelligence. volume 33, pages 7801-7808, 2019.
[75] Sebastian Farquhar, Ryan Carey, and Tom Everitt. Path-specific objectives for safer agent incentives. arXiv preprini arXiv:2204. 10018, 2022.
[76] Sander Beckers, Hana Chockler, and Joseph Y. Halpern. A quantitative account harm. 2022.
[77] Sander Beckers, Hana Chockler, and Joseph Y. Halpern. A causal analysis of harm. 2022.
[78] Judea Pearl. The seven tools of causal inference, with reflections on machine learning. Communications of the ACM, 62(3):54- 60, 2019.
[79] Nick Bostrom and Eliezer Yudkowsky. The ethics of artificial intelligence. The Cambridge handbook of artificial intelligence, 1:316-334 6344, 2014.
[80] Judea Pearl. Theoretical impediments to machine learning with seven sparks from the causal revolution. arXiv preprint arXiv: 1.0.vv00001.00010 2018.
[81] Chen Avin, lyy Shpitser, and Judea Pearl. Identifiability of path-specific effects. 2005.
[82] Erik Carlson, Jens Johansson, and Olle Risberg. Causal accounts of harming. Pacific Philo- sophical Quarterly, 2021.
[83] Michael R Waldmann and Jörn H Dieterich. Throwing a bomb person versus throwing a person on a bomb: Intervention myopia in moral intuitions. Psychological science, 18(3):
247-253, 2007.
[84] Erik Carlson, Jens Johansson, and Olle Risberg. Causal accounts of harming. Pacific Philo- sophical Quarterly, 103(2):420-44) 2022.
[85] Juan Correa and Elias Bareinboim. A calculus for stochastic interventions: Causal effect identification and surrogate experiments. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 10093-10100, 2020.
[86] Neil Feit. Harming by failing to benefit. Ethical theory and moral practice, 22(4):809-823,
2019.
[87] William Holden, Juhaeri Juhaeri, and Wanju Dai. Benefit-risk analysis: proposal using quantitative methods. Pharmacoepidemiology and drug safety, 12(7))611-616 2003.
[88] Joseph Halpern. A modification of the halpern-peat definition of causality. In Twenty-Fourth International Joint Conference on Artificial Intelligence, 2015.
[89] S Kay, Lewis A Opler, Abraham Fiszbein, PM Ramirez, MGA Opler, and I. White. Positive and negative syndrome scale. GROUP, 1:4, 1992.
[90] Andrew J Cutler, Ronald N Marcus, Sterling A Hardy, Amy O' O'Donnell, William H Carson, and Robert D McQuade. The efficacy and safety of lower doses of aripiprazole for the treatment of patients with acute exacerbation of schizophrenia. CNS spectrums, 119)):69-7722 2006.
[91] Joseph P McEvoy, David G Daniel, William H Carson Jr, Robert D McQuade, and Ronald N Marcus. A randomized, double-blind, placebo-controlled, study of the efficacy and safety of aripiprazole 10, 15 or 20 mg/day for the treatment of patients with acute exacerbations of schizophrenia. Journal of psychiatric research, 41 1):895)99995 9055,905,222 2007.
[92] John M Kane, William Carson, Anutosh R Saha, Robert McQuade, Gary G Ingenito. Dan L Zimbroff, and Mirza W Ali. Efficacy and safety of aripiprazole and haloperidol versus placebo in patients with schizophrenia and schizoaffective disorder. Journal of Clinical Psychiatry, 63(9):763-771, 2002.
[93] Steven G Potkin, Anutosh R Saha, Mary Kujawa, William H Carson, Mirza Ali, Elyse Stock, Joseph Stringfellow, Gary Ingenito, and Stephen R Marder. Aripiprazole, an antipsychotic with a novel mechanism of action, and risperidone vs placebo in patients with schizophrenia and schizoaffective disorder. Archives of general psychiatry, 60(7):681-690 2003.
[94] Erick H Turner, Daniel Knoepfimacher, and Lee Shapley. Publication bias in antipsychotic trials: an analysis of efficacy comparing the published literature to the us food and drug administration database. PLOS medicine, 933)ee0011 189, 2012.
[95] Niki Kilbertus, Philip J Ball, Matt J Kusner, Adrian Weller, and Ricardo Silva. The sensitivity of counterfactual fairness to unmeasured confounding. Uncertainty artificial intelligence, pages 616-626. PMLR, 2020.
14

---

[96] Junzhe Zhang, Jin Tian, and Elias Bareinboim. Partial counterfactual identification from observational and experimental data. In International Conference on Machine Learning, pages (6548-265588 PMLR, 2022.
[97] Trevor Hastie and Robert Tibshirani. Generalized additive models for medical research. Statistical methods in medical research, 4(3):177-166, 1995.
[98] Kai Epstude and Neal Roese. The functional theory of counterfactual thinking. Personality and social psychology review, 12(2):168-192, 2008.
[99] Tobias Gerstenberg, Noah D) Goodman, David A Lagnado, and Joshua B Tenenbaum. A counterfactual simulation model of causal judgments for physical events. Psychological Review, 128(5):936, 2021.
[100] Jacob C Reinhold, Aaron Carass, and Jerry L. Prince. A structural causal model for mr images of multiple sclerosis. International Conference on Medical Image Computing and Computer-sssisted Intervention, pages 782-792. Springer, 2021.
[101] Adrian Tousignant, Paul Lemaître, Doina Precup. Douglas L Arnold, and Tal Arbel. Prediction of disease progression in multiple sclerosis patients using deep learning analysis of mri data. In International conference on medical imaging with deep learming, pages 483-492. PMLR, 2019.
[102] Stratis Tsirtsis, Abir De, and Manuel Rodriguez. Counterfactual explanations in sequential decision making under uncertainty. Advances in Neural Information Processing Systems, 34: 30127-30139, 2021.
[103] Siqi Liu, Kay Choong See, Kee Yuan Ngiam, Leo Anthony Celi, Xingzhi Sun, Mengling Feng, et al. Reinforcement learning for clinical decision support in critical care: comprehensive review. Journal of medical Internet research, 22(7):e18477, 2020.
[104] Chao Yu, Jiming Liu, Shamim Nemati. and Guosheng Yin. Reinforcement learning in healthcare: A survey. ACM Computing Surveys (CSUR), 55(1):1-36, 2021.
[105] Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Concrete problems in ai safety. arXiv preprint arXiv:1606. 06565, 2016.
[106] Laura Weidinger, John Mellor, Mariheth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang, Myra Cheng, Mia Glaese, Borja Balle, Atoosa Kasirzadeh, et al. Ethical and social risks of harm from language models. arXiv preprint
2021.
[107] Ninareh Mehrabi, Fred Morstatter, Nripsuta Saxena, Kristina Lerman, and Aram Galstyan. A survey on bias and fairness in machine learning. ACM Computing Surveys (CSUR), 54(6): 1-35, 2021.
[108] Robert Challen, Joshua Denny, Martin Pitt, Luke Gompels, Tom Edwards, and Krasimira -tanava-Aaaasova. Artificial intelligence, bias and clinical safety. BMJ Quality & Safety, 28 (3):231-237, 2019.
[109] David Leslie. Understanding artificial intelligence ethics and safety. arXiv preprint 056,44 2019.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]. In the final paragraph the introduction we list the results and the sections they are derived in. These are reflected in the abstract.
(b) Did you describe the limitations of your work? [Yes]. In section 2.3 describe setup and how our results cover the simplest case: single agents and single decisions.
(c) Did you discuss any potential negative societal impacts of your work? [Yes]. In Appendix L.I
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you including theoretical results...
15

---

(a) Did you state the full set of assumptions of all theoretical results? [Yes]. Section 2 is dedicated to the assumptions underlying our framework. In section 5 present the assumptions underlying our no-go theorem. Any further assumptions used to derive our results are outlined in the corresponding appendices for each result.
(b) Did you include complete proofs of all theoretical results? [Yes] These proofs are included in Appendices C- H.
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either in the supplemental material as a URL)? [N/A] Our work reports no experimental results. In section 6 we use three parameter parametric equation as an example, and cite the corresponding experimental paper where these parameters were leamnd.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [N/A] Our study does not use any datasets.
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [N/A]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [N/A] All results were derived by hand.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [N/A] As mentioned in checklist 3 a), section 6 cite the study that created the generalized additive model whose parameters are referenced in one of our examples. This is a three parameter regression model and we cite the paper that published these parameters. We do not use any implementation of their model or any of their code or data.
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A] We have not used any human subjects crowd-soureing.
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals. applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
16


Reviews:
[{"review_id": "Lg0U1xPK_hL", "reviewer": "Reviewer_tHHv", "summary": "The paper argues that harm, evasion of which is key for AI to be safe and ethical when deployed, is intrinsically a counterfactual quantity. Further, that conventional ML is guaranteed to end up with harmful policies if unable to perform counterfactual reasoning. As contribution, it claims to be the first to produce a statistical definition of harm, and have derived a set of counterfactual objective functions to robustly mitigate harm. Finally, using a statistical model for indentifying optimal drug doses, the paper demonstrates its framework that integrates harm into algorithmic decisions can dentify doses that are significantly less harmful while not sacrificing efficacy whilst standard algorithms yield significant harm.", "questions": "If indeed tha paper elucidated or proved that the proposed counterfactual-based harm-aversed framework can account for more complex harmful situations, it would be good for the authors to make this explicit. The paper also showed how outcome-dependent optimal policy learning happens within the framework. How about the framework being able to learn amid contextual, P(x; M), changes? In otherwords, when distribution shift is not just on the outcome distribution, but also on the input (e.g., covariate shifts) or to both (concept drift)?  Lastly, some support (proof) to qualify certain claims (assumptions), as follows:\n1. The  claim that agents trained to maximize factual objective functions are guaranteed to proceed with harmful policies, albeit they are allowed to retrain, amid distributional shifts (L332~L334). For one, Scholkopf et al. [1] argues that change in data distribution \"may lead to (arbitrarily) in accurate predictions\". Thus, this claim should be situated on the premise that the characterized objective function is pertinent only to domains where there is vulnerability to harm. Secondly, however, why would this claim hold even if the agent can retrain itself?\n2. Why was the requirement that agents have some fixed harm aversion relaxed, and tapered to requiring only that the agents do not take needlessly harmful actions?", "limitations": "On the two brief mentions on limitation, namely, (1) that it remains \"an open question as to how counterfactual reasoning can be achieved with current implementations\", and (2) \"there is growing interest in counterfactual reasoning with deep learning models\", it would help to discuss this further in a section on related works.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is very well written. The social significance of the research in general, together with its arguments, research questions and formal elucidations, are well positioned and articulated.\n\nHowever, inherent in formal frameworks and models is to abstract or simplify potentially required strong ties to more complex real-world contexts whilst claim generalizability. While a generalized additive model was used to illustrate the counterfactual framework for dose-response predictions, how to scale up to a rather complex domain (e.g., hazardous environment in which human and AI teamp up to navigate the situation) is worth discussing. Further, it is unclear how the causal constructs elucidated here accounted for or will account for human feedback, e.g., Alice's preferences, clinician's sense of values or human demonstrations in general (objective functions, which for example provably beneficial AI methods account for).  While it is not a question for me that this is the first to  provide a statistical defintion of harm and render  it a counterfactual framework, the discussion on related works seems to miss the built up on why the current literature has not addressed it until this time, whether previous works have sought or alluded this reasearch direction and might be falling short, and how close (or far) current deep learning frameworks on counterfactual reasoning relate to this work. \n\n \n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "HvwfbZFrdfG", "reviewer": "Reviewer_jo7e", "summary": "This paper uses a structural causal modeling (SCM) framework to define a statistical notion of harm which the authors name counterfactual harm (CH). The statistical definition extends a philosophical one named the comparative counterfactual account, which states that an action causes harm to a person if that person would have been better off had the action not occurred. The paper introduces a harm penalized utility (HPU) objective and compares the optimal actions according to this objective with others including an unpenalized utility and an objective that formalizes risk-aversion (penalizing the second moment) across several illustrative examples. The paper argues that avoiding harm requires causal reasoning, and specifically counterfactual reasoning, and shows that optimal actions according to other objectives can lead to causing harm. It presents several definitions and theorems to generalize this argument from the examples- the theorems show that various non-CH objectives can cause harm under a distribution shift in the data. The paper also illustrates its approach by comparison on a real data example involving dose responses for a drug. ", "questions": "Q1: Is it really necessary to focus on counterfactuals to define an adequate notion of harm? What if we replace the counterfactual distribution in formula (6) with an interventional distribution? I believe the resulting definition would still be interesting and usable for many of the examples considered. An “interventional comparative account” of harm could say that an action will cause harm to a person if that person will (probably) be better off if a different action is chosen. Since many of the examples in the paper consider actions that may actually be feasible this definition may suffice for them. By contrast, the definition of counterfactual fairness (for example) considers attributes which cannot feasibly be changed, hence it is necessarily counterfactual and could not be interventional.\n\nQ2: Is it too narrow to operationalize harm in a (simple) SCM framework? The model focuses on specific variables, while our intuition about harm usually involves a more open and wholistic idea of the world. Consider the two thieves example in Appendix D. The model specifies that Alice can only choose to rob or not rob Bob, while in the real world Alice could choose to simply warn Bob about Eve. Why should we apply the path-specific definition of harm that ignores an important way that Alice’s action can impact Bob’s wellbeing? I don’t think this resolves the paradox, but expanding the action space can. Similarly, a policy optimized to avoid harm according to a model built on one set of variables could cause harm involving other variables which were not included in the model. I think this is not just a curious theoretical issue because in algorithmic systems the model assumptions can be become built in and the data infrastructure can grow and increase in complexity, possibly cementing in place a harmful set of assumptions. However, the proposed narrow definition of harm would not address such issues.\n\nQ3: Are the authors certain about the claims in the paper regarding the \"first\" statistical definition of harm? It would require a truly massive literature search to establish this. It seems to me the contribution of the paper is interesting enough to speak for itself, without needing to be claimed as a first in kind. ", "limitations": "Some already highlighted under the weaknesses, like reliance on untestable assumptions and use of unobservable constructs, and most importantly the first two questions listed above.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper considers problems which are practically important. It is clear, interesting, and will likely generate a lot of discussion. A strength of its approach is the direct focus on things that truly matter, like utility, but a corresponding weakness is that much of the relevant quantities cannot be observed and require strong modeling assumptions, like the knowledge of an SCM. Note that this is not a specific weakness of the current paper, but one it shares with others that have been published in comparable conferences (including some of the cited related literature).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["Discrimination / Bias / Fairness Concerns", "Responsible Research Practice (e.g., IRB, documentation, research ethics)"]}, "extra_text": {}}, {"review_id": "9KGFy-sFHM", "reviewer": "Reviewer_8Y89", "summary": "This paper rigorously define the concept of \"counterfactual harm\" using statistical analysis. Overall, the authors quantify the \"benefit\" & \"harm\" based on a common definition of counterfactual comparative account (CCA). With the mathematical formulation of counterfactual harm, the authors further derive a family of counterfactual objective functions for mitigation. Solid theoretical analysis are provided with the structural causal modeling, and an empirical example illustration is shown to demonstrate the effectiveness of the proposed quantification of counterfactual harm. ", "questions": "1. I'm wondering how we can apply this definition (harm quantification) to a general ML setting, so as to mitigate some potential biases learned in models? Is the SCM a necessary setting for incorporating the proposed family of objective functions? \n\n2. In a more specific setting (e.g., fairness domain), can we equate or connect the harm quantification to some existing measurements (e.g., counterfactual fairness - Kusner, Matt J., et al. \"Counterfactual fairness.\" NeurIPS'17)? If yes, then what is the key difference for the proposed harm quantification? \n\n3. Is it possible to design some validation experiments on real-world data for this work?", "limitations": "1. The major theoretical analysis & results are purely based on the statistical causal analysis, which may not fit in the settings of standard ML scenarios. Personally, I think the proposed work could be limited since it is hard to directly employ such harm quantification to mitigate the \"harmful\" patterns existing in commonly used models, such as neural nets. \n\n2. The effectiveness of this work on wider domains still needs further validation, especially with the experiments on real-world datasets. Current illustration part could be limited to fully evaluate the proposed harm measurements. \n\n3. The authors should conduct a more detailed discussion between this work and other existing frameworks for quantifying \"harmful\" concepts (e.g., fairness, privacy). It seems like the authors try to make the definition from a higher level, but we readers/researchers also need to know what is the generality brought by this paper. In the current version, this part is quite limited in my view. ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 1, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- The topic of this paper is important and significant, which can largely benefit the areas such as fairness AI & ethical AI. The provided quantification method is general, so many down-stream application scenarios could refer. \n\n- The equations in the main body of the paper is well introduced and clarified, which is easy to follow. \n\n- The authors provide many contexts and background information which is very useful for readers. The provided high-level examples can help quickly extract the main idea of this paper. \n\nWeakness:\n\n- The overall paper is not well structured. The section titles are kind of unclear. \n\n- The analysis setups are too specific, even though the harm definition is general by itself. This could be limited when applying such quantification scheme to a general ML settings. \n\n- The paper only provides a simple illustration for demonstration, without any real-world experiments & comparison. This makes the proposed work hard to be fully evaluated regarding to its overall performance. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["Inappropriate Potential Applications & Impact  (e.g., human rights concerns)"]}, "extra_text": {}}, {"review_id": "2nY7eeLywlc", "reviewer": "Reviewer_rBDL", "summary": "This paper considers defining harm as a counterfactual quantity involving user’s utility. The harm is defined based on the expected difference in the utility following the policy and the utility following the default action. Equipped with the definition of harm, the authors proposed harm penalized utility [HPU] and relate to its implications to decision making. Authors investigated a few properties of the definition(s) and provide a proof-of-concept simulation result in the context of dose-response showing that a policy can be optimized to lower harm depending on the user specified hyper-parameter. \n", "questions": "\n- In the example in the introduction, I understand that why doctors would favor Treatment 1 vs 2. This feels so much like a trolley problem where inaction can cause 5 people die and action (switching) can cause 1 people die. What’s your explanation? In HPU, U is -5, -1 and harm is -1. So with lambda 4, there is no difference between switching lanes?\n- Def 1, why should noise be mutually independent?\n- Is utility fixed over the entire population? A is the agent's action but U is user's utility function. User != Agent. (Y would also be user's outcome...) \n- What's the practical importance of Y in defining harm and benefit when you have utility U? Y exists only to be marginalized out. For decision making, how would you use Y? where Y is not observed for the current user and X is given...", "limitations": "Please see the weaknesses especially about the non-identifiability of the measure. One may show the applicability of new definition under a learned model, etc.\n=======\nI raise my score which now reflects better understanding of the paper.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I would like to thank the authors first for an intriguing notion for harm (and benefit).\n\nStrengths\n- As a causal inference researcher, seeing the definition of harm at the level of “counterfactual” not “intervention” is refreshing.\n\nWeaknesses\n- It is unclear why we should define harm and benefit separately by taking max(0, difference in utility)\n- Simply the P terms in Eq 6 and 7 are not identifiable not only from factual outcome but also from any experimental data. (Line 235–236)\n- For the same reason, the authors used an existing model not the data. What would be the implications of using a learned model? A learned model only reflects the factual outcome correctly and will estimate counterfactual harm incorrectly.\n- It is unclear how benefit or harm are defined in medical or related research.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}]

Author rebuttal:

Decision: Accept