A Causal Analysis of Harm
Authors: Sander Beckers
Hana Chockler
Joseph Halpern
Venue: NeurIPS 2022 Accept (2022)
Keywords: harm
causality
utility

Abstract: As autonomous systems rapidly become ubiquitous, there is a growing need for a legal and regulatory framework to
address when and how such a system harms someone. There have been several attempts within the philosophy literature to define harm, but none of them has proven capable of dealing with with the many examples that have been presented, leading some to suggest that the notion of harm should be abandoned and ``replaced by more well-behaved notions''. As harm is generally something that is caused, most of these definitions have involved causality at some level. Yet surprisingly, none of them makes use of causal models and the definitions of actual causality that they can express. In this paper we formally define a qualitative notion of harm that uses causal models and is based on a well-known definition of actual causality (Halpern, 2016). The key novelty of our definition is that it is based on contrastive causation and uses a default utility to which the utility of actual outcomes is compared. We show that our definition is able to handle the examples from the literature, and illustrate its importance for reasoning about situations involving autonomous systems.

A Causal Analysis of Harm
Sander Beckers
Hana Chockler
Cluster of Excellence in Machine Learning University of Tübingen and Munich Center for Mathematical Philosophy, LMU srekcebrednas @om
causaLens and Department of Informatics King's 's College London hana.chocklerde wer..chhocklr@kkl.c....k kclac.uk
Joseph Y. Halpern
Computer Science Department
Cornell University
halpern @ee.eec..coo @cw.coree...eeuu
Abstract
As autonomous systems rapidly become ubiquitous, there is growing need for legal and regulatory framework that addresses when and how such a system harms someone. There have been several attempts within the philosophy literature define harm, but none of them has proven capable of dealing with the many examples that have been presented, leading some to suggest that the notion of harm should be abandoned and "replaced by more well-behaved notions". As harm is generally something that is caused, most of these definitions have involved causality at some level. Yet surprisingly, none them makes use of causal models and the definitions of actual causality that they can express. In this paper we formally define a qualitative notion of harm that uses causal models and is based on well-known definition of actual causality [13] The key features of our definition are that it is based on contrastive causation and uses a default utility to which the utility of actual outcomes is compared. We show that our definition is able handle the examples from the literature, and illustrate its importance for reasoning about situations involving autonomous systems.
Introduction
The notion that one should not cause harm is central tenet in many religions; it is enshrined in the medical profession's 'ss Hippocratic Oath, which states explicitly "I will do no harm or injustice to [my patients]" 20 it is also critical element in the law. Not surprisingly, there have been many attempts in the philosophy literature to define harm. Motivated by the observation that we speak of "causing harm". most of these have involved causality at some level. All these attempts have encountered difficulties. Indeed, Bradley [4] says:
Unfortunately, when we look attempts to explain the nature of harm, we find mess. The most widely discussed account, the comparative account, faces counterexanples that seem fatal. But no alternative account has gained any currency. My diagnosis is that the notion of harm is Frankensteinian jumble ... It should be replaced by other more well-behaved notions.
The situation has not improved much since Bradley's paper (see, e.g., recent accounts like 5.8) Yet the legal and regulatory aspects of harm are becoming particularly important now, as autonomous systems become increasingly more prevalent. In fact, the new proposal for Europe's Al act M contains over 25 references to "harm" or "harmful", saying such things as "... it is appropriate to classify [AI systems] as high-risk if, in the light of their intended purpose, they pose high risk of
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

harm to the health and safety or the fundamental rights of persons... 7 Proposal preamble, clause (32)]. Moreover, the European Commission recognized that if harm is to play such crucial role, it must be defined carefully, saying "Stakeholders also highlighted that it is important to define 'harm' 7 Part 2, Section 3. I.. Legislative bodies in the UK are also discussing the question of harm and who caused harm in the case of accidents involving autonomous vehicles. The Law Commission of England and Wales and the Scottish Law Commission are recommending that drivers of self-driving cars should not be legally responsible for crashes; rather, the onus should lie with the manufacturer [6]. In particular, if there is harm then this is caused by the manufacturers. The manufacturers translate this recommendation to standard according to which the driver does not even have to pay attention while at the wheel. If a complex situation arises on the road requiring the driver's attention, the car will notify the driver, giving them 10 seconds to take control. If the driver does not react in time, the car will flash emergency lights, slow down, and eventually stop [18] Consider the following example (to which return later).
Example 1 (Autonomous Car) An autonomous car detects an unexpected stationary car in front it it on a highway. It could alert the driver Bob, who would then have to react within 10 seconds. However, 10 seconds is too long: the car will crash into the stationary car within 8 seconds. The autonomous car's algorithm directs it to crash into the safety fence on the side of the highway, injuring Bob. Bob claims that he was harmed by the car. Moreover, he also claims that, if alerted, he would have been able to find a better solution that would not have resulted in his being injured (e.g.. swerving into the incoming traffic then back to his own lane after passing the stationary car). We assume that if the autonomous car had done nothing and collided with the stationary car, both drivers would have been injured much more severely.
While the causal model depicting this story is fairly straightforward, the decision on whether harm was caused to Bob, and if yes, who what caused the harm, is far less clear. Indeed, the philosophy literature seems to suggest that trying to determine this systematically is a lost cause. But as this example illustrates, the stakes of having a well-defined notion of harm have become much higher with the advent of automated decision-making. In contrast to human agents, such systems do not have an informal understanding of harm that informs their actions; so we need a formal definition. Situations like that described in Example | are bound to arise frequently in the interaction of autonomous systems with human users, in a variety of domains. We briefly outline two of those.
Imagine a UAV used by the military has to decide whether or not it should bomb a suspected enemy encampment. The problem is that the target is not clearly identified, because there two camps close to each other: one consisting of civilian refugees, another consisting of a rebel group that is about to launch deadly attack on the refugee camp, killing all of its inhabitants. The UAV's decision is based only on the expected utility of the refugees, and therefore it bombs the camp. Tragically, as it turns out, the camp was that of the refugees. Here we have the intuition that the UAV harmed these refugees, despite the fact that both actions would have led to all the refugees being killed. Examples in which one event (the bombing) preempts another event (the attack) from causing an outcome known as Late Preemption examples in the causality literature; we discuss them later in the paper.
In the healthcare domain, autonomous systems are used for, among other things, classifying MRI brain images suspected of containing a tumor. If an image is classified as having a tumor, the system decides whether to recommend surgery. While the overall accuracy of the system is superior that of humans, in some instances the system overlooks an operable tumor. Imagine a patient who has such tumor and dies from brain cancer as the result of not undergoing surgery, leading to dispute between the patient's family and the hospital regarding whether the patient was harmed. Even if both parties agree that the patient would probably have been alive if the diagnosis had been performed by a human, the hospital might claim that using the system is the optimal policy, and therefore one should compare the actual outcome only to those that could have occurred under the policy.
Fortunately, the formal tools at our disposal to develop formal notion of harm have also improved over the past few years; we take full ad advantage of these developments in this paper. Concretely, we provide a formal definition of harm that we believe deals with all the concerns that have been raised, seems to match our intuitions well, and connects closely to work on decision theory and utility. Here we briefly give a high-level overview of the key features of our approach and how they deal with the problems raised in the earlier papers.
There is one set of problems that arise from using counterfactuals that also arise with causality, and can be dealt with using the by-now standard approaches in defining causality. For example, Carlson,

---

Johansson, and Risberg [5] raise a number of problems with defining harm causally that are solved by simply applying the defimition of actual causality given by Halpern [3] The issue of whether failing to take an action can be viewed as causing harm (e.g., can failing to water a neighbor's plants after promising to do so viewed as causing harm) can also be dealt with by using the standard definition of causality (which allows lack of an action to be cause).
We remark that Richens, Beard, and Thompson [22] (RBT from now on) also recently observed that using causality appropriately could deal with some of the problems raised in the harm literature !
They also give formal definition of harm that uses causal models, but it does not make use of sophisticated definition of actual causality such as the one given by Halpern [12 [3] (See Section S for comparison of our approach theirs and more discussion of this issue.) RBT focus on the more quantitative, probabilistic aspects of harm. We also believe that a quantitative account is extremely important; we offer such an account in (2). Conceptually though, the qualitative notion comes first: only after establishing whether or not there was harm does it make sense to ask how much harm occurred. Indeed, our quantitative account generalizes the qualitative account we develop here in
several ways (see Section 6
In any case, just applying the definition of causality does not deal with all problems. The other key step that we take is to assume that there exists a default utility. Roughly speaking, we define an event to cause harm whenever causes the utility of the outcome to be lower than the default utility. The default may be context- deeeedeteeeeeeett and there may be disagreement about what the default should be. We view that as feature of our definition. For example, we can capture the fact that people disagree about whether a doctor euthanizing patient in great pain causes harm by taking it to be disagreement about what the appropriate default should be. Likewise, the dispute between the family and the hospital described above can be modeled as a disagreement about the right default. Moreover, by explicitly bringing utility into the picture, we can connect issues that that have been discussed at length regarding utility (e.g.. what the appropriate discount factor to apply to the utility of future generations is) to issues of harm.
Causal Models and Actual Causality
We start with a review of causal models [15), since they play a critical role in our definition of harm. The material in this section is largely taken from [13. We assume that the world is described in terms of variables and their values. Some variables may have a causal influence on others. This influence is modeled by a set of structural equations. It is conceptually useful split the variables into two sets: the exogenous variables, whose values are determined by factors outside the model, and the endogenous variables, whose values are ultimately determined by the exogenous variables. The structural equations describe how these values are determined.
Formally, a causal model M is pair (8,F), where S is signature, which explicitly lists the endogenous and exogenous variables and characterizes their possible values, and F defines a set of (modifiable) structural equations, relating the values of the variables. A signature S is tuple (U,VV R), where U is set of exogenous variables, V is a set of endogenous variables, and R associates with every variable Y E U SUV V a nonempty set R(Y) of possible values for Y (i.e., the set of values over which Y ranges). For simplicity, we assume here that V is finite, as is R(Y) for every endogenous variable Y E V.. F associates with each endogenous variable X E V a function denoted Fx (i.e., Fx F(X)) such that Fx (XEER((U)) x (Xyev- (X) R(Y)) - R(X). This mathematical notation just makes precise the fact that Fx determines the value of X. given the values of all the other variables in 2 V.. The structural equations define what happens in the presence of external interventions. Setting the value of some set X of variables to Z in causal model M = (S, F) results in a new causal model, denoted M M which is identical to M, except that the equations for X in J are replaced by X = T.
Note that the causal models we consider here are deterministic. In general, one can also consider probabilistic causal models. A probabilistic causal model is tuple M = (S,F,
F,,r),,, Pr), where (S, F)
a causal model, and Pr is probability on contexts. Deterministic models suffice for offering a qualitative notion of harm, but we use probabilistic causal models for our quantitative generalization
Indeed, a talk by Jonathan Richens that discussed these issues (which are also discussed in 22 was attended by one of the authors of this paper, and it motivated us to look carefully harm.
3

---

The dependencies between variables in a causal model M = V.,. F) can be described using a causal network (or causal graph), whose nodes are labeled by the endogenous and exogenous variables in M, with one node for each variable in 4 U.. The roots of the graph are (labeled by) the exogenous variables. There is a directed edge from variable X to Y if depends on X; this the case if there is some setting of all the variables in 24 U V other than X and Y such that varying the value of X in that setting results in a variation in the value of Y; that is, there is a setting Z of the variables other than X and Y and values and x' of X such that Fy (z. ž) + Fy 2). A causal model M is recursive (or acyelic) if its causal graph is acyclic. It should be clear that if M is an acyclic causal model, then given context, that is, a setting iž for the exogenous variables in U, the values of all the other variables are determined (i.e., there is a unique solution to all the equations). We can determine these values by starting at the top of the graph and working way down. In this paper, following the literature, we restrict to recursive models.
We call pair (M, ū)) consisting of causal model M and a context iī a (causal) setting. A causal formula & is true or false in a setting. We write (M, ū)) = ab if the causal formula d is true in the setting (M, ū). The - relation is defined inductively. (M, u)) | X = r if the variable X has value a in the unique (since we are dealing with acyclic models) solution to the equations in M in context (that is, the unique vector of values for the exogenous variables that simultaneously satisfies all
equations in M with the variables in V/ set to ū). Finally, (M, ū + I <- 5ه if (My-gr ū)) F y.
A standard use of causal models is to define actual causation: that is, what it means for some particular event that occurred to cause another particular event. There have been a number of definitions of actual causation given for acyclic models (e.g.. M we say in the remainder of the paper applies without change to other definitions of actual causality in causal models, for definiteness, we focus here what has been called the modified Halpern- Pearl definition 13), which we briefly review. (See [3] for more intuition and motivation.)
15 13 16 17 26 27
Although most of what
The events that can be causes are arbitrary conjunctions of primitive events (formulas of the form X r);; the events that can be caused are arbitrary Boolean combinations of primitive events. To relate the definition of causality to the (contrastive) definition of harm, we give contrastive variant of the definition of actual causality; rather than defining what it means for X E to be (actual) cause of é, we define what it means for X = F rather than X it' to be cause of & rather than d'
Definition X = E rather than X = i' is an actual cause of o rather than o' in (M, i)) if the following three conditions hold:
ACI. (M, ū) -
o.
F AC2. There is a set W of variables in V and a setting i of the variables in W such that (M, it)
W- v and (M, -= =| - F, w w]ss where 6' da is valid.
AC3. X is minimal: there no strict subset X" of X such that X F" can replace X 3², where Htl is the restriction of i to the variables in X"
ACI just says that X = H cannot be considered cause of o unless both X z and o actually happen. AC3 is a minimality condition, which says that a cause has no irrelevant conjuncts. AC2 captures the standard but-for condition (X- = H rather than X = I' is a cause of o if, had X beem 4² rather than Z. Φ would not have happened) but allows us to apply it while keeping fixed some variables to the value that they had in the actual setting (M, Z)) If X H is an actual cause of & and there are two or more conjuncts in X = F, one of which is X = xr, then X = x is part of a cause of 6. In the special case that W = 0, we get the standard but-for definition of causality: if X = z had not occurred (because X was F' instead) & would not have occurred (because it would have been Φ').
The reader can easily verify that X = / is an actual cause of Φ according to the standard non- contrastive definition [13] iff there exist x² and ø' such that X = ã rather than X = x" is an actual cause of & rather than ø' according to our contrastive definition.
3 Defining Harm
Many definitions of harm have been considered in the literature. The ones most relevant to us are those involving causality and counterfactuals, which have been split into two groups, called the causal

---

account of harm and the counterfactual comparative account account of harm. Carlson et al. [ discuss many variants of the causal account; they all have the following structure:
An event e harms an agent ag if and only if there is a state of affairs 8 such that (i) e causes 8 to obtain, and (ii) is harm for ag.
The definitions differ in how they interpret the second clause. We note that although these definitions use the word "cause", it is never defined formally. "Harm" is also not always defined, although in some cases the second clause is replaced by phrases that are intended to be easier to interpret. For example, what Suits [24] calls the causal- intrinsii badness account takes to be a harm for ag if is "intrinsically bad" for ag.
The causal-counterfactue account (see, e.g., 9 the same, but now the second clause is replaced by a phrase involving counterfactuals, In its simplest version, this can be formulated as follows: s is a harm for ag if and only if ag would have been better off had 8 not obtained.
19 25
has the same structure; the first clause is
Even closer to our account is what has been called the contrastive causal-counterfauuue account. For example, Bontly 3 proposed the following:
An event e harms a person ag if and only if there is a state of affairs s and a contrast state of affairs s' such that (i) rather than a contrast event e' causes rather than s' to obtain, and (ii) ag is worse off in s than in s'.
Our formal definition of harm is quite close to Bontly's. s. We replace "state of affairs" by "outcomes", and associate with each outcome utility. This is essentially the standard model in decision theory, where actions map states to outcomes, which have associated utilities. Besides allowing us connect our view to the standard decision-theoretic view (see, e.g., [21 (31), this choice means that we can benefit from all the work done on utility by decision theorists.
To define harm formally in our framework, we need to both extend and specialize causal models: We specialize causal models by assuming that they include a special endogenous variable O for outcome. The various values of the outcome value will be assigned utility. We often think of an action as affecting many variables, whose values together constitute the outcome. The decision to "package up" all these variables into a single variable O here is deliberate; we do not want to consider the causal impact of some variables that make up the outcome on other variables that make up the outcome (and so do not want to allow interventions on individual variables that make up an outcome; we allow only interventions on complete outcomes). On the other hand, we extend causal models by assigning utility value to outcomes (i.e., on values of the outcome variable), and by having a default utility.
We thus take causal utility model to be one of the form M- = (, F.. u. d), where (2, V. F) is causal model one of whose endogenous variables O, u : R(O) + [o. 1] is a utility function on outcomes (for simplicity, we assume that utilities are normalized so that the best utility is and the worst utility 0), and d E [o, 1] is default utility ( As before, we call a pair (M, ii), where now M is causal utility model and if E RL)),, setting.
Just like causality, we define harm relative to setting. Whether or not an event X = F harms an agent in given setting will depend very much on the choice of utility function and default value. Thus, to justify particular ascription of harm, we will have to justify both these choices. In the examples we consider, we typically view the utility function to be ag's utility function, but we are not committed to this choice (c.g., when deciding whether harm is caused by parent not giving child ice cream, we may use the parent's 's definition of utility, rather than the child's one). The choice of a default value is more complicated, and will be discussed when get to examples; for the definition itself, we assume that we are just given the model, including utility function and default value.
The second clause of our definition is formalization of Bontly's 's definition, using the definition of causality given in Section 2 where the events for us, as in standard causal models, have the form X = and the alternative events have the form X = I", and they cause outcomes O = o and O o', respectively. Unlike Bontly's 's definition (and others), not only do we require that ag is worse off in outcome o (the analogue of state of affairs s) than in outcome (where "worse off" is formalized by
As we said in the introduction, in general, we think of the default utility as being context- -eeeexeteeeedeent so we really want function from contexts to default utilities. However, in all the examples we consider in this paper, a single default utility suffices, so for ease of exposition, we make this simplification here.
5

---

taking the utility to be lower), we also require the utility of to be lower than the default utility. There is also issue to whether consider there to be harm if X = I results in worse outcome than o. Since intuitions may differ here, we formalize this requirement a third clause, H3, and use it to distinguish between harm and strict harm. We will see the effects of our modifications to Bontly's definition when we consider examples in Section I
Definition 2 X = I harms ag in (M, ū) where M = (21, (,) F, u, there exist E R(O)
and T' R(X) such that
HI. u(O o) < d; and
H2. there exists df R(O) such that X E rather than X = I" causes O = o rather than = d and u(O o) < u(O d').
X strictly harms ag in (M, 1) if. addition,
HB. u(O = o) S u(O = d") ) for the unique d" E R(O) such that (M, 2)) - 1= In the special case where Definition 2 is satisfied for some value of appearing H2 such that u -
u(O o'), we say that X causes ag's utility be lower than the default.
is important to point out that it is quite rare for harm and strict harm to come apart. For one,
requires causation and but-for causation come apart (otherwise of d"). In addition, it requires O to have at least three values (otherwise again of o"). Lastly, even if these two conditions are met, we also need that u(O o") u(O = o) < u(O d). We discuss one example in the supplementary material (see Section ??) in which these conditions are met.
As with most concepts in actual causality, deciding whether harm occurred is intractable. Indeed, it is easy to see that it is at least as hard as causality, which is DP-complete [12]. However, this is unlikely to be problem in practice, since expect that the causal models that arise when want to deal with harm will have few variables, which take on few possible values (or will involve many individuals that can all be described with by small causal model), we decide harm by simply checking all possibilities.
It is useful to compare our definition with the counterfactual comparative account of harm. Here is, translated into our notation:
Definition 3 X H counterfactually harms ag in (M, i). where M ((U, V.,, d. there exist 0, o' R(O) and j' R((X) such that
CL. (M, 7) = X 7 =
C2. (M, d) = [ + (0
C3. u(O = o) < cu(0
That is, X = - counterfactually harms ag if, for some x' and o', X 5 is what actually happens (CI), O o' would have happened had X been set 3" (C2), and ag gets higher utility from of than from (C3). C1 and C2 together are equivalent to ACI and AC2 in the special case that W = 0. That is. CI and C2 essentially amount to but-for causality. C3 differs from our conditions by not taking into account the default value.
Note that Definition 3 has no analogue of AC3, but all the examples focus on cases where X is actually singleton, so AC3 is trivially satisfied. The key point from our perspective is that the counterfactual comparative account considers only but-for causality, and does not consider a default value. The examples in the next section show how critical these distinctions are.
As mentioned earlier, RBT recently developed a formal account of harm using causal models. While their account is probabilistic and quantitative, we can consider the special case where everything is deterministic and qualitative. When we do this, their account reduces to strengthening Definition 3 that brings it somewhat closer to our account: they also suggest using defaults, but have default
actions rather than default utilities. In their version of Definition 3 X is taken to be the variable representing the action(s) performed and x' the default action. order deal with the limitations of but-for causality, RBT offer more general account (see their Appendix A) that uses path-specific causality, instead of actual causation. This makes their account different from ours in some significant respects; see Section 5

---

4 Examples
We now analyse several examples to illustrate how our definition handles the most prominent issues that have been raised in the literature on harm. Bradley 4 p. 398] identifies two such issues that strike him "as very serious", namely the problem of preemption, and the problem of distinguishing harm from merely failing to benefit. These problems therefore serve as a good starting point.
4.1 Preemption
To anyone familiar with the literature on actual causation what follows will not come as a surprise. I Lewis used examples of preemption to argue that there can be causation without counterfactual dependence (i.e., we need to go beyond but-for causality); this conclusion is now universally accepted. Essentially the same examples show up in the literature on harm: cases of preemption show that an event can cause harm even though the agent's well-being does not counterfactually depend on it. Thus, the counterfactual comparative account of harm fails for the same reason it failed for causality. The good news is that the formal definition of causation (by design) handles problems like preemption well; moreover, the solution carries over directly to our definition of harm. The following vignette is due to Bradley [4, but issues of preemption show up in many papers on causality [ 15 13 17 [6]; all can be dealt with essentially the same way.
Example 2 (Late Preemption) Suppose Batman drops dead of a heart attack. A millisecond after his death, his body's is hit by a flaming cannonball. The cannonball would have killed Batman if he had still been alive. So the counterfactual account entails that the heart attack was not harmful to Batman. It didn't make things go worse for him. But intuitively, the heart attack was harmful. The fact that he would have been harmed by the flaming cannonball anyway does not seem relevant to whether the heart attack was actually harmful.
In terms of the formal definition, we take H to represent whether Batman has a heart attack (H- = 0 if he doesn'ts tt; H = 1 if he does), C' to represent if Batman is hit by cannonball, and D to represent whether Batman dies. Let be the context where H = 1. Even without describing the equations, according to the story, (M, ii) + H = 1A D = 1 A H = ( = 1): Batman has a heart attack and he dies, but he would have died even if he did not have a heart attack (since he would have been hit by the cannon ball). Thus, C3 does not hold, since o d': the outcome is the same whether or not Batman has a heart attack.
The standard causal account handles this problem by introducing two new variables: K, for "Batman is killed by the cannonball", and S, for "Batman died of a heart attack", to take into account the temporal asymmetry between death due to a heart attack and death due cannonball. (We could also deal with this asymmetry by having -tampe" variables that talk about when Batman is alive. For more details on incorporating temporal information by using time- stamped variables, see [3. The causal model has the following equations: D = V K (i.e., D = 1 if either S = 1 or K 1: Batman dies if he has a heart attack the canonball kills him); S H (Batman's heart stops if he has a heart attack); and K = - A C (Batman is killed by the canonball if the canonball hits him and his heart is still beating). We now get that Batman's 's heart attack rather than its absence is a cause of him being alive rather than dead. Clearly (M, i7) + H = I A D = 1. If we fix K = O (its actual value, since the cannonball in fact does not kill Batman; he already dead by the time the cannonball hits him), then we have that (M, 7)) }= [ = 0, K = Oj(D = 0). so AC2 holds. Thus, the causal part of H2 holds. (See 13 Example 2.3.3] for a detailed discussion of an isomorphic example.)
If we further assume, quite reasonably, that Batman prefers being alive to being dead (so the utility of being alive is higher than that of being dead) and that the default utility is that of him being alive, then H1 and H2 hold. Thus, our definition of harm avoids the counterintuitive conclusion by observing that Batman's 's heart attack caused his death, thereby causing the utility to be lower than the default. t
Our analysis of preemption is indicative of the more general point that many of the issues plaguing the literature on harm can be resolved by making use of causal models and the definitions of causation that they allow. Causal models allow a more precise and explicit representation of the relevant causal structure, thereby forcing a modeler to make modeling choices that resolve the inherent ambiguity that comes with an informal and underspecified causal scenario. Obviously such modeling choices can be the subject of debate (see [14] for discussion of these modeling choices). The point is not that using causal models by itself determines a unique verdict on whether harm has occurred, but rather that such debate cannot even be had without being explicit about the underlying causal structure.
7

---

4.2 Failing to Benefit
One of the central challenges in defining harm is to distinguish it from merely failing to benefit. Although most authors define benefit simply as the symmetric counterpart to harm, we do not believe that this is always appropriate; we return to this issue in 22 where we consider more quantitative notions of harm. But for the current discussion, we can set this issue aside: what matters is that merely failing to make someone better off does not in itself suffice to say that there was harm. Carlson et al. present the following well-known scenario to illustrate the point.
Example 3 (Golf Clubs) Batman contemplates giving set of golf clubs to Robin, but eventually decides to keep them. If he had not decided to keep them, he would have given the clubs to Robin, which would have made Robin better off.
By keeping the golf clubs, Batman clearly failed to make Robin better off. The counterfactual account considers any such failure to result in harm. Indeed, it is easy to see that C1-C3 hold. If we take GGC to represent whether Batman gives the golf clubs Robin (GGC = 1 if he does; GGC 0 if he doesn') 1) and the outcome O to represent whether Robin gets the golf clubs (O = 1 if he does; O O if he doesn't), t), then GGC further assume that Robin's 's utility of getting the golf clubs is higher than his utility of not getting them, then C3 holds. Yet sounds counterintuitive claim that Batman harmed Robin on this occasion.
is a but-for cause of GGC O, so CI and C2 hold. If we
Although H2 holds in our account of harm (for the same reason that C1-C3 hold), we avoid the counterintuitive conclusion by assuming that the default utility is u(O = 0). so HI does not hold. This seems to us reasonable; there is nothing in the story that suggests that Robin is entitled to expect golf clubs. On the other hand, if we learn that Batman is a professional golfer, Robin has been his reliable caddy for many years, and that at the start of every past season Batman has purchased a set of golf clubs for Robin, then it sounds quite plausible that the default is for Robin to receive set of golf clubs. With this default, HI does hold, and our definition concludes that Robin has been harmed. Thus our account can offer different verdicts depending on the choice of default utility, As we said in the introduction, we view this flexibility as a feature of our account. This point is highlighted in the following, arguably more realistic, scenario. (RBT make exactly the same point as we do when they analyze such examples [2 p. 15).)
Example 4 (Tip) Batman contemplates giving tip to his waiter, but eventually decides to keep the extra money for himself. If he had not decided to keep it, he would have given it to the waiter, which would have made the waiter better off.
To those living in the US does not at all sound counterintuitive to claim that Batman harmed the waiter, for his income substantially depends on receiving tips and he almost always does receive a tip. Indeed, if we take the default utility to be that of receiving a tip, then in this example, the waiter is harmed by Batman not giving a tip. By way of contrast, in countries in Europe where a tip would not be expected, seems to us reasonable to take the default utility to be that of not receiving a tip. In this case, the waiter would not be harmed.
Examples and 4 are isomorphic as far as the causal structure goes; we can take the utilities to be the same as well. This means that we need additional structure to be able to claim that the agent is harmed in one case and not the other. That additional structure in our framework, which we would argue quite natural, the choice of default utility. Note that neither scenario explicitly mentions what the default utility should be. We thus need to rely on further background information to make a case for a particular choice. There can be many factors that go into determining a good default. We therefore do not give general recipe for doing so. Indeed, as we pointed out in the introduction with the euthanasia example, reasonable people can disagree about the appropriate default (and thus reach different conclusions regarding harm).
4.3 Preventing Worse
There exist situations in which the actual event rather than an alternative event causes a bad outcome rather than good outcome, but the alternative results in even worse outcome. Because of the latter, we do not consider these situations to be cases of strict harm, due to condition H3 in Definition From the perspective of the manufacturer, this is precisely what is going on in starting Example I but Bob might disagree. We now take a closer look at this example to bring out the conflicting perspectives.
2
8

---

Example 5 (Autonomous Car) Let O be three-valued variable capturing the outcome for Bob, with the utility defined as equal to the value of O. O = 0.5 stands for the injury resulting from crashing into the safety fence, and potentially more severe injury resulting from crashing into the deshinggg
stationary car is captured by O = Bob not being injured is O =1.
Recall that the system has the built-in standard that the driver's 's reaction time is 10 seconds, which too long to avoid colliding into the stationary car. Imagine the manufacturer implemented this standard by restricting the system's 's actions in such cases to two possibilities: do not intervene
(F = 0) or drive
drive into the fence (F = our Late Preemption example (Example 2), for hitting the fence preempts the collision with the stationarty car. We therefore add a variable capture the asymmetry between hitting the fence and hitting the stationary car: FH and CH respectively. The equation for O is then such that O = 1 if
1). =). This means that the causal structure is very similar to
CH = 0, O = 0.5 if F H 1, and = 0 if CH = and FH = 0. FH - CH--
FH
CHH
Figure I: Causal graph for Ex. 5
As the autonomous car drives towards the fence only because there is a stationary car, the equation for F is F C' (where C' represents the presence of the car). The fact that hitting the fence prevents hitting the car is captured in the equation for CH: CH = C A FH.. Lastly, we have FH = F. The context is such that F = 1 and C = 1, and thus FH = 1, O = 0.5, and CHI =.
Did the system harm Bob? Carlson et al. [3] answer this in the negative for an example that is essentially the same as this one (see their MMany Threats" example) and use this verdict to argue against the nuusal-counterfactuu account, which reaches the opposite verdict. According to them, the contrastive causal causal-counterfin cuusal-counterfactull account does not reach this verdict because there is no contrastive causation here, but as they do not give definition of causation, it impossible to reconstruct how they arrive at this verdict. In any case, what matters for our purposes is that our definition of causation does consider the system's s hitting the fence rather than not intervening to cause Bob being mildly injured rather than not being injured at all. To see why, observe that taking W to be CH, we get that F = 1 rather than F = 0 causes O = 0.5 rather than O = 1: (M, 2)) + [F + 0, CH + 0jO 1.
Therefore, if we assume that the default utility is that of Bob not being injured, conditions H1 and H2 are satisfied and there is harm. Notice though that F = 1 rather than F = 0 is also but-for cause of O = 0.5 rather than O = 0, that is, Bob's being mildly injured rather than severely injured counterfactually depends on the system's 's action. This is where condition H3 kicks in: it ensures that we do not consider there to be strict harm caused if the alternative would have resulted in an even worse outcome. Thus, the car manufacturer could make the case that although their policy harmed Bob, it was justified in doing so.
More generally, it is an easy consequence of our definitions that in cases where X á rather than X H² causes harm but not strict harm, the alternative event would have resulted in strict harm, i.e., X = x" rather than X = ø would have caused strict harm. As a result, it is sensible in such cases for someone to argue that they were justified in causing harm, as the alternative would have been worse.
Bob, on the other hand, believes he has been strictly harmed, because he claims that he could have prevented the collision if he had been alerted. This disagreement can be captured formally by stating that Bob using a talued variable F instead of binary one, where the third option (F = 2) prnceppone to alerting Bob. varporatinng this variable into the model (and assuming that Bob is sneecctti regarding his driving skills) we would again get that = rather than F = 2 causes O = 0.5 rather than O 1, but with the important distinction that H3 is satisfied for these contrast values and thus the system's action does strictly harm Bob. Our analysis does not resolve the conflict (and is not meant to do so), instead it allows for precise formulation of the source of the disagrcement. D
correspor correct

---

In the supplementary material, we describe four more examples illustrating multiple contrasts for the outcome, the role of the choice and the range of variables and the choice of default, and our rationale for considering contrastive definition, rather than causal-counterfactuall one.
5 Comparison to RBT
In this section, we do a more careful comparison of our approach and that of RBT. RBT focus on choices made by an agent, where these are choices of what action to take, and assume that there is a default action, to which they compare the choice made by the agent. It follows easily from RBT's definition that if the agent performs the default action, there is no harm. Yet there are many instances in which doing what is morally preferable causes harm, albeit accidentally. Simply imagine a doctor prescribing medication to a patient, and the patient unfortunately suffering a very rare allergic reaction to the medication, where the reaction is far worse than the initial condition that the patient had. Then clearly the doctor harmed the patient. The most obvious choice of default action here is the actual action (and, in fact, RBT themselves mention following "clinical guidelines" as example of default action in their Appendix C). But this means that according to RBT's definition there would not be harm here. Although we use default utility, there is no need for this default utility to be the utility of a default action, so we do not have this problem.
Another significant difference between our approach and that of RBT is that, although RBT use causal models, unlike us, they do not use sophisticated definition of actual causality such as the one given by Halpern [12 1 In their Definition 3, RBT consider fut-for causality. Not surprisingly, this will not suffice to deal with problematic examples where the more general notion of causality is needed (see, e.g., Example 2) In their Definition 9, they generalize Definition 3 to allow for path-dependent causality. They do not explain how they choose which paths to consider, but both Example and the corresponding example of late preemption they consider, by choosing the appropriate paths, they can simulate the effects of AC2. (Specifically, they can simulate the effect of choosing set W of variables and fixing them to their actual values.) As a consequence, they get the same results as those obtained by Halpern's definition of actual causality. It is not clear whether this will always be the case. More importantly, the ability to determine harm relative to some choice of paths gives the modeler significant extra degree of freedom tailor the results obtained. We believe that if paths are going to be used, there needs to be a more principled analysis of how to go about choosing them.
6 Conclusion
We have defined a qualitative notion of harm, and shown that it deals well with the many problematic examples the philosophy literature. We believe that our definition will be widely applicable in the regulatory frameworks that we expect to be designed soon in order to deal with autonomous systems.
Of course, having qualitative notion of harm is only first step. For practical applications. we often need to quantify harm; for example, we may want to choose the least harmful of a set of possible interventions. As we said, we develop a quantitative notion of harm in 22. While one could just define a quantitive notion that considers the difference between the utility of the actual outcome and the default utility (this is essentially what RBT do), we believe that the actual problem is more nuanced. For example, even if we can agree on the degree of harm to an individual, if there are many people involved and there is probability of each one being harmed, should we just sum the individual harms, weighted by the probability? We argue that this is not always appropriate, and discuss alternatives, drawing on work from the decision-theory literature.
Acknowledgements
The authors would like to thank the NeurIPS reviewers for their detailed comments and Jonathan Richens for fruitful discussion of preliminary version of this paper. Sander Beckers was supported by the German Research Foundation (DFG) under Germany's 's Excellence Strategy EXC number 2064/1 - Project number 390727645, and by the Alexander von Humboldt Foundation. Hana Chockler was supported part by the UKRI Trust-worthy Autonomous Systems Hub (EP/V00784X/1) and the UKRI Strategic Priorities Fund to the UKRI Research Node on Trustworthy Autonomous Systems Governance and Regulation (EP/V026607/1). Joe Halpern was supported part by NSF grant IIS- 110-17,,44, ARO grant W91 INF-22-1-006) and MURI grant W911NF-19-1-0211
10

---

References
[1] S. Beckers. Causal sufficiency and actual causation. Journal of Philosophical Logic, 50:
1341-1374, 2021.
[2] S. Beckers, H. Chockler, and J.Y. Halpern. Quantifying harm. arXiv, 2022. URL https //arxiv.org. wrgg/b/////a/////// 15111
[3] Bontly, Causes, contrasts, and the non-identity problem. Philosophical Studies, 173:1233-
1251, 2016.
[4] B. Bradley. Doing away with harm. Philosophy and Phenomenological Research, 85:390-412, 2012.
[5] E. Carlson, Johansson, and O. Risberg. Causal accounts of harming. Pucific Philosophical Quarterly, 2021.
[6] Law Commission. automated vehicles announced.
Legal
reforms
to
allow safe
introduction of https://wwed wwwwom.wwmmmmmmo gov uk/
legal to--ttt-llllln of 2022.
announced/
[7] European Commission. Proposal for a regulation of the European parliament and the council laying down harmonised rules on artificial intelligence (Artificial Intelligence Act) and amending certain Union legislative acts, 2021. https//attiicallnelllieenaaaccut autt accessed Aug. 8, 2021.
[8] N. Feit. Harming by failing to benefit. Ethical Theory and Moral Practice, 22:809 823, 2019.
[9] M. Gardner. A harm-based solution to the non-identity problem. Ergo: An Open Access Journal of Philosophy, 2:427-444, 2015.
[10] C. Glymour and F. Wimberly. Actual causes and thought experiments. In J. Campbell, M. O'Rourke, and H. Silverstein, editors, Causation and Explanation, pages 43 67. MIT Press, Cambridge, MA, 2007.
[11] N. Hall. Structural equations and causation. Philosophical Studies, 132: 109-136,,, 2007.
[12] J. Y. Halpern. modification of the Halpern- Pearl definition causality. In Proc. 24th International Joint Conference on Artificial Intelligence (IJCAI 2015), pages 3022-3033, 2015.
[13] J. Y. Halpern. Actual Causality. MIT Press, Cambridge, MA, 2016.
[14] J. Y. Halpern and C. Hitchoock. Actual causation and the art of modeling. In R. Dechter, H. Geffner, and J.Y. Halpern, editors, Causality, Probability, and Heuristics: A Tribute to Judea Pearl, pages 383-406. College Publications, London, 2010.
[15] J. Y. Halpern and J. Pearl. Causes and explanations: British Journal for Philosophy of Science, 56(4):843-855 887, 2005.
moneturralmodell approach. Part I: Causes.
[16] C. Hitchcock. The intransitivity of causation revealed in equations and graphs. Journal of Philosophy, XC) 299, 2001.
[17] C. Hitchcock. Prevention, preemption, and the principle of sufficient reason. Philosophical Review, 11::495-532, 2007.
[18] BBC News. 'self-driving' cars to be allowed on uk roads this year. httpss//www.be uk tk///ees//eeccmooogg-5566655
2022.
[19] R. Northcott. Harm and causation. Utilitas, 2:147-164, 2015.
[20] National Library of Medicine. Hippocratic oath. https: nlm. nih. maih.gore www.:ni......nm.. www.hoogo/oom/gggggeeee / greek_ carkkkoath.heee html 2002.
[21] M. Resnik. Choices: Introduction Decision Theory. University of Minnesota Press, Minneapolis, 1987.

---

[22] J. G. Richens, R. Beard, and D. H. Thompson. Counterfactual harm. arXiv, 2022. URL
https://arxiv..nm http://rg//pp//////////// 12993. 219993.pd. pdf
[23] L. J. Savage. Foundations of Statistics. Wiley, New York, 1954.
[24] D. B. Suits. Why death is not bad for the one who died. American Philosophical Quarterly, 38: 69 84, 2001.
[25] J. Thomson. More the metaphysics of harm. Philosophy and Phenomenological Research, 82:436 454466448..10 2011.
[26] B. Weslake. A partial theory of actual causation. British Journal for the Philosophy of Science, 2015. To appear.
[27] Woodward. Making Things Happen: Theory of Causal Explanation. Oxford University Press, Oxford, U.K., 2003.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material URL)? [N/A]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [N/A]
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [N/A]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [N/A]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [N/A]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material as URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
12


Reviews:
[{"review_id": "yGK1en9oAbv", "reviewer": "Reviewer_7g7A", "summary": "This paper proposes a way to define **harm** in a causal way. The work in more on of a philosophical level, trying to criticize the fact that the definition of harm in philosophical studies is generally devoid of causality and putting forward the notion that the use of causality theory can define a more general definition of harm. Several examples are presented throughout the paper that show how the causal analysis and a new definition of harm can handle several standard examples from literature and can thus be used for effective reasoning in various domains and situations.", "questions": "In addition to some important points raised in the weaknesses section, I have a couple of specific questions:\n\n1. How will the definition of harm be extended/modified if we have dependent exogeneous variables in the causal graph?\n\n2. How are confounders handled in the causal definition of harm?", "limitations": "This is tricky. Although I do not see any specific negative societal impact, defining and constraining harm to a single definition might be a little bit risky. As I have mentioned in my review before, each can have his/her own interpretation with respect to the definition.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. The paper tackles a very important and sensitive problem of defining harm. Since this can have multiple interpretations and can be conceived by different people in unique ways, trying to  come up with a causal-standard definition is indeed a nice effort.\n\n2. The paper is written beautifully and the examples are crystal clear. In my opinion, even a person with little knowledge of causality will be able to follow the paper and get an overall idea about the main message of this work.\n\n3. The definition of **harm** in section 3 seems well thought out and well sketched.\n\nWeaknesses:\n\n1. The method seems to have a very heavy dependence on the default utility. \n\n2. I do not agree with the Batman and Robin example, where the definition concludes that Robin was harmed. Just because Robin receives a new set of clubs at start of every season, it still does not make him entitled to receive it the next year as well. There can be the effects of some hidden confounders that are being ignored here.\n\n3. The default utility of receiving or not receiving a tip in **Example 4** seems incomplete. For example, what about the cases where the exact amount of tip is also important? Say the waiter is better off if he/she receives 15% of the total bill as a tip instead of 5% in the US. \n\nOverall, this is an interesting take combining philosophical arguments with causality in a more explicit way. I would be eager to see the author's response to the raised points.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "mfOabWOLdZL", "reviewer": "Reviewer_f27v", "summary": "In this paper, the authors proposed the first qualitative causal framework that is used to define the harm and conceptually show their framework can solve various controversial real-life examples in terms of minimizing the harm. ", "questions": "The authors applied the framework of harm to several real-life scenarios. However, one thing in common is that all the scenarios consider defining harm for a single agent. However, in reality, the harm is usually multi-agents based. Without a clear quantitative suggestion for the utility function, it is hard to minimize the total harm. Even without considering the dependency among the agents, simply using the framework proposed by the author to make the decision could be challenging (Consider the Trolley problem). I suggest the authors consider how to address this issue or explicitly state this limitation.\n ", "limitations": "The utility function u which measures the goodness of the decision is a vague term. In reality, the way of defining such a utility function could invoke controversy and fairness issues, which have also been discussed by the author. ", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\nIt is by far the first literature, as far as I know, that attempts to discuss forming the framework for evaluating harm. It also discussed how to apply this framework to several real-life problems. \n\nWeakness:\n\nThough I understand the difficulty of quantitatively defining the utility function, the lack of discussion of it could discourage its application to reality, I will discuss it in detail in the questions session. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["Discrimination / Bias / Fairness Concerns"]}, "extra_text": {}}, {"review_id": "Yp8s1NQzAd4", "reviewer": "Reviewer_2wZ6", "summary": "[REMARK: This is an emergency review, that is, the reviewer did neither select autonomously into this paper out of interest/expertise nor was the reviewer granted with sufficient time (with respect to standard time frames) to appropriately provide a review. Please take the review with caution, the confidence level will be adjusted accordingly.]\n\nThe paper presents a formalization of the concept of \"harm\" (to harm someone) using structural causal models (SCM) as found in the counterfactual theory of causality vocally discussed by Judea Pearl. Specifically, the authors extend SCMs to cover a \"default utility\" (referred to as causal utility model) to then propose 3 conditions (H1 to H3) which captures (defines) whether an even $\\overset{\\rightarrow}{x}$ causes harm to an agent $\\mathbf{ag}$. The authors provide a motivation, a comparison to other definitions (in terms of their presented formalism) and finally discuss several philosophical key examples displaying the advantages of their proposed definition.", "questions": "The questions are derived mostly from aspects mentioned in the \"Weaknesses\" section. I'd invite the authors to answer these questions to overcome the observed weaknesses for improving the paper to ultimately increase the chances of both contribution and visibility to the community. The list of questions is unordered and they range in quality and speed of expected answer (some are minor, quickly answerable whereas others are more fundamental, crucial, maybe difficult to answer):\n* How can we cope with temporal notions?\n* How can we overcome intractability of harm computation, in the face of our necessity to compute since otherwise a formalized notion would have no practical implications?\n* Under which circumstances can our causal model guarantee that our harm computation will be approximately correct?", "limitations": "No contradictions or any sort of relevant mistake have been detected in the paper. The overall clarity of the paper is a clear advantage. Existing bodies of work are being referenced accordingly throughout the paper.\n\nThere is nothing to be reproduced, therefore, naturally, there is also no code. Societal impact is being discussed throughout.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\nDiscussing and overall researching into harm and how to capture/measure it is certainly timely in the advent of discussions around AGI alignment etc. The same holds for the intention of deriving a causal account since causal models allow for an explicable, interpretable modelling scheme. As the authors suggest, one can only debate about assumptions if said modelling assumption are recognizable. The presented formalization copes well with the discussed examples, which cover key philosophical dilemmas. Capturing formally, with thought through notions, otherwise intangible concepts like harm is an important step towards accountable intelligent systems.\n\n**Weaknesses**\n\nWhile one can question on a more general note whether discussing this topic within NeurIPS (especially in the presented form) is sensible, I believe the importance on a societal scale for such a work is undeniable and NeurIPS with both its breadth and influence would fit as a great platform. To comment on the paper itself more importantly, the presentation sets its focus on the formal account of harm, and while it does motivate generally and discuss specifically, the presentation (of otherwise very visual/intuitive ideas) is arguably dry. That is, to convey the ideas or spread the formalized notion (which is likely the key interest of the authors) alternate means of presentation using illustrations with figures but also visualizations of possibly synthesized versions of the examples would help in doing so. Regarding the mathematical account, the lack of analysis for induced properties (when is the computation feasible, what models do satisfy the condition in the first place, etc.) is a weakness (albeit a likely difficult one to fix since it is difficult enough already to capture a philosophical core problem that has existed for centuries mathematically).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "7iKzN2OQtSc", "reviewer": "Reviewer_6SwV", "summary": "The paper introduces a formal definition of harm that builds upon the modified Halpern-Pearl definition of actual causation. The main differences between the definition and previous definitions in the literature are the new concepts introduced in the *causal utility model*, which adds 1) a utility function $\\mathbf u : \\mathcal R(O) \\to [0, 1]$ for some $O \\in \\mathcal V$ and 2) a \"default utility\" $d \\in [0, 1]$ to the standard causal model. The authors introduce 7 examples of situations where an agent is or is not caused harm that are prominent in the literature. They argue that their definition of harm either correctly identifies the presence or absence of harm for each of these examples, or correctly identifies the precise source of ambiguity regarding the presence or absence of harm.", "questions": "Questions:\n1. Why is the causal utility model defined and used, rather than Halpern's extended causal model in the definition of harm? It appears to me that harm as defined in the paper is equivalent to causation under an extended causal model with a particular partial preorder.\n2. Is it possible for harm to occur without causation? If not, why is this? Is it possible to introduce a result demonstrating that harm should occur due to causation?\n3. It seems the notion of harm and blame should be related; do you have thoughts about this? Shouldn’t the definition of harm incorporate the mental model of the agent taking an action that may have harmed the agent being harmed and that of the agent being harmed?\n4. The authors emphasize that the definition of harm proposed in the paper is \"qualitative\", while RBT's definition is quantitative. I am not sure this is clear, can you explain this distinction and provide further elaborations on how to think about it?\n5. If the definition of harm is qualitative, why does the causal utility model use a utility function that maps to a number in $[0, 1]$, rather than one that induces an ordering (or partial preorder) over $O$? As the authors note, it is unclear how to combine the numerical utilities of multiple agents. Is there a reason a mapping to $[0, 1]$ must be used rather than an ordering?\n6. This definition of harm seems quite interesting, and I am a bit intrigued to know if counterexamples to the definition already exist (similar to the ones I showed in the limitations section). ", "limitations": "I do not see potential for negative social impact of the work, as the work is aimed at formally defining harm - research which may help autonomous systems reason about minimizing harm in the future. Thus, I list some limitations of the work I observed below.\n\n## Insufficient evaluation\n\nThe positive examples provided do not provide any guarantees that there are no counterexamples to the definition, and there are only 7 positive examples provided with no extensive human evaluation (by laypeople or by experts). I list two possible counterexamples below and would be curious to hear the authors’ thoughts.\n\n## The relationship between harm and responsibility of agents is not considered by the work\n\nThis criticism is aimed at the fact that the definition proposed in the work does not consider the intention of the agents.\n\nConsider the following example:\n\n> A driver honks her car horn at the driver in front of her, startling an apartment resident dangling his legs off of his balcony railing. The resident falls off the balcony in fright, becoming gravely injured upon hitting the ground.\n\nIt seems strange to claim that the driver harmed the resident, because 1) she could not have known that the resident was in danger of falling 2) even if she did know, the resident is partially at fault for willfully placing himself in a precarious situation. To make the example even more clear-cut:\n\n> An apartment resident decides that he will jump off his balcony upon the first car honk he hears. A driver honks her horn at the driver in front of her. Hearing this horn honk, the resident jumps off the balcony, becoming injured upon hitting the ground.\n\nNow, it seems even stranger to claim that the driver harmed the resident, because the resident took an action intended to harm himself, while the driver did not.\n\nThis issue could be resolved by using Halpern's concept of blame introduced in (Chockler and Halpern, 2014 - https://www.aaai.org/Papers/JAIR/Vol22/JAIR-2204.pdf). The extended causal model could also solve this problem.\n\nThe idea of blame is also touched upon in Example 7. If Victoria knew she was able to rescue Betty without breaking her arm ($P = 2$) and chose to break her arm while rescuing her ($P = 1$), she caused Betty harm, because this goes against social norms. However, if she did not know she'd be able to do so and accidentally broke her arm ($P = 2$ was not an option), it is hard to argue that she caused Betty harm, as attempting to rescue Betty is well within social norms, and Betty's utility increased overall.\n\n## Causing a decrease in utility in a socially unacceptable way (violating a default) is sufficient but not necessary for harm to occur\n\nThis criticism is aimed at the use of the \"default utility\" to represent all norms.\n\nThe work does not allow for situations where harm is done but reparations are made, and the agent in question is overall better off. For example: \n\n> A thief assaults and robs a passerby but later, out of a guilty conscience, gives the money back, adding some additional money to more-than compensate for any physical and emotional distress caused. The passerby is happier than they would have been otherwise, because they needed the money.\n\nWhile the passerby is better off by the end of the story, the thief did cause harm to the passerby in the intermediate step of assaulting and robbing them. While this issue *could* be addressed by stating that the default world is the world where the thief does not assault/rob and gives the passerby money for no reason, this seems to be a difficult argument to make, as the thief is expected by social norms only to not commit any crimes. Setting to default world to the world where the thief gives money to the passerby would mean that *everyone* is harming the passerby by not giving money to them.\n\nAn even more clear-cut version of this example follows:\n\n> A thief assaults and robs a passerby. After the local media reports on the crimes committed, many people donate money to the passerby in sympathy. The passerby agrees that he is better off now than he would have been had he not been robbed and assaulted.\n\nIt seems that the thief's actions, contrary to his expectations, helped the passerby overall. Is it still true that the thief did not cause harm to the passerby?\n\nThe issue could also be fixed by using the extended causal model to, regardless of utility, label any world where the thief commits a crime as less normal than a world where the thief does not commit a crime.", "rating": 3, "confidence": 4, "soundness": 1, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Originality**: The paper's main idea of introducing a causal definition of harm follows (Bontly, 2016). The novelty of the method lies in the fact that the definition introduced is formally stated using Halpern and Pearl's language of causal models. ~However, the novel components that the new definition introduces – default utility + the observation that preventing a worse outcome does not constitute harm – do not seem original, as they seem to be subsumed by the extended causal model for the purpose of determining whether or not harm was caused. (Halpern and Hitchcock, 2011 - https://arxiv.org/pdf/1309.1226.pdf). The concepts of the default world and utility function could be naturally included in the partial preorder if the previous definition of extended causal models were used. This would mean that harm may be defined as modified Halpern-Pearl causation under the extended model, using a particular partial preorder. Perhaps the authors could comment on and clarify this point, in case I am missing any subtlety here.~ (retracted, see \"Reply to reviewer's first response\", parts 1 and 2 below)\n\n**Quality**: ~The solution to the proposed research question ignores or misses prior work, as discussed above, and while~ While the solution to the proposed research question resolves an issue posed to a previous attempt in the literature made at defining harm using causation, the proposed definition lacks sufficient justification. ~Why would we need this specific definition instead of using the already existent one? Further, there~ There are no theoretical results introduced that demonstrate that counterexamples do not exist, and neither extensive human evaluation of the definition. Some potential counterexamples to the proposed definition are provided in the limitations section of this review.\n\n**Clarity**: Overall, the paper is clearly written. The motivation of the paper is clear, as is the proposed definition, the example formulation, and the comparison of the definition's evaluations on the examples to those of the prior work mentioned in the paper.\n\n**Significance**: Formalizations of harm become increasingly important as automated systems grow in ubiquity. Thus, the problem addressed by the paper and the idea of using actual causation to formalize the problem are important to the responsible development of large-scale AI systems. Overall, the motivation behind the paper is greatly appreciated. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "4bvF5KVbNgO", "reviewer": "Reviewer_RbEE", "summary": "The work proposes a qualitative definition of harm based on causal models in the context of dealing with autonomous AI systems. The authors state that the notion of harm is ill-defined and motivate their qualitative approach on the observation that much of the discussion of harm is qualitative. After reviewing causal models and describing the introduced definition of harm, the work applies and discusses it (including its limitations) on various examples.", "questions": "See W1 and W2 above.", "limitations": "The authors adequately addressed the limitations.", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n- S1: The definition is inspired by previous work (RBT, Bontly) and extends them to a causal definition.\n\n- S2: Besides some minor issues (see following weaknesses), the proposed definition is well described and understandable. \n\n- S3: The extensive examples, including the discussions contained in the main text and the supplement, give a great overview and intuition of the proposed definition.\n\n**Weaknesses**\n- W1: While building on Y. Halpern [11] and Y. Halpern and J. Pearl. [13] working with the concept of 'actual causality', the paper might be improved by briefly discussing differences to the common approach of probabilistic structural causal models [X1, X2], where variables are considered to be sampled from probability distributions with structural equations including random noise terms. Especially recursive deterministic computation of variables as described in the paper must assume that no noise terms are included in the system. The authors might consider to state this assumption explicitly.\n\n- W2: Furthermore, the relations between exogenous and endogenous variables (line 112) seems to be more strict than the standard definition using 'Pearlian causality'. E.g. compare to [X2] p.23 “Endogeneous variables are those that the modeler tries to understand, while exogenous ones are determined by factors outside the model, and are taken as given.” While similar in the intention, Peters et al. give no definition of how to treat the relationship between endogenous and exogenous variables. Assuming endogenous variables to be fully determined by exogenous factors is a strong assumption that, again, only holds under the assumption that all exogenous variables are known and observed and no additional random noise terms are influencing the system.\n\n\n[X1] Pearl, J. (2009). Causality (Cambridge university press)\n\n[X2] Peters, J., Janzing, D., and Schölkopf, B. (2017). Elements of causal inference: foundations and learning algorithms (The MIT Press)\n\n\n**Minor comments**\n\n- Typo line 104 \"that that\"\n- Typo line 166 \"beem\"", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}]

Author rebuttal:

Decision: Accept