SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Sander Beckers
sanderbeckers.com
srekcebrednas@gmail.com
As more and more decisions that have a significant ethical dimension are being outsourced to AI systems, it is important to have a definition of moral responsibility that can be applied to AI systems. Moral responsibility for an outcome of an agent who performs some action is commonly taken to involve both a causal condition and an epistemic condition: the action should cause the outcome, and the agent should have been aware – in some form or other – of the possible moral consequences of their action. This paper presents a formal definition of both conditions within the framework of causal models. I compare my approach to the existing approaches of Graham and van Hees (BvH) and of Halpern and Kleiman-Weiner (HK). I then generalize my definition into a degree of responsibility.
As more and more decisions that have a significant ethical dimension are being outsourced to AI systems, it is important to have a definition of responsibility that can be applied to the decisions of AI systems, and that can be used by AI systems in the process of its decision-making [8]. To meet the first condition, such a definition should require only a minimal notion of agency and instead focus on those aspects of responsibility that are readily applicable to (current) AI systems. To meet the second condition, such a definition should be formulated in a language that can be implemented into an AI system, so that it can integrate judgments of responsibility into its decision-making. This paper sets out to propose such a definition using the well-established framework of causal models [22, 23].
There exist different notions of moral responsibility that one might be interested in, and here we restrict attention to just one of them, namely responsibility for consequences, meaning the responsibility one has for a particular outcome that is the result of performing a particular action. This can be expressed more clearly by saying that the action caused the outcome, and therefore the first condition of concern here is the causal condition on responsibility [12, 26, 21, 5]. The past two decades have seen immense progress on offering formal definitions of actual causation by way of using causal models, and the definition here developed takes maximal advantage of this progress by comparing some recent proposals and choosing the one that correctly handles several complicated cases to be considered [33, 16, 2].
Our actions can cause all kinds of outcomes for which we are clearly not morally responsible: if a train crashes into a car that illegally crosses the railroad then the train conductor is not responsible for the car driver's death, if you turn on a light switch in a hotel room then you are not responsible if a short-circuit follows, etc.. The standard intuition that we have in such cases is that the agent "could not have known" that their action would cause the outcome. This is why definitions of responsibility also invoke an epistemic condition, stating roughly that the agent should have been able to foresee that they are performing an action which could result in them being responsible for the outcome [5, 27, 25].
In addition to the causal and the epistemic conditions, it is standard to demand that responsibility also requires the fulfilment of a control condition (sometimes also called freedom condition), which expresses the fact that the agent had the right sort of control whilst performing their action [25]. Due to its close connection to issues of free will and determinism, this condition is heavily debated within philosophy. Within the context of (current) AI systems, however, the control condition can take on a more mundane form: any action that was a result of the correct operation of its program can be viewed as being under the AI's control. Therefore I simply take there to be a specific action variable that ranges over a set of possible actions, and assume that whenever the AI system is running successfully it has control over the value that this variable takes.
My approach proceeds along the same lines as that of Graham and van Hees (BvH) [5]. They offer the most influential formalization of moral responsibility that incorporates both the causal and the epistemic conditions, and therefore their work forms an appropriate point of comparison. Although I agree with the spirit of their approach, I disagree with its formulation. First, their causal condition defines causation as being a Necessary Element of a Sufficient Set (NESS). However, their use of game-theory instead of causal models results in an overly simplistic view of NESS-causation that cannot handle indirect causation. Therefore I first formulate their definition using causal models, and then show how to modify it so that it can overcome this limitation. Second, I disagree with the particulars of both their causal and their epistemic conditions. I argue for replacing the NESS definition of causation with my recently developed Counterfactual NESS (CNESS) definition [2]. Their epistemic condition states that the agent should minimize the probability of causation. I argue for giving that condition a secondary role: minimizing the probability of causing the outcome is subservient to minimizing the probability of the outcome simpliciter. I analyze several examples to illustrate the superiority of my conditions.
More recently, Halpern & Kleiman-Weiner (HK) [17] used causal models to propose definitions of several concepts that are closely related to moral responsibility. Although they do not explicitly define moral responsibility, they do suggest using the modified Halpern & Pearl (HP) definition of causation for the causal condition [16]. The HP definition correctly handles most of the counterexamples to the NESS definition here presented, but I discuss two types of example for which it fails (whereas the CNESS definition does not). HK also offer a definition of "degree of blameworthiness" that for all intents and purposes is very similar to an epistemic condition: it measures the extent to which the agent minimized the probability of the outcome. I present a case in which the epistemic conditions of BvH and HK conflict in order to argue that a more elaborate epistemic condition is required. My epistemic condition combines that of HK with that of BvH by demanding that an agent minimizes the probability of the outcome, but if possible also minimizes the probability of causation.<sup>1</sup>
Here is the general schema that encompasses all definitions of responsibility that I aim to consider.
Responsibility Schema. An agent who performs  $A = a$  is responsible for outcome  $O = o$  if:
- (Control Condition) The agent had control over  $A = a$ .
- (Causal Condition)  $A = a$  causes  $O = o$ .
- (Epistemic Condition) The agent believes that they could have avoided being responsible for  $O = o$  by performing some alternative action  $A = a'$ .
As mentioned, I simply assume that the Control Condition is always met (as do BvH, who call it the Agency Condition). For sake of brevity, I leave it implicit from now on.
Formalizing the Causal Condition comes down to settling the discussion on how to formalize actual causation, which has received considerable attention over the past two decades [22, 31, 15, 30, 16, 1]. A full discussion of causation would be too ambitious for the present purposes. Instead, I evaluate the suitability of several definitions of causation within the context of responsibility by presenting examples that bring across how they differ. On the basis of this evaluation I suggest adopting the CNESS definition and refer the reader to [2] for a more general motivation.
The Epistemic Condition requires settling the question: what does it take for the agent to believe that performing  $A = a'$  allows them to avoid responsibility? Since this condition uses the notion of
<sup>1</sup>Note that the epistemic conditions of HK and BvH are not necessarily inconsistent: if one simply defines causation as an increase in the probability of the outcome occurring, they become equivalent. Except for the fact that he uses objective probabilities rather than those of the agent, this is roughly the proposal defended in [29]. As exemplified by the examples to be discussed (and as exemplified by browsing the recent literature on causation) such a naive probabilistic approach to causation is unable to deliver sensible verdicts.
responsibility, our Schema is circular. There exist different ways of filling it in so that it no longer is circular, and it is this flexibility that makes filling in the condition interesting. One possible suggestion is to demand that the agent believed  $A = a'$  would not result in the outcome  $O = o$ , another weaker suggestion is to demand that the agent believed  $A = a'$  would not cause  $O = o$ , etc..
A note of clarification is in order before we proceed. The current work does not aim to offer a complete theory of moral responsibility for AI systems, but rather zooms in on the above conditions whilst ignoring certain others. Concretely, here are some important issues that I set aside in this paper.
1.1 Some Limitations and Related Work
There exist forms of moral responsibility that do not (always) involve causation, such as those that follow from certain societal norms and expectations. For example, a captain is responsible for everything that happens on their ship, a parent is responsible for the behavior of their child, etc.. More generally, assigning responsibility to AI systems should itself be seen as just one part of the wider discussion on accountability that arises from the introduction of such systems into our society [19, 7].
Relatedly, responsibility is often associated with the morally stronger notions of blame and praise. I take responsibility to be a weaker notion that necessarily precedes judgments of blame and praise: one cannot be blameworthy for an outcome unless one is responsible for it, and similarly for praise. To develop definitions of blame and praise requires bringing into view both the absolute moral valence of the outcome  $O = o$  (was it good or bad?) and its relative valence (was it better/worse than an alternative which it prevented?), as well as the costs incurred by the agent when performing an action. As the vast literature on trolley cases and other moral dilemmas illustrates, these issues make matters significantly more complex [10, 20].
One condition in particular that seems highly relevant to assigning blame (resp. praise) is to consider whether the outcome caused by the agent is harmful (resp. beneficial) or not. Indeed, one natural way of implementing a formal definition of responsibility within AI systems is to demand that it tries to avoid becoming responsible for harmful outcomes. This is confirmed by the recent European AI Act, which categorizes the risk that an AI system poses based on how likely it is to cause harm [11]. Beckers et. al. recently proposed a causal analysis of harm that is also formalized using causal models, and thus it could easily be integrated into the present proposal [3, 4]. In the present paper, however, I choose to focus exclusively on defining responsibility, thereby paving the way for future definitions of blame and praise.
Duijf presents a formalization of moral responsibility for outcomes that is likewise inspired by, but not an endorsement of, BvH [9]. Rather than defending an alternative definition of responsibility as I do, he presents a broad landscape of completely formal conditions for responsibility that one might consider and analyzes their logical relations. As with BvH, his definition of NESS causation is formulated using game-theory, and thus it is likewise restricted to applications of direct causation.
The next section introduces the formalism of causal models that will be used to express all candidate definitions and related notions. Section 3 presents the BvH and HK definitions of responsibility and their respective definitions of causation. Section 4 discusses the Causal Condition by introducing two more definitions of causation and offers some examples to argue in favor of adopting the CNESS definition. (Some further examples are offered in the appendix.) We move on to a discussion of the Epistemic Condition in Section 5, which leads the way to my definition of moral responsibility. Since responsibility is often taken to come in degrees, in Section 6 I define the degree of responsibility and sketch how it helps interpret recent empirical work in psychology on responsibility judgments.
2 Causal Models
This section reviews the definition of causal models as understood in the structural modeling tradition started by Pearl [22], where I use the notation from Halpern [16].
Definition 1. A signature  $\mathcal{S}$  is a tuple  $(\mathcal{U},\mathcal{V},\mathcal{R})$ , where  $\mathcal{U}$  is a set of exogenous variables,  $\mathcal{V}$  is a set of endogenous variables, and  $\mathcal{R}$  a function that associates with every variable  $Y\in \mathcal{U}\cup \mathcal{V}$  a nonempty
<sup>2</sup>I should note that they use the HP-definition of causation, which I criticize below. However, they state explicitly that their approach applies just as well to other definitions of causation.
set  $\mathcal{R}(Y)$  of possible values for  $Y$  (i.e., the set of values over which  $Y$  ranges). If  $\vec{X} = (X_1, \ldots, X_n)$ ,  $\mathcal{R}(\vec{X})$  denotes the crossproduct  $\mathcal{R}(X_1) \times \dots \times \mathcal{R}(X_n)$ .
Exogenous variables represent unobserved factors whose causal origins are outside the scope of the causal model, such as background conditions and noise. The values of the endogenous variables, on the other hand, are causally determined by other variables within the model.
Definition 2. A causal model  $M$  is a pair  $(S, \mathcal{F})$ , where  $S$  is a signature and  $\mathcal{F}$  defines a function that associates with each endogenous variable  $Y$  a structural equation  $F_{Y}$  giving the value of  $Y$  in terms of the values of other endogenous and exogenous variables. Formally, the equation  $F_{Y}$  maps  $\mathcal{R}(\mathcal{U} \cup \mathcal{V} - \{Y\})$  to  $\mathcal{R}(Y)$ , so  $F_{Y}$  determines the value of  $Y$ , given the values of all the other variables in  $\mathcal{U} \cup \mathcal{V}$ .
We usually write the equation for an endogenous variable as  $Y = f(\vec{X})$ , where  $\vec{X}$  are called the parents of  $Y$  (and  $Y$  is called a child of each variable in  $\vec{X}$ ), and the function  $f$  is such that it only depends on the values of  $\vec{X}$ . The ancestor relation is the transitive closure of the parent relation. In this paper we restrict attention to acyclic models, that is, models where no variable is an ancestor of itself. A (directed) path is a sequence of variables in which each element is a child of the previous element. In this manner an acyclic causal model induces a unique DAG, i.e., a Directed Acyclic Graph, which is simply a graphical representation of all the ancestral relations.
An intervention has the form  $\vec{X} \gets \vec{x}$ , where  $\vec{X}$  is a set of endogenous variables. Intuitively, this means that the values of the variables in  $\vec{X}$  are set to the values  $\vec{x}$ . The equations define what happens in the presence of interventions. The intervention  $\vec{X} \gets \vec{x}$  in a causal model  $M = (S, \mathcal{F})$  results in a new causal model, denoted  $M_{\vec{X} \gets \vec{x}}$ , which is identical to  $M$ , except that  $\mathcal{F}$  is replaced by  $\mathcal{F}^{\vec{X} \gets \vec{x}}$ : for each variable  $Y \notin \vec{X}$ ,  $F_{Y}^{\vec{X} \gets \vec{x}} = F_{Y}$  (i.e., the equation for  $Y$  is unchanged), while for each  $X'$  in  $\vec{X}$ , the equation  $F_{X'}$  for  $X'$  is replaced by  $X' = x'$  (where  $x'$  is the value in  $\vec{x}$  corresponding to  $X'$ ).
Given a signature  $\mathcal{S} = (\mathcal{U},\mathcal{V},\mathcal{R})$ , an atomic formula is a formula of the form  $X = x$ , for  $X \in \mathcal{V}$  and  $x \in \mathcal{R}(X)$ . A causal formula (over  $\mathcal{S}$ ) is one of the form  $[Y_1 \leftarrow y_1, \ldots, Y_k \leftarrow y_k] \phi$ , where
-  $\phi$  is a Boolean combination of atomic formulas,
-  $Y_{1}, \ldots, Y_{k}$  are distinct variables in  $\mathcal{V}$ , and
-  $y_{i} \in \mathcal{R}(Y_{i})$  for each  $1 \leq i \leq k$ .
Such a formula is abbreviated as  $[\vec{Y} \leftarrow \vec{y}]\phi$ . The special case where  $k = 0$  is abbreviated as  $\phi$ . Intuitively,  $[Y_1 \leftarrow y_1, \ldots, Y_k \leftarrow y_k]\phi$  says that  $\phi$  would hold if  $Y_i$  were set to  $y_i$ , for  $i = 1, \ldots, k$ .
We call a setting  $\vec{u} \in \mathcal{R}(\mathcal{U})$  of values of exogenous variables a context. A causal formula  $\psi$  is true or false in a causal setting, which is a causal model given a context. As usual, we write  $(M, \vec{u}) \models \psi$  if the causal formula  $\psi$  is true in the causal setting  $(M, \vec{u})$ . The  $\vDash$  relation is defined inductively.  $(M, \vec{u}) \models X = x$  if the variable  $X$  has value  $x$  in the unique (since we are dealing with recursive models) solution to the equations in  $M$  in context  $\vec{u}$  (i.e., the unique vector of values that simultaneously satisfies all equations in  $M$  with the variables in  $\mathcal{U}$  set to  $\vec{u}$ ). The truth of conjunctions and negations is defined in the standard way. Finally,  $(M, \vec{u}) \models [\vec{Y} \leftarrow \vec{y}] \phi$  if  $(M_{\vec{Y} \leftarrow \vec{y}}, \vec{u}) \models \phi$ .
In addition to the causal setting  $(M, \vec{u})$  that describes both the objective causal relations and their actual realization, we also need to represent the agent's beliefs regarding what could possibly happen in order to fill in the Epistemic Condition. I do so in the same manner as proposed by HK: we take Pr to be a probability distribution over a set of causal settings  $\mathcal{K}$ , so that Pr expresses the agent's subjective probabilities before the agent performs their action. As do HK, I assume for simplicity that all the causal models appearing in  $\mathcal{K}$  have the same signature (i.e., the same exogenous and endogenous variables). We define an epistemic state of an agent to consist of a pair  $\mathcal{E} = (\mathrm{Pr}, \mathcal{K})$ , and define a responsibility setting  $(M, \vec{u}, \mathcal{E})$  as the combination of a causal setting and an epistemic state.
3 The BvH and HK Definitions
BvH [5] work within a game-theoretic framework and do not use causal models, so in order to compare their approach to mine we need to first translate it into the language of causal models. I do not delve into the details but rather offer a rough sketch of such a translation.
Similar to causal models, BvH represent the agents' influence on the outcome  $O$  by way of a function. Yet instead of letting some endogenous variables  $\vec{A}$  represent the actions of agents directly, they use variables to represent the strategies that each agent can adopt to guide their actions. Aside from that, the main difference between the two formalisms is that theirs is unable to represent indirect causal relations. In general, the equations of a causal model allow for an unlimited number of intermediate ancestors between variables  $\vec{A}$  and an outcome variable  $O$ , so that causal influence from an agent's action can be passed on along intermediate variables to the outcome variable. BvH's outcome function on the other hand abstracts away from any mediated form of causal influence, so that the strategies causally determine the outcome directly. As a result, their games are to be interpreted as a single-equation causal model of the form  $O = f_{O}(\vec{A})$ . (Since the variables  $\vec{A}$  are determined directly by the context  $\vec{u}$ , I adopt the standard practice of leaving their equations implicit.)
BvH use the famous NESS definition of causation that was proposed by Wright [32, 33] – and also formed the inspiration for the Halpern & Pearl (HP) definitions [22, 15, 16] – which states that causes are Necessary Elements of a Sufficient Set for the effect. Taking into account the previous remarks, it is more accurate to speak of the direct NESS definition. I here present my recent formalization of both the direct and the indirect NESS definitions using causal models [2]. First we need to define causal sufficiency. As do BvH, I take it to mean that a set guarantees the effect regardless of the values of the variables outside of the set.
Definition 3 (Sufficiency). We say that  $\vec{X} = \vec{x}$  is sufficient for  $Y = y$  w.r.t.  $(M, \vec{u})$  if  $Y \notin \vec{X}$  and for all values  $\vec{z} \in \mathcal{R}(\vec{Z})$  where  $\vec{Z} = \mathcal{V} - (\vec{X} \cup \{Y\})$ , it holds that  $(M, \vec{u}) \models [\vec{X} \leftarrow \vec{x}, \vec{Z} \leftarrow \vec{z}] Y = y$ .
Direct NESS-causation is then defined by stating that:
- the candidate cause and the effect actually occurred;
- the candidate cause is a member of a sufficient set;
- and it is necessary for the set to be sufficient.
Definition 4 (Direct NESS).  $X = x$  directly NESS-causes  $Y = y$  w.r.t.  $(M, \vec{u})$  if there exists a  $\vec{W} = \vec{w}$  so that the following conditions hold:
DN1.  $(M,\vec{u})\vDash X = x\wedge \vec{W} = \vec{w}\wedge Y = y.$
DN2.  $X = x\wedge \vec{W} = \vec{w}$  is sufficient for  $Y = y$  w.r.t.  $(M,\vec{u})$
DN3.  $\vec{W} = \vec{w}$  is not sufficient for  $Y = y$  w.r.t.  $(M,\vec{u})$
We can now formulate the counterpart of the BvH definition using causal models by filling in their conditions into our Responsibility Schema.
Definition 5 (BvH Responsibility). An agent who performs  $A = a$  is responsible for outcome  $O = o$  w.r.t. a responsibility setting  $(M, \vec{u}, \mathcal{E})$  if:
- (Causal Condition)  $A = a$  directly NESS-causes  $O = o$  w.r.t.  $(M, \vec{u})$ .
- (Epistemic Condition) There exists  $a' \in \mathcal{R}(A)$  so that  $\operatorname*{Pr}(A = a \text{ directly NESS-causes } O = o) > \operatorname*{Pr}(A = a' \text{ directly NESS-causes } O = o)$ .<sup>4</sup>
Informally, the BvH definition of responsibility requires that an agent's action directly NESS-caused the outcome, and that the agent believes they failed to minimize the probability of their action causing the outcome. The following example (taken from BvH) illustrates their definition.
Example 1. Two assassins (Assassin<sub>1</sub> and Assassin<sub>2</sub>), in place as snipers, shoot and kill Victim, with each of the bullets fatally piercing Victim's heart at exactly the same moment. Although neither of them could have prevented the outcome, each of them is clearly responsible for Victim's death.
Let  $V$  stand for Victim's death ( $V = 1$ ) or survival ( $V = 0$ ), and let  $A_{1}, A_{2}$  stand for the actions of the two assassins, where  $A_{i} = 1$  if and only if  $\text{Assassin}_i$  shoots. We can then capture this example with the single equation  $V = A_{1} \vee A_{2}$ , and a context  $\vec{u}$  such that  $A_{1} = 1$  and  $A_{2} = 1$ .
<sup>3</sup>As I said, this is a rough sketch. Technically, one should distinguish between games in normal form, which is the form considered by BvH, and games in extensive form, from which the normal form games have been derived. Games in extensive form do allow for indirect relations, and thus there might be a way of representing indirect causal relations in game theory after all.
4Of course these probabilities have to be read as being conditioned on the corresponding action, i.e., as "the agent's probability that the action would cause the outcome if it were performed".
Does the BvH definition (Definition 5) succeed in establishing that each of the assassins is responsible for Victim's death? To find out, we first need to evaluate whether  $A_{1} = 1$  (resp.  $A_{2} = 1$ ) directly NESS-causes  $V = 1$  (Def. 4). We can choose  $\tilde{W} = \emptyset$  to get the desired result, as follows.